v1.2版本改为通过xbox系列手柄进行遥控,并增加智能模式和手动模式
This commit is contained in:
myperworld committed
1 parent
76603d3c31
commit
1a97bc32c3
1839 files changed
+478852
-3
No files matched your search
@@ -0,0 +1,66 @@
|
||||
# Advanced Development with FastLED
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## GDB On Unit Tests
|
||||
|
||||
Yes, we have step through debugging with FastLED.
|
||||
|
||||
* VSCode
|
||||
* Install Plugin: GDB Debugger - Beyond
|
||||
* Navigate to one of the tests in `tests/` and open in
|
||||
* Hit `F5`
|
||||
|
||||
If the Python Debugger pops up, then manually switch the VSCode debugger using `Launch(gdb)`
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Enabling 3-second compile times using our `web-compiler`
|
||||
|
||||
* You must have `docker` installed for fastest compile times. It's free.
|
||||
* `cd <FASTLED FOLDER>`
|
||||
* `pip install fastled`
|
||||
* `fastled examples/Blink/Blink.ino`
|
||||
* Cpp changes to the fastled source can be compiled by the live fastled compiler.
|
||||
|
||||
## Testing your changes
|
||||
|
||||
Most of this is in the basic CONTRIBUTING.md guide. But as a reminder
|
||||
* Unit Testing: `./test`
|
||||
* Linting: `./lint`
|
||||
* Compiling on platforms `./compile uno,teensy41,esp32s3 --examples Blink,Apa102HD`
|
||||
|
||||
## Enabling AI coding
|
||||
|
||||
`aider.chat` is available for advanced and high velocity coding with FastLED. To use it, have your open-ai or Anthropic api key ready. It's recommended to use Anthropic as it's performance is much better than Open-AI for coding.
|
||||
|
||||
At the root of the project type:
|
||||
|
||||
`./ai` and follow the prompts. Once the key is installed you will get a prompt that looks like this:
|
||||
|
||||
```bash
|
||||
architect>
|
||||
```
|
||||
|
||||
There are two modes to use this AI, a watch mode which watches your files for changes and launches automatically, and a slow way which you add target files then instruct it to make changes:
|
||||
|
||||
* watch mode (best for implimenting a function or two)
|
||||
* Edit any file in the repo. Add a comment with `AI!` at the end. The ai will see this and start implementing what you just typed.
|
||||
* Example: Edit `src/fl/vector.h` and put in a comment `// Add more comments AI!`, then say yes to the changes in the prompt.
|
||||
* Slow mode (much better for bigger changes across the file or several)
|
||||
* While you are in the `architect> ` prompt you will add a file to the chat
|
||||
* `/add src/fl/vector.h`
|
||||
* Now tell the AI what you want it to do, and it will do it.
|
||||
* Making the AI fix it's own problems it introduced.
|
||||
* At the AI command prompt, have it run the following
|
||||
* Linux/Mac: `/run ./test`
|
||||
* On Windows: `/run uv run test.py`
|
||||
* After the test concludes, the AI will ask you if you want to add the output back into the chat, agree to it then let it try to correct it's mistakes.
|
||||
|
||||
Every single time you do a change, make sure and thoroughly check it. I recommend VSCodes built in git diff tool.
|
||||
|
||||
Although the AI is pretty amazing, it will inject entropy into your code and this is the source of a lot of problems. So watch all changes it makes very thoroughly. Under almost all circumstances you will have to revert unnecessary changes (like comments) line by line.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Special Notes on APA102 and the 'High Definition' Mode in FastLED
|
||||
|
||||
The APA102 LED driver includes a 5-bit per-LED brightness component. Previously, this feature was not fully utilized, except through a workaround that defined a global brightness affecting all LEDs uniformly rather than individually.
|
||||
|
||||
In FastLED the APA102 chipset will have extra resolution in comparison to the WS2812 RGB8 mode.
|
||||
|
||||
There are two modes:
|
||||
* APA102 "Regular Mode"
|
||||
* Has enhanced color resolution when using the "global brightness" factor
|
||||
* APA102HD Mode
|
||||
* Applies automatic gamma correction at the driver level using "pseudo 13-bit" color mixing.
|
||||
|
||||
**APA102HD Mode**
|
||||
|
||||
[example: examples/APA102HD](examples/Apa102HD/)
|
||||
|
||||
By introducing a 5-bit gamma bit-shift algorithm, we now effectively leverage this per-LED brightness control. Faced with the decision to either rewrite the entire `CRGB` library to expose the 5-bit brightness—including adaptations for formats like RGBW—or to retain the existing RGB8 format used by FastLED and implement the enhancement at the driver level, the latter option was chosen. This approach avoids widespread changes and maintains compatibility; if RGB8 suffices for game development, it is adequate for LED development as well.
|
||||
|
||||
The term "Pseudo-13-bit" arises because the additional resolution becomes significant only when all color components are at low values. For example, colors like `CRGB(255, 255, 254)` or `CRGB(255, 1, 1)` do not benefit from increased resolution due to the dominance of the brighter components. However, in low-light conditions with colors such as `CRGB(8, 8, 8)`, where the maximum component value is low, the pseudo-13-bit algorithm significantly enhances resolution—precisely where increased resolution is most desired.
|
||||
|
||||
Gamma correction is applied to preserve the RGB8 format and because future LEDs are expected to support gamma correction inherently. In game development, the 0-255 color values are based on the gamma scale rather than the linear power scale. LEDs like the WS2812 operate on a linear power scale, which results in washed-out, undersaturated colors when displaying captured video directly. Implementing software gamma correction for RGB8 severely reduces color resolution.
|
||||
|
||||
To address this, an internal gamma scale mapping is applied:
|
||||
|
||||
```
|
||||
RGB8 → RGB16 + 5-bit gamma → RGB8 + 5-bit gamma
|
||||
```
|
||||
|
||||
During the conversion back to RGB8, the brightness from the 5-bit gamma is optimally distributed using a closed-form mathematical algorithm. Rather than the iterative bit-shifting approach used in earlier versions, the current implementation uses a sophisticated quantization and scaling method that provides superior accuracy.
|
||||
|
||||
**Closed-Form Algorithm (Current Implementation)**
|
||||
|
||||
The algorithm works by:
|
||||
|
||||
1. **Finding the maximum component**: Determines which RGB component requires the highest brightness level
|
||||
2. **Quantizing to 5-bit scale**: Computes the optimal 5-bit brightness value that accommodates the maximum component
|
||||
3. **Applying pre-computed scaling**: Uses a lookup table of precise scaling factors to convert the 16-bit gamma-corrected values back to 8-bit RGB components
|
||||
|
||||
The key insight is the quantization formula:
|
||||
```
|
||||
scale = (max_component + (2047 - (max_component >> 5))) >> 11
|
||||
```
|
||||
|
||||
This ensures the 5-bit brightness value is always sufficient to represent the maximum component while minimizing quantization error.
|
||||
|
||||
The scaling factors in the lookup table are computed as:
|
||||
```
|
||||
bright_scale[i] = (i/31) * (255/65536) * 256
|
||||
```
|
||||
|
||||
These pre-computed values eliminate floating-point arithmetic while maintaining precision. The final RGB values are calculated with proper rounding:
|
||||
```
|
||||
rgb_out = (rgb16 * scale_factor + 0x808000) >> 24
|
||||
```
|
||||
|
||||
**Algorithm Performance**
|
||||
|
||||
This closed-form approach delivers exceptional accuracy:
|
||||
- **Worst-case error**: 0.2% over the full 16-bit input range
|
||||
- **Average error**: 0.05% across the range
|
||||
- **Perfect match**: to floating-point reference implementation
|
||||
|
||||
By eliminating iterative bit-shifting, this method also provides consistent performance regardless of input values and significantly reduces code size on embedded platforms.
|
||||
|
||||
**Version History**
|
||||
|
||||
In version 3.9.0, the algorithm was completely rewritten to function natively on 8-bit controllers like the `__AVR__` chipsets without significant performance loss. The current closed-form implementation replaced the previous iterative approach, providing both better accuracy and smaller code footprint.
|
||||
|
||||
**Further Enhancements in Version 3.9.0**
|
||||
|
||||
Additionally, version 3.9.0 separated the color temperature from the global brightness scale. Before this update, global brightness was pre-mixed with the component scales—a method suitable for the WS2812's RGB8 format but not for the APA102's RGB8 plus 5-bit brightness. The update saw the global brightness and color scales separated for non-AVR chipsets. While the WS2812 continues to use pre-mixed values for performance reasons on AVR chipsets, the APA102 now performs component mixing within the "pseudo-13-bit space."
|
||||
|
||||
Although APA102HD mode offers the highest dynamic range, the standard APA102 mode also benefits from increased resolution when adjusting global brightness. In this mode, instead of pre-mixing scales and multiplying them against each `CRGB` value, the global brightness is applied to the 5-bit brightness component, and only the color scales are multiplied against the `CRGB` values. This approach is superior because each component of the color scale typically exceeds 127, providing ample high-order bits to preserve color information.
|
||||
@@ -0,0 +1,168 @@
|
||||
# 🚨🚨🚨 **CRITICAL REQUIREMENTS FOR BACKGROUND AGENTS** 🚨🚨🚨
|
||||
|
||||
# ⚠️ **STOP! READ THIS FIRST!** ⚠️
|
||||
|
||||
## **MANDATORY TESTING REQUIREMENT - NO EXCEPTIONS!**
|
||||
|
||||
### 🔥 **RULE #1: ALWAYS RUN `bash test` BEFORE COMPLETION** 🔥
|
||||
|
||||
**THIS IS THE MOST IMPORTANT RULE FOR BACKGROUND AGENTS:**
|
||||
|
||||
- **🚨 EVERY BACKGROUND AGENT MUST RUN `bash test` BEFORE SAYING THEY ARE DONE**
|
||||
- **🚨 NO EXCEPTIONS - THIS IS MANDATORY AND NON-NEGOTIABLE**
|
||||
- **🚨 IF YOU DON'T RUN `bash test`, YOUR WORK IS INCOMPLETE**
|
||||
- **🚨 SAYING "DONE" WITHOUT RUNNING `bash test` IS BREAKING THE RULES**
|
||||
|
||||
### **WHY THIS IS CRITICAL:**
|
||||
- The `bash test` command runs the full test suite including unit tests and compilation checks
|
||||
- It catches breaking changes, compilation errors, and regressions
|
||||
- It ensures the codebase remains stable and functional
|
||||
- **WITHOUT THIS, YOU WILL BREAK THE PROJECT**
|
||||
|
||||
### **WHAT `bash test` DOES:**
|
||||
- Runs all C++ unit tests
|
||||
- Runs Python linting and formatting checks
|
||||
- Validates code compilation across platforms
|
||||
- Checks for syntax errors and warnings
|
||||
- Ensures all examples still compile
|
||||
|
||||
### 🚨 **MANDATORY STEPS FOR COMPLETION - FOLLOW EXACTLY:** 🚨
|
||||
|
||||
1. **🔥 RUN `bash test` COMMAND**
|
||||
- Execute: `bash test`
|
||||
- Wait for ALL tests to complete
|
||||
- Check that ALL tests PASS
|
||||
|
||||
2. **🔥 VERIFY TEST RESULTS**
|
||||
- If ANY test fails, you MUST fix the issues
|
||||
- Do NOT ignore warnings or errors
|
||||
- Do NOT proceed until ALL tests pass
|
||||
|
||||
3. **🔥 USE MCP SERVER VALIDATION (RECOMMENDED)**
|
||||
- Use the `validate_completion` tool from MCP server: `uv run mcp_server.py`
|
||||
- This tool runs `bash test` and validates results
|
||||
- Only indicate completion if this tool returns SUCCESS
|
||||
|
||||
4. **🔥 ZERO TOLERANCE FOR FAILURES**
|
||||
- If ANY test fails, the background agent MUST fix the issues
|
||||
- Do NOT indicate completion with failing tests
|
||||
- Do NOT ignore test errors or warnings
|
||||
- Fix the code and re-run `bash test` until ALL tests pass
|
||||
|
||||
### **COMPLETION CHECKLIST - MUST COMPLETE ALL ITEMS:**
|
||||
- [ ] ✅ All code changes have been made
|
||||
- [ ] ✅ **`bash test` has been run and ALL tests PASS**
|
||||
- [ ] ✅ No compilation errors or warnings remain
|
||||
- [ ] ✅ MCP server `validate_completion` tool shows SUCCESS (if used)
|
||||
- [ ] ✅ All test output has been reviewed for any issues
|
||||
- [ ] ✅ **ONLY THEN** indicate task completion
|
||||
|
||||
### 🔥 **CONSEQUENCES OF NOT FOLLOWING THIS RULE:** 🔥
|
||||
|
||||
- **YOUR CHANGES WILL BREAK THE PROJECT**
|
||||
- **OTHER DEVELOPERS WILL HAVE TO FIX YOUR MISTAKES**
|
||||
- **THE CONTINUOUS INTEGRATION SYSTEM WILL FAIL**
|
||||
- **YOU WILL CAUSE REGRESSIONS AND BUGS**
|
||||
|
||||
### 🚨 **FINAL WARNING:** 🚨
|
||||
|
||||
**IF YOU ARE A BACKGROUND AGENT AND YOU INDICATE COMPLETION WITHOUT RUNNING `bash test`, YOU HAVE FAILED TO FOLLOW THE MOST BASIC REQUIREMENT OF THIS PROJECT.**
|
||||
|
||||
**NO EXCEPTIONS. NO SHORTCUTS. ALWAYS RUN `bash test` FIRST.**
|
||||
|
||||
---
|
||||
|
||||
## **Examples of CORRECT Background Agent Behavior:**
|
||||
|
||||
### ✅ **GOOD Example:**
|
||||
```
|
||||
1. Made code changes
|
||||
2. Ran `bash test`
|
||||
3. All tests passed
|
||||
4. Indicated completion: "Task completed successfully. All tests pass."
|
||||
```
|
||||
|
||||
### ❌ **BAD Example:**
|
||||
```
|
||||
1. Made code changes
|
||||
2. Skipped running tests
|
||||
3. Indicated completion: "Task completed successfully."
|
||||
```
|
||||
**^ THIS IS WRONG AND BREAKS THE PROJECT**
|
||||
|
||||
### ✅ **GOOD Example with Test Failures:**
|
||||
```
|
||||
1. Made code changes
|
||||
2. Ran `bash test`
|
||||
3. Some tests failed
|
||||
4. Fixed the failing code
|
||||
5. Ran `bash test` again
|
||||
6. All tests passed
|
||||
7. Indicated completion: "Task completed successfully. All tests pass."
|
||||
```
|
||||
|
||||
### ❌ **BAD Example with Test Failures:**
|
||||
```
|
||||
1. Made code changes
|
||||
2. Ran `bash test`
|
||||
3. Some tests failed
|
||||
4. Ignored failures and indicated completion anyway
|
||||
```
|
||||
**^ THIS IS WRONG AND BREAKS THE PROJECT**
|
||||
|
||||
---
|
||||
|
||||
## **HOW TO RUN `bash test`:**
|
||||
|
||||
### **Command:**
|
||||
```bash
|
||||
bash test
|
||||
```
|
||||
|
||||
### **Expected Output (Success):**
|
||||
```
|
||||
Running C++ tests...
|
||||
All C++ tests passed ✅
|
||||
Running Python linting...
|
||||
All Python checks passed ✅
|
||||
Running compilation checks...
|
||||
All compilation checks passed ✅
|
||||
```
|
||||
|
||||
### **What to do if tests fail:**
|
||||
1. **READ THE ERROR MESSAGES CAREFULLY**
|
||||
2. **FIX THE CODE THAT'S CAUSING THE FAILURES**
|
||||
3. **RUN `bash test` AGAIN**
|
||||
4. **REPEAT UNTIL ALL TESTS PASS**
|
||||
5. **ONLY THEN INDICATE COMPLETION**
|
||||
|
||||
---
|
||||
|
||||
## **MCP SERVER VALIDATION TOOL:**
|
||||
|
||||
### **How to use:**
|
||||
```bash
|
||||
# Start MCP server
|
||||
uv run mcp_server.py
|
||||
|
||||
# Use the validate_completion tool
|
||||
# This automatically runs `bash test` and validates results
|
||||
```
|
||||
|
||||
### **Benefits:**
|
||||
- Automatically runs `bash test`
|
||||
- Provides clear pass/fail results
|
||||
- Ensures you don't forget to run tests
|
||||
- **RECOMMENDED FOR ALL BACKGROUND AGENTS**
|
||||
|
||||
---
|
||||
|
||||
## 🚨 **REMEMBER: THIS IS NOT OPTIONAL** 🚨
|
||||
|
||||
**EVERY BACKGROUND AGENT WORKING ON THIS PROJECT MUST:**
|
||||
1. **RUN `bash test` BEFORE COMPLETION**
|
||||
2. **ENSURE ALL TESTS PASS**
|
||||
3. **FIX ANY FAILURES**
|
||||
4. **ONLY THEN INDICATE COMPLETION**
|
||||
|
||||
**NO SHORTCUTS. NO EXCEPTIONS. ALWAYS TEST FIRST.**
|
||||
@@ -0,0 +1,47 @@
|
||||
# FastLED AI Agent Guidelines
|
||||
|
||||
## Quick Reference
|
||||
|
||||
This project uses directory-specific agent guidelines. See:
|
||||
|
||||
- **CI/Build Tasks**: `ci/AGENTS.md` - Python build system, compilation, MCP server tools
|
||||
- **Testing**: `tests/AGENTS.md` - Unit tests, test execution, validation requirements
|
||||
- **Examples**: `examples/AGENTS.md` - Arduino sketch compilation, .ino file rules
|
||||
|
||||
## Key Commands
|
||||
|
||||
- `uv run test.py` - Run all tests
|
||||
- `uv run test.py --cpp` - Run C++ tests only
|
||||
- `uv run test.py TestName` - Run specific C++ test (e.g., `uv run test.py xypath`)
|
||||
- `bash lint` - Run code formatting/linting
|
||||
- `uv run ci/ci-compile.py uno --examples Blink` - Compile examples for specific platform
|
||||
- `uv run ci/wasm_compile.py examples/Blink --just-compile` - Compile Arduino sketches to WASM
|
||||
- `uv run mcp_server.py` - Start MCP server for advanced tools
|
||||
|
||||
### QEMU Commands
|
||||
- `uv run ci/install-qemu.py` - Install QEMU for ESP32 emulation (standalone)
|
||||
- `uv run test.py --qemu esp32s3` - Run QEMU tests (installs QEMU automatically)
|
||||
- `FASTLED_QEMU_SKIP_INSTALL=true uv run test.py --qemu esp32s3` - Skip QEMU installation step
|
||||
|
||||
## Core Rules
|
||||
|
||||
### Command Execution (ALL AGENTS)
|
||||
- **Python**: Always use `uv run python script.py` (never just `python`)
|
||||
- **Stay in project root** - never `cd` to subdirectories
|
||||
- **Git-bash compatibility**: Prefix commands with space: `bash test`
|
||||
|
||||
### C++ Code Standards
|
||||
- **Use `fl::` namespace** instead of `std::`
|
||||
- **If you want to use a stdlib header like <type_traits>, look check for equivalent in `fl/type_traits.h`
|
||||
- **Use proper warning macros** from `fl/compiler_control.h`
|
||||
- **Follow existing code patterns** and naming conventions
|
||||
|
||||
### JavaScript Code Standards
|
||||
- **After modifying any JavaScript files**: Always run `bash lint --js` to ensure proper formatting
|
||||
|
||||
### Memory Refresh Rule
|
||||
**🚨 ALL AGENTS: Read the relevant AGENTS.md file before concluding work to refresh memory about current project rules and requirements.**
|
||||
|
||||
---
|
||||
|
||||
*This file intentionally kept minimal. Detailed guidelines are in directory-specific AGENTS.md files.*
|
||||
@@ -0,0 +1,26 @@
|
||||
# FastLED
|
||||
# https://github.com/FastLED/FastLED
|
||||
# MIT License
|
||||
|
||||
cmake_minimum_required(VERSION 3.5)
|
||||
|
||||
# Collect all source files
|
||||
file(GLOB FastLED_SRCS "src/*.cpp")
|
||||
file(GLOB FastLED_FL_SRCS "src/fl/*.cpp")
|
||||
file(GLOB FastLED_SENSORS_SRCS "src/sensors/*.cpp")
|
||||
file(GLOB FastLED_PLATFORM_ARDUINO_SRCS "src/platforms/arduino/*.cpp")
|
||||
file(GLOB FastLED_FX_SRCS "src/fx/*.cpp" "src/fx/**/*.cpp")
|
||||
|
||||
file(GLOB ESP32_SRCS "src/platforms/esp/32/*.cpp" "src/platforms/esp/32/rmt_5/*.cpp")
|
||||
file(GLOB ESP32_THIRD_PARTY_SRCS "src/third_party/**/src/*.c" "src/third_party/**/src/*.cpp")
|
||||
file(GLOB ESP32_LED_STRIP_SRCS "src/third_party/espressif/led_strip/src/*.c")
|
||||
|
||||
# Combine all source files into a single list
|
||||
list(APPEND FastLED_SRCS ${FastLED_FL_SRCS} ${FastLED_SENSORS_SRCS} ${FastLED_FX_SRCS} ${ESP32_SRCS} ${ESP32_THIRD_PARTY_SRCS} ${ESP32_LED_STRIP_SRCS} ${FastLED_PLATFORM_ARDUINO_SRCS})
|
||||
|
||||
# Register the component with ESP-IDF
|
||||
idf_component_register(SRCS ${FastLED_SRCS}
|
||||
INCLUDE_DIRS "src" "src/third_party/espressif/led_strip/src"
|
||||
REQUIRES arduino-esp32 esp_driver_rmt esp_lcd driver)
|
||||
|
||||
project(FastLED)
|
||||
@@ -0,0 +1,243 @@
|
||||
## Contributing
|
||||
|
||||
The most important part about contributing to FastLED is knowing how to test your changes.
|
||||
|
||||
The FastLED library includes a powerful cli that can compile to any device. It will run if you have either [python](https://www.python.org/downloads/) or [uv](https://github.com/astral-sh/uv) installed on the system.
|
||||
|
||||
## FastLED compiler cli
|
||||
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_clone_and_compile.yml)
|
||||
|
||||
The FastLED compiler cli can be invoked at the project root.
|
||||
|
||||
```bash (MacOS/Linux, windows us git-bsh or compile.bat)
|
||||
git clone https://github.com/fastled/fastled
|
||||
cd fastled
|
||||
./compile uno --examples Blink # linux/macos/git-bash
|
||||
# compile.bat # Windows.
|
||||
```
|
||||
|
||||
## Linting and Unit Testing
|
||||
|
||||
```bash
|
||||
./lint
|
||||
./test # runs unit tests
|
||||
# Note that you do NOT need to install the C++ compiler toolchain
|
||||
# for compiling + running unit tests via ./test. If `gcc` is not
|
||||
# found in your system `PATH` then the `ziglang` clang compiler
|
||||
# will be swapped in automatically.
|
||||
````
|
||||
|
||||
|
||||
### Testing a bunch of platforms at once.
|
||||
|
||||
```
|
||||
./compile teensy41,teensy40 --examples Blink
|
||||
./compile esp32dev,esp32s3,esp32c3,esp32c6,esp32s2 --examples Blink,Apa102HD
|
||||
./compiles uno,digix,attiny85 --examples Blink,Apa102HD
|
||||
```
|
||||
|
||||
## Unit Tests
|
||||
|
||||
Shared code is unit-tested on the host machine. They can be found at `tests/` at the root of the repo. Unit testing only requires either `python` or `uv` to be installed. The C++ compiler toolchain will be installed automatically.
|
||||
|
||||
The easiest way to run the tests is just use `./test`
|
||||
|
||||
Alternatively, tests can be built and run for your development machine with CMake:
|
||||
|
||||
```bash
|
||||
cmake -S tests -B tests/.build
|
||||
ctest --test-dir tests/.build --output-on-failure
|
||||
# Not that this will fail if you do not have gcc installed. When in doubt
|
||||
# use ./test to compile the unit tests, as a compiler is guaranteed to be
|
||||
# available via this tool.
|
||||
```
|
||||
|
||||
## QEMU Emulation Testing
|
||||
|
||||
FastLED supports testing ESP32-S3 examples in QEMU emulation, providing a powerful way to validate code without physical hardware.
|
||||
|
||||
### Running ESP32-S3 Examples in QEMU
|
||||
|
||||
```bash
|
||||
# Test default examples (BlinkParallel, RMT5WorkerPool)
|
||||
./test --qemu esp32s3
|
||||
|
||||
# Test specific examples
|
||||
./test --qemu esp32s3 BlinkParallel
|
||||
./test --qemu esp32s3 RMT5WorkerPool BlinkParallel
|
||||
|
||||
# Quick validation test (setup verification only)
|
||||
FASTLED_QEMU_QUICK_TEST=true ./test --qemu esp32s3
|
||||
```
|
||||
|
||||
### What QEMU Testing Does
|
||||
|
||||
1. **Automatic QEMU Installation**: Downloads and sets up ESP32-S3 QEMU emulator
|
||||
2. **Cross-Platform Compilation**: Builds examples for ESP32-S3 target architecture
|
||||
3. **Emulated Execution**: Runs compiled firmware in QEMU virtual environment
|
||||
4. **Automated Validation**: Monitors execution for success/failure indicators
|
||||
|
||||
### QEMU Test Output
|
||||
|
||||
The QEMU tests provide detailed feedback:
|
||||
- **Build Status**: Compilation success/failure for each example
|
||||
- **Execution Results**: Runtime behavior in emulated environment
|
||||
- **Summary Statistics**: Pass/fail counts and timing information
|
||||
- **Error Details**: Specific failure reasons when tests don't pass
|
||||
|
||||
### Supported Platforms
|
||||
|
||||
Currently supported QEMU platforms:
|
||||
- **esp32s3**: ESP32-S3 SoC emulation
|
||||
|
||||
Future platforms may include additional ESP32 variants as QEMU support expands.
|
||||
|
||||
### Advanced QEMU Usage
|
||||
|
||||
```bash
|
||||
# Run with verbose output to see detailed build and execution logs
|
||||
./test --qemu esp32s3 --verbose
|
||||
|
||||
# Test in non-interactive mode (useful for CI/CD)
|
||||
./test --qemu esp32s3 --no-interactive
|
||||
```
|
||||
|
||||
## VSCode
|
||||
|
||||
We also support VSCode and IntelliSense auto-completion when the free [platformio](https://marketplace.visualstudio.com/items?itemName=platformio.platformio-ide) extension is installed. The development sketch to test library changes can be found at [dev/dev.ino](dev/dev.ino).
|
||||
|
||||
* Make sure you have [platformio](https://marketplace.visualstudio.com/items?itemName=platformio.platformio-ide) installed.
|
||||
* Click the compile button.
|
||||
|
||||
|
||||
*Changes in non platform specific code can be tested quickly in our webcompiler by invoking the script `./wasm` at the project root*
|
||||
|
||||
## VSCode Debugging Guide
|
||||
|
||||
FastLED includes comprehensive VSCode debugging support with GDB and Clang-generated debug symbols, providing professional-grade step-through debugging capabilities for the test suite.
|
||||
|
||||
### Quick Start
|
||||
|
||||
1. **Prerequisites**: Ensure you have VSCode with the C/C++ extension installed
|
||||
2. **Open a test file**: e.g., `tests/test_allocator.cpp`
|
||||
3. **Set breakpoints**: Click the left margin next to line numbers
|
||||
4. **Press F5**: Automatically debugs the corresponding test executable
|
||||
5. **Debug**: Use F10 (step over), F11 (step into), F5 (continue)
|
||||
|
||||
### Available Debug Configurations
|
||||
|
||||
#### 🎯 **"Debug FastLED Test (Current File)"** ⭐ *Most Common*
|
||||
- **When to use**: When you have any test file open (e.g., `test_allocator.cpp`)
|
||||
- **What it does**: Automatically detects and debugs the corresponding test executable
|
||||
- **How to use**: Open any `test_*.cpp` file and press `F5`
|
||||
|
||||
#### 🔧 **Specific Test Configurations**
|
||||
- **"Debug test_allocator"** - Memory allocation and deallocation debugging
|
||||
- **"Debug test_math"** - Mathematical functions and color calculations
|
||||
- **"Debug test_fastled"** - Core FastLED functionality and API
|
||||
- **"Debug test_hsv16"** - Color space conversions and accuracy
|
||||
- **"Debug test_corkscrew"** - LED layout and geometric calculations
|
||||
|
||||
#### 🎛️ **Advanced Configurations**
|
||||
- **"Debug with Specific Test Filter"** - Run only specific test cases
|
||||
- Example: Filter "allocator_inlined" to debug only those tests
|
||||
- **"Debug with Custom Args"** - Pass custom command-line arguments
|
||||
- Example: `--verbose`, `--list-test-cases`, etc.
|
||||
|
||||
### Debugging Features
|
||||
|
||||
#### Step Commands
|
||||
- **F5**: Continue execution
|
||||
- **F10**: Step over (execute current line)
|
||||
- **F11**: Step into (enter function calls)
|
||||
- **Shift+F11**: Step out (exit current function)
|
||||
- **Ctrl+Shift+F5**: Restart debugging
|
||||
- **Shift+F5**: Stop debugging
|
||||
|
||||
#### Variable Inspection
|
||||
- **Variables panel**: See all local variables and their values
|
||||
- **Watch panel**: Add expressions to monitor continuously
|
||||
- **Hover inspection**: Hover over variables to see values
|
||||
- **Debug console**: Type expressions to evaluate
|
||||
|
||||
### Build Tasks for Debugging
|
||||
|
||||
Access via `Ctrl+Shift+P` → "Tasks: Run Task":
|
||||
|
||||
- **"Build FastLED Tests"** - Quick incremental build (default: `Ctrl+Shift+B`)
|
||||
- **"Build FastLED Tests (Full)"** - Complete build including Python tests
|
||||
- **"Build Single Test"** - Build and run specific test only
|
||||
- **"Clean Build"** - Remove all build artifacts and rebuild
|
||||
|
||||
### Debugging Tips for FastLED
|
||||
|
||||
#### Memory Issues
|
||||
```cpp
|
||||
// Use test_allocator for memory debugging
|
||||
// Set breakpoints on allocate/deallocate functions
|
||||
// Watch pointer values in Variables panel
|
||||
// Monitor memory patterns for corruption
|
||||
```
|
||||
|
||||
#### Color Conversion Issues
|
||||
```cpp
|
||||
// Use test_hsv16 for color space debugging
|
||||
// Watch RGB/HSV values during conversion
|
||||
// Step through algorithms with F11
|
||||
// Compare expected vs actual in Watch panel
|
||||
```
|
||||
|
||||
#### Template Debugging
|
||||
```cpp
|
||||
// Clang generates excellent template debug info
|
||||
// Step into template functions with F11
|
||||
// Watch template parameters in Variables panel
|
||||
// Use Call Stack to understand instantiation chain
|
||||
```
|
||||
|
||||
### Technical Setup
|
||||
|
||||
#### Clang + GDB Benefits
|
||||
This setup provides the **best of both worlds**:
|
||||
- **Clang's superior symbol generation**: Better template debugging, modern C++ support
|
||||
- **GDB's mature debugging features**: Robust breakpoint handling, memory inspection
|
||||
- **Cross-platform compatibility**: Works on Linux, macOS, Windows
|
||||
- **FastLED optimization**: Unified compilation testing with `FASTLED_ALL_SRC=1`
|
||||
|
||||
#### Debug Build Configuration
|
||||
The FastLED test system automatically uses optimal debug settings:
|
||||
- **Compiler**: Clang (when available) or GCC fallback
|
||||
- **Debug info**: `-g3` (full debug information including macros)
|
||||
- **Optimization**: `-O0` (no optimization for accurate debugging)
|
||||
- **Frame pointers**: `-fno-omit-frame-pointer` (for accurate stack traces)
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
#### "Program not found" Error
|
||||
1. **Build first**: Run "Build FastLED Tests" task
|
||||
2. **Check executable exists**: `ls tests/.build/bin/test_*`
|
||||
3. **Verify path**: Ensure executable path in launch.json is correct
|
||||
|
||||
#### Breakpoints Not Hit
|
||||
1. **Check file paths**: Ensure source file matches executable
|
||||
2. **Verify compilation**: Code might be optimized out
|
||||
3. **Try function breakpoints**: Sometimes more reliable than line breakpoints
|
||||
|
||||
#### Variables Show "Optimized Out"
|
||||
1. **Use debug build**: Already configured with `-O0` (no optimization)
|
||||
2. **Check variable scope**: Variable might be out of scope
|
||||
3. **Try different breakpoint**: Move breakpoint to where variable is active
|
||||
|
||||
For complete debugging documentation, see [DEBUGGING.md](DEBUGGING.md).
|
||||
|
||||
|
||||
## Once you are done
|
||||
* run `./test`
|
||||
* run `./lint`
|
||||
* Then submit your code via a git pull request.
|
||||
|
||||
|
||||
## Going deeper
|
||||
|
||||
[ADVANCED_DEVELOPMENT.md](https://github.com/FastLED/FastLED/blob/master/ADVANCED_DEVELOPMENT.md)
|
||||
@@ -0,0 +1,203 @@
|
||||
# Corkscrew Pipeline: computeTile and multiSample Integration
|
||||
|
||||
## Overview
|
||||
|
||||
The FastLED corkscrew allows the user to write to a regular rectangular buffer and have it displayd on a dense corkscrew of LEDs.
|
||||
|
||||
Dense 144LED @ 3.28 are cheap and readily avialable. They are cheap, have high density.
|
||||
|
||||
## Pipeline Components
|
||||
|
||||
### 1. User Paints to XY Grid
|
||||
|
||||
Users create patterns on a 2D rectangular grid (`fl::Grid<CRGB>`) using standard XY coordinates.
|
||||
|
||||
```cpp
|
||||
// Grid dimensions calculated from corkscrew parameters
|
||||
uint16_t width = input.calculateWidth(); // LEDs per turn
|
||||
uint16_t height = input.calculateHeight(); // Total vertical segments
|
||||
fl::Grid<CRGB> sourceGrid(width, height);
|
||||
```
|
||||
|
||||
### 2. LED Projection: Corkscrew → XY Grid
|
||||
|
||||
Each LED in the corkscrew has a corresponding floating-point position on the XY grid:
|
||||
|
||||
```cpp
|
||||
// From corkscrew.cpp - calculateLedPositionExtended
|
||||
vec2f calculateLedPosition(uint16_t ledIndex, uint16_t numLeds, uint16_t width) {
|
||||
const float ledProgress = static_cast<float>(ledIndex) / static_cast<float>(numLeds - 1);
|
||||
const uint16_t row = ledIndex / width; // Which turn (vertical position)
|
||||
const uint16_t remainder = ledIndex % width; // Position within turn
|
||||
const float alpha = static_cast<float>(remainder) / static_cast<float>(width);
|
||||
|
||||
const float width_pos = ledProgress * numLeds;
|
||||
const float height_pos = static_cast<float>(row) + alpha;
|
||||
|
||||
return vec2f(width_pos, height_pos);
|
||||
}
|
||||
```
|
||||
|
||||
### 3. computeTile: Splat Pixel Rendering
|
||||
|
||||
The `splat` function implements the "computeTile" concept by converting floating-point positions to `Tile2x2_u8` structures representing neighbor intensities:
|
||||
|
||||
```cpp
|
||||
// From splat.cpp
|
||||
Tile2x2_u8 splat(vec2f xy) {
|
||||
// 1) Get integer cell indices
|
||||
int16_t cx = static_cast<int16_t>(floorf(xy.x));
|
||||
int16_t cy = static_cast<int16_t>(floorf(xy.y));
|
||||
|
||||
// 2) Calculate fractional offsets [0..1)
|
||||
float fx = xy.x - cx;
|
||||
float fy = xy.y - cy;
|
||||
|
||||
// 3) Compute bilinear weights for 4 neighbors
|
||||
float w_ll = (1 - fx) * (1 - fy); // lower-left
|
||||
float w_lr = fx * (1 - fy); // lower-right
|
||||
float w_ul = (1 - fx) * fy; // upper-left
|
||||
float w_ur = fx * fy; // upper-right
|
||||
|
||||
// 4) Build Tile2x2_u8 with weights as intensities [0..255]
|
||||
Tile2x2_u8 out(vec2<int16_t>(cx, cy));
|
||||
out.lower_left() = to_uint8(w_ll);
|
||||
out.lower_right() = to_uint8(w_lr);
|
||||
out.upper_left() = to_uint8(w_ul);
|
||||
out.upper_right() = to_uint8(w_ur);
|
||||
|
||||
return out;
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Tile2x2_u8: Neighbor Intensity Representation
|
||||
|
||||
The `Tile2x2_u8` structure represents the sampling strength from the four nearest neighbors:
|
||||
|
||||
```cpp
|
||||
class Tile2x2_u8 {
|
||||
uint8_t mTile[2][2]; // 4 neighbor intensities [0..255]
|
||||
vec2<int16_t> mOrigin; // Base grid coordinate (cx, cy)
|
||||
|
||||
// Access methods for the 4 neighbors:
|
||||
uint8_t& lower_left(); // (0,0) - weight for pixel at (cx, cy)
|
||||
uint8_t& lower_right(); // (1,0) - weight for pixel at (cx+1, cy)
|
||||
uint8_t& upper_left(); // (0,1) - weight for pixel at (cx, cy+1)
|
||||
uint8_t& upper_right(); // (1,1) - weight for pixel at (cx+1, cy+1)
|
||||
};
|
||||
```
|
||||
|
||||
### 5. Cylindrical Wrapping with Tile2x2_u8_wrap
|
||||
|
||||
For corkscrew mapping, the tile needs cylindrical wrapping:
|
||||
|
||||
```cpp
|
||||
// From corkscrew.cpp - at_wrap()
|
||||
Tile2x2_u8_wrap Corkscrew::at_wrap(float i) const {
|
||||
Tile2x2_u8 tile = at_splat_extrapolate(i); // Get base tile
|
||||
Tile2x2_u8_wrap::Entry data[2][2];
|
||||
vec2i16 origin = tile.origin();
|
||||
|
||||
for (uint8_t x = 0; x < 2; x++) {
|
||||
for (uint8_t y = 0; y < 2; y++) {
|
||||
vec2i16 pos = origin + vec2i16(x, y);
|
||||
// Apply cylindrical wrapping to x-coordinate
|
||||
pos.x = fmodf(pos.x, static_cast<float>(mState.width));
|
||||
data[x][y] = {pos, tile.at(x, y)}; // {position, intensity}
|
||||
}
|
||||
}
|
||||
return Tile2x2_u8_wrap(data);
|
||||
}
|
||||
```
|
||||
|
||||
### 6. multiSample: Weighted Color Sampling
|
||||
|
||||
The `readFromMulti` method implements the "multiSample" concept by using tile intensities to determine sampling strength:
|
||||
|
||||
```cpp
|
||||
// From corkscrew.cpp - readFromMulti()
|
||||
void Corkscrew::readFromMulti(const fl::Grid<CRGB>& source_grid) const {
|
||||
for (size_t led_idx = 0; led_idx < mInput.numLeds; ++led_idx) {
|
||||
// Get wrapped tile for this LED position
|
||||
Tile2x2_u8_wrap tile = at_wrap(static_cast<float>(led_idx));
|
||||
|
||||
uint32_t r_accum = 0, g_accum = 0, b_accum = 0;
|
||||
uint32_t total_weight = 0;
|
||||
|
||||
// Sample from each of the 4 neighbors
|
||||
for (uint8_t x = 0; x < 2; x++) {
|
||||
for (uint8_t y = 0; y < 2; y++) {
|
||||
const auto& entry = tile.at(x, y);
|
||||
vec2i16 pos = entry.first; // Grid position
|
||||
uint8_t weight = entry.second; // Sampling intensity [0..255]
|
||||
|
||||
if (inBounds(source_grid, pos)) {
|
||||
CRGB sample_color = source_grid.at(pos.x, pos.y);
|
||||
|
||||
// Weighted accumulation
|
||||
r_accum += static_cast<uint32_t>(sample_color.r) * weight;
|
||||
g_accum += static_cast<uint32_t>(sample_color.g) * weight;
|
||||
b_accum += static_cast<uint32_t>(sample_color.b) * weight;
|
||||
total_weight += weight;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Final color = weighted average
|
||||
CRGB final_color = CRGB::Black;
|
||||
if (total_weight > 0) {
|
||||
final_color.r = static_cast<uint8_t>(r_accum / total_weight);
|
||||
final_color.g = static_cast<uint8_t>(g_accum / total_weight);
|
||||
final_color.b = static_cast<uint8_t>(b_accum / total_weight);
|
||||
}
|
||||
|
||||
mCorkscrewLeds[led_idx] = final_color;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Complete Pipeline Flow
|
||||
|
||||
```
|
||||
1. User draws → XY Grid (CRGB values at integer coordinates)
|
||||
↓
|
||||
2. LED projection → vec2f position on grid (floating-point)
|
||||
↓
|
||||
3. computeTile → Tile2x2_u8 (4 neighbor intensities)
|
||||
(splat) ↓
|
||||
4. Wrap for → Tile2x2_u8_wrap (cylindrical coordinates + intensities)
|
||||
cylinder ↓
|
||||
5. multiSample → Weighted sampling from 4 neighbors
|
||||
(readFromMulti) ↓
|
||||
6. Final LED color → CRGB value for corkscrew LED
|
||||
```
|
||||
|
||||
## Key Insights
|
||||
|
||||
### Sub-Pixel Accuracy
|
||||
The system achieves sub-pixel accuracy by:
|
||||
- Using floating-point LED positions on the grid
|
||||
- Converting to bilinear weights for 4 nearest neighbors
|
||||
- Performing weighted color sampling instead of nearest-neighbor
|
||||
|
||||
### Cylindrical Mapping
|
||||
- X-coordinates wrap around the cylinder circumference
|
||||
- Y-coordinates represent vertical position along the helix
|
||||
- Width = LEDs per turn, Height = total vertical segments
|
||||
|
||||
### Anti-Aliasing
|
||||
The weighted sampling naturally provides anti-aliasing:
|
||||
- Sharp grid patterns become smoothly interpolated on the corkscrew
|
||||
- Reduces visual artifacts from the discrete→continuous→discrete mapping
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
- **Memory**: O(W×H) for grid (O(N) for corkscrew LEDs where O(N) <= O(WxH) == O(WxW))
|
||||
- **Computation**: O(N) with 4 samples per LED (constant factor)
|
||||
- **Quality**: Sub-pixel accurate with built-in anti-aliasing
|
||||
|
||||
|
||||
## Future Work
|
||||
|
||||
Often led strips are soldered together. These leaves a gap between the other
|
||||
leds on the strip. This gab should be accounted for to maximize spatial accuracy with rendering straight lines (e.g. text).
|
||||
@@ -0,0 +1,368 @@
|
||||
# FastLED Audio Reactive Component Analysis & Improvement Recommendations
|
||||
|
||||
## Executive Summary
|
||||
|
||||
After comprehensive analysis of the FastLED codebase's audio reactive components and research into industry best practices, significant opportunities exist to enhance performance, accuracy, and musical responsiveness. The current implementation provides a solid foundation but lacks advanced algorithms for rhythm analysis, perceptual audio weighting, and embedded system optimization.
|
||||
|
||||
## Current Implementation Analysis
|
||||
|
||||
### Strengths
|
||||
|
||||
**1. Clean Architecture**
|
||||
- Well-structured `AudioReactive` class with clear separation of concerns
|
||||
- WLED-compatible 16-bin frequency analysis
|
||||
- Proper attack/decay smoothing with configurable parameters
|
||||
- Support for various audio input sources through `AudioSample` abstraction
|
||||
|
||||
**2. Core Processing Pipeline**
|
||||
- FFT-based frequency analysis using optimized KissFFT implementation
|
||||
- Automatic Gain Control (AGC) with sensitivity adjustment
|
||||
- Volume and peak detection with RMS computation
|
||||
- Basic beat detection with cooldown period
|
||||
|
||||
**3. Embedded-Friendly Design**
|
||||
- Fixed-size frequency bins (16 channels)
|
||||
- Stack-allocated FFT buffers using appropriate FastLED containers
|
||||
- No dynamic memory allocation in audio processing path
|
||||
- Configurable sample rates and processing parameters
|
||||
|
||||
### Critical Limitations
|
||||
|
||||
**1. Primitive Beat Detection Algorithm**
|
||||
```cpp
|
||||
// Current implementation (lines 184-214 in audio_reactive.cpp)
|
||||
void AudioReactive::detectBeat(fl::u32 currentTimeMs) {
|
||||
// Simple threshold-based detection
|
||||
if (currentVolume > mPreviousVolume + mVolumeThreshold &&
|
||||
currentVolume > 5.0f) {
|
||||
mCurrentData.beatDetected = true;
|
||||
}
|
||||
}
|
||||
```
|
||||
This approach suffers from:
|
||||
- False positives on sustained loud passages
|
||||
- Poor detection of complex rhythmic patterns
|
||||
- No tempo tracking or beat prediction
|
||||
- Inability to distinguish kick drums from other transients
|
||||
|
||||
**2. Lack of Perceptual Audio Weighting**
|
||||
- No psychoacoustic modeling for human auditory perception
|
||||
- Linear frequency binning doesn't match musical octaves
|
||||
- Missing A-weighting or loudness compensation
|
||||
- No consideration of critical bands or masking effects
|
||||
|
||||
**3. Limited Spectral Analysis**
|
||||
- Fixed 16-bin resolution may miss important frequency details
|
||||
- No spectral flux calculation for onset detection
|
||||
- Missing harmonic analysis for musical note detection
|
||||
- No adaptive frequency mapping based on musical content
|
||||
|
||||
**4. Insufficient Real-Time Optimization**
|
||||
- Missing SIMD optimization for FFT calculations
|
||||
- No circular buffer implementation for streaming
|
||||
- Potential cache misses in frequency bin processing
|
||||
- No memory layout optimization for embedded constraints
|
||||
|
||||
## Industry Best Practices Research
|
||||
|
||||
### Advanced Beat Detection Algorithms
|
||||
|
||||
**1. Spectral Flux-Based Onset Detection**
|
||||
- Analyzes changes in frequency domain magnitude over time
|
||||
- Significantly more accurate than simple amplitude thresholding
|
||||
- Reduces false positives by 60-80% compared to current method
|
||||
|
||||
**2. Multi-Band Onset Detection**
|
||||
- Separate onset detection for bass, mid, and treble frequencies
|
||||
- Enables detection of polyrhythmic patterns
|
||||
- Better handling of complex musical arrangements
|
||||
|
||||
**3. Tempo Tracking & Beat Prediction**
|
||||
- Autocorrelation-based tempo estimation
|
||||
- Predictive beat scheduling for smoother visual sync
|
||||
- Adaptive tempo tracking for live performances
|
||||
|
||||
### Perceptual Audio Processing
|
||||
|
||||
**1. Psychoacoustic Weighting**
|
||||
- Equal-loudness contours (ISO 226) for frequency weighting
|
||||
- Critical band analysis matching human auditory filters
|
||||
- Masking models to emphasize perceptually significant content
|
||||
|
||||
**2. Musical Frequency Mapping**
|
||||
- Logarithmic frequency scales matching musical octaves
|
||||
- Mel-scale or bark-scale frequency distribution
|
||||
- Adaptive binning based on musical key detection
|
||||
|
||||
### Embedded System Optimization
|
||||
|
||||
**1. Memory Management**
|
||||
- Circular buffers for audio streaming (single-threaded)
|
||||
- Stack-allocated processing with compile-time sizing
|
||||
- Preallocated object pools for temporary calculations
|
||||
- Memory alignment for efficient access patterns
|
||||
|
||||
**2. Platform-Specific Optimization**
|
||||
- ARM DSP instructions for FFT acceleration (Teensy 3.x)
|
||||
- AVR assembly optimizations for 8-bit platforms
|
||||
- ESP32 dual-core utilization for audio processing isolation
|
||||
- Cache-friendly data structures for modern ARM cores
|
||||
|
||||
## Specific Improvement Recommendations
|
||||
|
||||
### Phase 1: Enhanced Beat Detection (High Impact, Low Risk)
|
||||
|
||||
**1. Implement Spectral Flux Algorithm**
|
||||
```cpp
|
||||
class SpectralFluxDetector {
|
||||
#ifdef SKETCH_HAS_LOTS_OF_MEMORY
|
||||
fl::array<float, 16> mPreviousMagnitudes;
|
||||
fl::array<float, 32> mFluxHistory; // For advanced smoothing
|
||||
#else
|
||||
fl::array<float, 16> mPreviousMagnitudes;
|
||||
// No history on memory-constrained platforms
|
||||
#endif
|
||||
float mFluxThreshold;
|
||||
|
||||
bool detectOnset(const FFTBins& current, const FFTBins& previous);
|
||||
float calculateSpectralFlux(const FFTBins& current, const FFTBins& previous);
|
||||
};
|
||||
```
|
||||
|
||||
**2. Multi-Band Beat Detection**
|
||||
```cpp
|
||||
struct BeatDetectors {
|
||||
#ifdef SKETCH_HAS_LOTS_OF_MEMORY
|
||||
SpectralFluxDetector bass; // 20-200 Hz
|
||||
SpectralFluxDetector mid; // 200-2000 Hz
|
||||
SpectralFluxDetector treble; // 2000-20000 Hz
|
||||
#else
|
||||
SpectralFluxDetector combined; // Single detector for memory-constrained
|
||||
#endif
|
||||
};
|
||||
```
|
||||
|
||||
**3. Adaptive Threshold Algorithm**
|
||||
- Dynamic threshold based on recent audio history
|
||||
- Separate thresholds for different frequency bands
|
||||
- Tempo-aware cooldown periods
|
||||
|
||||
### Phase 2: Perceptual Audio Enhancement (Medium Impact, Medium Risk)
|
||||
|
||||
**1. A-Weighting Implementation**
|
||||
```cpp
|
||||
class PerceptualWeighting {
|
||||
static constexpr fl::array<float, 16> A_WEIGHTING_COEFFS = {
|
||||
// Frequency-dependent weighting factors
|
||||
};
|
||||
|
||||
#ifdef SKETCH_HAS_LOTS_OF_MEMORY
|
||||
fl::array<float, 16> mLoudnessHistory; // For dynamic compensation
|
||||
#endif
|
||||
|
||||
void applyAWeighting(AudioData& data) const;
|
||||
void applyLoudnessCompensation(AudioData& data, float referenceLevel) const;
|
||||
};
|
||||
```
|
||||
|
||||
**2. Musical Frequency Mapping**
|
||||
- Replace linear frequency bins with logarithmic musical scale
|
||||
- Implement note detection and harmonic analysis
|
||||
- Add chord progression recognition for advanced effects
|
||||
|
||||
**3. Dynamic Range Enhancement**
|
||||
- Intelligent compression based on musical content
|
||||
- Frequency-dependent AGC with musical awareness
|
||||
- Adaptive noise gate with spectral subtraction
|
||||
|
||||
### Phase 3: Embedded Performance Optimization (High Impact, Medium Risk)
|
||||
|
||||
**1. SIMD Acceleration**
|
||||
```cpp
|
||||
#ifdef ARM_NEON
|
||||
void processFFTBins_NEON(const FFTBins& input, AudioData& output);
|
||||
#endif
|
||||
|
||||
#ifdef __AVR__
|
||||
void processFFTBins_AVR_ASM(const FFTBins& input, AudioData& output);
|
||||
#endif
|
||||
```
|
||||
|
||||
**2. Circular Buffer for Audio Streaming**
|
||||
```cpp
|
||||
template<typename T, fl::size Size>
|
||||
class CircularBuffer {
|
||||
fl::size mWriteIndex{0};
|
||||
fl::size mReadIndex{0};
|
||||
fl::array<T, Size> mBuffer;
|
||||
|
||||
#ifdef SKETCH_HAS_LOTS_OF_MEMORY
|
||||
fl::array<T, Size> mBackupBuffer; // For advanced buffering strategies
|
||||
#endif
|
||||
|
||||
public:
|
||||
void push(const T& item);
|
||||
bool pop(T& item);
|
||||
bool full() const;
|
||||
bool empty() const;
|
||||
};
|
||||
```
|
||||
|
||||
**3. Cache-Friendly Data Layout**
|
||||
```cpp
|
||||
// Structure of Arrays for better cache locality
|
||||
struct AudioDataSoA {
|
||||
alignas(32) fl::array<float, 16> frequencyBins;
|
||||
alignas(32) fl::array<float, 16> previousBins;
|
||||
|
||||
#ifdef SKETCH_HAS_LOTS_OF_MEMORY
|
||||
alignas(32) fl::array<float, 16> spectralFlux;
|
||||
alignas(32) fl::array<float, 32> tempoHistory;
|
||||
#endif
|
||||
};
|
||||
```
|
||||
|
||||
### Phase 4: Advanced Features (Variable Impact, High Risk)
|
||||
|
||||
**1. Machine Learning Integration** (Only available with `SKETCH_HAS_LOTS_OF_MEMORY`)
|
||||
- Lightweight neural network for beat classification
|
||||
- Real-time genre detection for adaptive processing
|
||||
- Learned tempo tracking with user feedback
|
||||
|
||||
**2. Multi-Channel Analysis**
|
||||
- Stereo phase analysis for spatial effects
|
||||
- Mid/Side processing for enhanced separation
|
||||
- Cross-correlation for echo and reverb detection
|
||||
|
||||
**3. Musical Structure Analysis**
|
||||
- Verse/chorus detection for macro-level effects
|
||||
- Build-up and drop detection for electronic music
|
||||
- Key and chord progression analysis
|
||||
|
||||
## Implementation Priority Matrix
|
||||
|
||||
| Feature | Impact | Embedded Feasibility | Implementation Risk | Priority |
|
||||
|---------|--------|---------------------|-------------------|----------|
|
||||
| Spectral Flux Beat Detection | High | High | Low | 1 |
|
||||
| Multi-Band Onset Detection | High | High | Low | 2 |
|
||||
| A-Weighting & Loudness | Medium | High | Low | 3 |
|
||||
| SIMD Optimization | High | Medium | Medium | 4 |
|
||||
| Musical Frequency Mapping | Medium | Medium | Medium | 5 |
|
||||
| Lock-Free Buffers | High | Medium | High | 6 |
|
||||
| Tempo Tracking | Medium | Medium | High | 7 |
|
||||
| Machine Learning Features | Low | Low | High | 8 |
|
||||
|
||||
## Technical Specifications
|
||||
|
||||
### Memory Requirements
|
||||
- **Current**: ~2KB for AudioReactive instance
|
||||
- **Phase 1**: +1KB for enhanced beat detection
|
||||
- **Phase 2**: +2KB for perceptual processing
|
||||
- **Phase 3**: +4KB for optimization buffers
|
||||
|
||||
### Performance Targets
|
||||
- **Current**: ~200 MIPS on ARM Cortex-M4
|
||||
- **Optimized**: <150 MIPS with platform-specific optimizations
|
||||
- **Latency**: <5ms end-to-end processing (single-threaded)
|
||||
- **Beat Detection Accuracy**: >90% (vs ~60% current)
|
||||
|
||||
### Platform Compatibility
|
||||
- **Platforms with `SKETCH_HAS_LOTS_OF_MEMORY`**: Full feature support including ML and advanced buffering
|
||||
- ESP32, Teensy 4.x, STM32F4+ with sufficient RAM
|
||||
- **Memory-Constrained Platforms**: Basic feature set optimized for minimal memory usage
|
||||
- Arduino UNO, ATtiny, basic STM32, older Teensy models
|
||||
- **Feature Selection**: Automatic based on `SKETCH_HAS_LOTS_OF_MEMORY` define rather than manual platform detection
|
||||
|
||||
## Conclusion
|
||||
|
||||
The FastLED audio reactive component has solid foundations but significant room for improvement. The recommended phased approach balances feature enhancement with embedded system constraints, prioritizing high-impact, low-risk improvements first. Implementation of spectral flux-based beat detection alone would dramatically improve musical responsiveness while maintaining compatibility with existing hardware platforms.
|
||||
|
||||
The proposed enhancements align with industry best practices for real-time audio processing while respecting the unique constraints of embedded LED controller applications. Each phase provides measurable improvements in audio analysis accuracy and visual synchronization quality.
|
||||
|
||||
## C++ Design Audit Findings
|
||||
|
||||
### Critical Design Issues Identified
|
||||
|
||||
**1. Namespace and Container Usage Corrections**
|
||||
- **Issue**: Original recommendations used `std::` containers instead of FastLED's `fl::` namespace
|
||||
- **Fix**: All code examples now use `fl::array<T, N>` instead of `std::array<T, N>`
|
||||
- **Impact**: Ensures compatibility with FastLED's embedded-optimized container implementations
|
||||
|
||||
**2. Single-Threaded Architecture Recognition**
|
||||
- **Issue**: Original document included lock-free and atomic programming recommendations inappropriate for single-threaded embedded systems
|
||||
- **Fix**: Removed `std::atomic` and lock-free suggestions, replaced with single-threaded circular buffers
|
||||
- **Impact**: Reduces complexity and memory overhead while matching actual deployment constraints
|
||||
|
||||
**3. Platform-Specific Optimization Alignment**
|
||||
- **Issue**: Recommended AVX2 optimizations not available on embedded platforms
|
||||
- **Fix**: Focused on ARM DSP instructions (Teensy 3.x) and AVR assembly optimizations
|
||||
- **Impact**: Provides realistic performance improvements for actual target hardware
|
||||
|
||||
**4. Memory Management Best Practices**
|
||||
- **Issue**: Initial recommendations didn't fully leverage FastLED's memory management patterns
|
||||
- **Fix**: Emphasized `fl::array` for fixed-size allocations and proper alignment macros
|
||||
- **Impact**: Better cache performance and reduced memory fragmentation
|
||||
|
||||
### Recommended Design Patterns
|
||||
|
||||
**1. Memory-Aware Container Strategy**
|
||||
```cpp
|
||||
// Preferred: Fixed-size arrays for predictable memory usage
|
||||
fl::array<float, 16> mFrequencyBins;
|
||||
|
||||
#ifdef SKETCH_HAS_LOTS_OF_MEMORY
|
||||
// Advanced features with additional memory
|
||||
fl::array<float, 64> mExtendedHistory;
|
||||
#endif
|
||||
|
||||
// Avoid: Dynamic vectors in real-time audio path
|
||||
// fl::vector<float> mFrequencyBins; // Don't use for real-time
|
||||
```
|
||||
|
||||
**2. Memory-Based Feature Selection**
|
||||
```cpp
|
||||
#ifdef SKETCH_HAS_LOTS_OF_MEMORY
|
||||
// Full feature set for high-memory platforms
|
||||
#define AUDIO_ENABLE_SPECTRAL_FLUX 1
|
||||
#define AUDIO_ENABLE_MULTI_BAND 1
|
||||
#define AUDIO_ENABLE_TEMPO_TRACKING 1
|
||||
#else
|
||||
// Basic features only for memory-constrained platforms
|
||||
#define AUDIO_ENABLE_SPECTRAL_FLUX 0
|
||||
#define AUDIO_ENABLE_MULTI_BAND 0
|
||||
#define AUDIO_ENABLE_TEMPO_TRACKING 0
|
||||
#endif
|
||||
```
|
||||
|
||||
**3. Memory-Aligned Data Structures**
|
||||
```cpp
|
||||
struct AudioProcessingData {
|
||||
alignas(32) fl::array<float, 16> frequencyBins;
|
||||
alignas(16) fl::array<float, 16> previousBins;
|
||||
|
||||
#ifdef SKETCH_HAS_LOTS_OF_MEMORY
|
||||
alignas(32) fl::array<float, 16> spectralFluxBuffer;
|
||||
alignas(16) fl::array<float, 8> tempoTracker;
|
||||
#endif
|
||||
// Alignment ensures efficient memory access patterns
|
||||
};
|
||||
```
|
||||
|
||||
### Implementation Priority Adjustments
|
||||
|
||||
| Original Priority | Adjusted Priority | Reason |
|
||||
|------------------|------------------|---------|
|
||||
| Lock-Free Buffers (6) | Removed | Single-threaded architecture |
|
||||
| SIMD Optimization (4) | Platform-Specific Opt (4) | Match actual hardware capabilities |
|
||||
| Machine Learning (8) | Simplified ML (8) | Embedded memory constraints |
|
||||
|
||||
### Final Recommendations
|
||||
|
||||
**1. Use `SKETCH_HAS_LOTS_OF_MEMORY` for all feature gating instead of platform-specific defines**
|
||||
**2. Start with Phase 1 implementations using FastLED containers**
|
||||
**3. Leverage existing FastLED SIMD patterns from `lib8tion` for guidance**
|
||||
**4. Test on memory-constrained platforms (Arduino UNO) first to ensure baseline functionality**
|
||||
**5. Use ESP32 dual-core architecture for audio processing isolation when available**
|
||||
**6. Always provide fallback implementations for memory-constrained platforms**
|
||||
|
||||
The corrected design approach ensures all recommendations are implementable within FastLED's existing architecture while providing meaningful performance improvements for real-world embedded applications.
|
||||
@@ -0,0 +1,494 @@
|
||||
# FastLED Native Platform Configuration
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the comprehensive `platform.json` configuration for FastLED's native build platform. The design leverages the Python `ziglang` package to provide a unified, cross-platform compilation environment that integrates seamlessly with the existing `build_unit.toml` structure.
|
||||
|
||||
## Key Design Principles
|
||||
|
||||
### 1. **Unified Toolchain Integration**
|
||||
- **Ziglang Foundation**: Uses `python -m ziglang` commands matching the current `build_unit.toml` configuration
|
||||
- **Cross-Platform Consistency**: Single toolchain across Linux, macOS, Windows with automatic target detection
|
||||
- **Direct TOML Mapping**: Toolchain commands directly map between `platform.json` and `build_unit.toml`
|
||||
|
||||
### 2. **Comprehensive Architecture Support**
|
||||
- **x86_64 Targets**: Linux, macOS, Windows with appropriate system-specific flags
|
||||
- **ARM64 Targets**: Linux (generic), macOS (Apple Silicon), Windows with optimized CPU settings
|
||||
- **RISC-V 64**: Linux support for emerging architecture
|
||||
- **Auto-Detection**: Intelligent target selection for simplified development workflow
|
||||
|
||||
### 3. **Arduino Package Format Compliance**
|
||||
- **PlatformIO Integration**: Full compatibility with PlatformIO package management
|
||||
- **Framework Support**: Both FastLED-stub and Arduino-stub frameworks
|
||||
- **Package Dependencies**: Proper toolchain, framework, and tool dependency management
|
||||
- **Versioning Strategy**: Semantic versioning with appropriate version constraints
|
||||
|
||||
### 4. **Build System Flexibility**
|
||||
- **Flag Inheritance**: Base, debug, release, and test flag configurations
|
||||
- **Platform-Specific Linking**: Automatic application of OS-specific link flags
|
||||
- **Compiler Caching**: Optional ccache/sccache integration for performance
|
||||
- **Build System Support**: CMake, Ninja, and direct compilation compatibility
|
||||
|
||||
## Complete Platform.json Structure
|
||||
|
||||
The full `platform.json` includes:
|
||||
|
||||
1. **Board Definitions** (8 comprehensive targets):
|
||||
- `native-x64-linux`, `native-x64-macos`, `native-x64-windows`
|
||||
- `native-arm64-linux`, `native-arm64-macos`, `native-arm64-windows`
|
||||
- `native-riscv64-linux`, `native-auto` (intelligent detection)
|
||||
|
||||
2. **Toolchain Configuration**:
|
||||
- Complete ziglang tool mapping (CC, CXX, AR, LD, etc.)
|
||||
- Build flag hierarchies (base, debug, release, test)
|
||||
- Platform-specific link flags for each OS
|
||||
|
||||
3. **Package Management**:
|
||||
- Toolchain dependencies with proper versioning
|
||||
- Optional tool packages (cmake, ninja, caching)
|
||||
- Framework packages for Arduino compatibility
|
||||
|
||||
4. **Development Features**:
|
||||
- Integrated debugger support (GDB for Linux/Windows, LLDB for macOS)
|
||||
- Environment variable customization support
|
||||
- Target compatibility specifications
|
||||
|
||||
## Platform.json Configuration
|
||||
|
||||
The complete platform.json follows Arduino package format with these key sections:
|
||||
|
||||
### Core Metadata
|
||||
```json
|
||||
{
|
||||
"name": "FastLED Native Platform",
|
||||
"title": "Native Host Platform with Zig Toolchain",
|
||||
"description": "Cross-platform native compilation using Zig's bundled Clang",
|
||||
"version": "1.0.0",
|
||||
"packageType": "platform",
|
||||
"spec": {
|
||||
"owner": "fastled",
|
||||
"id": "native-zig",
|
||||
"name": "native-zig"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Framework Definitions
|
||||
```json
|
||||
{
|
||||
"frameworks": {
|
||||
"fastled-stub": {
|
||||
"package": "framework-fastled-stub",
|
||||
"script": "builder/frameworks/fastled-stub.py"
|
||||
},
|
||||
"arduino-stub": {
|
||||
"package": "framework-arduino-stub",
|
||||
"script": "builder/frameworks/arduino-stub.py"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Board Configurations
|
||||
Each board includes complete configuration:
|
||||
- **Hardware specs**: MCU, RAM, Flash, CPU frequency
|
||||
- **Target triple**: Zig compilation target
|
||||
- **Build flags**: Platform-specific preprocessor defines
|
||||
- **Debug tools**: Appropriate debugger (GDB/LLDB) configuration
|
||||
- **Framework support**: Compatible frameworks list
|
||||
|
||||
Example board definition:
|
||||
```json
|
||||
{
|
||||
"native-x64-linux": {
|
||||
"name": "Native x86_64 Linux",
|
||||
"mcu": "x86_64",
|
||||
"fcpu": "3000000000L",
|
||||
"ram": 17179869184,
|
||||
"flash": 1099511627776,
|
||||
"vendor": "Generic",
|
||||
"product": "Native x86_64",
|
||||
"frameworks": ["fastled-stub", "arduino-stub"],
|
||||
"build": {
|
||||
"target": "x86_64-linux-gnu",
|
||||
"toolchain": "ziglang",
|
||||
"cpu": "x86_64",
|
||||
"mcu": "native",
|
||||
"f_cpu": "3000000000L",
|
||||
"extra_flags": [
|
||||
"-DNATIVE_PLATFORM=1",
|
||||
"-DSTUB_PLATFORM=1",
|
||||
"-DFASTLED_NATIVE_X64=1",
|
||||
"-DFASTLED_NATIVE_LINUX=1"
|
||||
]
|
||||
},
|
||||
"upload": { "protocol": "native" },
|
||||
"debug": {
|
||||
"tools": {
|
||||
"gdb": {
|
||||
"server": {
|
||||
"executable": "gdb",
|
||||
"arguments": ["$PROG_PATH"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Toolchain Mapping
|
||||
Direct integration with ziglang commands:
|
||||
```json
|
||||
{
|
||||
"toolchain-ziglang": {
|
||||
"CC": "python -m ziglang cc",
|
||||
"CXX": "python -m ziglang c++",
|
||||
"AR": "python -m ziglang ar",
|
||||
"LD": "python -m ziglang c++",
|
||||
"OBJCOPY": "python -m ziglang objcopy",
|
||||
"NM": "python -m ziglang nm",
|
||||
"STRIP": "python -m ziglang strip",
|
||||
"RANLIB": "python -m ziglang ranlib"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Build Flag Hierarchies
|
||||
```json
|
||||
{
|
||||
"build_flags": {
|
||||
"base": [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-Wno-deprecated-register",
|
||||
"-Wno-backslash-newline-escape",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
"-DSTUB_PLATFORM=1",
|
||||
"-DNATIVE_PLATFORM=1"
|
||||
],
|
||||
"debug": [
|
||||
"-g3",
|
||||
"-Og",
|
||||
"-DDEBUG=1",
|
||||
"-DFASTLED_DEBUG=1"
|
||||
],
|
||||
"release": [
|
||||
"-O3",
|
||||
"-DNDEBUG=1",
|
||||
"-flto"
|
||||
],
|
||||
"test": [
|
||||
"-DFASTLED_UNIT_TEST=1",
|
||||
"-DFASTLED_FORCE_NAMESPACE=1",
|
||||
"-DFASTLED_TESTING=1",
|
||||
"-DFASTLED_USE_PROGMEM=0",
|
||||
"-DARDUINO=10808",
|
||||
"-DFASTLED_USE_STUB_ARDUINO=1",
|
||||
"-DSKETCH_HAS_LOTS_OF_MEMORY=1",
|
||||
"-DFASTLED_STUB_IMPL=1",
|
||||
"-DFASTLED_USE_JSON_UI=1",
|
||||
"-DFASTLED_NO_AUTO_NAMESPACE=1",
|
||||
"-DFASTLED_NO_PINMAP=1",
|
||||
"-DHAS_HARDWARE_PIN_SUPPORT=1",
|
||||
"-DFASTLED_DEBUG_LEVEL=1",
|
||||
"-DFASTLED_NO_ATEXIT=1",
|
||||
"-DDOCTEST_CONFIG_NO_EXCEPTIONS_BUT_WITH_ALL_ASSERTS=1"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Platform-Specific Linking
|
||||
```json
|
||||
{
|
||||
"link_flags": {
|
||||
"base": [],
|
||||
"linux": [
|
||||
"-pthread",
|
||||
"-lm",
|
||||
"-ldl"
|
||||
],
|
||||
"macos": [
|
||||
"-pthread",
|
||||
"-lm"
|
||||
],
|
||||
"windows": [
|
||||
"-mconsole",
|
||||
"-nodefaultlibs",
|
||||
"-unwindlib=libunwind",
|
||||
"-nostdlib++",
|
||||
"-lc++",
|
||||
"-lkernel32",
|
||||
"-luser32",
|
||||
"-lgdi32",
|
||||
"-ladvapi32"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with build_unit.toml
|
||||
|
||||
The `platform.json` is designed to seamlessly integrate with the existing `build_unit.toml` configuration:
|
||||
|
||||
### Toolchain Mapping
|
||||
|
||||
The `[tools]` section in `build_unit.toml` directly maps to the ziglang toolchain defined in `platform.json`:
|
||||
|
||||
- `cpp_compiler = ["python", "-m", "ziglang", "c++"]` → `CXX: "python -m ziglang c++"`
|
||||
- `linker = ["python", "-m", "ziglang", "c++"]` → `LD: "python -m ziglang c++"`
|
||||
- `archiver = ["python", "-m", "ziglang", "ar"]` → `AR: "python -m ziglang ar"`
|
||||
|
||||
### Build Flag Integration
|
||||
|
||||
The platform flags complement the TOML configuration:
|
||||
|
||||
- **Base flags**: Defined in both `platform.json` and `build_unit.toml` for consistency
|
||||
- **Platform-specific flags**: Windows-specific flags from TOML are automatically applied
|
||||
- **Test flags**: Unit test defines from TOML are included in the platform test configuration
|
||||
|
||||
### Target Selection
|
||||
|
||||
The platform supports automatic target detection or explicit target selection:
|
||||
|
||||
```bash
|
||||
# Auto-detect target
|
||||
platformio run -e native-auto
|
||||
|
||||
# Explicit target selection
|
||||
platformio run -e native-x64-linux
|
||||
platformio run -e native-arm64-macos
|
||||
platformio run -e native-x64-windows
|
||||
```
|
||||
|
||||
## Integration Benefits
|
||||
|
||||
### Seamless TOML Integration
|
||||
The design ensures perfect compatibility with existing `build_unit.toml`:
|
||||
- Toolchain commands match exactly: `["python", "-m", "ziglang", "c++"]`
|
||||
- Build flags complement rather than duplicate TOML configuration
|
||||
- Platform-specific flags automatically applied based on detection
|
||||
|
||||
### Enhanced Development Workflow
|
||||
- **Single Command Compilation**: `platformio run -e native-auto`
|
||||
- **Cross-Compilation Support**: Build for different targets from any host
|
||||
- **Testing Integration**: Full unit test support with proper namespace isolation
|
||||
- **Debug Builds**: Integrated debugging with platform-appropriate tools
|
||||
|
||||
### Future-Proof Architecture
|
||||
- **Extensible Design**: Easy addition of new architectures and targets
|
||||
- **WebAssembly Ready**: Structure supports future WASM integration
|
||||
- **Container Builds**: Foundation for Docker/Podman integration
|
||||
- **Static Analysis**: Framework for clang-tidy and analyzer integration
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Compilation
|
||||
|
||||
```bash
|
||||
# Compile for current platform (auto-detect)
|
||||
platformio run -e native-auto
|
||||
|
||||
# Compile for specific platform
|
||||
platformio run -e native-x64-linux
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Run unit tests
|
||||
platformio test -e native-auto
|
||||
|
||||
# Run tests with specific build flags
|
||||
platformio test -e native-x64-linux --test-port=native
|
||||
```
|
||||
|
||||
### Cross-Compilation
|
||||
|
||||
```bash
|
||||
# Compile for ARM64 Linux from x86_64
|
||||
platformio run -e native-arm64-linux
|
||||
|
||||
# Compile for Windows from Linux
|
||||
platformio run -e native-x64-windows
|
||||
```
|
||||
|
||||
### Debug Builds
|
||||
|
||||
```bash
|
||||
# Debug build with symbols
|
||||
platformio run -e native-auto --build-type=debug
|
||||
|
||||
# Release build with optimization
|
||||
platformio run -e native-auto --build-type=release
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Custom Board Definition
|
||||
|
||||
```json
|
||||
{
|
||||
"native-custom": {
|
||||
"name": "Custom Native Target",
|
||||
"mcu": "custom",
|
||||
"build": {
|
||||
"target": "x86_64-linux-musl",
|
||||
"extra_flags": [
|
||||
"-DCUSTOM_DEFINE=1",
|
||||
"-static"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
The platform respects environment variables for customization:
|
||||
|
||||
- `ZIG_TARGET`: Override target triple (e.g., `x86_64-linux-musl`)
|
||||
- `ZIG_CPU`: Override CPU type (e.g., `x86_64_v3`)
|
||||
- `FASTLED_NATIVE_FLAGS`: Additional compiler flags
|
||||
|
||||
### Build System Integration
|
||||
|
||||
The platform integrates with multiple build systems:
|
||||
|
||||
- **PlatformIO**: Primary integration with comprehensive board support
|
||||
- **CMake**: Compatible with existing CMake configurations
|
||||
- **Direct Compilation**: Can be used directly with ziglang commands
|
||||
|
||||
## Complete Board Definitions
|
||||
|
||||
The platform.json includes 8 comprehensive board definitions:
|
||||
|
||||
1. **native-x64-linux**: x86_64 Linux with GNU target
|
||||
2. **native-x64-macos**: x86_64 macOS with Apple-specific configuration
|
||||
3. **native-x64-windows**: x86_64 Windows with MinGW compatibility
|
||||
4. **native-arm64-linux**: ARM64 Linux with Cortex-A76 optimization
|
||||
5. **native-arm64-macos**: ARM64 macOS with Apple M1 optimization
|
||||
6. **native-arm64-windows**: ARM64 Windows with appropriate runtime
|
||||
7. **native-riscv64-linux**: RISC-V 64-bit Linux support
|
||||
8. **native-auto**: Automatic target detection and selection
|
||||
|
||||
## Tool Dependencies
|
||||
|
||||
The platform includes comprehensive tool support:
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": {
|
||||
"python-ziglang": {
|
||||
"type": "compiler",
|
||||
"version": "^0.13.0",
|
||||
"description": "Zig toolchain with bundled Clang/LLVM",
|
||||
"install": { "pip": "ziglang" }
|
||||
},
|
||||
"cmake": {
|
||||
"type": "build-system",
|
||||
"version": "^3.21.0",
|
||||
"optional": true
|
||||
},
|
||||
"ninja": {
|
||||
"type": "build-system",
|
||||
"version": "^1.11.0",
|
||||
"optional": true
|
||||
},
|
||||
"ccache": {
|
||||
"type": "compiler-cache",
|
||||
"version": "^4.0.0",
|
||||
"optional": true
|
||||
},
|
||||
"sccache": {
|
||||
"type": "compiler-cache",
|
||||
"version": "^0.7.0",
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Target Compatibility
|
||||
|
||||
Platform compatibility specifications ensure proper deployment:
|
||||
|
||||
```json
|
||||
{
|
||||
"target_compatibility": {
|
||||
"x86_64-linux-gnu": {
|
||||
"glibc": ">=2.31",
|
||||
"kernel": ">=5.4.0"
|
||||
},
|
||||
"x86_64-macos-none": {
|
||||
"macos": ">=11.0"
|
||||
},
|
||||
"x86_64-windows-gnu": {
|
||||
"windows": ">=10"
|
||||
},
|
||||
"aarch64-linux-gnu": {
|
||||
"glibc": ">=2.31",
|
||||
"kernel": ">=5.4.0"
|
||||
},
|
||||
"aarch64-macos-none": {
|
||||
"macos": ">=11.0"
|
||||
},
|
||||
"aarch64-windows-gnu": {
|
||||
"windows": ">=10"
|
||||
},
|
||||
"riscv64-linux-gnu": {
|
||||
"glibc": ">=2.31",
|
||||
"kernel": ">=5.10.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Ziglang not found**: Ensure `ziglang` Python package is installed
|
||||
2. **Target not supported**: Check `zig targets` for available targets
|
||||
3. **Linking errors**: Verify platform-specific link flags in configuration
|
||||
|
||||
### Debug Information
|
||||
|
||||
```bash
|
||||
# Check available targets
|
||||
zig targets
|
||||
|
||||
# Verify toolchain installation
|
||||
python -m ziglang version
|
||||
|
||||
# Test compilation
|
||||
python -m ziglang c++ --version
|
||||
```
|
||||
|
||||
## Implementation Status
|
||||
|
||||
This design is **ready for implementation** and provides:
|
||||
- ✅ **Complete board definitions** for all major native platforms
|
||||
- ✅ **Full toolchain integration** with existing ziglang setup
|
||||
- ✅ **Comprehensive build configurations** for all use cases
|
||||
- ✅ **Arduino package format compliance** for PlatformIO compatibility
|
||||
- ✅ **Debug and development tool integration** for professional development
|
||||
- ✅ **Future-proof extensible architecture** for additional platforms
|
||||
|
||||
The configuration can be deployed alongside the existing build system for gradual validation and testing before full migration.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **WebAssembly Support**: Integration with FastLED WASM compilation
|
||||
- **Static Analysis**: Integration with clang-tidy and static analyzers
|
||||
- **Performance Profiling**: Built-in profiling support for optimization
|
||||
- **Container Builds**: Docker/Podman integration for reproducible builds
|
||||
- **Package Registry**: Publication to Arduino package index for distribution
|
||||
|
||||
## Conclusion
|
||||
|
||||
This platform.json design provides a comprehensive, production-ready configuration for FastLED's native build platform. It seamlessly integrates with the existing `build_unit.toml` while providing the standardized Arduino package format required for broader ecosystem compatibility. The design is thoroughly tested conceptually and ready for implementation with proper validation and testing procedures.
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,414 @@
|
||||
# FastLED UI Layout Management System Design
|
||||
|
||||
## Overview
|
||||
|
||||
This document outlines the design for a comprehensive, responsive UI layout management system for the FastLED WebAssembly compiler interface. The system provides dynamic layout calculation, state management, and container coordination to ensure optimal UI presentation across different screen sizes and orientations.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ UILayoutPlacementManager │
|
||||
│ (Main Orchestrator) │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ • Coordinates all layout operations │
|
||||
│ • Applies CSS styling and grid layouts │
|
||||
│ • Handles layout mode transitions │
|
||||
│ • Manages container visibility │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌───────────────┼───────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────────┐ ┌─────────────────┐ ┌─────────────────────┐
|
||||
│ LayoutStateManager │ │ ResizeCoordinator│ │ ColumnValidator │
|
||||
│ (State & Logic) │ │ (Event Handling) │ │ (Optimization) │
|
||||
├─────────────────────┤ ├─────────────────┤ ├─────────────────────┤
|
||||
│ • Viewport tracking │ │ • Resize events │ │ • Layout validation │
|
||||
│ • Layout calculation│ │ • Debouncing │ │ • Performance check │
|
||||
│ • State transitions │ │ • Event dispatch │ │ • Efficiency metrics│
|
||||
│ • Container states │ │ • Coordination │ │ • Optimization hints│
|
||||
└─────────────────────┘ └─────────────────┘ └─────────────────────┘
|
||||
```
|
||||
|
||||
## Component Specifications
|
||||
|
||||
### 1. UILayoutPlacementManager
|
||||
|
||||
**Purpose**: Main orchestrator for all layout operations
|
||||
**Location**: `src/platforms/wasm/compiler/modules/ui_layout_placement_manager.js`
|
||||
|
||||
**Key Responsibilities:**
|
||||
- Apply layout styles to DOM elements
|
||||
- Coordinate between state manager and UI components
|
||||
- Handle layout mode transitions (mobile, tablet, desktop, ultrawide)
|
||||
- Manage container visibility and grid configurations
|
||||
|
||||
**Public API:**
|
||||
```javascript
|
||||
class UILayoutPlacementManager {
|
||||
constructor()
|
||||
applyLayout()
|
||||
forceLayoutUpdate()
|
||||
validateAndOptimizeLayout()
|
||||
getLayoutInfo()
|
||||
updateContentMetrics(totalGroups, totalElements)
|
||||
}
|
||||
```
|
||||
|
||||
### 2. LayoutStateManager
|
||||
|
||||
**Purpose**: Unified state management for layout calculations
|
||||
**Location**: `src/platforms/wasm/compiler/modules/layout_state_manager.js`
|
||||
|
||||
**Key Features:**
|
||||
- Centralized layout state storage
|
||||
- Viewport and breakpoint detection
|
||||
- Canvas and UI column calculations
|
||||
- Container state tracking
|
||||
- Atomic state updates with change events
|
||||
|
||||
**State Schema:**
|
||||
```javascript
|
||||
{
|
||||
mode: 'mobile' | 'tablet' | 'desktop' | 'ultrawide',
|
||||
viewportWidth: number,
|
||||
availableWidth: number,
|
||||
canvasSize: number,
|
||||
uiColumns: number,
|
||||
uiColumnWidth: number,
|
||||
uiTotalWidth: number,
|
||||
canExpand: boolean,
|
||||
container2Visible: boolean,
|
||||
totalGroups: number,
|
||||
totalElements: number
|
||||
}
|
||||
```
|
||||
|
||||
### 3. ResizeCoordinator
|
||||
|
||||
**Purpose**: Coordinated resize event handling
|
||||
**Location**: `src/platforms/wasm/compiler/modules/resize_coordinator.js`
|
||||
|
||||
**Features:**
|
||||
- Debounced resize event handling
|
||||
- Cross-component resize coordination
|
||||
- Performance-optimized event dispatch
|
||||
- Race condition prevention
|
||||
|
||||
### 4. ColumnValidator
|
||||
|
||||
**Purpose**: Layout optimization and validation
|
||||
**Location**: `src/platforms/wasm/compiler/modules/column_validator.js`
|
||||
|
||||
**Capabilities:**
|
||||
- Layout efficiency analysis
|
||||
- Column width optimization
|
||||
- Content density validation
|
||||
- Performance recommendations
|
||||
|
||||
## Layout Modes
|
||||
|
||||
### Mobile (≤ 768px)
|
||||
- **Layout**: Single column (1×N grid)
|
||||
- **Canvas**: Full width, centered
|
||||
- **UI**: Stacked below canvas
|
||||
- **Containers**: Only primary UI container visible
|
||||
|
||||
### Tablet (769px - 1199px)
|
||||
- **Layout**: Two columns (2×N grid)
|
||||
- **Canvas**: Left side, fixed width
|
||||
- **UI**: Right side, flexible width
|
||||
- **Containers**: Primary UI container only
|
||||
|
||||
### Desktop (1200px - 1599px)
|
||||
- **Layout**: Two columns (2×N grid)
|
||||
- **Canvas**: Left side, larger fixed width
|
||||
- **UI**: Right side, flexible width
|
||||
- **Containers**: Primary UI container only
|
||||
|
||||
### Ultrawide (≥ 1600px)
|
||||
- **Layout**: Three columns (3×N grid)
|
||||
- **Canvas**: Center, fixed width
|
||||
- **UI**: Left and right sides, flexible widths
|
||||
- **Containers**: Both primary and secondary UI containers
|
||||
|
||||
## Configuration
|
||||
|
||||
### Layout Constants
|
||||
```javascript
|
||||
{
|
||||
minCanvasSize: 320,
|
||||
maxCanvasSize: 800,
|
||||
minUIColumnWidth: 280,
|
||||
maxUIColumnWidth: 400,
|
||||
horizontalGap: 40,
|
||||
verticalGap: 20,
|
||||
containerPadding: 40,
|
||||
maxUIColumns: 3,
|
||||
preferredUIColumnWidth: 320,
|
||||
canvasExpansionRatio: 0.6,
|
||||
minContentRatio: 0.4
|
||||
}
|
||||
```
|
||||
|
||||
### Breakpoints
|
||||
```javascript
|
||||
{
|
||||
mobile: { max: 768 },
|
||||
tablet: { min: 769, max: 1199 },
|
||||
desktop: { min: 1200, max: 1599 },
|
||||
ultrawide: { min: 1600 }
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Todos
|
||||
|
||||
### Phase 1: Core Infrastructure ✅
|
||||
- [x] Create LayoutStateManager with unified state handling
|
||||
- [x] Implement ResizeCoordinator for event management
|
||||
- [x] Build ColumnValidator for layout optimization
|
||||
- [x] Refactor UILayoutPlacementManager to use new components
|
||||
|
||||
### Phase 2: State Management ✅
|
||||
- [x] Implement atomic state updates with change events
|
||||
- [x] Add container state tracking and visibility management
|
||||
- [x] Create viewport detection and breakpoint handling
|
||||
- [x] Add content metrics tracking (groups, elements)
|
||||
|
||||
### Phase 3: Layout Calculation ✅
|
||||
- [x] Implement responsive canvas sizing algorithms
|
||||
- [x] Add dynamic UI column width calculation
|
||||
- [x] Create layout mode transition logic
|
||||
- [x] Add grid template generation for CSS
|
||||
|
||||
### Phase 4: Event Coordination ✅
|
||||
- [x] Implement debounced resize handling
|
||||
- [x] Add cross-component event coordination
|
||||
- [x] Create performance-optimized event dispatch
|
||||
- [x] Add race condition prevention
|
||||
|
||||
### Phase 5: Optimization & Validation ✅
|
||||
- [x] Create layout efficiency analysis
|
||||
- [x] Implement column width optimization
|
||||
- [x] Add content density validation
|
||||
- [x] Create performance monitoring and recommendations
|
||||
|
||||
### Phase 6: Integration & Polish ✅
|
||||
- [x] Integrate with existing UI manager
|
||||
- [x] Add backward compatibility for legacy APIs
|
||||
- [x] Implement proper error handling and fallbacks
|
||||
- [x] Add comprehensive logging and debugging support
|
||||
|
||||
### Phase 7: Testing & Documentation
|
||||
- [ ] Create comprehensive unit tests for all components
|
||||
- [ ] Add integration tests for layout scenarios
|
||||
- [ ] Create visual regression tests for different screen sizes
|
||||
- [ ] Add performance benchmarks and optimization tests
|
||||
- [ ] Document API usage and configuration options
|
||||
|
||||
### Phase 8: Advanced Features
|
||||
- [ ] Add animation support for layout transitions
|
||||
- [ ] Implement custom breakpoint configuration
|
||||
- [ ] Add layout templates and presets
|
||||
- [ ] Create advanced grid layout options
|
||||
- [ ] Add accessibility features and ARIA support
|
||||
|
||||
### Phase 9: Performance Optimization
|
||||
- [ ] Implement layout caching for repeated calculations
|
||||
- [ ] Add virtual scrolling for large UI element lists
|
||||
- [ ] Optimize DOM manipulation and CSS application
|
||||
- [ ] Add lazy loading for non-visible UI components
|
||||
- [ ] Implement progressive enhancement for slower devices
|
||||
|
||||
### Phase 10: Extensibility
|
||||
- [ ] Create plugin system for custom layout algorithms
|
||||
- [ ] Add theme and styling customization APIs
|
||||
- [ ] Implement layout export/import functionality
|
||||
- [ ] Create developer tools for layout debugging
|
||||
- [ ] Add real-time layout editing capabilities
|
||||
|
||||
## Benefits
|
||||
|
||||
### Performance
|
||||
- **Debounced Events**: Prevents excessive layout recalculations
|
||||
- **Atomic Updates**: Eliminates partial state inconsistencies
|
||||
- **Efficient DOM**: Minimizes CSS recalculations and reflows
|
||||
- **Optimized Calculations**: Smart caching and memoization
|
||||
|
||||
### Maintainability
|
||||
- **Separation of Concerns**: Each component has a single responsibility
|
||||
- **Unified State**: Single source of truth for layout information
|
||||
- **Modular Design**: Components can be tested and modified independently
|
||||
- **Clear APIs**: Well-defined interfaces between components
|
||||
|
||||
### User Experience
|
||||
- **Responsive Design**: Seamless adaptation to any screen size
|
||||
- **Smooth Transitions**: Coordinated layout changes without flicker
|
||||
- **Optimal Layouts**: Intelligent space utilization across devices
|
||||
- **Fast Rendering**: Performance-optimized for real-time applications
|
||||
|
||||
### Developer Experience
|
||||
- **Type Safety**: Comprehensive JSDoc annotations for IDE support
|
||||
- **Debugging Tools**: Built-in logging and state inspection
|
||||
- **Extensible**: Easy to add new layout modes and features
|
||||
- **Documentation**: Clear API documentation and usage examples
|
||||
|
||||
## JSON UI Persistent State
|
||||
|
||||
### State Management Architecture
|
||||
The system now includes comprehensive JSON-based state persistence implemented in commit `86d9b7d5d`. This enables:
|
||||
|
||||
- **Persistent Layout State**: All layout configurations stored as JSON with atomic updates
|
||||
- **UI Element State Tracking**: Complete state management for UI elements with change events
|
||||
- **Recording/Playback System**: Full UI interaction recording and replay capabilities
|
||||
- **Container State Management**: Visibility and configuration state for all UI containers
|
||||
|
||||
### Core State Components
|
||||
|
||||
#### LayoutStateManager
|
||||
**Location**: `src/platforms/wasm/compiler/modules/layout_state_manager.js:487`
|
||||
|
||||
Provides centralized JSON state management with:
|
||||
```javascript
|
||||
{
|
||||
mode: 'mobile' | 'tablet' | 'desktop' | 'ultrawide',
|
||||
viewportWidth: number,
|
||||
availableWidth: number,
|
||||
canvasSize: number,
|
||||
uiColumns: number,
|
||||
uiColumnWidth: number,
|
||||
uiTotalWidth: number,
|
||||
canExpand: boolean,
|
||||
container2Visible: boolean,
|
||||
totalGroups: number,
|
||||
totalElements: number,
|
||||
containers: {
|
||||
'ui-controls': { visible: boolean, columns: number, width: number },
|
||||
'ui-controls-2': { visible: boolean, columns: number, width: number }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### UIRecorder
|
||||
**Location**: `src/platforms/wasm/compiler/modules/ui_recorder.js:517`
|
||||
|
||||
Records UI state changes as JSON events:
|
||||
```javascript
|
||||
{
|
||||
recording: {
|
||||
version: "1.0",
|
||||
startTime: "ISO8601",
|
||||
endTime: "ISO8601",
|
||||
metadata: { recordingId: string, layoutMode: string, totalDuration: number }
|
||||
},
|
||||
events: [
|
||||
{
|
||||
timestamp: number,
|
||||
type: 'add' | 'update' | 'remove',
|
||||
elementId: string,
|
||||
data: { elementType?: string, value?: any, previousValue?: any, elementConfig?: Object }
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### UIPlayback
|
||||
**Location**: `src/platforms/wasm/compiler/modules/ui_playback.js:596`
|
||||
|
||||
Reconstructs UI from JSON state with:
|
||||
- Event-by-event playback with timing control
|
||||
- Speed control and pause/resume functionality
|
||||
- Element state validation and restoration
|
||||
- Progress tracking and timeline navigation
|
||||
|
||||
### UI Reconstruction Benefits
|
||||
|
||||
Instead of manipulating DOM objects directly, the JSON state system enables:
|
||||
|
||||
1. **State-Driven Reconstruction**: Rebuild entire UI from current JSON state
|
||||
2. **Atomic Updates**: All changes applied atomically to prevent inconsistencies
|
||||
3. **Time Travel**: Navigate to any point in UI history via recorded states
|
||||
4. **State Validation**: Verify UI consistency against recorded state
|
||||
5. **Reproducible Layouts**: Exact recreation of UI configurations
|
||||
|
||||
### Usage Pattern for Reconstruction
|
||||
|
||||
```javascript
|
||||
// Get current UI state
|
||||
const currentState = layoutManager.stateManager.getState();
|
||||
|
||||
// Reconstruct UI from state instead of DOM manipulation
|
||||
function reconstructUI(state) {
|
||||
// Clear existing UI
|
||||
clearAllContainers();
|
||||
|
||||
// Rebuild from JSON state
|
||||
applyLayoutFromState(state);
|
||||
recreateElementsFromState(state);
|
||||
restoreContainerVisibility(state);
|
||||
}
|
||||
|
||||
// Apply state changes
|
||||
layoutManager.stateManager.updateState(newStateData);
|
||||
```
|
||||
|
||||
## Integration Points
|
||||
|
||||
### UI Manager Integration
|
||||
```javascript
|
||||
// In JsonUiManager constructor
|
||||
this.layoutManager = new UILayoutPlacementManager();
|
||||
|
||||
// Listen for layout changes
|
||||
this.layoutManager.stateManager.addStateChangeListener((changeEvent) => {
|
||||
const { state } = changeEvent;
|
||||
this.onLayoutChange(state.mode);
|
||||
});
|
||||
```
|
||||
|
||||
### Container Management
|
||||
```javascript
|
||||
// Check container visibility before placing elements
|
||||
const container2State = this.layoutManager.stateManager.getContainerState('ui-controls-2');
|
||||
if (container2State && container2State.visible) {
|
||||
// Use multi-container layout
|
||||
} else {
|
||||
// Use single-container layout
|
||||
}
|
||||
```
|
||||
|
||||
### Content Updates
|
||||
```javascript
|
||||
// Update content metrics when UI elements change
|
||||
this.layoutManager.updateContentMetrics(totalGroups, totalElements);
|
||||
|
||||
// Force layout recalculation when needed
|
||||
this.layoutManager.forceLayoutUpdate();
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Advanced Layout Modes
|
||||
- **Split-screen**: Side-by-side canvas and code editor
|
||||
- **Picture-in-picture**: Floating canvas with overlay UI
|
||||
- **Full-screen**: Immersive canvas mode with minimal UI
|
||||
- **Multi-monitor**: Extended layout across multiple displays
|
||||
|
||||
### Dynamic Content
|
||||
- **Adaptive UI**: UI elements that resize based on content
|
||||
- **Collapsible sections**: Expandable/collapsible UI groups
|
||||
- **Floating panels**: Draggable and dockable UI components
|
||||
- **Custom layouts**: User-defined layout configurations
|
||||
|
||||
### Accessibility
|
||||
- **Screen reader support**: Proper ARIA labels and navigation
|
||||
- **Keyboard navigation**: Full keyboard accessibility
|
||||
- **High contrast**: Theme support for visual accessibility
|
||||
- **Responsive text**: Scalable fonts and UI elements
|
||||
|
||||
---
|
||||
|
||||
*This design provides a robust foundation for responsive UI layout management in the FastLED WebAssembly compiler, ensuring optimal user experience across all device types and screen sizes.*
|
||||
@@ -0,0 +1,20 @@
|
||||
The MIT License (MIT)
|
||||
|
||||
Copyright (c) 2013 FastLED
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
|
||||
the Software, and to permit persons to whom the Software is furnished to do so,
|
||||
subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
||||
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
||||
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
@@ -0,0 +1,56 @@
|
||||
Platform Porting Guide
|
||||
==========================
|
||||
|
||||
# Fast porting for a new board on existing hardware
|
||||
|
||||
Sometimes "porting" FastLED simply consists of supplying new pin definitions for the given platform. For example, platforms/avr/fastpin_avr.h contains various pin definitions for all the AVR variant chipsets/boards that FastLED supports. Defining a set of pins involves setting up a set of definitions - for example here's one full set from the avr fastpin file:
|
||||
|
||||
```
|
||||
#elif defined(__AVR_ATmega1284P__) || defined(__AVR_ATmega644P__)
|
||||
|
||||
_FL_IO(A); _FL_IO(B); _FL_IO(C); _FL_IO(D);
|
||||
|
||||
#define MAX_PIN 31
|
||||
_FL_DEFPIN(0, 0, B); _FL_DEFPIN(1, 1, B); _FL_DEFPIN(2, 2, B); _FL_DEFPIN(3, 3, B);
|
||||
_FL_DEFPIN(4, 4, B); _FL_DEFPIN(5, 5, B); _FL_DEFPIN(6, 6, B); _FL_DEFPIN(7, 7, B);
|
||||
_FL_DEFPIN(8, 0, D); _FL_DEFPIN(9, 1, D); _FL_DEFPIN(10, 2, D); _FL_DEFPIN(11, 3, D);
|
||||
_FL_DEFPIN(12, 4, D); _FL_DEFPIN(13, 5, D); _FL_DEFPIN(14, 6, D); _FL_DEFPIN(15, 7, D);
|
||||
_FL_DEFPIN(16, 0, C); _FL_DEFPIN(17, 1, C); _FL_DEFPIN(18, 2, C); _FL_DEFPIN(19, 3, C);
|
||||
_FL_DEFPIN(20, 4, C); _FL_DEFPIN(21, 5, C); _FL_DEFPIN(22, 6, C); _FL_DEFPIN(23, 7, C);
|
||||
_FL_DEFPIN(24, 0, A); _FL_DEFPIN(25, 1, A); _FL_DEFPIN(26, 2, A); _FL_DEFPIN(27, 3, A);
|
||||
_FL_DEFPIN(28, 4, A); _FL_DEFPIN(29, 5, A); _FL_DEFPIN(30, 6, A); _FL_DEFPIN(31, 7, A);
|
||||
|
||||
#define HAS_HARDWARE_PIN_SUPPORT 1
|
||||
```
|
||||
|
||||
The ```_FL_IO``` macro is used to define the port registers for the platform while the ```_FL_DEFPIN``` macro is used to define pins. The parameters to the macro are the pin number, the bit on the port that represents that pin, and the port identifier itself. On some platforms, like the AVR, ports are identified by letter. On other platforms, like arm, ports are identified by number.
|
||||
|
||||
The ```HAS_HARDWARE_PIN_SUPPORT``` define tells the rest of the FastLED library that there is hardware pin support available. There may be other platform specific defines for things like hardware SPI ports and such.
|
||||
|
||||
## Setting up the basic files/folders
|
||||
|
||||
* Create platform directory (e.g. platforms/arm/kl26)
|
||||
* Create configuration header led_sysdefs_arm_kl26.h:
|
||||
* Define platform flags (like FASTLED_ARM/FASTLED_TEENSY)
|
||||
* Define configuration parameters re: interrupts, or clock doubling
|
||||
* Include extar system header files if needed
|
||||
* Create main platform include, fastled_arm_kl26.h
|
||||
* Include the various other header files as needed
|
||||
* Modify led_sysdefs.h to conditionally include platform sysdefs header file
|
||||
* Modify platforms.h to conditionally include platform fastled header
|
||||
|
||||
## Porting fastpin.h
|
||||
|
||||
The heart of the FastLED library is the fast pin access. This is a templated class that provides 1-2 cycle pin access, bypassing digital write and other such things. As such, this will usually be the first bit of the library that you will want to port when moving to a new platform. Once you have FastPIN up and running then you can do some basic work like testing toggles or running bit-bang'd SPI output.
|
||||
|
||||
There's two low level FastPin classes. There's the base FastPIN template class, and then there is FastPinBB which is for bit-banded access on those MCUs that support bitbanding. Note that the bitband class is optional and primarily useful in the implementation of other functionality internal to the platform. This file is also where you would do the pin to port/bit mapping defines.
|
||||
|
||||
Explaining how the macros work and should be used is currently beyond the scope of this document.
|
||||
|
||||
## Porting fastspi.h
|
||||
|
||||
This is where you define the low level interface to the hardware SPI system (including a writePixels method that does a bunch of housekeeping for writing led data). Use the fastspi_nop.h file as a reference for the methods that need to be implemented. There are ofteh other useful methods that can help with the internals of the SPI code, I recommend taking a look at how the various platforms implement their SPI classes.
|
||||
|
||||
## Porting clockless.h
|
||||
|
||||
This is where you define the code for the clockless controllers. Across ARM platforms this will usually be fairly similar - though different arm platforms will have different clock sources that you can/should use.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,510 @@
|
||||
# FastLED - The Universal LED Library
|
||||
|
||||
**Transform any microcontroller into an LED powerhouse.**
|
||||
|
||||
Drive **30,000+ LEDs** on high-end devices • **Sub-$1 compatibility** on tiny chips • **Background rendering** for responsive apps • **Nearly every LED chipset supported** • [**#2 most popular Arduino library**](https://docs.arduino.cc/libraries/)
|
||||
|
||||
[](https://www.ardu-badge.com/FastLED) [](https://github.com/FastLED/FastLED/actions/workflows/build.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_unit_test.yml) [](http://fastled.io/docs) [](https://www.reddit.com/r/FastLED/)
|
||||
|
||||
|
||||
## ⚡ Get Blinking in 30 Seconds
|
||||
|
||||
```cpp
|
||||
#include <FastLED.h>
|
||||
#define NUM_LEDS 60
|
||||
CRGB leds[NUM_LEDS];
|
||||
|
||||
void setup() {
|
||||
FastLED.addLeds<WS2812, 6>(leds, NUM_LEDS);
|
||||
}
|
||||
|
||||
void loop() {
|
||||
leds[0] = CRGB::Red; FastLED.show(); delay(500);
|
||||
leds[0] = CRGB::Blue; FastLED.show(); delay(500);
|
||||
}
|
||||
```
|
||||
|
||||
**✅ Works on Arduino, ESP32, Teensy, Raspberry Pi, and 50+ other platforms**
|
||||
|
||||
## 🚀 Why FastLED?
|
||||
|
||||
| **Massive Scale** | **Tiny Footprint** | **Background Rendering** | **Universal** |
|
||||
|-------------------|---------------------|--------------------------|---------------|
|
||||
| Drive 30,000 LEDs on Teensy 4.1 | Runs on $0.50 ATtiny chips | ESP32/Teensy render while you code | Works on 50+ platforms |
|
||||
| 50 parallel strips on Teensy | <2KB on Arduino Uno | Never miss user input | Nearly every LED chipset |
|
||||
|
||||
**🎯 Performance**: Zero-cost global brightness • High-performance 8-bit math, memory efficient on platforms that need it.
|
||||
**🔧 Developer Experience**: Quick platform switching • Extensive examples • Active community support
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [🆕 Latest Feature](#-latest-feature)
|
||||
- [⭐ Community Growth](#-community-growth)
|
||||
- [🆕 Latest Features](#-latest-features)
|
||||
- [🌍 Platform Support](#-platform-support)
|
||||
- [📦 Installation](#-installation)
|
||||
- [📚 Documentation & Support](#-documentation--support)
|
||||
- [🎮 Advanced Features](#-advanced-features)
|
||||
- [🤝 Contributing](#-contributing)
|
||||
|
||||
📊 <strong>Detailed Build Status</strong>
|
||||
|
||||
### Arduino Family
|
||||
**Core Boards:** [](https://github.com/FastLED/FastLED/actions/workflows/build_uno.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_nano_every.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_uno_r4_wifif.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_yun.yml)
|
||||
|
||||
**ARM Boards:** [](https://github.com/FastLED/FastLED/actions/workflows/build_due.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_digix.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_giga_r1.yml)*
|
||||
|
||||
**ATtiny Series:** [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny85.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny88.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny1604.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny1616.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny4313.yml)**
|
||||
|
||||
*Notes: * Giga-R1 support added in 3.9.14 • ** ATtiny4313 has limited memory (WS2812 Blink + APA102 examples only)
|
||||
|
||||
### Teensy Series
|
||||
**Standard Models:** [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy30.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy31.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensyLC.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy40.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy41.yml)
|
||||
|
||||
**Special Features:** [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy_octo.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy41_ofled.yml)
|
||||
|
||||
### NRF52 (Nordic)
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_adafruit_feather_nrf52840_sense.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_nrf52840_dk.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_adafruit_xiaoblesense.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_nrf52_xiaoblesense.yml)
|
||||
|
||||
*Note: NRF52 XiaoBLE board has mbed engine but doesn't compile against Arduino.h for unknown reasons.
|
||||
|
||||
### Apollo3 (Ambiq)
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_apollo3_red.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_apollo3_thing_explorable.yml)
|
||||
|
||||
*Beta support added in 3.10.2
|
||||
|
||||
### STM32 (STMicroelectronics)
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_bluepill.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_blackpill_stm32f4.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_maple_map.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_stm103tb.yml)
|
||||
|
||||
*Note: STM103TB has limited PlatformIO support
|
||||
|
||||
### Silicon Labs (SiLabs)
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_mgm240s_thingplusmatter.yml)
|
||||
|
||||
*MGM240 (EFR32MG24) support for Arduino Nano Matter and SparkFun Thing Plus Matter boards
|
||||
|
||||
### Raspberry Pi Pico
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_rp2040.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_rp2350.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_rp2350B.yml)
|
||||
|
||||
### ESP32 (Espressif)
|
||||
**ESP8266:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp8622.yml)
|
||||
|
||||
**ESP32 Classic:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32dev.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32wroom.yml)
|
||||
|
||||
**ESP32 S-Series:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32s2.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32s3.yml)
|
||||
|
||||
**ESP32 C-Series:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32c2.yml)* [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32c3.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32c5.yml)[](https://github.com/FastLED/FastLED/actions/workflows/build_esp32c6.yml)
|
||||
|
||||
**ESP32 Advanced:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32h2.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32p4.yml)
|
||||
|
||||
**Special Features:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32_i2s_ws2812.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp_extra_libs.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32dev_namespace.yml)
|
||||
|
||||
**QEMU Tests:** [](https://github.com/FastLED/FastLED/actions/workflows/qemu_esp32dev_test.yml) [](https://github.com/FastLED/FastLED/actions/workflows/qemu_esp32c3_test.yml) [](https://github.com/FastLED/FastLED/actions/workflows/qemu_esp32s3_test.yml)
|
||||
|
||||
**Legacy:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32dev_idf3.3.yml)
|
||||
|
||||
*Notes: * ESP32-C2 support added in 3.9.10 • [Espressif compatibility evaluation](https://github.com/espressif/arduino-esp32/blob/gh-pages/LIBRARIES_TEST.md)
|
||||
|
||||
### Specialty Platforms
|
||||
**x86:** [](https://github.com/FastLED/FastLED/actions/workflows/build_linux.yml)
|
||||
|
||||
**WebAssembly:** [](https://github.com/FastLED/FastLED/actions/workflows/build_wasm.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_wasm_compilers.yml)
|
||||
|
||||
### Library Size Validation
|
||||
**Core Platforms:** [](https://github.com/FastLED/FastLED/actions/workflows/check_attiny85.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_uno_size.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_esp32_size.yml)
|
||||
|
||||
**Teensy Platforms:** [](https://github.com/FastLED/FastLED/actions/workflows/check_teensylc_size.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_teensy30_size.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_teensy31_size.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_teensy41_size.yml)
|
||||
|
||||
|
||||
## ⭐ Community Growth
|
||||
|
||||
<a href="https://star-history.com/#fastled/fastled&Date">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=fastled/fastled&type=Date&theme=dark" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=fastled/fastled&type=Date" />
|
||||
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=fastled/fastled&type=Date" />
|
||||
</picture>
|
||||
</a>
|
||||
|
||||
## 🆕 Latest Features
|
||||
|
||||
## **FastLED 3.10.2: Corkscrew Mapping**
|
||||
|
||||
See [examples/FestivalStick/FestivalStick.ino](examples/FestivalStick/FestivalStick.ino)
|
||||
|
||||
https://github.com/user-attachments/assets/19af4b54-c189-482b-b13b-2d7f73efa4e0
|
||||
|
||||
### **FastLED 3.10.0: Animartrix Out of Beta**
|
||||
Advanced animation framework for complex LED visualizations and effects.
|
||||
|
||||
[](https://fastled.github.io/assets/fastled_3_10_animartrix.mp4)
|
||||
|
||||
### **FastLED 3.9.16: WaveFx / Multi Layer Compositing**
|
||||
Multi-layer compositing & time-based animation control for tech-artists.
|
||||
|
||||
|
||||
https://github.com/user-attachments/assets/ff8e0432-3e0d-47cc-a444-82ce27f562af
|
||||
|
||||
| **3.9.13** | **3.9.10** | **3.9.8** | **3.9.2** |
|
||||
|------------|------------|-----------|-----------|
|
||||
| [**HD107 Turbo**](#new-in-3913-hd107-turbo-40mhz-led-support)<br/>40MHz LED support | [**ESP32 SPI**](#new-in-3910-super-stable-ws2812-spi-driver-for-esp32)<br/>Super stable WS2812 driver | [**Massive Teensy**](#new-in-398---massive-teensy-41--40-ws2812-led-output)<br/>50 parallel pins on 4.1 | [**WS2812 Overclock**](#new-in-392---overclocking-of-ws2812)<br/>Up to 70% speed boost |
|
||||
|
||||
| **3.7.7** | **More Features** |
|
||||
|-----------|-------------------|
|
||||
| [**RGBW Support**](#new-in-377---rgbw-led-strip-support)<br/>White channel LED strips | [📋 Full Changelog](https://github.com/FastLED/FastLED/releases)<br/>[📺 Demo Videos](https://zackees.github.io/fastled-wasm/) |
|
||||
|
||||
[**📺 Live Demos**](https://zackees.github.io/fastled-wasm/) • [**📋 Full Changelog**](https://github.com/FastLED/FastLED/releases)
|
||||
|
||||
<details>
|
||||
<summary>📖 <strong>Detailed Feature Information</strong> (Click to expand)</summary>
|
||||
|
||||
## New in 3.10.2: Corkscrew Mapping
|
||||
|
||||
See [examples/FestivalStick/FestivalStick.ino](examples/FestivalStick/FestivalStick.ino)
|
||||
|
||||
https://github.com/user-attachments/assets/19af4b54-c189-482b-b13b-2d7f73efa4e0
|
||||
|
||||
## New in 3.10.0: Animartrix out of beta
|
||||
|
||||
[](https://fastled.github.io/assets/fastled_3_10_animartrix.mp4)
|
||||
|
||||
|
||||
## New in 3.9.16: WaveFx / Multi Layer Compositing / Time-based animation control
|
||||
|
||||
Video:
|
||||
|
||||
|
||||
|
||||
https://github.com/user-attachments/assets/ff8e0432-3e0d-47cc-a444-82ce27f562af
|
||||
|
||||
|
||||
|
||||
#### Major release for tech-artists!
|
||||
|
||||
Lots of improvements in this release, read the full [change list here](https://github.com/FastLED/FastLED/releases/tag/3.9.16)
|
||||
|
||||
#### Links
|
||||
|
||||
* This demo -> [FxWave2d](https://github.com/FastLED/FastLED/blob/master/examples/FxWave2d/FxWave2d.ino)
|
||||
* [Wave Simulation Library](https://github.com/FastLED/FastLED/blob/master/src/fl/wave_simulation.h)
|
||||
* [FireCylinder](https://github.com/FastLED/FastLED/blob/master/examples/FireCylinder/FireCylinder.ino)
|
||||
* Wraps around so that (0,y) ~= (width-1,y)
|
||||
* [TimeAlpha](https://github.com/FastLED/FastLED/blob/master/src/fl/time_alpha.h)
|
||||
* Precision control of animations with time-based alpha transition.
|
||||
|
||||
|
||||
|
||||
## New in 3.9.13: HD107 "Turbo" 40Mhz LED Support
|
||||
|
||||
|
||||
|
||||
## New in 3.9.12: WS2816 "HD" LED support
|
||||
|
||||
|
||||
|
||||
## New in 3.9.10: Super Stable WS2812 SPI driver for ESP32
|
||||
|
||||
|
||||
|
||||
## New in 3.9.9: 16-way Yves I2S parallel driver for the ESP32-S3
|
||||
|
||||
|
||||
|
||||
*Note some users find that newer versions of the ESP32 Arduino core (3.10) don't work very well, but older versions do, see [issue 1903](https://github.com/FastLED/FastLED/issues/1903)
|
||||
|
||||
## New in 3.9.8 - Massive Teensy 4.1 & 4.0 WS2812 LED output
|
||||
|
||||
* *Teensy 4.1: 50 parallel pins*
|
||||
* *Teensy 4.0: 42 parallel pins*
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## New in 3.9.2 - Overclocking of WS2812
|
||||
|
||||
Update: max overclock has been reported at +70%: https://www.reddit.com/r/FastLED/comments/1gkcb6m/fastled_FASTLED_OVERCLOCK_17/
|
||||
|
||||
### New in 3.7.7 - RGBW LED Strip Support
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
## 🌍 Platform Support
|
||||
|
||||
### Platform Categories
|
||||
| **Arduino Family** | **ESP32 Series** | **Teensy** | **ARM** | **Specialty** |
|
||||
|--------------------|------------------|------------|---------|---------------|
|
||||
| Uno, Nano, Mega<br/>Due, Giga R1, R4 | ESP32, S2, S3, C3<br/>C6, H2, P4 | 3.0, 3.1, 4.0, 4.1<br/>LC + OctoWS2811 | STM32, NRF52<br/>Apollo3, Silicon Labs | Raspberry Pi<br/>WASM, x86 |
|
||||
|
||||
**FastLED supports 50+ platforms!** From sub-$1 ATtiny chips to high-end Teensy 4.1 with 50 parallel outputs.
|
||||
|
||||
## 📦 Installation
|
||||
|
||||
### Quick Install Options
|
||||
|
||||
| **Arduino IDE** | **PlatformIO** | **Package Managers** |
|
||||
|-----------------|----------------|----------------------|
|
||||
| Library Manager → Search "FastLED" → Install | `lib_deps = fastled/FastLED` | `pio pkg install --library "fastled/FastLED"` |
|
||||
| Or install [latest release .zip](https://github.com/FastLED/FastLED/releases/latest/) | Add to `platformio.ini` | Command line installation |
|
||||
|
||||
### Template Projects
|
||||
|
||||
- **🎯 [Arduino + PlatformIO Starter](https://github.com/FastLED/PlatformIO-Starter)** - Best of both worlds, works with both IDEs
|
||||
- **🚀 [FastLED Examples](examples)** - 100+ ready-to-run demos
|
||||
- **🌐 [Web Compiler](https://zackees.github.io/fastled-wasm/)** - Test in your browser
|
||||
|
||||
<details>
|
||||
<summary><strong>Arduino IDE Setup Instructions</strong> (Click to expand)</summary>
|
||||
|
||||
After installing the Arduino IDE, add FastLED through the Library Manager:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## 📚 Documentation & Support
|
||||
|
||||
| **📖 Documentation** | **💬 Community** | **🐛 Issues** | **📺 Examples** |
|
||||
|---------------------|------------------|---------------|-----------------|
|
||||
| [API Reference](http://fastled.io/docs) | [Reddit r/FastLED](https://www.reddit.com/r/FastLED/) | [GitHub Issues](http://fastled.io/issues) | [Live Demos](https://zackees.github.io/fastled-wasm/) |
|
||||
| [Doxygen Docs](https://fastled.io/docs/files.html) | 1000s of users & solutions | Bug reports & feature requests | [GitHub Examples](examples) |
|
||||
|
||||
**Need Help?** Visit [r/FastLED](https://reddit.com/r/FastLED) - thousands of knowledgeable users and extensive solution history!
|
||||
|
||||
## 🎮 Advanced Features
|
||||
|
||||
### Performance Leaders: Parallel Output Records
|
||||
|
||||
| **Platform** | **Max Parallel Outputs** | **Performance Notes** |
|
||||
|--------------|---------------------------|----------------------|
|
||||
| **Teensy 4.1** | 50 parallel strips | Current record holder - [Example](examples/TeensyMassiveParallel/TeensyMassiveParallel.ino) |
|
||||
| **Teensy 4.0** | 42 parallel strips | High-performance ARM Cortex-M7 |
|
||||
| **ESP32DEV** | 24 via I2S + 8 via RMT | [I2S Example](examples/EspI2SDemo/EspI2SDemo.ino) |
|
||||
| **ESP32-S3** | 16 via I2S + 4 via RMT | [S3 Example](examples/Esp32S3I2SDemo/Esp32S3I2SDemo.ino) |
|
||||
|
||||
*Note: Some ESP32 Arduino core versions (3.10+) have compatibility issues. Older versions work better - see [issue #1903](https://github.com/FastLED/FastLED/issues/1903)*
|
||||
|
||||
### Exotic Setups (120+ Outputs!)
|
||||
**Custom shift register boards** can achieve extreme parallel output:
|
||||
- **ESP32DEV**: [120-output virtual driver](https://github.com/hpwit/I2SClocklessVirtualLedDriver)
|
||||
- **ESP32-S3**: [120-output S3 driver](https://github.com/hpwit/I2SClockLessLedVirtualDriveresp32s3)
|
||||
|
||||
### Parallel WS2812 Drivers
|
||||
|
||||
FastLED supports several drivers for parallel WS2812 output.
|
||||
|
||||
#### Teensy
|
||||
|
||||
The following drivers are available for Teensy boards:
|
||||
|
||||
##### WS2812Serial driver
|
||||
|
||||
The `WS2812Serial` driver leverages serial ports for LED data transmission on Teensy boards.
|
||||
|
||||
###### Usage
|
||||
To use this driver, you must define `USE_WS2812SERIAL` before including the FastLED header.
|
||||
|
||||
```cpp
|
||||
#define USE_WS2812SERIAL
|
||||
#include <FastLED.h>
|
||||
```
|
||||
|
||||
Then, use `WS2812SERIAL` as the chipset type in your `addLeds` call. The data pin is not used for this driver.
|
||||
|
||||
```cpp
|
||||
FastLED.addLeds<WS2812SERIAL, /* DATA_PIN */, GRB>(leds, NUM_LEDS);
|
||||
```
|
||||
|
||||
###### Supported Pins & Serial Ports
|
||||
|
||||
| Port | Teensy LC | Teensy 3.2 | Teensy 3.5 | Teensy 3.6 | Teensy 4.0 | Teensy 4.1 |
|
||||
| :------ | :---------: | :--------: | :--------: | :--------: | :--------: | :--------: |
|
||||
| Serial1 | 1, 4, 5, 24 | 1, 5 | 1, 5, 26 | 1, 5, 26 | 1 | 1, 53 |
|
||||
| Serial2 | | 10, 31 | 10 | 10 | 8 | 8 |
|
||||
| Serial3 | | 8 | 8 | 8 | 14 | 14 |
|
||||
| Serial4 | | | 32 | 32 | 17 | 17 |
|
||||
| Serial5 | | | 33 | 33 | 20, 39 | 20, 47 |
|
||||
| Serial6 | | | 48 | | 24 | 24 |
|
||||
| Serial7 | | | | | 29 | 29 |
|
||||
| Serial8 | | | | | | 35 |
|
||||
|
||||
##### ObjectFLED Driver
|
||||
|
||||
The `ObjectFLED` driver is an advanced parallel output driver specifically optimized for Teensy 4.0 and 4.1 boards when using WS2812 LEDs. It is designed to provide high-performance, multi-pin output.
|
||||
|
||||
By default, FastLED automatically uses the `ObjectFLED` driver for WS2812 LEDs on Teensy 4.0/4.1 boards.
|
||||
|
||||
###### Disabling ObjectFLED (Reverting to Legacy Driver)
|
||||
If you encounter compatibility issues or wish to use the standard clockless driver instead of `ObjectFLED`, you can disable it by defining `FASTLED_NOT_USES_OBJECTFLED` before including the FastLED header:
|
||||
|
||||
```cpp
|
||||
#define FASTLED_NOT_USES_OBJECTFLED
|
||||
#include <FastLED.h>
|
||||
```
|
||||
|
||||
###### Re-enabling ObjectFLED (Explicitly)
|
||||
While `ObjectFLED` is the default for Teensy 4.0/4.1, you can explicitly enable it (though not strictly necessary) using:
|
||||
|
||||
```cpp
|
||||
#define FASTLED_USES_OBJECTFLED
|
||||
#include <FastLED.h>
|
||||
```
|
||||
|
||||
#### ESP32
|
||||
|
||||
The following drivers are available for ESP32 boards:
|
||||
|
||||
##### ESP32-S3 I2S Driver
|
||||
|
||||
The `ESP32-S3 I2S` driver leverages the I2S peripheral for high-performance parallel WS2812 output on ESP32-S3 boards. This driver is a dedicated clockless implementation.
|
||||
|
||||
###### Usage
|
||||
To use this driver, you must define `FASTLED_USES_ESP32S3_I2S` before including the FastLED header.
|
||||
|
||||
```cpp
|
||||
#define FASTLED_USES_ESP32S3_I2S
|
||||
#include <FastLED.h>
|
||||
```
|
||||
|
||||
Then, use `WS2812` as the chipset type in your `addLeds` call. The data pin will be configured by the I2S driver.
|
||||
|
||||
```cpp
|
||||
FastLED.addLeds<WS2812, DATA_PIN, GRB>(leds, NUM_LEDS);
|
||||
```
|
||||
|
||||
**Note:** This driver requires a compatible Arduino-ESP32 core/IDF.
|
||||
|
||||
##### Generic ESP32 I2S Driver
|
||||
|
||||
The generic `ESP32 I2S` driver provides parallel WS2812 output for various ESP32 boards (e.g., ESP32-DevKitC). It uses the I2S peripheral for efficient data transmission.
|
||||
|
||||
###### Usage
|
||||
To use this driver, you must define `FASTLED_ESP32_I2S` before including the FastLED header.
|
||||
|
||||
```cpp
|
||||
#define FASTLED_ESP32_I2S
|
||||
#include <FastLED.h>
|
||||
```
|
||||
|
||||
Then, use `WS2812` as the chipset type in your `addLeds` call. The data pin will be configured by the I2S driver.
|
||||
|
||||
```cpp
|
||||
FastLED.addLeds<WS2812, DATA_PIN, GRB>(leds, NUM_LEDS);
|
||||
```
|
||||
|
||||
###### DMA Buffer Configuration
|
||||
For improved resilience under interrupt load (e.g., Wi-Fi activity), you can increase the number of I2S DMA buffers by defining `FASTLED_ESP32_I2S_NUM_DMA_BUFFERS`. A value of `4` is often recommended.
|
||||
|
||||
```cpp
|
||||
#define FASTLED_ESP32_I2S_NUM_DMA_BUFFERS 4
|
||||
```
|
||||
|
||||
**Note:** All I2S lanes must share the same chipset/timings. If per-lane timing differs, consider using the RMT driver instead.
|
||||
|
||||
### Supported LED Chipsets
|
||||
|
||||
FastLED supports virtually every LED chipset available:
|
||||
|
||||
| **Clockless (3-wire)** | **SPI-based (4-wire)** | **Specialty** |
|
||||
|------------------------|------------------------|---------------|
|
||||
| **WS281x Family**: WS2811, WS2812 (NeoPixel), WS2812-V5B, WS2815 | **APA102 / DotStars**: Including HD107s (40MHz turbo) | **SmartMatrix Panels** |
|
||||
| **TM180x Series**: TM1809/4, TM1803 | **High-Speed SPI**: LPD8806, WS2801, SM16716 | **DMX Output** |
|
||||
| **Other 3-wire**: UCS1903, GW6205, SM16824E | **APA102HD**: Driver-level gamma correction | **P9813 Total Control** |
|
||||
|
||||
**RGBW Support**: WS2816 and other white-channel LED strips • **Overclocking**: WS2812 up to 70% speed boost
|
||||
|
||||
More details: [Chipset Reference Wiki](https://github.com/FastLED/FastLED/wiki/Chipset-reference)
|
||||
|
||||
### APA102 High Definition Mode
|
||||
```cpp
|
||||
#define LED_TYPE APA102HD // Enables hardware gamma correction
|
||||
void setup() {
|
||||
FastLED.addLeds<LED_TYPE, DATA_PIN, CLOCK_PIN, RGB>(leds, NUM_LEDS);
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
|
||||
Read more: [APA102 HD Documentation](APA102.md) • [Rust Implementation](https://docs.rs/apa102-spi/latest/apa102_spi/)
|
||||
|
||||
## 🛠️ Development & Contributing
|
||||
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_clone_and_compile.yml)
|
||||
|
||||
**Ready to contribute?** FastLED welcomes code contributions, platform testing, documentation improvements, and community support.
|
||||
|
||||
<details>
|
||||
<summary>🔧 <strong>Development Setup & Contributing Guide</strong> (Click to expand)</summary>
|
||||
|
||||
### Quick Development Setup
|
||||
Zero pain setup - can be done from command line in seconds with `uv` or `python`:
|
||||
|
||||
|
||||
|
||||
**Steps:**
|
||||
1. Fork the repository on GitHub
|
||||
2. Clone your fork locally: `git clone https://github.com/yourusername/FastLED.git`
|
||||
3. Test compilation: `bash compile` (select your target platform)
|
||||
4. Make your changes and test
|
||||
5. Push to your fork and create a pull request
|
||||
|
||||
**See our detailed [Contributing Guide](CONTRIBUTING.md) for more information.**
|
||||
|
||||
### Testing Other Devices
|
||||
Use the `compile` script to test on 20+ platforms:
|
||||
- Teensy series, ESP32 variants, Arduino family, ARM boards, and more
|
||||
- Automated testing ensures compatibility across all supported platforms
|
||||
|
||||
### How to Help
|
||||
|
||||
| **📋 How to Help** | **🔗 Resources** |
|
||||
|-------------------|------------------|
|
||||
| **Code contributions**: Bug fixes, new features, optimizations | [Contributing Guide](CONTRIBUTING.md) |
|
||||
| **Platform support**: Help test on new/existing platforms | [Platform Testing](compile) |
|
||||
| **Documentation**: Improve examples, fix typos, add tutorials | [Documentation](http://fastled.io/docs) |
|
||||
| **Community**: Answer questions on Reddit, report issues | [r/FastLED](https://reddit.com/r/FastLED) |
|
||||
|
||||
**Platform Porting**: Information on porting FastLED to new platforms: [PORTING.md](PORTING.md)
|
||||
|
||||
### About FastLED
|
||||
|
||||
**What's in the name?** Originally "FastSPI_LED" focused on high-speed SPI, but evolved to support all LED types and became "FastLED" - everything fast, for LEDs.
|
||||
|
||||
**Official Site**: [fastled.io](http://fastled.io) - documentation, issues, and news
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Daniel Garcia, Founder of FastLED
|
||||
|
||||
In Memory of Daniel Garcia
|
||||
Daniel Garcia, the brilliant founder of FastLED, tragically passed away in September 2019 in the Conception dive boat fire alongside his partner, Yulia. This heartbreaking loss was felt deeply by the maker and developer community, where Daniel's contributions had left an indelible mark.
|
||||
|
||||
Daniel was more than just a talented programmer; he was a passionate innovator who transformed the way creators interacted with LED technology. His work on FastLED brought high-performance LED control to countless projects, empowering developers to craft breathtaking installations.
|
||||
|
||||
In his personal life, Daniel was known for his kindness and creativity. His pride in FastLED and the vibrant community it fostered was a testament to his dedication to open-source development and his commitment to helping others bring light into the world.
|
||||
|
||||
While Daniel is no longer with us, his legacy continues through the FastLED library and the countless makers who use it. The community he built serves as a living tribute to his ingenuity, generosity, and the joy he found in sharing his work with the world.
|
||||
|
||||
## About the Current Contributor
|
||||
|
||||
Zach Vorhies, the current main contributor to FastLED, briefly worked with Dan in 2014 in San Francisco and was an avid user of the FastLED library for over 13 years. After Daniel Garcia’s untimely passing, Zach stepped up to ensure FastLED’s continued growth and development.
|
||||
|
||||
Zach has this to say about FastLED:
|
||||
|
||||
*"The true power of FastLED lies in its ability to transform programmers into LED artists. Free space becomes their canvas; bending light is their medium. FastLED is a collective effort by programmers who want to manifest the world that science fiction writers promised us. -- To contribute code to FastLED is to leave behind a piece of something immortal."*
|
||||
|
||||
|
||||
---
|
||||
|
||||
**💫 In memory of Daniel Garcia and the vision that transforms programmers into LED artists**
|
||||
|
||||
*To stay updated on the latest feature releases, please click the `Watch` button in the upper right*
|
||||
""
|
||||
@@ -0,0 +1,39 @@
|
||||
# FastLED Release howto
|
||||
|
||||
*Pushing a fastled release, the short version, last updated May 2024*
|
||||
|
||||
## Example
|
||||
|
||||
https://github.com/FastLED/FastLED/commit/4444758ffaf853ba4f8deb973532548c9c1ee231
|
||||
|
||||
## How to
|
||||
|
||||
Edit these files to update the version number
|
||||
* library.json
|
||||
* library.properties
|
||||
* src/FastLED.h
|
||||
* docs/Doxyfile
|
||||
* RELEASE.md
|
||||
* This file: update instructions with the current release.
|
||||
|
||||
|
||||
Edit this file with release notes and version number.
|
||||
* release_notes.md
|
||||
|
||||
Release notes should list highlight changes (not necessarily all minor bug fixes) and thank people for their help.
|
||||
|
||||
Git commands to commit and tag release'
|
||||
```bash
|
||||
$ git commit -am "Rev 3.10.3"
|
||||
$ git tag 3.10.3 master
|
||||
$ git push
|
||||
$ git push origin 3.10.3
|
||||
```
|
||||
|
||||
Then use the GitHub UI to make a new “Release”:
|
||||
|
||||
https://github.com/FastLED/FastLED/releases/new
|
||||
|
||||
Announce new version on subreddit, highlighting major changes and thanking people for helping.
|
||||
|
||||
That’s it.
|
||||
@@ -0,0 +1,806 @@
|
||||
Do a deep research on fastled's RMT5 driver and the persistance of flickering.
|
||||
|
||||
The RMT4 driver is MORE resistant to flickering. The RMT5 driver is pretty bad.
|
||||
|
||||
Cause: WIFI takes high prioritiy interrupts and starves the the RMT driver.
|
||||
|
||||
Potention solutions:
|
||||
|
||||
create an interrupt than level 3, but this requires us to create a asm shim to trampoline into the c function handler and do proper stack cleanup.
|
||||
|
||||
ESP32 & ESp32c3 and esp32s3 might have different cores. ones an xtensa, whats the other?
|
||||
|
||||
list them.
|
||||
|
||||
Another solution is to try and ping pong the rmt buffer like we do on the rmt4. However we must have better documentation
|
||||
to see if this is even possible.
|
||||
|
||||
Investigate the documentation and list it here on the RMT device for idf 5 and higher.
|
||||
|
||||
Then do a feasability status to see if the ping pong is p0ossible.
|
||||
|
||||
These are the two solutions. Both need to be investigated. But your investigation into RMT5.md
|
||||
|
||||
|
||||
## Deep Research results
|
||||
|
||||
FastLED RMT5 Driver Flickering Investigation
|
||||
Background: FastLED’s original ESP32 driver (“RMT4”) used Espressif’s low-level RMT APIs, with one RMT memory block per channel and built‐in double-buffering for non-PSRAM chips. The newer “RMT5” driver (used on ESP32-S3, C3, etc.) wraps Espressif’s led_strip component (IDF’s higher-level RMT LED driver)
|
||||
GitHub
|
||||
. Under heavy Wi‑Fi load, the RMT5 approach has proven much more prone to gaps: Wi‑Fi interrupts at high priority can delay the RMT ISR, causing the RMT buffer to run dry and LEDs to flicker
|
||||
esp32.com
|
||||
esp-idf-lib.readthedocs.io
|
||||
. (Indeed, the esp-idf led_strip documentation warns that “simultaneous use with Wi‑Fi” causes RMT bugs
|
||||
esp-idf-lib.readthedocs.io
|
||||
.)
|
||||
ESP32 Core Architectures
|
||||
The ESP32 family spans multiple architectures: classic ESP32 and ESP32-S3 use Tensilica Xtensa cores, while newer chips like ESP32-C3/C6/H2 use 32-bit RISC‑V cores
|
||||
en.wikipedia.org
|
||||
. For example, the ESP32-S3 is a dual-core Xtensa LX7, whereas the ESP32-C3 is a single-core RISC‑V. This matters for RMT timing (the underlying CPU instruction timing differs), but the flicker issue itself arises on any core when Wi‑Fi interrupts interfere with the RMT ISR.
|
||||
RMT Peripheral (IDF v5+) Documentation
|
||||
Espressif’s IDF documentation describes the RMT as a hardware peripheral that transmits sequences of “symbols” via GPIO, optionally using DMA. Because the RMT memory block is limited, long data transfers are split in a ping-pong fashion: the encoder callback can be invoked multiple times to refill the buffer as it empties
|
||||
docs.espressif.com
|
||||
. The docs note that these encoders run in ISR context and must be IRAM-safe (so that cache-disabled delays do not postpone interrupts)
|
||||
docs.espressif.com
|
||||
docs.espressif.com
|
||||
. In particular, IDF has a CONFIG_RMT_ISR_IRAM_SAFE option that ensures RMT ISRs can run even when flash cache is off, to avoid unpredictable timing when using ping-pong buffering
|
||||
docs.espressif.com
|
||||
. The RMT API also allows setting the ISR priority explicitly: in rmt_tx_channel_config_t, the field intr_priority chooses the interrupt level (default “0” picks a medium priority 1–3) or a user-specified priority
|
||||
docs.espressif.com
|
||||
. This means one can elevate the RMT interrupt to a higher level (4 or 5) than Wi‑Fi (which normally runs at a high but fixed level).
|
||||
Flickering Issue – Interrupt Starvation
|
||||
Users consistently report that with RMT5, enabling Wi‑Fi causes periodic color glitches. Under a light web server load, one user measured ~50 µs jitter in what should be a ~35 µs ISR interval
|
||||
esp32.com
|
||||
, due to Wi‑Fi activity stealing CPU time. When this jitter exceeds the tolerance of the WS2812 timing, the LED strip “runs out” of data and flickers
|
||||
esp32.com
|
||||
. In contrast, the older RMT4 code (especially when using two memory blocks) could often absorb such delays. Espressif’s community notes that any Wi‑Fi use with the RMT LED driver can trigger glitches, and one workaround is to run the RMT tasks on the second CPU core
|
||||
esp-idf-lib.readthedocs.io
|
||||
. In practice, however, the flicker appears because high-priority Wi‑Fi interrupts delay the RMT driver’s buffer refill callbacks.
|
||||
Potential Solutions
|
||||
Increase RMT ISR Priority: Configure the RMT channel to use a higher interrupt level than Wi‑Fi. In IDF, setting intr_priority=4 or 5 in rmt_tx_channel_config_t will raise the ISR above normal Wi‑Fi interrupts
|
||||
docs.espressif.com
|
||||
. This requires writing the RMT encoding callback in IRAM and may require an assembly shim if using priority 5. (On Xtensa, level‑5 interrupts cannot call C routines directly, so one writes a small ASM stub that calls into a safe C function.) The FastLED RMT code would need to initialize the RMT channel with a high intr_priority and mark its handler IRAM-safe (via CONFIG_RMT_ISR_IRAM_SAFE)
|
||||
docs.espressif.com
|
||||
. A community example confirms that raising to level‑5 can eliminate most Wi‑Fi jitter, but notes you must modify the ISR entry (often with inline ASM) to avoid stack/ISR constraints
|
||||
esp32.com
|
||||
.
|
||||
Ping-Pong (Double-Buffer) Mode: Use two RMT memory blocks per channel so the peripheral can alternate between them, thus buffering extra data. The IDF RMT encoder model explicitly supports ping‑pong buffering: the encoder callback is invoked each time a block fills, allowing the next symbols to be written
|
||||
docs.espressif.com
|
||||
. In FastLED, one enables this by allocating 2 blocks (e.g. setting mem_block_symbols to twice the usual) or enabling MEM_BLOCK_NUM=2. In practice, simply doubling the memory (using two blocks) absorbs Wi‑Fi-induced gaps: a forum user found that switching from 1 to 2 blocks removed all LED glitches by covering up ~120 µs of jitter
|
||||
esp32.com
|
||||
. The FastLED release notes even mention this behavior: on non-DMA chips (ESP32, C3, etc.), the RMT5 driver is configured to “double its memory” (i.e. use 2 blocks) by default, mimicking RMT4’s approach
|
||||
GitHub
|
||||
. Thus, ping-pong is technically supported. It requires verifying that the FastLED led_strip setup indeed uses two blocks (and if not, forcing it via FASTLED_RMT5_RECYCLE=0 or similar flags).
|
||||
Feasibility and Status
|
||||
Both strategies have merit. High-Priority ISR: requires low-level tweaking (setting intr_priority and possibly writing an IRAM ISR shim). This could be complex (especially on Xtensa) but would ensure timely buffer refills. Espressif docs confirm the ability to change RMT interrupt level
|
||||
docs.espressif.com
|
||||
and their IRAM-safe option mitigates deferred ISRs
|
||||
docs.espressif.com
|
||||
. However, once set, the interrupt priority can’t be changed without deleting the channel
|
||||
docs.espressif.com
|
||||
, so it must be done at RMT init. Ping-Pong Buffering: is simpler to implement since it leverages the existing RMT mechanism. The IDF RMT hardware natively supports multi-block transfers
|
||||
docs.espressif.com
|
||||
, and FastLED’s own code (via the led_strip component) already allows mem_block_symbols setting. Doubling the blocks is a one-line config change and was empirically validated by users
|
||||
esp32.com
|
||||
. FastLED even defaults to 2 blocks on chips without external RAM
|
||||
GitHub
|
||||
. In summary, raising the interrupt level could almost eliminate flicker at its source, but is invasive; enabling double-buffering is much easier and likely sufficient to cover typical Wi‑Fi spikes. Sources: Espressif’s RMT documentation (IDF v5+) and community forums provide detailed guidance on interrupt priorities, IRAM-safe handlers, and ping-pong transfers
|
||||
docs.espressif.com
|
||||
docs.espressif.com
|
||||
docs.espressif.com
|
||||
. FastLED release notes and forum tests confirm that Wi‑Fi interrupts cause RMT underruns and that using two RMT blocks removes glitches
|
||||
esp32.com
|
||||
esp32.com
|
||||
GitHub
|
||||
. Core architectures are listed for completeness
|
||||
en.wikipedia.org
|
||||
.
|
||||
Citations
|
||||
GitHub
|
||||
idf5_rmt.cpp
|
||||
|
||||
https://github.com/chaosgoo/mcompass/blob/d9d37398995c8e2b1806a6c90af03a1b8bc1c47e/Firmware/lib/FastLED/src/platforms/esp/32/idf5_rmt.cpp#L70-L78
|
||||
Amount of jitter on LEVEL3 interrupt? - ESP32 Forum
|
||||
|
||||
https://esp32.com/viewtopic.php?t=17206
|
||||
|
||||
led_strip - RMT-based driver for WS2812B/SK6812/APA106 LED strips — esp-idf-lib 1.0 documentation
|
||||
|
||||
https://esp-idf-lib.readthedocs.io/en/latest/groups/led_strip.html
|
||||
|
||||
ESP32 - Wikipedia
|
||||
|
||||
https://en.wikipedia.org/wiki/ESP32
|
||||
|
||||
Remote Control Transceiver (RMT) - ESP32 - — ESP-IDF Programming Guide v5.0 documentation
|
||||
|
||||
https://docs.espressif.com/projects/esp-idf/en/v5.0/esp32/api-reference/peripherals/rmt.html
|
||||
|
||||
Remote Control Transceiver (RMT) - ESP32 - — ESP-IDF Programming Guide v5.0 documentation
|
||||
|
||||
https://docs.espressif.com/projects/esp-idf/en/v5.0/esp32/api-reference/peripherals/rmt.html
|
||||
|
||||
Remote Control Transceiver (RMT) - ESP32 - — ESP-IDF Programming Guide v5.0 documentation
|
||||
|
||||
https://docs.espressif.com/projects/esp-idf/en/v5.0/esp32/api-reference/peripherals/rmt.html
|
||||
|
||||
Remote Control Transceiver (RMT) - ESP32 - — ESP-IDF Programming Guide v5.5.1 documentation
|
||||
|
||||
https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/peripherals/rmt.html
|
||||
Amount of jitter on LEVEL3 interrupt? - ESP32 Forum
|
||||
|
||||
https://esp32.com/viewtopic.php?t=17206
|
||||
Amount of jitter on LEVEL3 interrupt? - ESP32 Forum
|
||||
|
||||
https://esp32.com/viewtopic.php?t=17206
|
||||
GitHub
|
||||
release_notes.md
|
||||
|
||||
https://github.com/FastLED/FastLED/blob/0c22d156836639119fd6d0dc84d3c42f8bca7e0f/release_notes.md#L42-L47
|
||||
All Sources
|
||||
|
||||
## Audit Comments and Criticisms (2025-09-16)
|
||||
|
||||
### Structural Issues
|
||||
|
||||
1. **Incomplete Research Questions**: The initial questions (lines 1-23) remain unanswered in parts:
|
||||
- The ASM shim solution for level 5 interrupts is mentioned but lacks implementation details
|
||||
- Core architectures are listed but the impact on RMT timing differences isn't fully explored
|
||||
|
||||
2. **Missing Concrete Implementation Path**: While two solutions are proposed (high-priority ISR and ping-pong buffering), there's no clear recommendation on which to pursue first or implementation timeline.
|
||||
|
||||
3. **Documentation Gaps**:
|
||||
- No code examples showing how to implement either solution in FastLED
|
||||
- Missing comparison table of RMT4 vs RMT5 architectural differences
|
||||
- No benchmarks showing actual flicker frequency/severity metrics
|
||||
|
||||
### Technical Concerns
|
||||
|
||||
4. **Incomplete Testing Coverage**:
|
||||
- No mention of testing methodology to validate fixes
|
||||
- Missing list of affected board configurations (which ESP32 variants + which IDF versions)
|
||||
- No quantification of "pretty bad" flickering (frequency, duration, visual impact)
|
||||
|
||||
5. **Risk Assessment Missing**:
|
||||
- Level 5 interrupt solution needs warning about potential system stability impacts
|
||||
- No discussion of power consumption implications of either solution
|
||||
- Missing compatibility matrix with other FastLED features
|
||||
|
||||
### Content Issues
|
||||
|
||||
6. **Redundant Citations**: Multiple duplicate citations to the same ESP32 forum thread and documentation pages (lines 103-140)
|
||||
|
||||
7. **Inconsistent Formatting**:
|
||||
- Mix of bullet points and paragraph styles in solutions section
|
||||
- Inconsistent use of technical terms (RMT4 vs "RMT4", Wi-Fi vs WiFi)
|
||||
|
||||
### Missing Critical Information
|
||||
|
||||
8. **User Impact Analysis**:
|
||||
- No guidance on which use cases are most affected
|
||||
- Missing workaround recommendations for users experiencing issues now
|
||||
- No mention of whether this affects all LED types or specific protocols
|
||||
|
||||
9. **Configuration Details**:
|
||||
- The ping-pong solution mentions "FASTLED_RMT5_RECYCLE=0" but doesn't explain what this does
|
||||
- No clear documentation of all relevant FastLED compile-time flags
|
||||
|
||||
10. **Priority and Severity**:
|
||||
- No clear statement on whether this is a critical bug or edge case
|
||||
- Missing timeline for fix deployment
|
||||
- No mention of whether this is a regression from previous versions
|
||||
|
||||
### Recommendations for Next Steps
|
||||
|
||||
1. **Immediate**: Test ping-pong buffering solution as it's simpler and less risky
|
||||
2. **Short-term**: Create reproducible test case with measurable flicker metrics
|
||||
3. **Medium-term**: Document all FastLED RMT configuration options in one place
|
||||
4. **Long-term**: Consider abstracting RMT driver to allow runtime switching between implementations
|
||||
|
||||
### New Issues Identified
|
||||
|
||||
- **Thread Safety**: No discussion of thread safety when RMT ISR runs at different priority levels
|
||||
- **Memory Usage**: No analysis of memory impact when doubling RMT blocks
|
||||
- **Backwards Compatibility**: No mention of how fixes will affect existing code using RMT5
|
||||
- **Error Handling**: No discussion of graceful degradation when solutions fail
|
||||
|
||||
---
|
||||
|
||||
## QEMU TESTING FRAMEWORK (2025-09-16)
|
||||
|
||||
### Overview
|
||||
This section outlines comprehensive test criteria for validating RMT interrupt solutions using QEMU emulation on ESP32-S3 and ESP32-C3 platforms. QEMU provides a controlled environment to reproduce Wi-Fi interference patterns and measure interrupt latency without physical hardware variations.
|
||||
|
||||
### Test Infrastructure Requirements
|
||||
|
||||
#### QEMU Setup Commands
|
||||
```bash
|
||||
# Install QEMU for ESP32 emulation
|
||||
uv run ci/install-qemu.py
|
||||
|
||||
# Run QEMU tests with ESP32-S3 (Xtensa)
|
||||
uv run test.py --qemu esp32s3
|
||||
|
||||
# Run QEMU tests with ESP32-C3 (RISC-V)
|
||||
uv run test.py --qemu esp32c3
|
||||
|
||||
# Skip QEMU installation if already present
|
||||
FASTLED_QEMU_SKIP_INSTALL=true uv run test.py --qemu esp32s3
|
||||
```
|
||||
|
||||
#### Platform Coverage Matrix
|
||||
|
||||
| Platform | Architecture | RMT Version | Max Official Priority | Max Custom Priority | Test Priority |
|
||||
|----------|-------------|-------------|----------------------|-------------------|---------------|
|
||||
| ESP32-S3 | Xtensa LX7 | RMT5 | 3 | 5 | High |
|
||||
| ESP32-C3 | RISC-V | RMT5 | 3 | 7 | High |
|
||||
| ESP32-C6 | RISC-V | RMT5 | 3 | 7 | Medium |
|
||||
| ESP32 | Xtensa LX6 | RMT4 | N/A | N/A | Reference |
|
||||
|
||||
### Test Scenarios
|
||||
|
||||
#### 1. Baseline RMT Timing Tests
|
||||
**Objective**: Establish baseline RMT performance without interference
|
||||
|
||||
**Test Cases**:
|
||||
- **T1.1**: RMT5 default priority (level 3) LED refresh timing
|
||||
- **T1.2**: Memory buffer utilization patterns (single vs double block)
|
||||
- **T1.3**: WS2812B timing compliance under no-load conditions
|
||||
- **T1.4**: Multiple channel simultaneous operation
|
||||
|
||||
**Success Criteria**:
|
||||
- WS2812B timing within ±150ns tolerance
|
||||
- Zero dropped frames for 300 LED array
|
||||
- Consistent ~35µs refresh intervals
|
||||
- No interrupt overruns in baseline conditions
|
||||
|
||||
**QEMU Implementation**:
|
||||
```cpp
|
||||
// Test harness in tests/qemu_rmt_baseline.cpp
|
||||
void test_rmt_baseline_timing() {
|
||||
// Configure RMT channel with default priority 3
|
||||
rmt_channel_config_t config = {
|
||||
.gpio_num = TEST_GPIO_PIN,
|
||||
.clk_div = 2,
|
||||
.mem_block_num = 1, // Single block baseline
|
||||
.intr_priority = 3 // Official maximum
|
||||
};
|
||||
|
||||
// Measure timing consistency across 1000 refresh cycles
|
||||
for (int i = 0; i < 1000; i++) {
|
||||
uint64_t start = esp_timer_get_time();
|
||||
fastled_rmt_refresh_leds(test_pattern, 300);
|
||||
uint64_t elapsed = esp_timer_get_time() - start;
|
||||
|
||||
// Verify timing bounds
|
||||
assert(elapsed >= 34500 && elapsed <= 35500); // ±500µs tolerance
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. Wi-Fi Interference Simulation
|
||||
**Objective**: Reproduce the Wi-Fi starvation issue that causes RMT5 flickering
|
||||
|
||||
**Test Cases**:
|
||||
- **T2.1**: Simulated Wi-Fi interrupt bursts at level 4 priority
|
||||
- **T2.2**: Sustained Wi-Fi traffic during LED refresh cycles
|
||||
- **T2.3**: Timing jitter measurement under interference
|
||||
- **T2.4**: Buffer underrun detection and quantification
|
||||
|
||||
**QEMU Wi-Fi Simulation**:
|
||||
```cpp
|
||||
// Simulate Wi-Fi interrupt load in QEMU
|
||||
void simulate_wifi_interference() {
|
||||
// Install level 4 interrupt to simulate Wi-Fi
|
||||
esp_intr_alloc(ETS_WIFI_MAC_INTR_SOURCE,
|
||||
ESP_INTR_FLAG_LEVEL4 | ESP_INTR_FLAG_IRAM,
|
||||
wifi_interference_isr, NULL, &wifi_handle);
|
||||
|
||||
// Generate periodic 40-50µs interrupt bursts
|
||||
esp_timer_create(&wifi_timer_config, &wifi_timer);
|
||||
esp_timer_start_periodic(wifi_timer, 100); // 100µs period
|
||||
}
|
||||
|
||||
void IRAM_ATTR wifi_interference_isr(void* arg) {
|
||||
// Simulate 40µs Wi-Fi processing time
|
||||
uint64_t start = esp_timer_get_time();
|
||||
while ((esp_timer_get_time() - start) < 40) {
|
||||
// Busy wait to consume CPU time
|
||||
__asm__ volatile ("nop");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Success Criteria for Problem Reproduction**:
|
||||
- Measured jitter > ±10µs with Wi-Fi simulation active
|
||||
- Buffer underruns detected in RMT interrupt handler
|
||||
- Visual flicker correlation with Wi-Fi interrupt timing
|
||||
|
||||
#### 3. High-Priority Interrupt Solution Validation
|
||||
**Objective**: Test level 4/5 interrupt priority solutions
|
||||
|
||||
**Test Cases**:
|
||||
- **T3.1**: ESP32-S3 level 4 interrupt implementation (experimental)
|
||||
- **T3.2**: ESP32-S3 level 5 interrupt with assembly shim
|
||||
- **T3.3**: ESP32-C3 level 4-7 interrupt testing (RISC-V advantage)
|
||||
- **T3.4**: Interrupt latency measurement and comparison
|
||||
|
||||
**Implementation Reference**: `src/platforms/esp/32/interrupts/esp32_s3.hpp:149-161`
|
||||
|
||||
**Test Implementation**:
|
||||
```cpp
|
||||
// Test level 5 interrupt solution (ESP32-S3 only)
|
||||
void test_level5_interrupt_solution() {
|
||||
// Use custom high-priority RMT initialization
|
||||
esp_err_t err = fastled_esp32s3_rmt_init_custom(
|
||||
RMT_CHANNEL_0,
|
||||
TEST_GPIO_PIN,
|
||||
40000000, // 40MHz resolution
|
||||
128, // Single block
|
||||
5 // Level 5 priority - EXPERIMENTAL
|
||||
);
|
||||
|
||||
assert(err == ESP_OK);
|
||||
|
||||
// Activate Wi-Fi interference simulation
|
||||
simulate_wifi_interference();
|
||||
|
||||
// Measure timing stability with level 5 priority
|
||||
for (int i = 0; i < 1000; i++) {
|
||||
uint64_t start = esp_timer_get_time();
|
||||
fastled_rmt_refresh_leds(test_pattern, 300);
|
||||
uint64_t elapsed = esp_timer_get_time() - start;
|
||||
|
||||
// Level 5 should maintain tight timing despite Wi-Fi
|
||||
assert(elapsed >= 34800 && elapsed <= 35200); // ±200µs tolerance
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 4. Double-Buffer Solution Validation
|
||||
**Objective**: Test ping-pong buffering approaches (limited by LED strip API)
|
||||
|
||||
**Test Cases**:
|
||||
- **T4.1**: Memory block doubling via `mem_block_symbols` parameter
|
||||
- **T4.2**: Custom low-level RMT implementation with true ping-pong
|
||||
- **T4.3**: Buffer refill timing analysis
|
||||
- **T4.4**: Comparison with RMT4 dual-buffer behavior
|
||||
|
||||
**Critical Finding**: LED strip API limitation prevents true ping-pong (`RMT.md:254-258`)
|
||||
|
||||
**Test Implementation**:
|
||||
```cpp
|
||||
// Test doubled memory blocks (not true ping-pong)
|
||||
void test_doubled_memory_blocks() {
|
||||
rmt_channel_config_t config = {
|
||||
.gpio_num = TEST_GPIO_PIN,
|
||||
.clk_div = 2,
|
||||
.mem_block_num = 2, // Double the memory
|
||||
.intr_priority = 3
|
||||
};
|
||||
|
||||
// This creates larger buffer, NOT ping-pong
|
||||
// Test if larger buffer helps with Wi-Fi jitter
|
||||
simulate_wifi_interference();
|
||||
|
||||
// Measure improvement (if any)
|
||||
uint32_t underrun_count = 0;
|
||||
for (int i = 0; i < 1000; i++) {
|
||||
if (fastled_rmt_refresh_with_underrun_detection(test_pattern, 300)) {
|
||||
underrun_count++;
|
||||
}
|
||||
}
|
||||
|
||||
// Expect fewer underruns than single block, but not elimination
|
||||
printf("Double block underruns: %lu (should be < single block)\n", underrun_count);
|
||||
}
|
||||
```
|
||||
|
||||
#### 5. Cross-Architecture Comparison Tests
|
||||
**Objective**: Compare RMT behavior between Xtensa and RISC-V cores
|
||||
|
||||
**Test Cases**:
|
||||
- **T5.1**: Identical test patterns on ESP32-S3 (Xtensa) vs ESP32-C3 (RISC-V)
|
||||
- **T5.2**: Interrupt latency differences between architectures
|
||||
- **T5.3**: Assembly shim requirement validation (Xtensa only)
|
||||
- **T5.4**: Priority level effectiveness comparison
|
||||
|
||||
**Architecture-Specific Considerations**:
|
||||
- **Xtensa (ESP32-S3)**: Requires assembly shims for level 4+ (`esp32_s3.hpp:316-368`)
|
||||
- **RISC-V (ESP32-C3)**: C handlers work at all priority levels (`esp32_c3_c6.hpp:180-194`)
|
||||
|
||||
### Automated Test Execution
|
||||
|
||||
#### Test Harness Integration
|
||||
```bash
|
||||
# Add to existing test framework
|
||||
uv run test.py RMTInterruptValidation --qemu esp32s3
|
||||
uv run test.py RMTInterruptValidation --qemu esp32c3
|
||||
|
||||
# Specific test scenarios
|
||||
uv run test.py RMTBaseline --qemu esp32s3
|
||||
uv run test.py RMTWiFiInterference --qemu esp32s3
|
||||
uv run test.py RMTHighPriority --qemu esp32s3
|
||||
uv run test.py RMTDoubleBuffer --qemu esp32s3
|
||||
```
|
||||
|
||||
#### Expected Test Results
|
||||
|
||||
| Test Scenario | ESP32-S3 Expected | ESP32-C3 Expected | Pass/Fail Criteria |
|
||||
|---------------|-------------------|-------------------|-------------------|
|
||||
| T1.1 Baseline | ±200µs jitter | ±200µs jitter | PASS if within tolerance |
|
||||
| T2.1 Wi-Fi Interference | ±10-50µs jitter | ±10-50µs jitter | PASS if problem reproduced |
|
||||
| T3.1 Level 4 Priority | ±500µs improvement | ±500µs improvement | PASS if jitter reduced |
|
||||
| T3.2 Level 5 Priority | ±1000µs improvement | N/A (max level 7) | PASS if jitter eliminated |
|
||||
| T4.1 Double Buffer | Partial improvement | Partial improvement | PASS if some reduction |
|
||||
|
||||
#### Performance Benchmarks
|
||||
|
||||
**Target Metrics**:
|
||||
- **Baseline RMT5**: 35±10µs refresh time (no Wi-Fi)
|
||||
- **RMT5 + Wi-Fi**: 35±50µs refresh time (problem case)
|
||||
- **Level 5 Priority**: 35±2µs refresh time (solution target)
|
||||
- **Double Buffer**: 35±20µs refresh time (partial solution)
|
||||
|
||||
### Test Data Collection
|
||||
|
||||
#### Logging and Metrics
|
||||
```cpp
|
||||
// Timing data collection structure
|
||||
typedef struct {
|
||||
uint64_t timestamp;
|
||||
uint32_t refresh_time_us;
|
||||
uint32_t interrupt_latency_us;
|
||||
uint8_t priority_level;
|
||||
bool wifi_active;
|
||||
bool buffer_underrun;
|
||||
} rmt_timing_sample_t;
|
||||
|
||||
// Collect 10,000 samples per test scenario
|
||||
#define TIMING_SAMPLES 10000
|
||||
rmt_timing_sample_t timing_data[TIMING_SAMPLES];
|
||||
```
|
||||
|
||||
#### Statistical Analysis
|
||||
- **Mean refresh time** and standard deviation
|
||||
- **95th percentile latency** measurements
|
||||
- **Buffer underrun frequency** (occurrences per 1000 refreshes)
|
||||
- **Jitter distribution** histograms
|
||||
|
||||
### QEMU Limitations and Considerations
|
||||
|
||||
#### QEMU Timing Accuracy
|
||||
- QEMU provides relative timing comparison, not absolute accuracy
|
||||
- Interrupt priority simulation may not perfectly match hardware
|
||||
- Focus on comparative analysis between configurations
|
||||
|
||||
#### Hardware Validation Requirements
|
||||
- QEMU results must be validated on physical ESP32-S3/C3 hardware
|
||||
- Real Wi-Fi interference patterns may differ from simulation
|
||||
- Oscilloscope measurements required for WS2812B timing verification
|
||||
|
||||
---
|
||||
|
||||
## FEASIBILITY AUDIT (2025-09-16)
|
||||
|
||||
### Executive Summary
|
||||
After analyzing the FastLED RMT5 driver implementation and ESP32 IDF documentation, **both proposed solutions are technically feasible**. The ping-pong buffering approach is recommended as the primary solution due to its simplicity and proven effectiveness.
|
||||
|
||||
### Core Architecture Analysis
|
||||
|
||||
#### ESP32 Variants and Impact
|
||||
- **ESP32/ESP32-S3**: Xtensa LX6/LX7 cores - Support level 5 interrupts but require ASM shims
|
||||
- **ESP32-C3/C6**: RISC-V cores - Different interrupt architecture but same priority levels
|
||||
- **Impact on RMT**: Core architecture affects interrupt entry overhead (~10-15% difference) but doesn't fundamentally change the Wi-Fi starvation issue
|
||||
|
||||
### Solution 1: High-Priority Interrupt (Level 5)
|
||||
|
||||
#### Current State
|
||||
- FastLED RMT5 currently uses default priority 3 (found in `strip_rmt.cpp:31`)
|
||||
- Wi-Fi typically runs at priority 4, causing starvation
|
||||
- Infrastructure exists: `interrupt_priority` parameter already plumbed through (`strip_rmt.h:23`)
|
||||
|
||||
#### Implementation Requirements
|
||||
1. **Xtensa ASM Shim** (ESP32/S3):
|
||||
```asm
|
||||
; Required for level 5 interrupts on Xtensa
|
||||
; Must save/restore registers manually
|
||||
; Call C handler with proper stack alignment
|
||||
```
|
||||
2. **IRAM Safety**: All ISR code must be marked `IRAM_ATTR`
|
||||
3. **Configuration**: Set `CONFIG_RMT_ISR_IRAM_SAFE=y` in IDF
|
||||
|
||||
#### Feasibility: **MEDIUM COMPLEXITY**
|
||||
- ✅ Technically possible - IDF supports priority levels 1-5
|
||||
- ✅ Infrastructure partially exists
|
||||
- ⚠️ Requires platform-specific ASM code
|
||||
- ⚠️ Risk of system instability if not done correctly
|
||||
- ⚠️ Cannot change priority after initialization
|
||||
|
||||
### Solution 2: Ping-Pong Buffering (Double Memory Blocks)
|
||||
|
||||
#### Current State
|
||||
- RMT5 uses ESP-IDF's `led_strip` API which abstracts away direct RMT control
|
||||
- `mem_block_symbols` parameter increases buffer size but **NOT true ping-pong**
|
||||
- RMT4 had direct access to RMT hardware registers for manual buffer switching
|
||||
- LED strip API handles encoding internally, no access to `fillNext()` mechanism
|
||||
|
||||
#### Critical Issue: **LED Strip API Limitation**
|
||||
The `led_strip` API in IDF v5 doesn't expose ping-pong buffer control:
|
||||
- RMT4: Direct hardware access, manual buffer refill via `fillNext()` interrupt
|
||||
- RMT5: High-level API, encoder runs in ISR but no manual buffer control
|
||||
- Increasing `mem_block_symbols` just makes a larger single buffer
|
||||
- **This is NOT equivalent to RMT4's dual-buffer ping-pong**
|
||||
|
||||
#### Implementation Challenges
|
||||
1. **No Manual Buffer Control**:
|
||||
```cpp
|
||||
// RMT4 had this in interrupt handler:
|
||||
pController->fillNext(true); // Manually refill next buffer half
|
||||
|
||||
// RMT5 LED strip API has no equivalent
|
||||
led_strip_refresh_async(); // Fire and forget, no buffer control
|
||||
```
|
||||
|
||||
2. **Encoder Abstraction**:
|
||||
- Encoder callback runs automatically in ISR context
|
||||
- No way to manually trigger buffer refills
|
||||
- Can't implement true ping-pong without bypassing LED strip API
|
||||
|
||||
#### Feasibility: **COMPLEX - REQUIRES MAJOR REFACTOR**
|
||||
- ❌ NOT a simple one-line change
|
||||
- ❌ LED strip API doesn't support true ping-pong
|
||||
- ⚠️ Would need to bypass LED strip API entirely
|
||||
- ⚠️ Requires reimplementing low-level RMT control like RMT4
|
||||
- ⚠️ Loses benefits of IDF's maintained LED strip driver
|
||||
|
||||
### Memory Impact Analysis
|
||||
|
||||
#### Current Memory Usage
|
||||
- **Without DMA**: 48-64 symbols × 4 bytes = 192-256 bytes per channel
|
||||
- **With DMA**: 1024 symbols × 4 bytes = 4KB per channel
|
||||
|
||||
#### With Double Buffering
|
||||
- **Without DMA**: 384-512 bytes per channel (+192-256 bytes)
|
||||
- **With DMA**: No change (already large)
|
||||
|
||||
For typical 8-channel setup: +1.5-2KB total RAM usage
|
||||
|
||||
### Testing Validation Required
|
||||
|
||||
1. **Reproduce Issue**:
|
||||
- Enable Wi-Fi AP mode + web server
|
||||
- Drive 300+ LEDs on multiple pins
|
||||
- Measure flicker frequency with oscilloscope
|
||||
|
||||
2. **Validate Fix**:
|
||||
- Test both solutions independently
|
||||
- Measure interrupt latency improvement
|
||||
- Verify no regressions on non-Wi-Fi use cases
|
||||
|
||||
### Implementation Priority
|
||||
|
||||
1. **Phase 1 (Immediate)**: Implement ping-pong buffering
|
||||
- Low risk, high reward
|
||||
- Can ship in patch release
|
||||
|
||||
2. **Phase 2 (Future)**: Consider high-priority ISR for extreme cases
|
||||
- Only if ping-pong proves insufficient
|
||||
- Requires extensive platform testing
|
||||
|
||||
### Configuration Recommendations
|
||||
|
||||
Add these defines to allow user tuning:
|
||||
```cpp
|
||||
#define FASTLED_RMT5_MEM_BLOCKS_MULTIPLIER 2 // Double buffer by default
|
||||
#define FASTLED_RMT5_INTERRUPT_PRIORITY 3 // Keep default, allow override
|
||||
#define FASTLED_RMT5_FORCE_IRAM_SAFE 0 // Optional IRAM safety
|
||||
```
|
||||
|
||||
### Solution 3: Bypass LED Strip API (Use Low-Level RMT)
|
||||
|
||||
#### Approach
|
||||
- Abandon `led_strip` API entirely
|
||||
- Use IDF v5's low-level RMT TX API directly
|
||||
- Implement custom encoder with manual buffer management
|
||||
- Port RMT4's `fillNext()` logic to IDF v5 RMT API
|
||||
|
||||
#### Implementation Requirements
|
||||
```cpp
|
||||
// Use raw RMT API instead of led_strip
|
||||
rmt_new_tx_channel(&config, &channel);
|
||||
rmt_new_bytes_encoder(&encoder_config, &encoder);
|
||||
// Implement custom ISR with buffer refill logic
|
||||
```
|
||||
|
||||
#### Feasibility: **MODERATE - MOST VIABLE**
|
||||
- ✅ Gives full control over buffer management
|
||||
- ✅ Can implement true ping-pong like RMT4
|
||||
- ⚠️ Requires significant code rewrite
|
||||
- ⚠️ Must maintain custom encoder implementation
|
||||
- ✅ Proven approach (RMT4 uses this successfully)
|
||||
|
||||
### Revised Conclusion
|
||||
|
||||
**The initial assessment was incorrect.** The LED strip API abstraction in RMT5 fundamentally prevents implementing true ping-pong buffering as a "simple one-line change."
|
||||
|
||||
**Recommended approach:**
|
||||
1. **Immediate**: Increase interrupt priority (Solution 1) - simplest to implement
|
||||
2. **Short-term**: Bypass LED strip API and use low-level RMT (Solution 3) - most effective
|
||||
3. **Long-term**: Work with Espressif to add ping-pong support to LED strip API
|
||||
|
||||
The flickering issue is real and significant, but the solution requires more substantial changes than initially thought. Simply increasing `mem_block_symbols` only creates a larger buffer, not the dual-buffer ping-pong mechanism that made RMT4 resistant to Wi-Fi interference.
|
||||
|
||||
---
|
||||
|
||||
## INTERRUPT IMPLEMENTATION ANALYSIS (2025-09-16)
|
||||
|
||||
### Overview of Current FastLED Interrupt Infrastructure
|
||||
|
||||
Based on analysis of the interrupt header files (`src/platforms/esp/32/interrupts/`), FastLED has already implemented comprehensive infrastructure for high-priority interrupt handling on both Xtensa (ESP32-S3) and RISC-V (ESP32-C3/C6) architectures.
|
||||
|
||||
### Architecture-Specific Implementations
|
||||
|
||||
#### ESP32-S3 (Xtensa LX7) - `esp32_s3.hpp`
|
||||
|
||||
**Key Capabilities**:
|
||||
- **Official Support**: Priority levels 1-3 via `rmt_tx_channel_config_t`
|
||||
- **Experimental Support**: Priority levels 4-5 with custom assembly shims
|
||||
- **Assembly Trampoline Macro**: `FASTLED_ESP_XTENSA_ASM_INTERRUPT_TRAMPOLINE` (`esp32_s3.hpp:316-368`)
|
||||
- **Interrupt Installation Functions**: Pre-built helpers for level 3-5 priority installation
|
||||
|
||||
**Critical Requirements for Level 4-5 Implementation**:
|
||||
```cpp
|
||||
// Level 5 interrupt requires assembly shim (lines 149-161)
|
||||
extern void xt_highint5(void);
|
||||
void IRAM_ATTR fastled_esp32s3_level5_handler(void);
|
||||
|
||||
// Installation helper already available
|
||||
esp_err_t fastled_esp32s3_install_level5_interrupt(
|
||||
int source, void *arg, esp_intr_handle_t *handle);
|
||||
```
|
||||
|
||||
**Build Configuration Required**:
|
||||
- `CONFIG_RMT_ISR_IRAM_SAFE=y`
|
||||
- Assembly `.S` files in build system
|
||||
- `-mlongcalls` compiler flag
|
||||
|
||||
#### ESP32-C3/C6 (RISC-V) - `esp32_c3_c6.hpp`
|
||||
|
||||
**Key Advantages**:
|
||||
- **Simplified Implementation**: C handlers work at any priority level (1-7)
|
||||
- **No Assembly Required**: Standard `IRAM_ATTR` functions sufficient
|
||||
- **PLIC Integration**: Platform-Level Interrupt Controller handles arbitration
|
||||
- **Maximum Priority**: Level 7 (vs. Xtensa level 5)
|
||||
|
||||
**Implementation Benefits**:
|
||||
```cpp
|
||||
// RISC-V can use C handlers directly at high priority
|
||||
void IRAM_ATTR fastled_riscv_experimental_handler(void *arg);
|
||||
|
||||
// Simpler installation (lines 286-292)
|
||||
esp_err_t fastled_riscv_install_interrupt(
|
||||
int source, int priority, void (*handler)(void *),
|
||||
void *arg, esp_intr_handle_t *handle);
|
||||
```
|
||||
|
||||
### Testing Infrastructure Integration
|
||||
|
||||
#### Current Test Framework Compatibility
|
||||
|
||||
The interrupt implementations are designed to integrate with the existing QEMU test infrastructure:
|
||||
|
||||
```bash
|
||||
# ESP32-S3 Xtensa testing
|
||||
uv run test.py --qemu esp32s3
|
||||
|
||||
# ESP32-C3 RISC-V testing
|
||||
uv run test.py --qemu esp32c3
|
||||
```
|
||||
|
||||
#### Proposed Test Case Implementation
|
||||
|
||||
Based on the interrupt header analysis, the test framework should include:
|
||||
|
||||
1. **Level 3 Baseline Tests** (Official FastLED support)
|
||||
- Verify standard `rmt_tx_channel_config_t` integration
|
||||
- Test maximum officially supported priority
|
||||
- Validate against existing FastLED RMT driver
|
||||
|
||||
2. **Level 4-5 Experimental Tests** (ESP32-S3)
|
||||
- Test assembly shim functionality (`FASTLED_ESP_XTENSA_ASM_INTERRUPT_TRAMPOLINE`)
|
||||
- Validate custom RMT initialization bypassing official driver
|
||||
- Measure Wi-Fi interference immunity
|
||||
|
||||
3. **Level 4-7 RISC-V Tests** (ESP32-C3/C6)
|
||||
- Test C handler high-priority interrupts
|
||||
- Compare RISC-V vs Xtensa interrupt latency
|
||||
- Validate PLIC priority arbitration
|
||||
|
||||
### Implementation Readiness Assessment
|
||||
|
||||
#### What's Already Available
|
||||
|
||||
**✅ Complete Infrastructure**:
|
||||
- Assembly trampoline macros for Xtensa high-priority interrupts
|
||||
- C handler support for RISC-V high-priority interrupts
|
||||
- Interrupt installation helper functions for both architectures
|
||||
- Build configuration documentation
|
||||
- IRAM safety considerations documented
|
||||
|
||||
**✅ Test Integration Points**:
|
||||
- QEMU support for both ESP32-S3 and ESP32-C3
|
||||
- Existing test framework can be extended
|
||||
- Performance measurement infrastructure available
|
||||
|
||||
#### What Needs Implementation
|
||||
|
||||
**🔨 RMT Driver Integration**:
|
||||
- `fastled_esp32s3_rmt_init_custom()` function implementation
|
||||
- `fastled_riscv_rmt_init_experimental()` function implementation
|
||||
- Bridge between interrupt infrastructure and RMT hardware configuration
|
||||
|
||||
**🔨 Custom RMT Control**:
|
||||
- Bypass LED strip API for true ping-pong buffering
|
||||
- Low-level RMT register manipulation
|
||||
- Custom encoder implementation for buffer management
|
||||
|
||||
### Recommended Implementation Sequence
|
||||
|
||||
#### Phase 1: High-Priority Interrupt (Ready to Implement)
|
||||
```cpp
|
||||
// ESP32-S3 Level 5 Implementation - infrastructure exists
|
||||
void IRAM_ATTR rmt_level5_handler(void *arg) {
|
||||
// Custom RMT buffer refill logic
|
||||
// Bypass official LED strip API
|
||||
}
|
||||
|
||||
// Use existing trampoline macro
|
||||
FASTLED_ESP_XTENSA_ASM_INTERRUPT_TRAMPOLINE(rmt_level5_isr, rmt_level5_handler)
|
||||
|
||||
// Install with existing helper
|
||||
fastled_esp32s3_install_level5_interrupt(ETS_RMT_INTR_SOURCE, NULL, &handle);
|
||||
```
|
||||
|
||||
#### Phase 2: Custom RMT Implementation
|
||||
```cpp
|
||||
// Bypass led_strip API entirely - use raw RMT API
|
||||
rmt_new_tx_channel(&channel_config, &channel);
|
||||
rmt_new_bytes_encoder(&encoder_config, &encoder);
|
||||
|
||||
// Implement custom buffer management
|
||||
// Port RMT4 fillNext() logic to IDF v5 RMT API
|
||||
```
|
||||
|
||||
#### Phase 3: QEMU Validation
|
||||
- Implement test cases in `tests/qemu_rmt_interrupts.cpp`
|
||||
- Validate both Xtensa and RISC-V implementations
|
||||
- Performance comparison and regression testing
|
||||
|
||||
### Risk Assessment
|
||||
|
||||
**Low Risk (High-Priority Interrupts)**:
|
||||
- Infrastructure already exists and documented
|
||||
- Clear implementation path with existing helpers
|
||||
- QEMU testing available for validation
|
||||
|
||||
**Medium Risk (Custom RMT Implementation)**:
|
||||
- Requires bypassing official LED strip API
|
||||
- Must maintain custom encoder implementation
|
||||
- Compatibility with future IDF versions uncertain
|
||||
|
||||
**Critical Success Factors**:
|
||||
1. **IRAM Safety**: All high-priority code must be IRAM-resident
|
||||
2. **Timing Validation**: QEMU + hardware oscilloscope verification
|
||||
3. **Regression Testing**: Ensure no impact on normal priority operation
|
||||
4. **Documentation**: Clear usage guidelines for different priority levels
|
||||
|
||||
### Conclusion
|
||||
|
||||
The interrupt infrastructure analysis reveals that FastLED already has **comprehensive, production-ready infrastructure** for high-priority interrupt implementation on both Xtensa and RISC-V architectures. The main implementation work required is:
|
||||
|
||||
1. **Integration layer** between interrupt infrastructure and RMT hardware
|
||||
2. **Custom RMT driver** that bypasses LED strip API limitations
|
||||
3. **Test validation** using existing QEMU framework
|
||||
|
||||
This significantly reduces the implementation complexity compared to the initial assessment. The high-priority interrupt solution is **immediately implementable** using existing infrastructure, while the ping-pong buffering solution requires the more substantial custom RMT driver development.
|
||||
@@ -0,0 +1,261 @@
|
||||
# FastLED TEST_CASE Execution Guide
|
||||
|
||||
## Overview
|
||||
|
||||
FastLED uses the **doctest** framework for C++ unit testing. Tests are organized in files named `test_*.cpp` in the `tests/` directory. Each file can contain multiple `TEST_CASE` macros that define individual test functions.
|
||||
|
||||
## Test Structure
|
||||
|
||||
- **Test Files**: Located in `tests/test_*.cpp` (e.g., `test_algorithm.cpp`, `test_easing.cpp`)
|
||||
- **TEST_CASEs**: Individual test functions defined with `TEST_CASE("name")` macro
|
||||
- **SUBCASEs**: Nested test sections within TEST_CASEs using `SUBCASE("name")`
|
||||
- **Assertions**: `CHECK()`, `CHECK_EQ()`, `CHECK_LT()`, etc. for validation
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using the MCP Server (Recommended)
|
||||
|
||||
1. **Start the MCP server:**
|
||||
```bash
|
||||
uv run mcp_server.py
|
||||
```
|
||||
|
||||
2. **Available MCP tools:**
|
||||
- `run_tests`: Run tests with various options
|
||||
- `list_test_cases`: List available TEST_CASEs
|
||||
- `test_instructions`: Show detailed instructions
|
||||
|
||||
### Command Line Usage
|
||||
|
||||
1. **Run all tests:**
|
||||
```bash
|
||||
uv run test.py
|
||||
# or using the user rule:
|
||||
bash test
|
||||
```
|
||||
|
||||
2. **Run only C++ tests:**
|
||||
```bash
|
||||
uv run test.py --cpp
|
||||
```
|
||||
|
||||
3. **Run specific test file:**
|
||||
```bash
|
||||
uv run test.py --cpp algorithm
|
||||
# This runs test_algorithm.cpp
|
||||
```
|
||||
|
||||
## Running Specific TEST_CASEs
|
||||
|
||||
### Method 1: Direct Executable (Most Precise)
|
||||
|
||||
First, compile the tests, then run the specific executable with doctest filtering:
|
||||
|
||||
```bash
|
||||
# 1. Compile the test (if not already compiled)
|
||||
uv run test.py --cpp algorithm
|
||||
|
||||
# 2. Run specific TEST_CASE
|
||||
./tests/.build/bin/test_algorithm --test-case="reverse an int list"
|
||||
|
||||
# 3. Run with verbose output
|
||||
./tests/.build/bin/test_algorithm --test-case="reverse an int list" --verbose
|
||||
```
|
||||
|
||||
### Method 2: Using MCP Server
|
||||
|
||||
Use the `run_tests` tool with specific parameters:
|
||||
- `test_type`: "specific"
|
||||
- `specific_test`: "algorithm" (without test_ prefix)
|
||||
- `test_case`: "reverse an int list"
|
||||
- `verbose`: true (optional)
|
||||
|
||||
### Method 3: List Available TEST_CASEs First
|
||||
|
||||
```bash
|
||||
# Using MCP list_test_cases tool, or manually:
|
||||
grep -r "TEST_CASE" tests/test_algorithm.cpp
|
||||
```
|
||||
|
||||
## Common Test Files and Their Purpose
|
||||
|
||||
| File | Purpose | Example TEST_CASEs |
|
||||
|------|---------|-------------------|
|
||||
| `test_algorithm.cpp` | Algorithm utilities | "reverse an int list" |
|
||||
| `test_easing.cpp` | Easing functions | "8-bit easing functions", "easeInOutQuad16" |
|
||||
| `test_hsv16.cpp` | HSV color space | "HSV to RGB conversion", "HSV hue wraparound" |
|
||||
| `test_math.cpp` | Mathematical functions | "sin32 accuracy", "random number generation" |
|
||||
| `test_vector.cpp` | Vector container | "vector push_back", "vector iteration" |
|
||||
| `test_fx.cpp` | Effects framework | "fx engine initialization" |
|
||||
| `test_apa102_hd.cpp` | APA102 HD driver | "APA102 color mapping" |
|
||||
| `test_bitset.cpp` | Bitset operations | "bitset operations" |
|
||||
|
||||
## Example Workflows
|
||||
|
||||
### Debug a Failing Test
|
||||
|
||||
1. **List TEST_CASEs in a specific file:**
|
||||
```bash
|
||||
# Using MCP server list_test_cases tool with test_file: "easing"
|
||||
```
|
||||
|
||||
2. **Run with verbose output:**
|
||||
```bash
|
||||
uv run test.py --cpp easing --verbose
|
||||
```
|
||||
|
||||
3. **Run specific failing TEST_CASE:**
|
||||
```bash
|
||||
./tests/.build/bin/test_easing --test-case="8-bit easing functions" --verbose
|
||||
```
|
||||
|
||||
### Test Development Workflow
|
||||
|
||||
1. **Clean build and run:**
|
||||
```bash
|
||||
uv run test.py --cpp mytest --clean --verbose
|
||||
```
|
||||
|
||||
2. **Run only failed tests:**
|
||||
```bash
|
||||
cd tests/.build && ctest --rerun-failed
|
||||
```
|
||||
|
||||
3. **Test with different compilers:**
|
||||
```bash
|
||||
uv run test.py --cpp mytest --clang
|
||||
```
|
||||
|
||||
### Finding Specific Tests
|
||||
|
||||
1. **Search for TEST_CASEs by pattern:**
|
||||
```bash
|
||||
# Using MCP list_test_cases tool with search_pattern: "easing"
|
||||
```
|
||||
|
||||
2. **Manual search:**
|
||||
```bash
|
||||
grep -r "TEST_CASE.*easing" tests/
|
||||
```
|
||||
|
||||
## Doctest Command Line Options
|
||||
|
||||
When running test executables directly, you can use these doctest options:
|
||||
|
||||
- `--test-case=<name>`: Run specific TEST_CASE
|
||||
- `--test-case-exclude=<name>`: Exclude specific TEST_CASE
|
||||
- `--subcase=<name>`: Run specific SUBCASE
|
||||
- `--subcase-exclude=<name>`: Exclude specific SUBCASE
|
||||
- `--list-test-cases`: List all TEST_CASEs in the executable
|
||||
- `--list-test-suites`: List all test suites
|
||||
- `--verbose`: Show detailed output including successful assertions
|
||||
- `--success`: Show successful assertions too
|
||||
- `--no-colors`: Disable colored output
|
||||
- `--force-colors`: Force colored output
|
||||
- `--no-breaks`: Disable breaking into debugger
|
||||
- `--no-skip`: Don't skip any tests
|
||||
|
||||
### Examples:
|
||||
|
||||
```bash
|
||||
# List all TEST_CASEs in a test file
|
||||
./tests/.build/bin/test_easing --list-test-cases
|
||||
|
||||
# Run multiple TEST_CASEs with pattern
|
||||
./tests/.build/bin/test_easing --test-case="*quad*"
|
||||
|
||||
# Exclude specific TEST_CASEs
|
||||
./tests/.build/bin/test_easing --test-case-exclude="*16-bit*"
|
||||
|
||||
# Run with full verbose output
|
||||
./tests/.build/bin/test_easing --success --verbose
|
||||
```
|
||||
|
||||
## Understanding Test Output
|
||||
|
||||
### Successful Test:
|
||||
```
|
||||
[doctest] run with "--help" for options
|
||||
===============================================================================
|
||||
TEST_CASE: reverse an int list
|
||||
===============================================================================
|
||||
tests/test_algorithm.cpp:12
|
||||
all tests passed!
|
||||
===============================================================================
|
||||
[doctest] test cases: 1 | 1 passed | 0 failed | 0 skipped
|
||||
[doctest] assertions: 5 | 5 passed | 0 failed |
|
||||
[doctest] Status: SUCCESS!
|
||||
```
|
||||
|
||||
### Failed Test:
|
||||
```
|
||||
===============================================================================
|
||||
TEST_CASE: reverse an int list
|
||||
===============================================================================
|
||||
tests/test_algorithm.cpp:12
|
||||
tests/test_algorithm.cpp:20: ERROR: CHECK_EQ(vec[0], 6) is NOT correct!
|
||||
values: CHECK_EQ(5, 6)
|
||||
|
||||
===============================================================================
|
||||
[doctest] test cases: 1 | 0 passed | 1 failed | 0 skipped
|
||||
[doctest] assertions: 5 | 4 passed | 1 failed |
|
||||
[doctest] Status: FAILURE!
|
||||
```
|
||||
|
||||
## Tips and Best Practices
|
||||
|
||||
1. **Use `--verbose`** to see detailed test output and assertions
|
||||
2. **Use `--clean`** when testing after significant code changes
|
||||
3. **List TEST_CASEs first** to see what's available before running specific tests
|
||||
4. **Individual TEST_CASE execution** is useful for debugging specific functionality
|
||||
5. **Check test output carefully** - doctest provides detailed failure information with line numbers
|
||||
6. **Use pattern matching** with `--test-case="*pattern*"` to run related tests
|
||||
7. **Combine with debugger** - tests are compiled with debug symbols (`-g3`)
|
||||
|
||||
## Environment Setup
|
||||
|
||||
- Tests are compiled with Debug flags (`-g3`, `-O0`) for better debugging
|
||||
- GDB integration available for crash analysis
|
||||
- Static analysis warnings enabled as errors
|
||||
- Cross-platform support (Linux, macOS, Windows)
|
||||
- Uses zig/clang as the default compiler for consistency
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues:
|
||||
|
||||
1. **Test executable not found:**
|
||||
```bash
|
||||
# Make sure tests are compiled first
|
||||
uv run test.py --cpp <test_name>
|
||||
```
|
||||
|
||||
2. **TEST_CASE name not found:**
|
||||
```bash
|
||||
# List available TEST_CASEs
|
||||
./tests/.build/bin/test_<name> --list-test-cases
|
||||
```
|
||||
|
||||
3. **Compilation errors:**
|
||||
```bash
|
||||
# Clean build
|
||||
uv run test.py --cpp <test_name> --clean
|
||||
```
|
||||
|
||||
4. **Permission denied on executable:**
|
||||
```bash
|
||||
chmod +x tests/.build/bin/test_<name>
|
||||
```
|
||||
|
||||
## Integration with Development
|
||||
|
||||
### Git Hooks
|
||||
Consider adding a pre-commit hook to run relevant tests:
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Run tests for changed files
|
||||
uv run test.py --cpp
|
||||
```
|
||||
|
||||
### IDE Integration
|
||||
Most IDEs can be configured to run individual TEST_CASEs directly from the editor by configuring the test executable path and doctest arguments.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Testing
|
||||
|
||||
* REPO SYNC MODE: When there is a compiler error there isn't a hard failure. The compiler error needs to get propagate back to the top.
|
||||
|
||||
|
||||
* Esp32 testing
|
||||
* https://github.com/marketplace/actions/esp32-qemu-runner will run a sketch for X seconds and see's if it crashes
|
||||
* There's specific tests we'd like to run with this including the WS2812 and APA102 tests to test the clockless and clocked drivers
|
||||
|
||||
# Feature Enhancements
|
||||
|
||||
[ ] Adafruit converter driver
|
||||
[ ] NeoPixel converter driver
|
||||
|
||||
* I2S driver for ESP32 WS2812
|
||||
* https://github.com/hpwit/I2SClocklessLedDriver
|
||||
* Our copy is here: https://github.com/FastLED/FastLED/blob/master/src/platforms/esp/32/clockless_i2s_esp32.h
|
||||
* S3:
|
||||
* https://github.com/hpwit/I2SClockLessLedDriveresp32s3
|
||||
* Apparently, this driver allows MASSIVE parallelization for WS2812
|
||||
* Timing guide for reducing RMT frequency https://github.com/Makuna/NeoPixelBus/pull/795
|
||||
* ESp32 LED guide
|
||||
* web: https://components.espressif.com/components/espressif/led_strip
|
||||
* repo: https://github.com/espressif/idf-extra-components/tree/60c14263f3b69ac6e98ecae79beecbe5c18d5596/led_strip
|
||||
* adafruit conversation on RMT progress: https://github.com/adafruit/Adafruit_NeoPixel/issues/375
|
||||
|
||||
|
||||
* MIT Licensed SdFat library
|
||||
* https://github.com/greiman/SdFat
|
||||
* YVes LittleFS implementation for ESP
|
||||
* https://github.com/hpwit/ledOS/blob/main/src/fileSystem.h
|
||||
|
||||
* NimBLE for Arduino
|
||||
* https://github.com/h2zero/NimBLE-Arduino?tab=readme-ov-file
|
||||
|
||||
* Arduino test compile
|
||||
* https://github.com/hpwit/arduino-test-compile/blob/master/arduino-test-compile.sh
|
||||
|
||||
|
||||
# Misc:
|
||||
|
||||
* sutaburosu's guide to playing around with FastLED 4
|
||||
* https://github.com/sutaburosu/FastLED4-ESP32-playpen
|
||||
|
||||
|
||||
# Animartrix: try using sinf, cosf and other found trip instead of trying to go
|
||||
full double precision.
|
||||
@@ -0,0 +1,352 @@
|
||||
# Video Recording Feature for FastLED Web App
|
||||
|
||||
## Implementation Status: ✅ COMPLETE - FULLY IMPLEMENTED
|
||||
|
||||
**Last Updated**: September 18, 2025
|
||||
**Commit**: Refactored video recording code to fix bugs and improve reliability
|
||||
|
||||
### Overview
|
||||
A video recording feature has been implemented for the FastLED web application in `src/platforms/wasm/compiler/`. This feature allows users to capture both the canvas animations and audio output to create high-quality video recordings.
|
||||
|
||||
## Features Implemented
|
||||
|
||||
### 1. Recording Button UI
|
||||
- **Location**: Upper right corner of the canvas
|
||||
- **Visual States**:
|
||||
- **Idle**: Gray circular button with a record icon (⚫)
|
||||
- **Recording**: Red pulsing button with a stop icon (⬛)
|
||||
- **Animation**: Smooth pulse effect when recording
|
||||
- **Note**: Stdout button should be moved to upper left of canvas to avoid conflict with record button placement
|
||||
|
||||
### 2. Video Capture Specifications
|
||||
- **Default Format**: MP4 with H.264 video codec and AAC audio codec
|
||||
- **Fallback Formats**: WebM with VP9/VP8 video codec and Opus audio codec (auto-detected based on browser support)
|
||||
- **Video Bitrate**: 10 Mbps (high quality)
|
||||
- **Audio Bitrate**: 128 kbps
|
||||
- **Frame Rate**: 30 FPS default (configurable to 60 FPS)
|
||||
- **Resolution**: Captures at canvas native resolution
|
||||
- **Frame Logging**: Each frame insertion is logged to console (e.g., "frame 16")
|
||||
- **Automatic File Extension**: Correct file extension (.mp4 or .webm) based on selected codec
|
||||
- **Recording Statistics**: Logs frame count, duration, FPS, and file size upon completion
|
||||
- **Improved Error Handling**: Graceful fallback when MediaRecorder is unavailable
|
||||
|
||||
### 3. Audio Capture
|
||||
- **Source**: Web Audio API AudioContext
|
||||
- **Integration**: Automatically captures audio if AudioContext is available
|
||||
- **MP3 Player Integration**:
|
||||
- Captures audio from the existing MP3 player module
|
||||
- Audio streams from MP3 playback are routed through the AudioContext
|
||||
- Mixed audio (synthesized + MP3) is captured in the recording
|
||||
- **Audio Mixing**: Combines all audio sources (synthesized sounds + MP3 playback) into a single stream
|
||||
- **Fallback**: Records video-only if audio is unavailable
|
||||
- **Sync**: Audio and video are automatically synchronized
|
||||
|
||||
### 4. User Interaction
|
||||
- **Start Recording**: Click the record button or press `Ctrl+R` / `Cmd+R`
|
||||
- **Stop Recording**: Click the button again (now showing stop icon)
|
||||
- **File Save**: Automatic download prompt when recording stops
|
||||
- **Filename Format**: `fastled-recording-YYYY-MM-DD-HH-MM-SS.{mp4|webm}` (extension matches codec)
|
||||
- **Graceful Degradation**: Record button automatically hidden if MediaRecorder unsupported
|
||||
|
||||
## Frame Handling & Encoder Optimizations
|
||||
|
||||
### Dropped Frame Detection & Recovery
|
||||
The video recording system implements comprehensive dropped frame handling:
|
||||
|
||||
#### Detection Methods
|
||||
- **Frame Counter Monitoring**: Tracks expected vs. actual frame sequence
|
||||
- **Timestamp Analysis**: Compares MediaRecorder timestamps with canvas refresh rate
|
||||
- **Performance Observer**: Monitors frame presentation delays and skips
|
||||
|
||||
#### Recovery Strategies
|
||||
```javascript
|
||||
// Dropped frame detection
|
||||
let expectedFrameCount = 0;
|
||||
let actualFrameCount = 0;
|
||||
const frameDropThreshold = 2; // Allow 2 consecutive drops before intervention
|
||||
|
||||
mediaRecorder.addEventListener('dataavailable', (event) => {
|
||||
actualFrameCount++;
|
||||
const timestamp = performance.now();
|
||||
const expectedTimestamp = startTime + (expectedFrameCount * (1000 / fps));
|
||||
|
||||
if (timestamp - expectedTimestamp > frameDropThreshold * (1000 / fps)) {
|
||||
console.warn(`Dropped frames detected: expected ${expectedFrameCount}, got ${actualFrameCount}`);
|
||||
// Trigger recovery actions
|
||||
handleDroppedFrames(expectedFrameCount - actualFrameCount);
|
||||
}
|
||||
|
||||
console.log(`frame ${actualFrameCount}`);
|
||||
expectedFrameCount++;
|
||||
});
|
||||
|
||||
function handleDroppedFrames(droppedCount) {
|
||||
// Option 1: Duplicate last frame to maintain timing
|
||||
// Option 2: Reduce capture frame rate temporarily
|
||||
// Option 3: Skip frame interpolation for smooth playback
|
||||
}
|
||||
```
|
||||
|
||||
### Duplicate Frame Optimization
|
||||
|
||||
#### H.264 Encoder Optimizations
|
||||
The H.264 encoder in MP4 format provides several optimizations for static or repetitive content:
|
||||
|
||||
- **P-frames (Predicted frames)**: Encode only differences from previous frames
|
||||
- **B-frames (Bidirectional frames)**: Reference both previous and future frames for maximum compression
|
||||
- **Temporal Compression**: Automatically detects static regions and encodes efficiently
|
||||
- **Rate Control**: Adjusts bitrate dynamically based on content complexity
|
||||
|
||||
#### Implementation Details
|
||||
```javascript
|
||||
// Configure encoder for optimal duplicate frame handling
|
||||
const encoderConfig = {
|
||||
mimeType: 'video/mp4;codecs=h264,aac',
|
||||
videoBitsPerSecond: 10000000,
|
||||
audioBitsPerSecond: 128000,
|
||||
// H.264 specific optimizations
|
||||
keyFrameInterval: 30, // I-frame every 30 frames (1 second at 30fps)
|
||||
frameSkipping: true, // Allow encoder to skip identical frames
|
||||
contentHint: 'motion' // Optimize for animation content
|
||||
};
|
||||
|
||||
// Monitor encoding efficiency
|
||||
mediaRecorder.addEventListener('start', () => {
|
||||
console.log('Encoder supports duplicate frame optimization: ',
|
||||
mediaRecorder.mimeType.includes('h264'));
|
||||
});
|
||||
```
|
||||
|
||||
#### Fallback Encoder Behavior
|
||||
When MP4/H.264 is not available, the system falls back to WebM/VP9:
|
||||
|
||||
- **VP9 Optimization**: Similar temporal compression but different algorithm
|
||||
- **Opus Audio**: Efficient silence detection and compression
|
||||
- **WebM Container**: Supports variable frame rates for dropped frame scenarios
|
||||
|
||||
#### Performance Monitoring
|
||||
```javascript
|
||||
// Track encoding performance
|
||||
const encodingStats = {
|
||||
framesEncoded: 0,
|
||||
framesDuplicated: 0,
|
||||
bytesGenerated: 0,
|
||||
compressionRatio: 0
|
||||
};
|
||||
|
||||
mediaRecorder.addEventListener('dataavailable', (event) => {
|
||||
encodingStats.bytesGenerated += event.data.size;
|
||||
encodingStats.framesEncoded++;
|
||||
|
||||
// Estimate duplicate frame optimization
|
||||
const expectedSize = (canvas.width * canvas.height * 3) / 8; // RGB to bytes
|
||||
const actualSize = event.data.size;
|
||||
const efficiency = 1 - (actualSize / expectedSize);
|
||||
|
||||
if (efficiency > 0.8) {
|
||||
encodingStats.framesDuplicated++;
|
||||
console.log(`High compression frame ${encodingStats.framesEncoded} (${efficiency.toFixed(2)} efficiency)`);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
### Files Created/Modified
|
||||
|
||||
1. **`modules/video_recorder.js`** (NEW)
|
||||
- Core VideoRecorder class
|
||||
- MediaRecorder API integration
|
||||
- Canvas stream capture
|
||||
- Audio context integration with MP3 player support
|
||||
- Mixed audio capture (MP3 + synthesized sounds)
|
||||
- File save functionality
|
||||
|
||||
2. **`index.html`** (MODIFIED)
|
||||
- Added record button HTML structure
|
||||
- SVG icons for record/stop states
|
||||
|
||||
3. **`index.css`** (MODIFIED)
|
||||
- Record button styling
|
||||
- Animation effects (pulse, hover)
|
||||
- State transitions
|
||||
|
||||
4. **`index.js`** (MODIFIED)
|
||||
- Video recorder initialization
|
||||
- Button event handling
|
||||
- Keyboard shortcut support
|
||||
- Global debug functions
|
||||
|
||||
## API Documentation
|
||||
|
||||
### MediaRecorder API Usage
|
||||
|
||||
```javascript
|
||||
// Canvas capture
|
||||
canvas.captureStream(fps) // Creates MediaStream from canvas
|
||||
|
||||
// Audio capture with MP3 mixing
|
||||
audioContext.createMediaStreamDestination() // Creates audio destination node
|
||||
|
||||
// MP3 player audio routing
|
||||
mp3Player.connect(audioDestination) // Route MP3 audio to recording destination
|
||||
synthesizer.connect(audioDestination) // Route synthesized audio to destination
|
||||
|
||||
// Combined stream with mixed audio
|
||||
new MediaStream([
|
||||
...videoStream.getVideoTracks(),
|
||||
...audioDestination.stream.getAudioTracks() // Contains mixed audio from all sources
|
||||
])
|
||||
|
||||
// Recording with mixed audio
|
||||
mediaRecorder = new MediaRecorder(stream, {
|
||||
mimeType: 'video/mp4;codecs=h264,aac', // Default: MP4 with H.264 video and AAC audio
|
||||
videoBitsPerSecond: 10000000,
|
||||
audioBitsPerSecond: 128000 // Captures all mixed audio sources
|
||||
})
|
||||
|
||||
// Frame logging during recording
|
||||
mediaRecorder.addEventListener('dataavailable', (event) => {
|
||||
console.log(`frame ${++frameCounter}`); // Logs each frame insertion
|
||||
});
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **File Save Timing**: Files are saved AFTER recording stops (not when it starts)
|
||||
- Ensures precise start/stop timing for A/V sync
|
||||
- User sees save dialog after recording completes
|
||||
- No filename prompt interrupts recording start
|
||||
|
||||
2. **High Bitrate Choice**: 10 Mbps video bitrate
|
||||
- Ensures high quality capture
|
||||
- Suitable for 60 FPS content
|
||||
- Can be compressed later if needed
|
||||
|
||||
3. **MP4/H.264 Format**: Primary codec choice
|
||||
- Universal browser support for high-quality video
|
||||
- Good compression efficiency with AAC audio
|
||||
- Supports both video and audio tracks
|
||||
- WebM/VP9 fallback for browsers with limited MP4 support
|
||||
|
||||
## Usage Instructions
|
||||
|
||||
### For Users
|
||||
1. Click the gray record button in the upper-right corner of the canvas
|
||||
2. Button turns red and pulses to indicate recording
|
||||
3. Perform your LED animation
|
||||
4. Click the red button to stop recording
|
||||
5. Browser will prompt to save the video file
|
||||
|
||||
### For Developers
|
||||
|
||||
#### Global Functions (Console Access)
|
||||
```javascript
|
||||
// Get recorder instance
|
||||
window.getVideoRecorder()
|
||||
|
||||
// Start recording programmatically
|
||||
window.startVideoRecording()
|
||||
|
||||
// Stop recording programmatically
|
||||
window.stopVideoRecording()
|
||||
```
|
||||
|
||||
#### Keyboard Shortcuts
|
||||
- `Ctrl+R` / `Cmd+R`: Toggle recording on/off
|
||||
|
||||
## Browser Compatibility
|
||||
|
||||
- ✅ Chrome/Chromium (Full support)
|
||||
- ✅ Firefox (Full support)
|
||||
- ✅ Edge (Full support)
|
||||
- ⚠️ Safari (Limited codec support, uses fallback)
|
||||
|
||||
## Recent Bug Fixes & Improvements
|
||||
|
||||
### September 18, 2025 Refactor
|
||||
- **Fixed codec configuration**: Changed default from VP9 to H.264/MP4 for better compatibility
|
||||
- **Improved audio routing**: Enhanced audio connection logic for more reliable audio capture
|
||||
- **Dynamic file extensions**: Automatically uses correct extension (.mp4 or .webm) based on codec
|
||||
- **Enhanced error handling**: Added graceful fallback when MediaRecorder API is unavailable
|
||||
- **Better resource cleanup**: Improved disposal method to prevent memory leaks
|
||||
- **Frame logging implementation**: Added proper frame counting and logging as documented
|
||||
- **Recording statistics**: Added comprehensive logging of recording metrics (duration, FPS, file size)
|
||||
- **Canvas content detection**: Improved blank canvas detection without interfering with actual content
|
||||
- **UI positioning fix**: Moved stdout label to upper left to avoid conflicts with record button
|
||||
|
||||
### Bug Fixes Addressed
|
||||
1. **Inconsistent codec defaults** - Now prioritizes H.264/MP4 over VP9/WebM
|
||||
2. **Audio connection failures** - Improved audio routing with better error handling
|
||||
3. **Wrong file extensions** - Files now use correct extension matching the codec
|
||||
4. **Missing frame logging** - Frame counter now properly logs each frame
|
||||
5. **Memory leaks** - Enhanced cleanup in disposal and error scenarios
|
||||
6. **Canvas interference** - Removed test pattern drawing that could interfere with content
|
||||
7. **MediaRecorder detection** - Added proper feature detection and graceful degradation
|
||||
|
||||
## Future Enhancements (Optional)
|
||||
|
||||
- [ ] Settings menu for quality presets
|
||||
- [ ] Frame rate selector (30/60 FPS toggle)
|
||||
- [ ] Recording timer display
|
||||
- [ ] Pause/resume functionality
|
||||
- [ ] Custom filename input
|
||||
- [ ] Multiple format export options
|
||||
- [ ] Recording preview before save
|
||||
- [ ] Audio level meters for MP3 and synthesized audio
|
||||
- [ ] Separate audio track controls (mute individual sources)
|
||||
- [ ] Audio normalization for mixed sources
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No Audio in Recording
|
||||
- Check if browser has microphone permissions
|
||||
- Verify AudioContext is initialized
|
||||
- Some browsers require user interaction to start AudioContext
|
||||
- Ensure MP3 player audio nodes are properly connected to the recording destination
|
||||
- Verify that audio mixing is working (check browser console for audio node errors)
|
||||
|
||||
### Recording Button Not Appearing
|
||||
- Check browser console for errors
|
||||
- Verify canvas element exists with ID "myCanvas"
|
||||
- Ensure JavaScript modules are loading correctly
|
||||
|
||||
### Low Quality Video
|
||||
- Browser may be using fallback codec
|
||||
- Check available disk space
|
||||
- Try different browser for VP9 support
|
||||
|
||||
### Dropped Frames Issues
|
||||
- **High CPU Usage**: Reduce canvas complexity or frame rate
|
||||
- **Memory Pressure**: Monitor browser memory usage, restart if needed
|
||||
- **Background Tabs**: Ensure recording tab has focus for optimal performance
|
||||
- **Hardware Acceleration**: Enable GPU acceleration in browser settings
|
||||
|
||||
### Encoder Optimization Not Working
|
||||
- **H.264 Unavailable**: Check browser codec support with `MediaRecorder.isTypeSupported()`
|
||||
- **Poor Compression**: Verify content has static regions for optimization
|
||||
- **Frame Skipping Disabled**: Some browsers may not support frameSkipping parameter
|
||||
- **Content Hint Ignored**: Fallback to default encoding if contentHint not supported
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
### Basic Functionality
|
||||
- [x] Record button appears in correct position
|
||||
- [x] Button changes color when recording
|
||||
- [x] Pulse animation works during recording
|
||||
- [x] Canvas content is captured correctly
|
||||
- [x] Audio is captured (when available)
|
||||
- [x] File saves with correct timestamp
|
||||
- [x] Keyboard shortcut works
|
||||
- [x] Multiple start/stop cycles work correctly
|
||||
- [x] Memory is properly released after recording
|
||||
|
||||
### Frame Handling & Optimization
|
||||
- [x] Frame logging appears in console during recording
|
||||
- [x] Dropped frame detection triggers warnings when appropriate
|
||||
- [x] Static content shows high compression efficiency
|
||||
- [x] H.264 encoder optimization is active when supported
|
||||
- [x] Fallback to WebM/VP9 works when MP4 unavailable
|
||||
- [x] Performance monitoring tracks encoding statistics
|
||||
- [x] Recovery strategies activate during frame drops
|
||||
- [x] Variable frame rate handling works under load
|
||||
@@ -0,0 +1,57 @@
|
||||
I was dissatisfied with a gamma corrected ramp using the APA102HD mode (it seemed to visibly step at one or more points), and some analysis of the 8 bit rgb + 5 bit brightness function showed why:
|
||||
https://www.argyllcms.com/APA102_loglog.svg.
|
||||
|
||||
So here is a suggested replacement for five_bit_bitshift() in fl/five_bit_hd_gamma.cpp, that fixed the problem for me:
|
||||
|
||||
// Improved FastLED five_bit_bitshift() function
|
||||
// This appears to exactly match the floating point reference,
|
||||
// with a worst case resulting error of 0.2% over the full 16 bit input range,
|
||||
// and an average error of 0.05% over that range. Errors scale with maximum
|
||||
// magnitude.
|
||||
|
||||
// ix/31 * 255/65536 * 256 scaling factors, valid for indexes 1..31
|
||||
static uint32_t bright_scale[32] = {
|
||||
0, 2023680, 1011840, 674560, 505920, 404736, 337280, 289097,
|
||||
252960, 224853, 202368, 183971, 168640, 155668, 144549, 134912,
|
||||
126480, 119040, 112427, 106509, 101184, 96366, 91985, 87986,
|
||||
84320, 80947, 77834, 74951, 72274, 69782, 67456, 65280
|
||||
};
|
||||
|
||||
// Since the return value wasn't used, it has been omitted.
|
||||
// It's not clear what scale brightness is, or how it is to be applied,
|
||||
// so we assume 8 bits applied over the given rgb values.
|
||||
void five_bit_bitshift(uint16_t r16, uint16_t g16, uint16_t b16, uint8_t brightness,
|
||||
CRGB *out, uint8_t *out_power_5bit) {
|
||||
uint8_t r8 = 0, g8 = 0, b8 = 0;
|
||||
|
||||
// Apply any brightness setting (we assume brightness is 0..255)
|
||||
if (brightness != 0xff) {
|
||||
r16 = scale16by8(r16, brightness);
|
||||
g16 = scale16by8(g16, brightness);
|
||||
b16 = scale16by8(b16, brightness);
|
||||
}
|
||||
|
||||
// Locate the largest value to set the brightness/scale factor
|
||||
uint16_t scale = max3(r16, g16, b16);
|
||||
|
||||
if (scale == 0) {
|
||||
*out = CRGB(0, 0, 0);
|
||||
*out_power_5bit = 0;
|
||||
return;
|
||||
} else {
|
||||
uint32_t scalef;
|
||||
|
||||
// Compute 5 bit quantized scale that is at or above the maximum value.
|
||||
scale = (scale + (2047 - (scale >> 5))) >> 11;
|
||||
|
||||
// Adjust the 16 bit values to account for the scale, then round to 8 bits
|
||||
scalef = bright_scale[scale];
|
||||
r8 = (r16 * scalef + 0x808000) >> 24;
|
||||
g8 = (g16 * scalef + 0x808000) >> 24;
|
||||
b8 = (b16 * scalef + 0x808000) >> 24;
|
||||
|
||||
*out = CRGB(r8, g8, b8);
|
||||
*out_power_5bit = scale;
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,246 @@
|
||||
# FastLED CI/Build System Agent Guidelines
|
||||
|
||||
## 🔧 Python Build System
|
||||
|
||||
### Core Architecture
|
||||
FastLED uses a high-performance Python-based build system:
|
||||
|
||||
**Key Components:**
|
||||
- `ci/compiler/clang_compiler.py` - Core compiler with PCH and fingerprint cache
|
||||
- `ci/ci/fingerprint_cache.py` - 99% faster PCH rebuilds through intelligent caching
|
||||
- `ci/util/build_flags.py` - TOML-based build configuration management
|
||||
- `ci/compiler/test_example_compilation.py` - Example compilation system
|
||||
|
||||
**Performance Features:**
|
||||
- **PCH (Precompiled Headers)** - Dramatically faster compilation
|
||||
- **Fingerprint Cache** - 99% time reduction for unchanged dependencies
|
||||
- **Parallel Compilation** - Automatic CPU-optimized parallel builds
|
||||
- **Smart Rebuilds** - Only rebuild when source files actually change
|
||||
|
||||
### 🚨 Command Execution Rules
|
||||
|
||||
#### FOREGROUND AGENTS (Interactive Development)
|
||||
**Python Code Execution:**
|
||||
- ✅ **Create/modify tmp.py** in project root (NOT subdirectories)
|
||||
- ✅ **Always run: `uv run python tmp.py`** (never just `python`)
|
||||
- ✅ **Correct import paths:** `from ci.ci.module` for ci.ci modules, `from ci.module` for top-level ci modules
|
||||
- ❌ **NEVER cd to subdirectories** - stay in project root
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
# tmp.py (created in project root)
|
||||
from ci.ci.clang_compiler import Compiler # CORRECT import path
|
||||
import subprocess
|
||||
result = subprocess.run(['git', 'status'], capture_output=True, text=True)
|
||||
print(result.stdout)
|
||||
```
|
||||
Then run: `uv run python tmp.py`
|
||||
|
||||
#### BACKGROUND AGENTS (Automated/CI Environments)
|
||||
**Restrictions:**
|
||||
- ❌ **NO tmp.py files** (forbidden for background agents)
|
||||
- ✅ **Use existing scripts:** `uv run python ci/ci-compile.py uno --examples Blink`
|
||||
- ✅ **Use MCP server tools** for programmatic operations
|
||||
- ✅ **Always use `uv run python`** with existing scripts
|
||||
|
||||
## 🤖 MCP Server Configuration
|
||||
|
||||
**Start server:** `uv run mcp_server.py`
|
||||
|
||||
**Available Tools:**
|
||||
- Running tests with various options
|
||||
- Compiling examples for different platforms
|
||||
- Code fingerprinting and change detection
|
||||
- Linting and formatting
|
||||
- **Build info analysis** for platform-specific defines, compiler flags, toolchain information
|
||||
- **Symbol analysis** for binary optimization (all platforms)
|
||||
- Stack trace setup for enhanced debugging
|
||||
- **🌐 FastLED Web Compiler** with Playwright (FOREGROUND AGENTS ONLY)
|
||||
- **🚨 `validate_completion` tool** (MANDATORY for background agents)
|
||||
|
||||
### FastLED Web Compiler (FOREGROUND AGENTS ONLY)
|
||||
|
||||
**Prerequisites:**
|
||||
- `pip install fastled` - FastLED web compiler
|
||||
- `pip install playwright` - Browser automation
|
||||
- Docker (optional, for faster compilation)
|
||||
|
||||
**Usage via MCP Server:**
|
||||
```python
|
||||
# Basic usage
|
||||
use run_fastled_web_compiler tool with:
|
||||
- example_path: "examples/Audio"
|
||||
- capture_duration: 30
|
||||
- headless: false
|
||||
- save_screenshot: true
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- Compiles Arduino sketches to WASM for browser execution
|
||||
- Captures console.log output with playwright automation
|
||||
- Takes screenshots of running visualizations
|
||||
- Monitors FastLED_onFrame calls to verify proper initialization
|
||||
|
||||
**🚫 BACKGROUND AGENT RESTRICTION:** This tool is COMPLETELY DISABLED for background agents and CI environments.
|
||||
|
||||
## 🤖 Linting Guidelines
|
||||
|
||||
### FOREGROUND AGENTS
|
||||
**🚨 ALWAYS USE `bash lint` - DO NOT RUN INDIVIDUAL LINTING SCRIPTS**
|
||||
|
||||
- ✅ **CORRECT:** `bash lint`
|
||||
- ❌ **FORBIDDEN:** `./lint-js`, `./check-js`, `python3 scripts/enhance-js-typing.py`
|
||||
- ❌ **FORBIDDEN:** `uv run ruff check`, `uv run black`, individual tools
|
||||
|
||||
**WHY:** `bash lint` provides comprehensive coverage of Python, C++, JavaScript, and enhancement analysis.
|
||||
|
||||
### BACKGROUND AGENTS
|
||||
**CAN USE FINE-GRAINED LINTING FOR SPECIFIC NEEDS**
|
||||
|
||||
Background agents may use individual linting scripts when needed:
|
||||
- `./lint-js` - JavaScript-only linting
|
||||
- `./check-js` - JavaScript type checking
|
||||
- `python3 scripts/enhance-js-typing.py` - JavaScript enhancement analysis
|
||||
- `uv run ruff check` - Python linting only
|
||||
- MCP server `lint_code` tool - Programmatic access
|
||||
|
||||
**BUT STILL PREFER `bash lint` FOR COMPREHENSIVE CHECKING**
|
||||
|
||||
## Build System Commands
|
||||
|
||||
**Unit Test Compilation:**
|
||||
```bash
|
||||
# Fast Python API (default)
|
||||
bash test --unit --verbose # Uses PCH optimization
|
||||
|
||||
# Disable PCH if needed
|
||||
bash test --unit --no-pch --verbose
|
||||
|
||||
# Legacy CMake system (8x slower)
|
||||
bash test --unit --legacy --verbose
|
||||
```
|
||||
|
||||
**Available Build Options:**
|
||||
- `--no-pch` - Disable precompiled headers
|
||||
- `--clang` - Use Clang compiler (recommended for speed)
|
||||
- `--clean` - Force full rebuild
|
||||
- `--verbose` - Show detailed compilation output
|
||||
- `--legacy` - Use old CMake system (discouraged)
|
||||
|
||||
**Platform Compilation:**
|
||||
```bash
|
||||
# Compile examples for specific platforms
|
||||
uv run ci/ci-compile.py uno --examples Blink
|
||||
uv run ci/ci-compile.py esp32dev --examples Blink
|
||||
uv run ci/ci-compile.py teensy31 --examples Blink
|
||||
```
|
||||
|
||||
**WASM Compilation:**
|
||||
```bash
|
||||
# Compile any sketch directory to WASM
|
||||
uv run ci/wasm_compile.py path/to/your/sketch
|
||||
|
||||
# Quick compile test (compile only, no browser)
|
||||
uv run ci/wasm_compile.py examples/Blink --just-compile
|
||||
|
||||
# Compile and open browser
|
||||
uv run ci/wasm_compile.py examples/Blink -b --open
|
||||
```
|
||||
|
||||
## Build Troubleshooting
|
||||
|
||||
**For Linker Issues:**
|
||||
1. Check `tests/cmake/LinkerCompatibility.cmake` first
|
||||
2. Look for lld-link detection and compatibility functions
|
||||
3. Check GNU→MSVC flag translation logic
|
||||
4. Verify warning suppression settings
|
||||
|
||||
**For Compiler Issues:**
|
||||
1. Check `tests/cmake/CompilerDetection.cmake` for detection logic
|
||||
2. Review `tests/cmake/CompilerFlags.cmake` for flag conflicts
|
||||
3. Verify optimization settings in `OptimizationSettings.cmake`
|
||||
|
||||
**For Build Performance:**
|
||||
1. Check `tests/cmake/ParallelBuild.cmake` for parallel settings
|
||||
2. Review LTO configuration in `OptimizationSettings.cmake`
|
||||
3. Verify linker selection (mold, lld, default)
|
||||
|
||||
## 🚨 Critical User Feedback Corrections
|
||||
|
||||
**Always Use `uv run python` - NEVER just `python` or `uv run`:**
|
||||
- ❌ **WRONG**: `python -c "..."`
|
||||
- ❌ **WRONG**: `uv run -c "..."`
|
||||
- ✅ **CORRECT**: `uv run python -c "..."`
|
||||
|
||||
**Correct Import Paths:**
|
||||
- ✅ **CORRECT**: `from ci.ci.clang_compiler import ...` (for ci.ci modules)
|
||||
- ✅ **CORRECT**: `from ci.clang_compiler import ...` (for top-level ci modules, if they exist)
|
||||
|
||||
**Never Change Directory:**
|
||||
- ❌ **WRONG**: `cd ci && python ...`
|
||||
- ❌ **WRONG**: `cd ci/ci && python ...`
|
||||
- ✅ **CORRECT**: Stay in project root, use full paths: `uv run python ci/script.py`
|
||||
|
||||
**Git-Bash Terminal Compatibility:**
|
||||
Use leading space for git-bash compatibility:
|
||||
- ✅ **CORRECT**: ` bash test` (note the leading space)
|
||||
- ❌ **INCORRECT**: `bash test` (may get truncated to `ash test`)
|
||||
|
||||
## Platform Build Information Analysis
|
||||
|
||||
**Generate Build Info:**
|
||||
```bash
|
||||
# Compile a platform to generate build_info.json
|
||||
uv run ci/ci-compile.py uno --examples Blink
|
||||
uv run ci/ci-compile.py esp32dev --examples Blink
|
||||
```
|
||||
|
||||
**Analyze Platform Defines:**
|
||||
```bash
|
||||
# Command line tool
|
||||
python3 ci/ci/build_info_analyzer.py --board uno --show-defines
|
||||
|
||||
# Via MCP server
|
||||
uv run mcp_server.py
|
||||
# Use build_info_analysis tool with: board="uno", show_defines=true
|
||||
```
|
||||
|
||||
**Compare Platforms:**
|
||||
```bash
|
||||
# Command line
|
||||
python3 ci/ci/build_info_analyzer.py --compare uno esp32dev
|
||||
|
||||
# Via MCP
|
||||
# Use build_info_analysis tool with: board="uno", compare_with="esp32dev"
|
||||
```
|
||||
|
||||
## Symbol Analysis for Binary Optimization
|
||||
|
||||
**Analyze specific platform:**
|
||||
```bash
|
||||
uv run ci/ci/symbol_analysis.py --board uno
|
||||
uv run ci/ci/symbol_analysis.py --board esp32dev
|
||||
uv run ci/ci/symbol_analysis.py --board teensy31
|
||||
```
|
||||
|
||||
**Analyze all available platforms:**
|
||||
```bash
|
||||
uv run ci/demo_symbol_analysis.py
|
||||
```
|
||||
|
||||
**Using MCP Server:**
|
||||
```bash
|
||||
# Use MCP server tools
|
||||
uv run mcp_server.py
|
||||
# Then use symbol_analysis tool with board: "uno" or "auto"
|
||||
# Or use symbol_analysis tool with run_all_platforms: true
|
||||
```
|
||||
|
||||
**JSON Output:**
|
||||
```bash
|
||||
uv run symbol_analysis.py --board esp32dev --output-json
|
||||
# Results saved to: .build/esp32dev_symbol_analysis.json
|
||||
```
|
||||
|
||||
## Memory Refresh Rule
|
||||
**🚨 ALL AGENTS: Read ci/AGENTS.md before concluding CI/build work to refresh memory about current Python build system rules and MCP server requirements.**
|
||||
@@ -0,0 +1,228 @@
|
||||
# FastLED Python Build System - Implementation Summary
|
||||
|
||||
## Overview
|
||||
|
||||
Successfully implemented a new Python-based build system to replace the complex CMake approach for FastLED example compilation testing. The new system delivers on all design goals: **Simple, Fast, and Transparent**.
|
||||
|
||||
## Key Achievements
|
||||
|
||||
### ⚡ Performance Results
|
||||
- **4.3 examples/second** compilation rate
|
||||
- **90%+ faster** than CMake-based approach
|
||||
- **Sub-second builds** for incremental changes
|
||||
- **Intelligent caching** with 30%+ hit rates on repeated builds
|
||||
|
||||
### 🏗️ Architecture Implemented
|
||||
|
||||
```
|
||||
FastLED Python Build System
|
||||
├── ci/build_system/
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ ├── build_config.py # Configuration management
|
||||
│ ├── example_scanner.py # .ino file discovery & analysis
|
||||
│ ├── change_detector.py # Fast file change detection
|
||||
│ ├── compiler.py # Direct Clang compilation
|
||||
│ ├── cache_manager.py # Build cache management
|
||||
│ └── fastled_build.py # Main build engine
|
||||
├── ci/fastled_build # CLI entry point
|
||||
├── ci/test_example_compilation_python.py # New test script
|
||||
└── ci/test_example_compilation.py # Updated original (with --use-python-build flag)
|
||||
```
|
||||
|
||||
### 🎯 Core Features Delivered
|
||||
|
||||
#### 1. **Fast Example Discovery**
|
||||
- Scans 80+ examples in milliseconds
|
||||
- Categorizes by FastLED usage and PCH compatibility
|
||||
- Intelligent filtering of problematic examples
|
||||
|
||||
#### 2. **Content-Based Change Detection**
|
||||
- SHA-256 fingerprinting of source files
|
||||
- Dependency tracking for headers
|
||||
- Smart cache invalidation
|
||||
|
||||
#### 3. **Precompiled Header Optimization**
|
||||
- Automatic PCH generation for FastLED headers
|
||||
- PCH reuse across compatible examples
|
||||
- 50%+ compilation speed improvement
|
||||
|
||||
#### 4. **Intelligent Caching**
|
||||
- Content-addressable build cache
|
||||
- sccache integration for compiler-level caching
|
||||
- Automatic cleanup of stale entries
|
||||
|
||||
#### 5. **Parallel Compilation**
|
||||
- Multi-threaded compilation (16 cores utilized)
|
||||
- ThreadPoolExecutor-based job management
|
||||
- Optimal resource utilization
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### CLI Tool
|
||||
```bash
|
||||
# Quick status check
|
||||
./fastled_build status
|
||||
|
||||
# Build all examples
|
||||
./fastled_build build
|
||||
|
||||
# Build specific examples
|
||||
./fastled_build build Blink DemoReel100
|
||||
|
||||
# Clean build
|
||||
./fastled_build clean
|
||||
./fastled_build build
|
||||
|
||||
# System diagnostics
|
||||
./fastled_build diagnostics
|
||||
```
|
||||
|
||||
### Test Scripts
|
||||
```bash
|
||||
# Default Python build system (recommended)
|
||||
python test_example_compilation.py
|
||||
|
||||
# Alternative Python build system script
|
||||
python test_example_compilation_python.py
|
||||
|
||||
# Both scripts now use Python build system by default
|
||||
python test_example_compilation.py Blink DemoReel100 # Python system
|
||||
```
|
||||
|
||||
## Technical Implementation Details
|
||||
|
||||
### Example Categorization
|
||||
- **FastLED PCH Compatible**: Examples that can use precompiled headers
|
||||
- **FastLED Full Build**: Examples requiring full compilation
|
||||
- **Basic Examples**: Non-FastLED examples
|
||||
- **Platform Specific**: Hardware-dependent examples
|
||||
- **Problematic**: Known issues (excluded by default)
|
||||
|
||||
### Build Configuration
|
||||
- **Compiler**: Auto-detection (Clang preferred)
|
||||
- **Platform**: STUB implementation for testing
|
||||
- **Flags**: C++17, -O0 for fast compilation
|
||||
- **Arduino Compatibility**: WASM Arduino.h integration
|
||||
|
||||
### Performance Optimizations
|
||||
- **Fast Paths**: "No changes" detection in < 1 second
|
||||
- **PCH Reuse**: Share PCH across compatible examples
|
||||
- **Content Hashing**: Only hash changed files
|
||||
- **Parallel Processing**: All I/O and compilation parallelized
|
||||
|
||||
## Migration Status
|
||||
|
||||
### ✅ Phase 1: Parallel Implementation - COMPLETE
|
||||
- ✅ Implemented Python build system alongside CMake
|
||||
- ✅ Added `--use-python-build` flag to test script
|
||||
- ✅ Validated equivalent functionality
|
||||
- ✅ Performance testing shows 4x+ improvement
|
||||
|
||||
### 🚧 Phase 2: Feature Parity - IN PROGRESS
|
||||
- ✅ All CMake features replicated
|
||||
- ✅ Performance optimization completed
|
||||
- ✅ CLI and API documentation
|
||||
- ⏳ Extended platform testing
|
||||
|
||||
### ✅ Phase 3: Migration - COMPLETE
|
||||
- ✅ Switched default to Python build system
|
||||
- ✅ Removed CMake implementation completely
|
||||
- ✅ Updated `test_example_compilation.py` to use Python build system only
|
||||
- ✅ Maintained backward compatibility with same CLI interface
|
||||
|
||||
### 🧹 Phase 4: Cleanup - ⚠️ **INCOMPLETE**
|
||||
- ❌ Remove misleading "CMake" references in function names and error messages
|
||||
- ❌ Clean up dead code paths that reference non-existent CMake directories
|
||||
- ❌ Fix confusing system names (both systems are Python-based)
|
||||
- ❌ Update all documentation to reflect accurate system architecture
|
||||
- ⚠️ **Status**: Misleading naming caused confusion and bugs; cleanup required
|
||||
|
||||
## Performance Comparison
|
||||
|
||||
| Metric | CMake System | Python System | Improvement |
|
||||
|--------|--------------|---------------|-------------|
|
||||
| Cold start | 15-30s | 1-2s | **15x faster** |
|
||||
| Incremental (5 examples) | 8-15s | 1-3s | **5x faster** |
|
||||
| Full build (80 examples) | 45-60s | 10-15s | **4x faster** |
|
||||
| Cache hit rate | ~60% | ~90% | **50% better** |
|
||||
| Memory usage | 2-4GB | 200-500MB | **80% less** |
|
||||
|
||||
## Error Handling & Diagnostics
|
||||
|
||||
### Comprehensive Error Reporting
|
||||
- Clear, actionable error messages
|
||||
- Compilation error aggregation
|
||||
- Performance timing breakdown
|
||||
- Cache statistics and health
|
||||
|
||||
### Diagnostic Tools
|
||||
```bash
|
||||
./fastled_build diagnostics
|
||||
```
|
||||
Shows:
|
||||
- Compiler information and availability
|
||||
- Cache statistics and hit rates
|
||||
- sccache integration status
|
||||
- File change detection stats
|
||||
|
||||
## Integration with Existing Workflow
|
||||
|
||||
### Backward Compatibility
|
||||
- ✅ Original `test_example_compilation.py` still works with same interface
|
||||
- ✅ Same command-line arguments and exit codes
|
||||
- ✅ Same behavior and output format
|
||||
- ✅ Complete drop-in replacement (CMake removed)
|
||||
- ✅ Performance improvement with no API changes
|
||||
|
||||
### Testing Integration
|
||||
```python
|
||||
# Easy integration in existing test scripts
|
||||
from build_system import BuildConfig, FastLEDBuildEngine
|
||||
|
||||
config = BuildConfig.from_environment()
|
||||
engine = FastLEDBuildEngine(config)
|
||||
result = engine.build(examples=["Blink", "DemoReel100"])
|
||||
assert result.success
|
||||
```
|
||||
|
||||
## Benefits Realized
|
||||
|
||||
### 1. **Simplicity**
|
||||
- ✅ Direct compiler control (no CMake abstraction)
|
||||
- ✅ Python stack traces for easy debugging
|
||||
- ✅ Transparent, understandable build logic
|
||||
|
||||
### 2. **Performance**
|
||||
- ✅ 4x+ faster than CMake approach
|
||||
- ✅ Intelligent caching tailored to FastLED
|
||||
- ✅ Efficient incremental builds
|
||||
|
||||
### 3. **Maintainability**
|
||||
- ✅ Python codebase (team expertise)
|
||||
- ✅ Modular, testable design
|
||||
- ✅ Clear separation of concerns
|
||||
|
||||
### 4. **Flexibility**
|
||||
- ✅ Easy FastLED-specific optimizations
|
||||
- ✅ Simple tool integration
|
||||
- ✅ Consistent cross-platform behavior
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Extended Testing**: Test with more examples and edge cases
|
||||
2. **Documentation**: Update development docs to recommend Python system
|
||||
3. **CI Integration**: Consider switching CI to use Python build system
|
||||
4. **Feature Additions**: Add requested features (linking, size analysis, etc.)
|
||||
|
||||
## Conclusion
|
||||
|
||||
⚠️ **MIGRATION STATUS**: The Python build system has successfully replaced the CMake approach, but **Phase 4 cleanup was never completed**, leaving misleading function names and error messages that reference "CMake" when both systems are actually Python-based.
|
||||
|
||||
### Final Results
|
||||
- **🚀 13.3 examples/second** compilation rate (in cached scenarios)
|
||||
- **⚡ Ultra-fast builds** with intelligent caching
|
||||
- **🔄 100% backward compatibility** with existing workflows
|
||||
- **🧹 Clean codebase** with CMake complexity completely removed
|
||||
- **📊 Superior performance** across all metrics
|
||||
|
||||
The system delivers on all original design goals: **Simple, Fast, and Transparent**. However, **incomplete cleanup has left confusing naming** that makes the system less transparent than intended. The naming issues have been identified and are being addressed.
|
||||
@@ -0,0 +1,11 @@
|
||||
This is the Continuous Integration tool that builds the product in various configurations.
|
||||
|
||||
Running
|
||||
|
||||
Install the python `uv` tool.
|
||||
|
||||
`pip install uv`
|
||||
|
||||
Now run the python in this directory
|
||||
|
||||
`uv run ci-compile.py`
|
||||
@@ -0,0 +1,313 @@
|
||||
# FastLED Symbol Analysis
|
||||
|
||||
This document describes the new generic symbol analysis functionality that works across all supported platforms including UNO, ESP32, and others.
|
||||
|
||||
## Overview
|
||||
|
||||
The symbol analysis tool provides comprehensive analysis of compiled ELF files to help identify optimization opportunities and understand binary size allocation. Unlike the previous ESP32-specific implementation, this tool:
|
||||
|
||||
- ✅ **Works on ALL platforms** (UNO, ESP32, Teensy, STM32, etc.)
|
||||
- ✅ **Pulls ALL symbols** without filtering or classification
|
||||
- ✅ **No FastLED-specific filtering** - analyzes everything
|
||||
- ✅ **Generic and extensible** for any platform with build_info.json
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Analyze a specific platform
|
||||
|
||||
```bash
|
||||
cd ci
|
||||
uv run symbol_analysis.py --board uno
|
||||
uv run symbol_analysis.py --board esp32dev
|
||||
```
|
||||
|
||||
### Run analysis on all available platforms
|
||||
|
||||
```bash
|
||||
uv run demo_symbol_analysis.py
|
||||
```
|
||||
|
||||
### Use in GitHub Actions
|
||||
|
||||
```bash
|
||||
uv run symbol_analysis_runner.py --board uno --example Blink
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
- **`symbol_analysis.py`** - Main generic symbol analysis tool
|
||||
- **`symbol_analysis_runner.py`** - GitHub Actions wrapper
|
||||
- **`demo_symbol_analysis.py`** - Demonstration script for multiple platforms
|
||||
- **`SYMBOL_ANALYSIS.md`** - This documentation file
|
||||
|
||||
## Example Output
|
||||
|
||||
### UNO Platform Results
|
||||
|
||||
```
|
||||
================================================================================
|
||||
UNO SYMBOL ANALYSIS REPORT
|
||||
================================================================================
|
||||
|
||||
SUMMARY:
|
||||
Total symbols: 51
|
||||
Total symbol size: 3767 bytes (3.7 KB)
|
||||
|
||||
LARGEST SYMBOLS (all symbols, sorted by size):
|
||||
1. 1204 bytes - ClocklessController<...>::showPixels(...)
|
||||
2. 572 bytes - CFastLED::show(unsigned char) [clone .constprop.18]
|
||||
3. 460 bytes - main
|
||||
4. 204 bytes - CPixelLEDController<...>::show(...)
|
||||
5. 168 bytes - CPixelLEDController<...>::showColor(...)
|
||||
|
||||
SYMBOL TYPE BREAKDOWN:
|
||||
t: 19 symbols, 2710 bytes (2.6 KB)
|
||||
T: 14 symbols, 904 bytes (0.9 KB)
|
||||
b: 15 symbols, 89 bytes (0.1 KB)
|
||||
d: 3 symbols, 64 bytes (0.1 KB)
|
||||
```
|
||||
|
||||
### ESP32 Platform Results
|
||||
|
||||
```
|
||||
================================================================================
|
||||
ESP32DEV SYMBOL ANALYSIS REPORT
|
||||
================================================================================
|
||||
|
||||
SUMMARY:
|
||||
Total symbols: 2503
|
||||
Total symbol size: 237092 bytes (231.5 KB)
|
||||
|
||||
LARGEST SYMBOLS (all symbols, sorted by size):
|
||||
1. 12009 bytes - _vfprintf_r
|
||||
2. 11813 bytes - _svfprintf_r
|
||||
3. 8010 bytes - _vfiprintf_r
|
||||
4. 7854 bytes - _svfiprintf_r
|
||||
5. 4192 bytes - port_IntStack
|
||||
|
||||
SYMBOL TYPE BREAKDOWN:
|
||||
T: 1240 symbols, 156230 bytes (152.6 KB)
|
||||
t: 371 symbols, 45841 bytes (44.8 KB)
|
||||
d: 475 symbols, 13032 bytes (12.7 KB)
|
||||
D: 60 symbols, 9421 bytes (9.2 KB)
|
||||
W: 126 symbols, 5209 bytes (5.1 KB)
|
||||
```
|
||||
|
||||
### ESP32S3 Platform Results
|
||||
|
||||
```
|
||||
================================================================================
|
||||
ESP32S3 SYMBOL ANALYSIS REPORT
|
||||
================================================================================
|
||||
|
||||
SUMMARY:
|
||||
Total symbols: 2831
|
||||
Total symbol size: 260710 bytes (254.6 KB)
|
||||
|
||||
LARGEST SYMBOLS (all symbols, sorted by size):
|
||||
1. 11309 bytes - _vfprintf_r
|
||||
2. 11118 bytes - _svfprintf_r
|
||||
3. 7337 bytes - _svfiprintf_r
|
||||
4. 7265 bytes - _vfiprintf_r
|
||||
5. 4192 bytes - port_IntStack
|
||||
6. 1164 bytes - printBeforeSetupInfo()
|
||||
7. 828 bytes - fl::RmtController5::loadPixelData(PixelIterator&)
|
||||
|
||||
SYMBOL TYPE BREAKDOWN:
|
||||
T: 1416 symbols, 172591 bytes (168.5 KB)
|
||||
t: 429 symbols, 48854 bytes (47.7 KB)
|
||||
d: 529 symbols, 14189 bytes (13.9 KB)
|
||||
D: 72 symbols, 9831 bytes (9.6 KB)
|
||||
W: 122 symbols, 5457 bytes (5.3 KB)
|
||||
```
|
||||
|
||||
## Output Files
|
||||
|
||||
The tool generates JSON files with complete symbol data:
|
||||
|
||||
- `.build/{board}_symbol_analysis.json` - Complete analysis results
|
||||
|
||||
### JSON Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": {
|
||||
"board": "uno",
|
||||
"total_symbols": 51,
|
||||
"total_size": 3767,
|
||||
"largest_symbols": [...],
|
||||
"type_breakdown": {...},
|
||||
"dependencies": {...}
|
||||
},
|
||||
"all_symbols_sorted_by_size": [
|
||||
{
|
||||
"address": "00001240",
|
||||
"size": 1204,
|
||||
"type": "t",
|
||||
"name": "_ZN19ClocklessController...",
|
||||
"demangled_name": "ClocklessController<...>::showPixels(...)"
|
||||
},
|
||||
...
|
||||
],
|
||||
"dependencies": {
|
||||
"module.cpp.o": ["symbol1", "symbol2", ...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Supported Platforms
|
||||
|
||||
The tool automatically detects and works with any platform that has:
|
||||
- A `build_info.json` file in `.build/{platform}/`
|
||||
- Standard toolchain tools (nm, c++filt) in the build info aliases
|
||||
|
||||
Successfully tested platforms:
|
||||
- ✅ **UNO** (AVR) - 51 symbols, 3.7 KB
|
||||
- ✅ **ESP32DEV** (Xtensa) - 2503 symbols, 231.5 KB
|
||||
- ✅ **ESP32S3** (Xtensa) - 2831 symbols, 254.6 KB
|
||||
- ✅ **TEENSY31** (ARM Cortex-M4) - 238 symbols, 10.9 KB
|
||||
- ✅ **TEENSYLC** (ARM Cortex-M0+) - 241 symbols, 9.1 KB
|
||||
- ✅ **DIGIX** (ARM Cortex-M3) - 298 symbols, 17.9 KB
|
||||
- ✅ **STM32** (ARM Cortex-M3) - 311 symbols, 15.6 KB
|
||||
|
||||
## Key Improvements
|
||||
|
||||
### From ESP32-specific to Generic
|
||||
|
||||
**Before (ESP32 only):**
|
||||
```python
|
||||
# Hard-coded ESP32 board detection
|
||||
esp32_boards = ["esp32dev", "esp32", "esp32s2", ...]
|
||||
|
||||
# FastLED-specific filtering
|
||||
if any(keyword in search_text for keyword in [
|
||||
"fastled", "cfastled", "crgb", "hsv", ...]):
|
||||
fastled_symbols.append(symbol_info)
|
||||
```
|
||||
|
||||
**After (Generic):**
|
||||
```python
|
||||
# Auto-detect any platform with build_info.json
|
||||
for item in build_dir.iterdir():
|
||||
if item.is_dir() and (item / "build_info.json").exists():
|
||||
available_boards.append((build_info_file, item.name))
|
||||
|
||||
# No filtering - analyze ALL symbols
|
||||
symbols.append(symbol_info) # Everything included
|
||||
```
|
||||
|
||||
### Symbol Analysis Improvements
|
||||
|
||||
1. **No Classification** - All symbols are included without filtering
|
||||
2. **Complete Demangling** - All C++ symbols are properly demangled
|
||||
3. **Platform Agnostic** - Works with any toolchain (AVR, ARM, Xtensa, etc.)
|
||||
4. **Size-based Sorting** - All results sorted by symbol size for easy optimization targeting
|
||||
|
||||
## GitHub Actions Integration
|
||||
|
||||
The `symbol_analysis_runner.py` script provides GitHub Actions compatibility:
|
||||
|
||||
```yaml
|
||||
- name: Run Symbol Analysis
|
||||
run: python ci/symbol_analysis_runner.py --board uno --example Blink
|
||||
```
|
||||
|
||||
This runner:
|
||||
- Validates build_info.json exists
|
||||
- Handles errors gracefully with `--skip-on-failure`
|
||||
- Provides proper exit codes for CI/CD
|
||||
|
||||
## Migration from ESP32 Analysis
|
||||
|
||||
If you were using the old ESP32-specific analysis:
|
||||
|
||||
**Old command:**
|
||||
```bash
|
||||
cd ci/ci && python esp32_symbol_analysis.py
|
||||
```
|
||||
|
||||
**New command:**
|
||||
```bash
|
||||
cd ci && uv run symbol_analysis.py --board esp32dev
|
||||
```
|
||||
|
||||
The new tool provides all the same information plus:
|
||||
- Support for more platforms
|
||||
- ALL symbols instead of just FastLED ones
|
||||
- Better error handling
|
||||
- Cleaner output format
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Analyzing Custom Platforms
|
||||
|
||||
To add support for a new platform:
|
||||
|
||||
1. Ensure the platform builds correctly with PlatformIO
|
||||
2. Verify `build_info.json` is generated in `.build/{platform}/`
|
||||
3. Run: `uv run symbol_analysis.py --board {platform}`
|
||||
|
||||
### Batch Analysis
|
||||
|
||||
```bash
|
||||
# Analyze all available platforms
|
||||
uv run demo_symbol_analysis.py
|
||||
|
||||
# Analyze specific platforms in sequence
|
||||
for board in uno esp32dev teensy31; do
|
||||
uv run symbol_analysis.py --board $board
|
||||
done
|
||||
```
|
||||
|
||||
### Integration with Build Systems
|
||||
|
||||
```python
|
||||
# In your Python build script
|
||||
from ci.ci.symbol_analysis import main as analyze_symbols
|
||||
import sys
|
||||
|
||||
# Override arguments
|
||||
sys.argv = ["symbol_analysis.py", "--board", "uno"]
|
||||
analyze_symbols()
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **"build_info.json not found"**
|
||||
- Solution: Compile the platform first: `uv run ci/ci-compile.py {board} --examples Blink`
|
||||
|
||||
2. **"KeyError: 'board_name'"**
|
||||
- Solution: The board name in the directory doesn't match the key in build_info.json
|
||||
- Check: `cat .build/{board}/build_info.json | jq 'keys'`
|
||||
|
||||
3. **"Tool not found"**
|
||||
- Solution: The toolchain isn't installed or path is incorrect in build_info.json
|
||||
|
||||
### Debug Mode
|
||||
|
||||
To debug issues, run with verbose Python output:
|
||||
```bash
|
||||
uv run python -v symbol_analysis.py --board uno
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new features:
|
||||
|
||||
1. Maintain platform independence
|
||||
2. Don't add filtering logic
|
||||
3. Test on multiple platforms
|
||||
4. Update this documentation
|
||||
5. Add tests to verify functionality
|
||||
|
||||
## Performance
|
||||
|
||||
Platform analysis times (approximate):
|
||||
- **UNO**: < 1 second (51 symbols)
|
||||
- **Teensy**: < 2 seconds (238-241 symbols)
|
||||
- **STM32**: < 3 seconds (311 symbols)
|
||||
- **ESP32**: < 10 seconds (2503 symbols)
|
||||
|
||||
The tool scales well with symbol count and provides progress feedback during demangling.
|
||||
@@ -0,0 +1,13 @@
|
||||
"""FastLED CI tools and utilities
|
||||
|
||||
Module-level initialization to ensure consistent runtime behavior across tools.
|
||||
"""
|
||||
|
||||
# Configure UTF-8 console output on Windows globally for CI tools
|
||||
try:
|
||||
from ci.util.console_utf8 import configure_utf8_console
|
||||
|
||||
configure_utf8_console()
|
||||
except Exception:
|
||||
# Never fail import due to console configuration
|
||||
pass
|
||||
@@ -0,0 +1,45 @@
|
||||
|
||||
|
||||
|
||||
#define FASTLED_USE_ADAFRUIT_NEOPIXEL
|
||||
#include "FastLED.h"
|
||||
|
||||
// Test the non-template Adafruit bridge
|
||||
#define DATA_PIN 2
|
||||
#define NUM_LEDS 144
|
||||
|
||||
// Test LED arrays with different RGB orders
|
||||
CRGB leds_grb[NUM_LEDS];
|
||||
CRGB leds_rgb[NUM_LEDS];
|
||||
CRGB leds_bgr[NUM_LEDS];
|
||||
|
||||
void setup() {
|
||||
Serial.begin(115200);
|
||||
|
||||
// Test template controllers
|
||||
FastLED.addLeds<WS2812, DATA_PIN, GRB>(leds_grb, NUM_LEDS);
|
||||
FastLED.addLeds<WS2812, DATA_PIN+1, RGB>(leds_rgb, NUM_LEDS);
|
||||
FastLED.addLeds<WS2812, DATA_PIN+2, BGR>(leds_bgr, NUM_LEDS);
|
||||
|
||||
// Fill with test patterns
|
||||
fill_rainbow(leds_grb, NUM_LEDS, 0, 255/NUM_LEDS);
|
||||
fill_solid(leds_rgb, NUM_LEDS, CRGB::Red);
|
||||
fill_solid(leds_bgr, NUM_LEDS, CRGB::Blue);
|
||||
|
||||
FastLED.show();
|
||||
Serial.println("Adafruit bridge test initialized");
|
||||
}
|
||||
|
||||
void loop() {
|
||||
static uint8_t hue = 0;
|
||||
|
||||
// Update colors
|
||||
fill_rainbow(leds_grb, NUM_LEDS, hue, 255/NUM_LEDS);
|
||||
fill_solid(leds_rgb, NUM_LEDS, CHSV(hue, 255, 255));
|
||||
fill_solid(leds_bgr, NUM_LEDS, CHSV(hue + 128, 255, 255));
|
||||
|
||||
FastLED.show();
|
||||
|
||||
hue += 2;
|
||||
delay(50);
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
[platformio]
|
||||
src_dir = symlink://../../../src
|
||||
|
||||
[env:dev]
|
||||
platform = https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip
|
||||
board = esp32-s3-devkitc-1
|
||||
framework = arduino
|
||||
lib_deps =
|
||||
adafruit/Adafruit NeoPixel
|
||||
|
||||
lib_ldf_mode=deep
|
||||
build_flags=-DFASTLED_STUB_MAIN_INCLUDE_INO="../ci/adafruit_bridge/adafruit_bridge.ino.cpp"
|
||||
-DFASTLED_USE_ADAFRUIT_NEOPIXEL=1
|
||||
@@ -0,0 +1,769 @@
|
||||
# dataclasses
|
||||
|
||||
import json
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
# An open source version of the esp-idf 5.1 platform for the ESP32 that
|
||||
# gives esp32 boards the same build environment as the Arduino 2.3.1+.
|
||||
|
||||
# Set to a specific release, we may want to update this in the future.
|
||||
ESP32_IDF_5_1_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip"
|
||||
|
||||
# TODO: Upgrade toolkit to 5.3
|
||||
ESP32_IDF_5_3_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/53.03.10/platform-espressif32.zip"
|
||||
ESP32_IDF_5_4_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/54.03.20/platform-espressif32.zip"
|
||||
ESP32_IDF_5_5_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/55.03.30-2/platform-espressif32.zip"
|
||||
ESP32_IDF_LATEST_PIOARDUINO = ESP32_IDF_5_5_PIOARDUINO
|
||||
|
||||
ESP32_IDF_LATEST_PIOARDUINO = (
|
||||
"https://github.com/pioarduino/platform-espressif32.git#develop"
|
||||
)
|
||||
ESP32_IDF_4_4_LATEST = (
|
||||
"https://github.com/platformio/platform-espressif32/archive/refs/tags/v4.4.0.zip"
|
||||
)
|
||||
APOLLO3_2_2_0 = "https://github.com/nigelb/platform-apollo3blue"
|
||||
# Top of trunk.
|
||||
# ESP32_IDF_5_1_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32"
|
||||
|
||||
# Old fork that we were using
|
||||
# ESP32_IDF_5_1_PIOARDUINO = "https://github.com/zackees/platform-espressif32#Arduino/IDF5"
|
||||
|
||||
# ALL will be auto populated in the Board constructor whenever a
|
||||
# board is defined.
|
||||
ALL: list["Board"] = []
|
||||
|
||||
|
||||
class Auto:
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class Board:
|
||||
board_name: str
|
||||
real_board_name: str | None = None
|
||||
platform: str | None = None
|
||||
platform_needs_install: bool = False
|
||||
use_pio_run: bool = (
|
||||
False # some platforms like esp32-c2-devkitm-1 will only work with pio run
|
||||
)
|
||||
platform_packages: str | None = None
|
||||
framework: str | None = None
|
||||
board_build_mcu: str | None = None
|
||||
board_build_core: str | None = None
|
||||
board_build_filesystem_size: str | None = None
|
||||
build_flags: list[str] | None = None # Reserved for future use.
|
||||
build_unflags: list[str] | None = None # New: unflag options
|
||||
defines: list[str] | None = None
|
||||
customsdk: str | None = None
|
||||
board_partitions: str | None = None # Reserved for future use.
|
||||
no_board_spec: bool = (
|
||||
False # For platforms like 'native' that don't need a board specification
|
||||
)
|
||||
add_board_to_all: bool = True
|
||||
lib_compat_mode: str | None = (
|
||||
None # Library compatibility mode (e.g., 'off' for native platform)
|
||||
)
|
||||
lib_ldf_mode: str | None = (
|
||||
None # Library Dependency Finder mode (e.g., 'chain+' for enhanced dependency finding)
|
||||
)
|
||||
lib_ignore: list[str] | None = (
|
||||
None # Libraries to ignore during compilation (e.g., ['I2S'] for UNO R4 WiFi)
|
||||
)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Check if framework is set, warn and auto-set to arduino if missing (except for native/stub platforms)
|
||||
if self.framework is None and not self._is_native_or_stub_platform():
|
||||
self.framework = "arduino"
|
||||
|
||||
# Auto-detect and exclude problematic libraries based on environmental signals
|
||||
self._auto_detect_library_exclusions()
|
||||
|
||||
ALL.append(self)
|
||||
|
||||
def _is_native_or_stub_platform(self) -> bool:
|
||||
"""Check if this is a native or stub platform that doesn't need a framework"""
|
||||
# Check for native platform
|
||||
if self.platform and (
|
||||
"native" in self.platform.lower() or "stub" in self.platform.lower()
|
||||
):
|
||||
return True
|
||||
|
||||
# Check for stub build flags
|
||||
if self.build_flags:
|
||||
for flag in self.build_flags:
|
||||
if "FASTLED_STUB" in flag or "PLATFORM_NATIVE" in flag:
|
||||
return True
|
||||
|
||||
# Check no_board_spec flag (typically used for native platforms)
|
||||
if self.no_board_spec:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def _auto_detect_library_exclusions(self) -> None:
|
||||
"""Automatically detect and exclude libraries known to be broken on this platform."""
|
||||
|
||||
# Check for I2S library problems based on environmental signals
|
||||
if self._should_ignore_i2s_library():
|
||||
if not self.lib_ignore:
|
||||
self.lib_ignore = []
|
||||
if "I2S" not in self.lib_ignore:
|
||||
self.lib_ignore.append("I2S")
|
||||
|
||||
def _should_ignore_i2s_library(self) -> bool:
|
||||
"""Detect if I2S library should be ignored based on platform characteristics."""
|
||||
|
||||
# Known problematic platforms - Renesas RA family
|
||||
if self.platform == "renesas-ra":
|
||||
return True
|
||||
|
||||
# Known problematic board names
|
||||
problematic_boards = ["uno_r4_wifi", "uno_r4_minima"]
|
||||
if self.board_name in problematic_boards:
|
||||
return True
|
||||
|
||||
# Check for specific defines that indicate I2S problems
|
||||
if self.defines:
|
||||
problematic_defines = [
|
||||
"ARDUINO_UNOR4_WIFI",
|
||||
"ARDUINO_UNOR4_MINIMA",
|
||||
"ARDUINO_ARCH_RENESAS",
|
||||
"ARDUINO_ARCH_RENESAS_UNO",
|
||||
"_RENESAS_RA_",
|
||||
"ARDUINO_FSP",
|
||||
]
|
||||
for define in self.defines:
|
||||
# Check if define contains any problematic patterns
|
||||
if any(prob_define in define for prob_define in problematic_defines):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def clone(self) -> "Board":
|
||||
out = Board(
|
||||
board_name=self.board_name,
|
||||
add_board_to_all=False,
|
||||
)
|
||||
for field_name, field_info in self.__dataclass_fields__.items():
|
||||
field_value: Any = getattr(self, field_name)
|
||||
# Create deep copy for mutable types to avoid shared references
|
||||
if isinstance(field_value, (list, dict)):
|
||||
field_value = deepcopy(field_value) # type: ignore[misc]
|
||||
setattr(out, field_name, field_value)
|
||||
return out
|
||||
|
||||
def get_real_board_name(self) -> str:
|
||||
return self.real_board_name if self.real_board_name else self.board_name
|
||||
|
||||
def to_dictionary(self) -> dict[str, list[str]]:
|
||||
out: dict[str, list[str]] = {}
|
||||
if self.real_board_name:
|
||||
out[self.board_name] = [f"board={self.real_board_name}"]
|
||||
options = out.setdefault(self.board_name, [])
|
||||
|
||||
if self.platform:
|
||||
options.append(f"platform={self.platform}")
|
||||
|
||||
if self.platform_needs_install:
|
||||
options.append("platform_needs_install=true")
|
||||
if self.platform_packages:
|
||||
options.append(f"platform_packages={self.platform_packages}")
|
||||
if self.framework:
|
||||
options.append(f"framework={self.framework}")
|
||||
if self.board_build_core:
|
||||
options.append(f"board_build.core={self.board_build_core}")
|
||||
if self.board_build_mcu:
|
||||
options.append(f"board_build.mcu={self.board_build_mcu}")
|
||||
if self.board_build_filesystem_size:
|
||||
options.append(
|
||||
f"board_build.filesystem_size={self.board_build_filesystem_size}"
|
||||
)
|
||||
if self.defines:
|
||||
for define in self.defines:
|
||||
options.append(f"build_flags=-D{define}")
|
||||
|
||||
# Handle build_unflags
|
||||
if self.build_unflags:
|
||||
for uf in self.build_unflags:
|
||||
options.append(f"build_unflags={uf}")
|
||||
|
||||
# Handle explicit build_flags (added for native host compilation and other special cases)
|
||||
if self.build_flags:
|
||||
for bf in self.build_flags:
|
||||
options.append(f"build_flags={bf}")
|
||||
|
||||
if self.customsdk:
|
||||
options.append(f"custom_sdkconfig={self.customsdk}")
|
||||
|
||||
# Add board-specific build cache directory pointing via symlink directive
|
||||
# here = Path(__file__).parent
|
||||
# project_root = here.parent.parent # Move from ci/util/ to project root
|
||||
# cache_dir = project_root / ".pio_cache" / self.board_name
|
||||
# absolute_cache_dir = cache_dir.resolve()
|
||||
# options.append(f"build_cache_dir=symlink://{absolute_cache_dir}")
|
||||
|
||||
return out
|
||||
|
||||
def __repr__(self) -> str:
|
||||
json_str = json.dumps(self.to_dictionary(), indent=4, sort_keys=True)
|
||||
return json_str
|
||||
|
||||
def __hash__(self) -> int:
|
||||
data_str = self.__repr__()
|
||||
return hash(data_str)
|
||||
|
||||
def to_platformio_ini(
|
||||
self,
|
||||
additional_defines: list[str] | None = None,
|
||||
additional_include_dirs: list[str] | None = None,
|
||||
additional_libs: list[str] | None = None,
|
||||
include_platformio_section: bool = False,
|
||||
core_dir: str | None = None,
|
||||
packages_dir: str | None = None,
|
||||
project_root: str | None = None,
|
||||
build_cache_dir: str | None = None,
|
||||
extra_scripts: list[str] | None = None,
|
||||
) -> str:
|
||||
"""Return a `platformio.ini` snippet representing this board.
|
||||
|
||||
The output is suitable for directly appending to a *platformio.ini* file
|
||||
and follows the same semantics used by the PlatformIO CLI. Only
|
||||
parameters understood by PlatformIO are emitted – internal helper
|
||||
fields like ``platform_needs_install`` and ``use_pio_run`` are **not**
|
||||
included because they are consumed exclusively by the build helpers in
|
||||
the *ci/* folder and would be ignored (or flagged as errors) by
|
||||
PlatformIO itself.
|
||||
|
||||
Args:
|
||||
example: Example name for dynamic build flags (currently unused)
|
||||
additional_defines: Additional defines to merge with board defines
|
||||
additional_include_dirs: Additional include directories to merge with build flags
|
||||
include_platformio_section: Whether to include [platformio] section with core_dir/packages_dir
|
||||
core_dir: PlatformIO core directory path
|
||||
packages_dir: PlatformIO packages directory path
|
||||
project_root: FastLED project root for lib_deps symlink
|
||||
"""
|
||||
lines: list[str] = []
|
||||
|
||||
# Optional [platformio] section
|
||||
if include_platformio_section:
|
||||
lines.append("[platformio]")
|
||||
if build_cache_dir:
|
||||
lines.append(f"build_cache_dir = {build_cache_dir}")
|
||||
if core_dir:
|
||||
lines.append(f"core_dir = {core_dir}")
|
||||
if packages_dir:
|
||||
lines.append(f"packages_dir = {packages_dir}")
|
||||
lines.append("")
|
||||
|
||||
# Section header
|
||||
lines.append(f"[env:{self.board_name}]")
|
||||
if extra_scripts:
|
||||
lines.append(f"extra_scripts = {' '.join(extra_scripts)}")
|
||||
|
||||
# Board identifier (skip for platforms that don't need board specification)
|
||||
if not self.no_board_spec:
|
||||
lines.append(f"board = {self.get_real_board_name()}")
|
||||
|
||||
# Optional parameters -------------------------------------------------
|
||||
if self.platform:
|
||||
lines.append(f"platform = {self.platform}")
|
||||
|
||||
if self.platform_packages:
|
||||
lines.append(f"platform_packages = {self.platform_packages}")
|
||||
|
||||
if self.framework:
|
||||
lines.append(f"framework = {self.framework}")
|
||||
|
||||
if self.board_build_core:
|
||||
lines.append(f"board_build.core = {self.board_build_core}")
|
||||
|
||||
if self.board_build_mcu:
|
||||
lines.append(f"board_build.mcu = {self.board_build_mcu}")
|
||||
|
||||
if self.board_build_filesystem_size:
|
||||
lines.append(
|
||||
f"board_build.filesystem_size = {self.board_build_filesystem_size}"
|
||||
)
|
||||
|
||||
if self.board_partitions:
|
||||
lines.append(f"board_partitions = {self.board_partitions}")
|
||||
|
||||
# Build-time flags and unflags ---------------------------------------
|
||||
build_flags_elements: list[str] = []
|
||||
if self.defines:
|
||||
build_flags_elements.extend(f"-D{define}" for define in self.defines)
|
||||
|
||||
# Add additional defines
|
||||
if additional_defines:
|
||||
build_flags_elements.extend(f"-D{define}" for define in additional_defines)
|
||||
|
||||
# Use static build flags
|
||||
if self.build_flags:
|
||||
build_flags_elements.extend(self.build_flags)
|
||||
|
||||
# Add additional include directories
|
||||
if additional_include_dirs:
|
||||
build_flags_elements.extend(
|
||||
f"-I{include_dir}" for include_dir in additional_include_dirs
|
||||
)
|
||||
|
||||
if build_flags_elements:
|
||||
# Put each build flag on a separate line for better readability and debugging
|
||||
lines.append("build_flags =")
|
||||
for flag in build_flags_elements:
|
||||
lines.append(f" {flag}")
|
||||
|
||||
if self.build_unflags:
|
||||
# PlatformIO accepts multiple *build_unflags* separated by spaces.
|
||||
# Emit a single line for readability.
|
||||
lines.append(f"build_unflags = {' '.join(self.build_unflags)}")
|
||||
|
||||
# Custom ESP-IDF sdkconfig override (ESP32-family boards)
|
||||
if self.customsdk:
|
||||
lines.append(f"custom_sdkconfig = {self.customsdk}")
|
||||
|
||||
# Library compatibility mode (for platforms like native that need special handling)
|
||||
if self.lib_compat_mode:
|
||||
lines.append(f"lib_compat_mode = {self.lib_compat_mode}")
|
||||
|
||||
# Library Dependency Finder mode (for enhanced dependency finding)
|
||||
if self.lib_ldf_mode:
|
||||
lines.append(f"lib_ldf_mode = {self.lib_ldf_mode}")
|
||||
|
||||
# Libraries to ignore during compilation
|
||||
if self.lib_ignore:
|
||||
if len(self.lib_ignore) == 1:
|
||||
lines.append(f"lib_ignore = {self.lib_ignore[0]}")
|
||||
else:
|
||||
lines.append("lib_ignore =")
|
||||
for lib in self.lib_ignore:
|
||||
lines.append(f" {lib}")
|
||||
|
||||
# Add FastLED-specific configurations if project_root is provided
|
||||
if project_root:
|
||||
# Only add default lib_ldf_mode if board doesn't specify its own
|
||||
current_config = "\n".join(lines)
|
||||
if "lib_ldf_mode" not in current_config:
|
||||
lines.append("lib_ldf_mode = chain")
|
||||
lines.append("lib_archive = true")
|
||||
|
||||
# Build lib_deps with additional libs only (FastLED is copied to lib/FastLED)
|
||||
# PlatformIO supports multiple lib_deps entries on separate lines
|
||||
if additional_libs:
|
||||
# Format as multi-line lib_deps for proper PlatformIO parsing
|
||||
if len(additional_libs) == 1:
|
||||
lines.append(f"lib_deps = {additional_libs[0]}")
|
||||
else:
|
||||
lines.append("lib_deps =")
|
||||
for entry in additional_libs:
|
||||
lines.append(f" {entry}")
|
||||
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# [env:sparkfun_xrp_controller]
|
||||
# platform = https://github.com/maxgerhardt/platform-raspberrypi
|
||||
# board = sparkfun_xrp_controller
|
||||
# framework = arduino
|
||||
# lib_deps = fastled/FastLED @ ^3.9.16
|
||||
|
||||
|
||||
WEBTARGET = Board(
|
||||
board_name="web",
|
||||
)
|
||||
|
||||
# Native host compilation target using PlatformIO's "native" platform.
|
||||
# This allows compiling FastLED for the host machine (Linux/macOS/Windows)
|
||||
# which is useful for CI compile-tests and static analysis. We replicate
|
||||
# the build flags present in ci/native/platformio.ini so that the same
|
||||
# stub implementation and main-file inclusion are used.
|
||||
|
||||
NATIVE = Board(
|
||||
board_name="native",
|
||||
platform="platformio/native",
|
||||
no_board_spec=True, # Native platform doesn't need a board specification
|
||||
lib_compat_mode="off", # Disable library compatibility checking for native platform
|
||||
build_flags=[
|
||||
"-DFASTLED_STUB_IMPL", # Enable stub platform implementations
|
||||
"-DFASTLED_USE_STUB_ARDUINO", # Enable Arduino stub implementations
|
||||
"-DPLATFORM_NATIVE", # Enable native platform stub compilation
|
||||
"-std=c++17",
|
||||
"-I../../../src/platforms/stub", # Include path for Arduino.h and other stub headers (relative to project dir)
|
||||
"-I../../../src", # Include path for FastLED.h and other source headers (relative to project dir)
|
||||
],
|
||||
)
|
||||
|
||||
DUE = Board(
|
||||
board_name="due",
|
||||
platform="atmelsam",
|
||||
)
|
||||
|
||||
|
||||
SPARKFUN_XRP_CONTROLLER_2350B = Board(
|
||||
board_name="sparkfun_xrp_controller",
|
||||
platform="https://github.com/maxgerhardt/platform-raspberrypi",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
APOLLO3_RED_BOARD = Board(
|
||||
board_name="apollo3_red",
|
||||
real_board_name="SparkFun_RedBoard_Artemis_ATP",
|
||||
platform=APOLLO3_2_2_0,
|
||||
platform_packages="framework-arduinoapollo3@https://github.com/sparkfun/Arduino_Apollo3#v2.2.0",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
APOLLO3_SPARKFUN_THING_PLUS_EXPLORABLE = Board(
|
||||
board_name="apollo3_thing_explorable",
|
||||
real_board_name="SparkFun_Thing_Plus_expLoRaBLE",
|
||||
platform=APOLLO3_2_2_0,
|
||||
platform_packages="framework-arduinoapollo3@https://github.com/sparkfun/Arduino_Apollo3#v2.2.0",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
ESP32DEV = Board(
|
||||
board_name="esp32dev",
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32DEV_IDF3_3 = Board(
|
||||
board_name="esp32dev_idf33",
|
||||
real_board_name="esp32dev",
|
||||
platform="espressif32@1.11.2",
|
||||
)
|
||||
|
||||
ESP32DEV_IDF4_4 = Board(
|
||||
board_name="esp32dev_idf44",
|
||||
real_board_name="esp32dev",
|
||||
platform=ESP32_IDF_4_4_LATEST,
|
||||
)
|
||||
|
||||
GIGA_R1 = Board(
|
||||
board_name="giga_r1",
|
||||
platform="ststm32",
|
||||
framework="arduino",
|
||||
real_board_name="giga_r1_m7",
|
||||
)
|
||||
|
||||
# ESP01 = Board(
|
||||
# board_name="esp01",
|
||||
# platform=ESP32_IDF_5_1_PIOARDUINO,
|
||||
# )
|
||||
|
||||
ESP32_C2_DEVKITM_1 = Board(
|
||||
board_name="esp32c2",
|
||||
real_board_name="esp32-c2-devkitm-1",
|
||||
platform="https://github.com/pioarduino/platform-espressif32/releases/download/stable/platform-espressif32.zip",
|
||||
framework="arduino, espidf",
|
||||
)
|
||||
|
||||
ESP32_C3_DEVKITM_1 = Board(
|
||||
board_name="esp32c3",
|
||||
real_board_name="esp32-c3-devkitm-1",
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_C5_DEVKITC_1 = Board(
|
||||
board_name="esp32c5",
|
||||
real_board_name="esp32-c5-devkitc-1",
|
||||
platform=ESP32_IDF_5_5_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_C6_DEVKITC_1 = Board(
|
||||
board_name="esp32c6",
|
||||
real_board_name="esp32-c6-devkitc-1",
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_S3_DEVKITC_1 = Board(
|
||||
board_name="esp32s3",
|
||||
real_board_name="esp32-s3-devkitc-1",
|
||||
platform=ESP32_IDF_5_4_PIOARDUINO,
|
||||
framework="arduino",
|
||||
board_partitions="huge_app.csv",
|
||||
build_unflags=["-DFASTLED_RMT5=0", "-DFASTLED_RMT5"],
|
||||
)
|
||||
|
||||
ESP32_S2_DEVKITM_1 = Board(
|
||||
board_name="esp32s2",
|
||||
real_board_name="lolin_s2_mini ",
|
||||
board_build_mcu="esp32s2",
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_UPESY_WROOM = Board(
|
||||
board_name="upesy_wroom",
|
||||
real_board_name="upesy_wroom",
|
||||
platform="espressif32",
|
||||
)
|
||||
|
||||
ESP32H2 = Board(
|
||||
board_name="esp32h2",
|
||||
real_board_name="esp32-h2-devkitm-1",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_P4 = Board(
|
||||
board_name="esp32p4",
|
||||
real_board_name="esp32-p4-evboard",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
platform="https://github.com/pioarduino/platform-espressif32/releases/download/stable/platform-espressif32.zip",
|
||||
)
|
||||
|
||||
ADA_FEATHER_NRF52840_SENSE = Board(
|
||||
board_name="adafruit_feather_nrf52840_sense",
|
||||
platform="nordicnrf52",
|
||||
)
|
||||
|
||||
XIAOBLESENSE_ADAFRUIT_NRF52 = Board(
|
||||
board_name="xiaoblesense_adafruit",
|
||||
platform="https://github.com/maxgerhardt/platform-nordicnrf52",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
)
|
||||
|
||||
# Alias: handle common misspelling without the trailing 't'
|
||||
XIAOBLESENSE_ADAFRUI_ALIAS = Board(
|
||||
board_name="xiaoblesense_adafrui", # missing 't'
|
||||
real_board_name="xiaoblesense_adafruit", # map to the correct board name
|
||||
platform="https://github.com/maxgerhardt/platform-nordicnrf52",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
XIAOBLESENSE_NRF52 = Board(
|
||||
board_name="xiaoblesense",
|
||||
real_board_name="xiaoble_adafruit",
|
||||
platform="https://github.com/maxgerhardt/platform-nordicnrf52",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
# Correct nRF52840 DK board definition
|
||||
# The Nordic nRF52840 DK is directly supported by the default PlatformIO
|
||||
# `nordicnrf52` platform under the board name `nrf52840_dk`, so we don't
|
||||
# need a custom platform package or extra installation steps. Point the
|
||||
# Board definition at the stock platform and use the canonical board name.
|
||||
# This fixes compilation failures introduced during the build-system
|
||||
# migration where the board was temporarily mapped to the XIAO variant.
|
||||
NRF52840 = Board(
|
||||
board_name="nrf52840_dk",
|
||||
real_board_name="nrf52840_dk_adafruit", # Use Adafruit BSP variant which includes full Nordic SDK headers
|
||||
platform="nordicnrf52",
|
||||
framework="arduino",
|
||||
platform_needs_install=False,
|
||||
platform_packages="framework-arduinoadafruitnrf52@^1.10601.0",
|
||||
defines=[
|
||||
"FASTLED_USE_COMPILE_TESTS=0",
|
||||
],
|
||||
board_build_core="nRF5", # Ensure correct core directory
|
||||
)
|
||||
|
||||
RPI_PICO = Board(
|
||||
board_name="rpipico",
|
||||
platform="https://github.com/maxgerhardt/platform-raspberrypi.git",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
platform_packages="framework-arduinopico@https://github.com/earlephilhower/arduino-pico.git",
|
||||
framework="arduino",
|
||||
board_build_core="earlephilhower",
|
||||
board_build_filesystem_size="0.5m",
|
||||
)
|
||||
|
||||
RPI_PICO2 = Board(
|
||||
board_name="rpipico2",
|
||||
real_board_name="rpipico", # Use the existing Pico board definition until PlatformIO adds native Pico 2 support
|
||||
platform="https://github.com/maxgerhardt/platform-raspberrypi.git",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
platform_packages="framework-arduinopico@https://github.com/earlephilhower/arduino-pico.git",
|
||||
framework="arduino",
|
||||
board_build_core="earlephilhower",
|
||||
board_build_filesystem_size="0.5m",
|
||||
)
|
||||
|
||||
BLUEPILL = Board(
|
||||
board_name="bluepill",
|
||||
real_board_name="bluepill_f103c8",
|
||||
platform="ststm32",
|
||||
)
|
||||
|
||||
# maple_mini_b20
|
||||
MAPLE_MINI = Board(
|
||||
board_name="maple_mini",
|
||||
real_board_name="maple_mini_b20",
|
||||
platform="ststm32",
|
||||
)
|
||||
|
||||
HY_TINYSTM103TB = Board(
|
||||
board_name="hy_tinystm103tb",
|
||||
platform="ststm32",
|
||||
)
|
||||
|
||||
ATTINY88 = Board(
|
||||
board_name="attiny88",
|
||||
platform="atmelavr",
|
||||
)
|
||||
|
||||
# ATtiny1604
|
||||
ATTINY1604 = Board(
|
||||
board_name="ATtiny1604",
|
||||
platform="atmelmegaavr",
|
||||
)
|
||||
|
||||
# attiny4313
|
||||
ATTINY4313 = Board(
|
||||
board_name="attiny4313",
|
||||
platform="atmelavr",
|
||||
)
|
||||
|
||||
ATTINY1616 = Board(
|
||||
board_name="ATtiny1616",
|
||||
platform="atmelmegaavr",
|
||||
)
|
||||
|
||||
UNO_R4_WIFI = Board(
|
||||
board_name="uno_r4_wifi",
|
||||
platform="renesas-ra",
|
||||
# I2S library auto-excluded due to missing r_i2s_api.h header in Arduino Renesas framework
|
||||
)
|
||||
|
||||
NANO_EVERY = Board(
|
||||
board_name="nano_every",
|
||||
platform="atmelmegaavr",
|
||||
)
|
||||
|
||||
ESP32DEV_I2S = Board(
|
||||
board_name="esp32dev_i2s",
|
||||
real_board_name="esp32dev",
|
||||
platform=ESP32_IDF_4_4_LATEST,
|
||||
)
|
||||
|
||||
ESP32S3_RMT51 = Board(
|
||||
board_name="esp32rmt_51",
|
||||
real_board_name="esp32-s3-devkitc-1",
|
||||
platform_needs_install=True,
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
defines=[
|
||||
"FASTLED_RMT5=1",
|
||||
],
|
||||
)
|
||||
|
||||
# Teensy boards
|
||||
TEENSY_LC = Board(
|
||||
board_name="teensylc",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
TEENSY30 = Board(
|
||||
board_name="teensy30",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
TEENSY31 = Board(
|
||||
board_name="teensy31",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
TEENSY40 = Board(
|
||||
board_name="teensy40",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
TEENSY41 = Board(
|
||||
board_name="teensy41",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
# Basic Arduino boards
|
||||
UNO = Board(
|
||||
board_name="uno",
|
||||
platform="atmelavr",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
YUN = Board(
|
||||
board_name="yun",
|
||||
platform="atmelavr",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
DIGIX = Board(
|
||||
board_name="digix",
|
||||
real_board_name="due", # Digix is Arduino Due compatible
|
||||
platform="atmelsam",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
# ESP8266 boards
|
||||
ESP01 = Board(
|
||||
board_name="esp01",
|
||||
platform="espressif8266",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
# ATtiny boards
|
||||
ATTINY85 = Board(
|
||||
board_name="attiny85",
|
||||
platform="atmelavr",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
# Seeed XIAO ESP32S3 board – same platform, needs FASTLED_RMT5 macro removal
|
||||
XIAO_ESP32S3 = Board(
|
||||
board_name="seeed_xiao_esp32s3",
|
||||
real_board_name="seeed_xiao_esp32s3",
|
||||
platform=ESP32_IDF_5_4_PIOARDUINO,
|
||||
board_partitions="huge_app.csv",
|
||||
defines=None,
|
||||
build_unflags=["-DFASTLED_RMT5=0", "-DFASTLED_RMT5"],
|
||||
)
|
||||
|
||||
# STM32F4 Black Pill board - addresses GitHub issue #726
|
||||
BLACKPILL = Board(
|
||||
board_name="blackpill",
|
||||
real_board_name="blackpill_f411ce",
|
||||
platform="ststm32",
|
||||
)
|
||||
|
||||
# Silicon Labs MGM240S boards (Arduino Nano Matter, SparkFun Thing Plus Matter)
|
||||
# Uses Silicon Labs EFM32 platform with Arduino framework support
|
||||
# Based on EFR32MG24 SoC with ARM Cortex-M33 @ 78MHz
|
||||
MGM240S = Board(
|
||||
board_name="mgm240",
|
||||
real_board_name="sparkfun_thingplusmatter",
|
||||
platform="https://github.com/maxgerhardt/platform-siliconlabsefm32/archive/refs/heads/silabs-arduino.zip",
|
||||
platform_needs_install=True,
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
|
||||
def _make_board_map(boards: list[Board]) -> dict[str, Board]:
|
||||
# make board map, but assert on duplicate board names
|
||||
board_map: dict[str, Board] = {}
|
||||
for board in boards:
|
||||
assert board.board_name not in board_map, (
|
||||
f"Duplicate board name: {board.board_name}"
|
||||
)
|
||||
board_map[board.board_name] = board
|
||||
return board_map
|
||||
|
||||
|
||||
_BOARD_MAP: dict[str, Board] = _make_board_map(ALL)
|
||||
|
||||
|
||||
def create_board(board_name: str, no_project_options: bool = False) -> Board:
|
||||
board: Board
|
||||
if no_project_options:
|
||||
board = Board(board_name=board_name, add_board_to_all=False)
|
||||
if board_name not in _BOARD_MAP:
|
||||
# empty board without any special overrides, assume platformio will know what to do with it.
|
||||
board = Board(board_name=board_name, add_board_to_all=False)
|
||||
else:
|
||||
board = _BOARD_MAP[board_name]
|
||||
return board.clone()
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"build": {
|
||||
"core": "maple",
|
||||
"cpu": "cortex-m3",
|
||||
"extra_flags": "-DSTM32F103C8 -DSTM32F1 -DARDUINO_GENERIC_STM32F103C8 -DARDUINO_ARCH_STM32",
|
||||
"f_cpu": "72000000L",
|
||||
"hwids": [
|
||||
[
|
||||
"0x1EAF",
|
||||
"0x0003"
|
||||
]
|
||||
],
|
||||
"ldscript": "jtag_c8.ld",
|
||||
"mcu": "stm32f103c8",
|
||||
"product_line": "STM32F103xx",
|
||||
"variant": "STM32F103C8"
|
||||
},
|
||||
"connectivity": [
|
||||
"can"
|
||||
],
|
||||
"debug": {
|
||||
"jlink_device": "STM32F103C8",
|
||||
"openocd_board": "stm32f103c8_blue_pill",
|
||||
"openocd_target": "stm32f1x",
|
||||
"svd_path": "STM32F103.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"stm32cube",
|
||||
"libopencm3"
|
||||
],
|
||||
"name": "Blue Pill STM32F103C8",
|
||||
"upload": {
|
||||
"maximum_ram_size": 20480,
|
||||
"maximum_size": 65536,
|
||||
"protocol": "stlink",
|
||||
"protocols": [
|
||||
"stlink",
|
||||
"dfu",
|
||||
"serial",
|
||||
"blackmagic",
|
||||
"jlink"
|
||||
]
|
||||
},
|
||||
"url": "https://stm32-base.org/boards/STM32F103C8T6-Blue-Pill",
|
||||
"vendor": "Generic"
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"build": {
|
||||
"core": "esp32",
|
||||
"f_cpu": "120000000L",
|
||||
"f_flash": "60000000L",
|
||||
"flash_mode": "qio",
|
||||
"mcu": "esp32c2",
|
||||
"variant": "esp32c2"
|
||||
},
|
||||
"connectivity": [
|
||||
"wifi"
|
||||
],
|
||||
"debug": {
|
||||
"openocd_target": "esp32c2.cfg"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"espidf"
|
||||
],
|
||||
"name": "Espressif ESP32-C2-DevKitM-1",
|
||||
"upload": {
|
||||
"flash_size": "4MB",
|
||||
"maximum_ram_size": 278528,
|
||||
"maximum_size": 4194304,
|
||||
"require_upload_port": true,
|
||||
"speed": 460800
|
||||
},
|
||||
"espidf": {
|
||||
"custom_sdkconfig": [
|
||||
"CONFIG_IDF_TARGET=\"esp32c2\""
|
||||
]
|
||||
},
|
||||
"url": "https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp8684/esp8684-devkitm-1/user_guide.html",
|
||||
"vendor": "Espressif"
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"build": {
|
||||
"core": "esp32",
|
||||
"f_cpu": "160000000L",
|
||||
"f_flash": "80000000L",
|
||||
"flash_mode": "qio",
|
||||
"mcu": "esp32c6",
|
||||
"variant": "esp32c6"
|
||||
},
|
||||
"connectivity": [
|
||||
"bluetooth",
|
||||
"wifi"
|
||||
],
|
||||
"debug": {
|
||||
"openocd_target": "esp32c6.cfg"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"espidf"
|
||||
],
|
||||
"name": "Espressif ESP32-C6-DevKitC-1",
|
||||
"upload": {
|
||||
"flash_size": "8MB",
|
||||
"maximum_ram_size": 327680,
|
||||
"maximum_size": 8388608,
|
||||
"require_upload_port": true,
|
||||
"speed": 460800
|
||||
},
|
||||
"url": "https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp32c6/esp32-c6-devkitc-1/index.html",
|
||||
"vendor": "Espressif"
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"build": {
|
||||
"core": "esp32",
|
||||
"f_cpu": "96000000L",
|
||||
"f_flash": "64000000L",
|
||||
"f_image": "48000000L",
|
||||
"flash_mode": "qio",
|
||||
"mcu": "esp32h2",
|
||||
"variant": "esp32h2"
|
||||
},
|
||||
"connectivity": [
|
||||
"bluetooth"
|
||||
],
|
||||
"debug": {
|
||||
"openocd_target": "esp32h2.cfg"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"espidf"
|
||||
],
|
||||
"name": "Espressif ESP32-H2-DevKit",
|
||||
"upload": {
|
||||
"flash_size": "4MB",
|
||||
"maximum_ram_size": 327680,
|
||||
"maximum_size": 4194304,
|
||||
"require_upload_port": true,
|
||||
"speed": 460800
|
||||
},
|
||||
"url": "https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp32h2/esp32-h2-devkitm-1/index.html",
|
||||
"vendor": "Espressif"
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino": {
|
||||
"ldscript": "linker_script.ld",
|
||||
"flash_layout": "100_0"
|
||||
},
|
||||
"extra_flags": "-DARDUINO_GIGA -DARDUINO_GIGA_PINS -DGIGA_PINS -DGIGA -DCM4_BINARY_START=0x60000000 -DCM4_BINARY_END=0x60040000 -DCM4_RAM_END=0x60080000",
|
||||
"core": "arduino",
|
||||
"cpu": "cortex-m7",
|
||||
"f_cpu": "480000000L",
|
||||
"mcu": "stm32h747xih6",
|
||||
"variant": "GIGA",
|
||||
"product_line": "STM32H747xx",
|
||||
"hwids": [
|
||||
[
|
||||
"0x2341",
|
||||
"0x0266"
|
||||
],
|
||||
[
|
||||
"0x2341",
|
||||
"0x0366"
|
||||
],
|
||||
[
|
||||
"0x2341",
|
||||
"0x0466"
|
||||
]
|
||||
]
|
||||
},
|
||||
"connectivity": [
|
||||
"bluetooth",
|
||||
"wifi"
|
||||
],
|
||||
"debug": {
|
||||
"jlink_device": "STM32H747XI_M7",
|
||||
"openocd_target": "stm32h7x_dual_bank"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino"
|
||||
],
|
||||
"name": "Arduino Giga R1 Wifi",
|
||||
"upload": {
|
||||
"_maximum_ram_size": 294248,
|
||||
"_maximum_size": 1048576,
|
||||
"maximum_ram_size": 523624,
|
||||
"maximum_size": 1966080,
|
||||
"protocol": "dfu",
|
||||
"protocols": [
|
||||
"cmsis-dap",
|
||||
"dfu",
|
||||
"jlink",
|
||||
"stlink",
|
||||
"mbed"
|
||||
],
|
||||
"require_upload_port": true,
|
||||
"use_1200bps_touch": true,
|
||||
"wait_for_upload_port": true,
|
||||
"offset_address": "0x08100000"
|
||||
},
|
||||
"url": "https://docs.arduino.cc/hardware/giga-r1-wifi",
|
||||
"vendor": "Arduino"
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino":{
|
||||
"ldscript": "nrf52_xxaa.ld"
|
||||
},
|
||||
"core": "nRF5",
|
||||
"cpu": "cortex-m4",
|
||||
"extra_flags": "-DARDUINO_NRF52_DK",
|
||||
"f_cpu": "64000000L",
|
||||
"mcu": "nrf52840",
|
||||
"variant": "nRF52DK",
|
||||
"zephyr": {
|
||||
"variant": "nrf52840dk_nrf52840"
|
||||
}
|
||||
},
|
||||
"connectivity": [
|
||||
"bluetooth"
|
||||
],
|
||||
"debug": {
|
||||
"default_tools": [
|
||||
"jlink"
|
||||
],
|
||||
"jlink_device": "nRF52840_xxAA",
|
||||
"onboard_tools": [
|
||||
"cmsis-dap",
|
||||
"jlink"
|
||||
],
|
||||
"svd_path": "nrf52840.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"mbed",
|
||||
"zephyr"
|
||||
],
|
||||
"name": "Nordic nRF52840-DK",
|
||||
"upload": {
|
||||
"maximum_ram_size": 262144,
|
||||
"maximum_size": 1048576,
|
||||
"protocol": "jlink",
|
||||
"protocols": [
|
||||
"jlink",
|
||||
"nrfjprog",
|
||||
"stlink",
|
||||
"blackmagic",
|
||||
"cmsis-dap",
|
||||
"mbed"
|
||||
]
|
||||
},
|
||||
"url": "https://os.mbed.com/platforms/Nordic-nRF52840-DK/",
|
||||
"vendor": "Nordic"
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino": {
|
||||
"earlephilhower": {
|
||||
"boot2_source": "boot2_w25q080_2_padded_checksum.S",
|
||||
"usb_vid": "0x2E8A",
|
||||
"usb_pid": "0x000A"
|
||||
}
|
||||
},
|
||||
"core": "earlephilhower",
|
||||
"cpu": "cortex-m0plus",
|
||||
"extra_flags": "-DARDUINO_RASPBERRY_PI_PICO -DARDUINO_ARCH_RP2040 -DUSBD_MAX_POWER_MA=250 ",
|
||||
"f_cpu": "133000000L",
|
||||
"hwids": [
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x00C0"
|
||||
],
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x000A"
|
||||
]
|
||||
],
|
||||
"mcu": "rp2040",
|
||||
"variant": "rpipico"
|
||||
},
|
||||
"debug": {
|
||||
"jlink_device": "RP2040_M0_0",
|
||||
"openocd_target": "rp2040.cfg",
|
||||
"svd_path": "rp2040.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino"
|
||||
],
|
||||
"name": "Pico",
|
||||
"upload": {
|
||||
"maximum_ram_size": 262144,
|
||||
"maximum_size": 2097152,
|
||||
"require_upload_port": true,
|
||||
"native_usb": true,
|
||||
"use_1200bps_touch": true,
|
||||
"wait_for_upload_port": false,
|
||||
"protocol": "picotool",
|
||||
"protocols": [
|
||||
"blackmagic",
|
||||
"cmsis-dap",
|
||||
"jlink",
|
||||
"raspberrypi-swd",
|
||||
"picotool",
|
||||
"picoprobe",
|
||||
"pico-debug"
|
||||
]
|
||||
},
|
||||
"url": "https://www.raspberrypi.org/products/raspberry-pi-pico/",
|
||||
"vendor": "Raspberry Pi"
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino": {
|
||||
"earlephilhower": {
|
||||
"boot2_source": "boot2_generic_03h_2_padded_checksum.S",
|
||||
"usb_vid": "0x2E8A",
|
||||
"usb_pid": "0x000B"
|
||||
}
|
||||
},
|
||||
"core": "earlephilhower",
|
||||
"cpu": "cortex-m33",
|
||||
"extra_flags": "-D ARDUINO_RASPBERRY_PI_PICO_2 -DARDUINO_ARCH_RP2350 -DUSBD_MAX_POWER_MA=250",
|
||||
"f_cpu": "133000000L",
|
||||
"hwids": [
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x00C0"
|
||||
],
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x000B"
|
||||
]
|
||||
],
|
||||
"mcu": "rp2350",
|
||||
"variant": "rpipico2"
|
||||
},
|
||||
"debug": {
|
||||
"jlink_device": "RP2350_0",
|
||||
"openocd_target": "rp2350.cfg",
|
||||
"svd_path": "rp2350.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino"
|
||||
],
|
||||
"name": "Pico 2",
|
||||
"upload": {
|
||||
"maximum_ram_size": 524288,
|
||||
"maximum_size": 2097152,
|
||||
"require_upload_port": true,
|
||||
"native_usb": true,
|
||||
"use_1200bps_touch": true,
|
||||
"wait_for_upload_port": false,
|
||||
"protocol": "picotool",
|
||||
"protocols": [
|
||||
"blackmagic",
|
||||
"cmsis-dap",
|
||||
"jlink",
|
||||
"raspberrypi-swd",
|
||||
"picotool",
|
||||
"picoprobe",
|
||||
"pico-debug"
|
||||
]
|
||||
},
|
||||
"url": "https://www.raspberrypi.org/products/raspberry-pi-pico/",
|
||||
"vendor": "Raspberry Pi"
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino": {
|
||||
"earlephilhower": {
|
||||
"boot2_source": "boot2_w25q080_2_padded_checksum.S",
|
||||
"usb_vid": "0x2E8A",
|
||||
"usb_pid": "0xF00A"
|
||||
}
|
||||
},
|
||||
"core": "earlephilhower",
|
||||
"cpu": "cortex-m0plus",
|
||||
"extra_flags": "-DARDUINO_RASPBERRY_PI_PICO_W -DARDUINO_ARCH_RP2040 -DUSBD_MAX_POWER_MA=250 ",
|
||||
"f_cpu": "133000000L",
|
||||
"hwids": [
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x00C0"
|
||||
],
|
||||
[
|
||||
"0x2E8A",
|
||||
"0xF00A"
|
||||
]
|
||||
],
|
||||
"mcu": "rp2040",
|
||||
"variant": "rpipicow"
|
||||
},
|
||||
"debug": {
|
||||
"jlink_device": "RP2040_M0_0",
|
||||
"openocd_target": "rp2040.cfg",
|
||||
"svd_path": "rp2040.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino"
|
||||
],
|
||||
"name": "Pico W",
|
||||
"upload": {
|
||||
"maximum_ram_size": 262144,
|
||||
"maximum_size": 2097152,
|
||||
"require_upload_port": true,
|
||||
"native_usb": true,
|
||||
"use_1200bps_touch": true,
|
||||
"wait_for_upload_port": false,
|
||||
"protocol": "picotool",
|
||||
"protocols": [
|
||||
"blackmagic",
|
||||
"cmsis-dap",
|
||||
"jlink",
|
||||
"raspberrypi-swd",
|
||||
"picotool",
|
||||
"picoprobe",
|
||||
"pico-debug"
|
||||
]
|
||||
},
|
||||
"url": "https://www.raspberrypi.org/products/raspberry-pi-pico/",
|
||||
"vendor": "Raspberry Pi"
|
||||
}
|
||||
@@ -0,0 +1,643 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Unified Build API
|
||||
|
||||
This module provides a unified build system for both unit tests and examples,
|
||||
centralizing PCH and library building while providing a simple, elegant API.
|
||||
|
||||
Key Features:
|
||||
- Unified API for both unit tests and examples
|
||||
- Automatic PCH header building and reuse
|
||||
- Automatic libfastled.a building and reuse
|
||||
- Organized build directories under .build/
|
||||
- Efficient multiple target building
|
||||
- Simple, elegant interface
|
||||
|
||||
Usage:
|
||||
# Unit test building
|
||||
unit_builder = BuildAPI(
|
||||
config_file="ci/build_unit.toml",
|
||||
build_dir=".build/unit",
|
||||
build_type=BuildType.UNIT_TEST
|
||||
)
|
||||
unit_builder.build_targets(["test_json.cpp", "test_color.cpp"])
|
||||
|
||||
# Example building
|
||||
example_builder = BuildAPI(
|
||||
config_file="ci/build_example.toml",
|
||||
build_dir=".build/examples",
|
||||
build_type=BuildType.EXAMPLE
|
||||
)
|
||||
example_builder.build_targets(["Blink.ino", "DemoReel100.ino"])
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Sequence, Set, Tuple
|
||||
|
||||
from ci.ci.fingerprint_cache import FingerprintCache
|
||||
from ci.compiler.clang_compiler import (
|
||||
BuildFlags,
|
||||
Compiler,
|
||||
CompilerOptions,
|
||||
LibarchiveOptions,
|
||||
)
|
||||
from ci.compiler.test_example_compilation import get_fastled_core_sources
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
from ci.util.running_process import subprocess_run
|
||||
|
||||
|
||||
class BuildType(Enum):
|
||||
"""Build type enumeration"""
|
||||
|
||||
UNIT_TEST = "unit_test"
|
||||
EXAMPLE = "example"
|
||||
|
||||
|
||||
@dataclass
|
||||
class BuildTarget:
|
||||
"""Represents a single build target (test file or example)"""
|
||||
|
||||
source_file: Path
|
||||
name: str
|
||||
output_dir: Path
|
||||
|
||||
def get_output_executable(self) -> Path:
|
||||
"""Get the expected output executable path"""
|
||||
if self.source_file.suffix == ".cpp":
|
||||
# Unit test: test_name -> test_name.exe
|
||||
base_name = self.source_file.stem
|
||||
return self.output_dir / f"{base_name}.exe"
|
||||
else:
|
||||
# Example: Blink.ino -> Blink.exe
|
||||
base_name = self.source_file.stem
|
||||
return self.output_dir / f"{base_name}.exe"
|
||||
|
||||
|
||||
@dataclass
|
||||
class BuildResult:
|
||||
"""Result of a build operation"""
|
||||
|
||||
target: BuildTarget
|
||||
success: bool
|
||||
message: str
|
||||
build_time: float
|
||||
|
||||
|
||||
class BuildAPI:
|
||||
"""
|
||||
Unified build API for both unit tests and examples.
|
||||
|
||||
This class centralizes all build operations including PCH building,
|
||||
library creation, and target compilation. It provides a simple,
|
||||
elegant interface while handling the complexity internally.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
build_flags_toml: str | Path,
|
||||
build_dir: str | Path,
|
||||
build_type: BuildType,
|
||||
use_pch: bool = True,
|
||||
parallel: bool = True,
|
||||
clean: bool = False,
|
||||
):
|
||||
"""
|
||||
Initialize the build API.
|
||||
|
||||
Args:
|
||||
build_flags_toml: Path to build_flags.toml file (build_unit.toml or build_example.toml)
|
||||
build_dir: Directory for build outputs (e.g., .build/unit or .build/examples)
|
||||
build_type: Type of build (UNIT_TEST or EXAMPLE)
|
||||
use_pch: Whether to use precompiled headers
|
||||
parallel: Whether to enable parallel compilation
|
||||
clean: Whether to force a clean build
|
||||
"""
|
||||
self.build_flags_toml = Path(build_flags_toml)
|
||||
self.build_dir = Path(build_dir)
|
||||
self.build_type = build_type
|
||||
self.use_pch = use_pch
|
||||
self.parallel = parallel
|
||||
self.clean = clean
|
||||
|
||||
# Ensure build directory exists
|
||||
self.build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Set up subdirectories for organization
|
||||
self.artifacts_dir = self.build_dir / "artifacts" # PCH, lib files
|
||||
self.targets_dir = self.build_dir / "targets" # Individual target builds
|
||||
self.cache_dir = self.build_dir / "cache" # Fingerprint caches
|
||||
|
||||
for directory in [self.artifacts_dir, self.targets_dir, self.cache_dir]:
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Initialize internal state
|
||||
self._compiler: Optional[Compiler] = None
|
||||
self._pch_file: Optional[Path] = None
|
||||
self._library_file: Optional[Path] = None
|
||||
self._fingerprint_cache: Optional[FingerprintCache] = None
|
||||
|
||||
# Parse build flags from TOML file
|
||||
self._parse_build_flags()
|
||||
|
||||
# Create compiler from TOML configuration
|
||||
self._create_compiler()
|
||||
|
||||
def _parse_build_flags(self) -> None:
|
||||
"""Parse build flags from TOML file"""
|
||||
if not self.build_flags_toml.exists():
|
||||
raise RuntimeError(
|
||||
f"Build flags TOML file not found: {self.build_flags_toml}"
|
||||
)
|
||||
|
||||
print(f"[BUILD API] Parsing build flags from {self.build_flags_toml}")
|
||||
|
||||
# Parse build flags from TOML
|
||||
self.build_flags = BuildFlags.from_toml_file(
|
||||
self.build_flags_toml,
|
||||
quick_build=True, # Use quick build mode for faster iteration
|
||||
strict_mode=False,
|
||||
)
|
||||
|
||||
# Initialize fingerprint cache
|
||||
cache_file = self.cache_dir / "fingerprint_cache.json"
|
||||
self._fingerprint_cache = FingerprintCache(cache_file)
|
||||
|
||||
def _create_compiler(self) -> None:
|
||||
"""Create compiler from TOML configuration ONLY - no inline settings allowed"""
|
||||
from ci.compiler.clang_compiler import create_compiler_options_from_toml
|
||||
|
||||
# ALL settings must come from the TOML file - NO hardcoded/inline settings allowed
|
||||
print(f"[BUILD API] Creating compiler from TOML configuration")
|
||||
|
||||
# Set up PCH output path
|
||||
pch_output_path: Optional[str] = None
|
||||
if self.use_pch:
|
||||
pch_output_path = str(
|
||||
self.artifacts_dir / f"{self.build_type.value}_pch.hpp.pch"
|
||||
)
|
||||
|
||||
# Use the proper function to create CompilerOptions from TOML - NO inline settings
|
||||
self.compiler_settings = create_compiler_options_from_toml(
|
||||
toml_path=self.build_flags_toml,
|
||||
include_path=str(PROJECT_ROOT / "src"), # Base include path only
|
||||
quick_build=True, # Use quick build mode for faster iteration
|
||||
strict_mode=False,
|
||||
# Pass PCH and parallel settings from constructor
|
||||
use_pch=self.use_pch,
|
||||
pch_output_path=pch_output_path,
|
||||
parallel=self.parallel,
|
||||
)
|
||||
|
||||
# Create compiler instance with TOML-loaded settings
|
||||
self._compiler = Compiler(self.compiler_settings, self.build_flags)
|
||||
|
||||
def _ensure_pch_built(self) -> bool:
|
||||
"""
|
||||
Ensure PCH header is built and ready.
|
||||
|
||||
Returns:
|
||||
bool: True if PCH is ready, False otherwise
|
||||
"""
|
||||
if not self.use_pch or not self._compiler:
|
||||
return True
|
||||
|
||||
print(f"[BUILD API] Building PCH header for {self.build_type.value}")
|
||||
|
||||
# Check if we need to rebuild PCH
|
||||
if not self.clean and self._pch_file and self._pch_file.exists():
|
||||
# TODO: Add proper dependency checking here
|
||||
print(f"[BUILD API] Reusing existing PCH: {self._pch_file}")
|
||||
return True
|
||||
|
||||
# Build PCH
|
||||
success = self._compiler.create_pch_file()
|
||||
if success and self.compiler_settings.pch_output_path:
|
||||
self._pch_file = Path(self.compiler_settings.pch_output_path)
|
||||
print(f"[BUILD API] PCH built successfully: {self._pch_file}")
|
||||
else:
|
||||
print(f"[BUILD API] PCH build failed")
|
||||
|
||||
return success
|
||||
|
||||
def _ensure_library_built(self) -> bool:
|
||||
"""
|
||||
Ensure libfastled.a is built and ready.
|
||||
|
||||
Returns:
|
||||
bool: True if library is ready, False otherwise
|
||||
"""
|
||||
if not self._compiler:
|
||||
return False
|
||||
|
||||
lib_name = f"libfastled_{self.build_type.value}.a"
|
||||
self._library_file = self.artifacts_dir / lib_name
|
||||
|
||||
print(f"[BUILD API] Building FastLED library for {self.build_type.value}")
|
||||
|
||||
# Check if we need to rebuild library
|
||||
if not self.clean and self._library_file.exists():
|
||||
# TODO: Add proper dependency checking here
|
||||
print(f"[BUILD API] Reusing existing library: {self._library_file}")
|
||||
return True
|
||||
|
||||
# Prefer UNITY chunk build for faster library creation
|
||||
try:
|
||||
unity_dir = self.artifacts_dir / "unity"
|
||||
unity_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Collect FastLED core sources (consistent with example compilation)
|
||||
all_sources = [
|
||||
p for p in get_fastled_core_sources() if p.name != "stub_main.cpp"
|
||||
]
|
||||
if not all_sources:
|
||||
print("[BUILD API] No FastLED sources found for unity build")
|
||||
return False
|
||||
|
||||
# Deterministic sort relative to project root
|
||||
project_root = PROJECT_ROOT
|
||||
|
||||
def sort_key(p: Path) -> str:
|
||||
try:
|
||||
rel = p.relative_to(project_root)
|
||||
except Exception:
|
||||
rel = p
|
||||
return rel.as_posix()
|
||||
|
||||
all_sources = sorted(all_sources, key=sort_key)
|
||||
|
||||
# Partition into chunks (1..4 typical)
|
||||
total = len(all_sources)
|
||||
max_chunks = min(4, max(1, (os.cpu_count() or 2) // 2))
|
||||
chunks = max_chunks if total >= max_chunks else max(1, total)
|
||||
|
||||
# Prepare compile options (PCH off for unity)
|
||||
compile_opts = CompilerOptions(
|
||||
include_path=self._compiler.settings.include_path,
|
||||
compiler=self._compiler.settings.compiler,
|
||||
defines=self._compiler.settings.defines,
|
||||
std_version=self._compiler.settings.std_version,
|
||||
compiler_args=self._compiler.settings.compiler_args,
|
||||
use_pch=False,
|
||||
additional_flags=["-c"],
|
||||
parallel=self.parallel,
|
||||
)
|
||||
|
||||
# Compute chunk sizes
|
||||
base = total // chunks
|
||||
rem = total % chunks
|
||||
start = 0
|
||||
unity_objects: list[Path] = []
|
||||
|
||||
for i in range(chunks):
|
||||
size = base + (1 if i < rem else 0)
|
||||
end = start + size
|
||||
group = all_sources[start:end]
|
||||
start = end
|
||||
|
||||
if not group:
|
||||
continue
|
||||
|
||||
unity_cpp = unity_dir / f"unity{i + 1}.cpp"
|
||||
# Use synchronous path to simplify error handling here
|
||||
result = self._compiler._compile_unity_sync(
|
||||
compile_opts,
|
||||
group,
|
||||
unity_output_path=unity_cpp,
|
||||
)
|
||||
if not result.ok:
|
||||
print(f"[BUILD API] Unity chunk {i + 1} failed: {result.stderr}")
|
||||
return False
|
||||
unity_objects.append(unity_cpp.with_suffix(".o"))
|
||||
|
||||
if not unity_objects:
|
||||
print("[BUILD API] No unity objects produced")
|
||||
return False
|
||||
|
||||
# Archive unity objects into the final library
|
||||
archive_result = self._compiler.create_archive(
|
||||
unity_objects, self._library_file, LibarchiveOptions()
|
||||
).result()
|
||||
if not archive_result.ok:
|
||||
print(f"[BUILD API] Archive creation failed: {archive_result.stderr}")
|
||||
return False
|
||||
|
||||
print(
|
||||
f"[BUILD API] Library built successfully (UNITY): {self._library_file}"
|
||||
)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"[BUILD API] UNITY library build failed with exception: {e}")
|
||||
return False
|
||||
|
||||
def _create_static_library(
|
||||
self, object_files: List[Path], output_lib: Path
|
||||
) -> bool:
|
||||
"""
|
||||
Create static library from object files.
|
||||
|
||||
Args:
|
||||
object_files: List of object files to archive
|
||||
output_lib: Output library file path
|
||||
|
||||
Returns:
|
||||
bool: True if successful, False otherwise
|
||||
"""
|
||||
if not object_files:
|
||||
return False
|
||||
|
||||
# Use archiver from build flags
|
||||
archiver_cmd = self.build_flags.tools.archiver
|
||||
archive_flags = self.build_flags.archive.flags
|
||||
|
||||
# Build archiver command
|
||||
cmd = (
|
||||
archiver_cmd
|
||||
+ [archive_flags]
|
||||
+ [str(output_lib)]
|
||||
+ [str(f) for f in object_files]
|
||||
)
|
||||
|
||||
print(f"[BUILD API] Creating static library: {' '.join(cmd)}")
|
||||
|
||||
# Execute archiver command using stdout-pumping wrapper
|
||||
try:
|
||||
_completed = subprocess_run(
|
||||
command=cmd,
|
||||
cwd=None,
|
||||
check=True,
|
||||
timeout=900,
|
||||
enable_stack_trace=False,
|
||||
)
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"[BUILD API] Archiver failed: {e}")
|
||||
print(f"[BUILD API] Archiver stdout: {e.stdout}")
|
||||
print(f"[BUILD API] Archiver stderr: {e.stderr}")
|
||||
return False
|
||||
|
||||
def build_targets(self, target_files: Sequence[str | Path]) -> List[BuildResult]:
|
||||
"""
|
||||
Build multiple targets efficiently.
|
||||
|
||||
Args:
|
||||
target_files: List of source files to build
|
||||
|
||||
Returns:
|
||||
List[BuildResult]: Results for each target
|
||||
"""
|
||||
print(
|
||||
f"[BUILD API] Building {len(target_files)} targets for {self.build_type.value}"
|
||||
)
|
||||
|
||||
# Ensure shared artifacts are built first
|
||||
if not self._ensure_pch_built():
|
||||
return [
|
||||
BuildResult(
|
||||
target=BuildTarget(Path(f), Path(f).stem, self.targets_dir),
|
||||
success=False,
|
||||
message="PCH build failed",
|
||||
build_time=0.0,
|
||||
)
|
||||
for f in target_files
|
||||
]
|
||||
|
||||
if not self._ensure_library_built():
|
||||
return [
|
||||
BuildResult(
|
||||
target=BuildTarget(Path(f), Path(f).stem, self.targets_dir),
|
||||
success=False,
|
||||
message="Library build failed",
|
||||
build_time=0.0,
|
||||
)
|
||||
for f in target_files
|
||||
]
|
||||
|
||||
# Build individual targets
|
||||
results: List[BuildResult] = []
|
||||
for target_file in target_files:
|
||||
result = self._build_single_target(Path(target_file))
|
||||
results.append(result)
|
||||
|
||||
return results
|
||||
|
||||
def _build_single_target(self, target_file: Path) -> BuildResult:
|
||||
"""
|
||||
Build a single target.
|
||||
|
||||
Args:
|
||||
target_file: Source file to build
|
||||
|
||||
Returns:
|
||||
BuildResult: Result of the build
|
||||
"""
|
||||
import time
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Create build target
|
||||
target_dir = self.targets_dir / target_file.stem
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
target = BuildTarget(
|
||||
source_file=target_file, name=target_file.stem, output_dir=target_dir
|
||||
)
|
||||
|
||||
print(f"[BUILD API] Building target: {target.name}")
|
||||
|
||||
try:
|
||||
# Compile and link the target
|
||||
success = self._compile_and_link_target(target)
|
||||
|
||||
build_time = time.time() - start_time
|
||||
message = "Build successful" if success else "Build failed"
|
||||
|
||||
return BuildResult(
|
||||
target=target, success=success, message=message, build_time=build_time
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
build_time = time.time() - start_time
|
||||
return BuildResult(
|
||||
target=target,
|
||||
success=False,
|
||||
message=f"Build error: {e}",
|
||||
build_time=build_time,
|
||||
)
|
||||
|
||||
def _compile_and_link_target(self, target: BuildTarget) -> bool:
|
||||
"""
|
||||
Compile and link a single target.
|
||||
|
||||
Args:
|
||||
target: Target to build
|
||||
|
||||
Returns:
|
||||
bool: True if successful, False otherwise
|
||||
"""
|
||||
if not self._compiler:
|
||||
return False
|
||||
|
||||
# Compile target source file
|
||||
obj_file = target.output_dir / f"{target.name}.o"
|
||||
if target.source_file.suffix == ".cpp":
|
||||
success = self._compiler.compile_cpp_file(target.source_file, obj_file)
|
||||
else:
|
||||
# For .ino files
|
||||
success = self._compiler.compile_ino_file(target.source_file, obj_file)
|
||||
|
||||
if not success:
|
||||
print(f"[BUILD API] Failed to compile {target.source_file}")
|
||||
return False
|
||||
|
||||
# Link with library to create executable
|
||||
exe_file = target.get_output_executable()
|
||||
success = self._link_executable([obj_file], exe_file)
|
||||
|
||||
if not success:
|
||||
print(f"[BUILD API] Failed to link {exe_file}")
|
||||
return False
|
||||
|
||||
print(f"[BUILD API] Successfully built {exe_file}")
|
||||
return True
|
||||
|
||||
def _link_executable(self, object_files: List[Path], output_exe: Path) -> bool:
|
||||
"""
|
||||
Link object files into an executable.
|
||||
|
||||
Args:
|
||||
object_files: Object files to link
|
||||
output_exe: Output executable path
|
||||
|
||||
Returns:
|
||||
bool: True if successful, False otherwise
|
||||
"""
|
||||
if not self._library_file or not self._library_file.exists():
|
||||
print(f"[BUILD API] Library not available for linking")
|
||||
return False
|
||||
|
||||
# Build linker command
|
||||
linker_cmd = self.build_flags.tools.linker
|
||||
link_flags = self.build_flags.link_flags
|
||||
|
||||
cmd = (
|
||||
linker_cmd
|
||||
+ link_flags
|
||||
+ [str(f) for f in object_files]
|
||||
+ [str(self._library_file)]
|
||||
+ ["-o", str(output_exe)]
|
||||
)
|
||||
cmd_str = subprocess.list2cmdline(cmd)
|
||||
|
||||
print(f"[BUILD API] Linking: {cmd_str}")
|
||||
|
||||
# Execute linker using stdout-pumping wrapper
|
||||
try:
|
||||
_completed = subprocess_run(
|
||||
command=cmd,
|
||||
cwd=None,
|
||||
check=True,
|
||||
timeout=900,
|
||||
enable_stack_trace=False,
|
||||
)
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"[BUILD API] Linker failed: {e}")
|
||||
print(f"[BUILD API] Linker stdout: {e.stdout}")
|
||||
print(f"[BUILD API] Linker stderr: {e.stderr}")
|
||||
return False
|
||||
|
||||
def get_build_info(self) -> Dict[str, Any]:
|
||||
"""Get information about the current build configuration"""
|
||||
return {
|
||||
"build_flags_toml": str(self.build_flags_toml),
|
||||
"build_dir": str(self.build_dir),
|
||||
"build_type": self.build_type.value,
|
||||
"use_pch": self.use_pch,
|
||||
"parallel": self.parallel,
|
||||
"pch_file": str(self._pch_file) if self._pch_file else None,
|
||||
"library_file": str(self._library_file) if self._library_file else None,
|
||||
}
|
||||
|
||||
def clean_build_artifacts(self) -> None:
|
||||
"""Clean all build artifacts"""
|
||||
print(f"[BUILD API] Cleaning build artifacts in {self.build_dir}")
|
||||
|
||||
if self.build_dir.exists():
|
||||
shutil.rmtree(self.build_dir)
|
||||
self.build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Recreate subdirectories
|
||||
for directory in [self.artifacts_dir, self.targets_dir, self.cache_dir]:
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Reset internal state
|
||||
self._pch_file = None
|
||||
self._library_file = None
|
||||
if self._fingerprint_cache:
|
||||
# Clear the cache by resetting it
|
||||
cache_file = self.cache_dir / "fingerprint_cache.json"
|
||||
self._fingerprint_cache = FingerprintCache(cache_file)
|
||||
|
||||
|
||||
def create_unit_test_builder(
|
||||
build_dir: str | Path = ".build/unit",
|
||||
use_pch: bool = True,
|
||||
parallel: bool = True,
|
||||
clean: bool = False,
|
||||
) -> BuildAPI:
|
||||
"""
|
||||
Create a BuildAPI instance for unit tests.
|
||||
|
||||
Args:
|
||||
build_dir: Build directory for unit test outputs
|
||||
use_pch: Whether to use precompiled headers
|
||||
parallel: Whether to enable parallel compilation
|
||||
clean: Whether to force a clean build
|
||||
|
||||
Returns:
|
||||
BuildAPI: Configured build API for unit tests
|
||||
"""
|
||||
build_flags_toml = PROJECT_ROOT / "ci" / "build_unit.toml"
|
||||
return BuildAPI(
|
||||
build_flags_toml=build_flags_toml,
|
||||
build_dir=build_dir,
|
||||
build_type=BuildType.UNIT_TEST,
|
||||
use_pch=use_pch,
|
||||
parallel=parallel,
|
||||
clean=clean,
|
||||
)
|
||||
|
||||
|
||||
def create_example_builder(
|
||||
build_dir: str | Path = ".build/examples",
|
||||
use_pch: bool = True,
|
||||
parallel: bool = True,
|
||||
clean: bool = False,
|
||||
) -> BuildAPI:
|
||||
"""
|
||||
Create a BuildAPI instance for examples.
|
||||
|
||||
Args:
|
||||
build_dir: Build directory for example outputs
|
||||
use_pch: Whether to use precompiled headers
|
||||
parallel: Whether to enable parallel compilation
|
||||
clean: Whether to force a clean build
|
||||
|
||||
Returns:
|
||||
BuildAPI: Configured build API for examples
|
||||
"""
|
||||
build_flags_toml = PROJECT_ROOT / "ci" / "build_example.toml"
|
||||
return BuildAPI(
|
||||
build_flags_toml=build_flags_toml,
|
||||
build_dir=build_dir,
|
||||
build_type=BuildType.EXAMPLE,
|
||||
use_pch=use_pch,
|
||||
parallel=parallel,
|
||||
clean=clean,
|
||||
)
|
||||
@@ -0,0 +1,32 @@
|
||||
[all]
|
||||
# Universal defines and compiler flags for compile-commands generation
|
||||
defines = [
|
||||
"STUB_PLATFORM",
|
||||
"__EMSCRIPTEN__"
|
||||
]
|
||||
compiler_flags = [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra"
|
||||
]
|
||||
# Include flags are passed verbatim; use -I and -isystem as needed
|
||||
include_flags = [
|
||||
"-Isrc",
|
||||
"-isystemsrc/platforms/stub"
|
||||
]
|
||||
|
||||
[tools]
|
||||
# Tool configuration required by BuildFlags.parse
|
||||
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
linker = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"]
|
||||
c_compiler = ["uv", "run", "python", "-m", "ziglang", "cc"]
|
||||
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
|
||||
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
|
||||
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
|
||||
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
|
||||
|
||||
[archive]
|
||||
# Required by BuildFlags.parse; not used for compile-commands generation
|
||||
flags = "rcsD"
|
||||
@@ -0,0 +1,181 @@
|
||||
# ================================================================================================
|
||||
# FastLED Example Build Flags Configuration
|
||||
# ================================================================================================
|
||||
#
|
||||
# This file centralizes ALL compilation flags used by example compilation to ensure
|
||||
# they remain synchronized and compatible across the entire build system.
|
||||
#
|
||||
# Examples use the STUB platform for Arduino-compatible functionality without hardware
|
||||
# dependencies, but do NOT need unit test specific flags like FASTLED_FORCE_NAMESPACE.
|
||||
|
||||
[tools]
|
||||
# Build tool configuration - Full compiler command with arguments
|
||||
# CRITICAL: Must use 'uv run python' for subprocess execution (subprocess doesn't inherit uv environment)
|
||||
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
linker = ["uv", "run", "python", "-m", "ziglang", "c++"] # Separate linker config for consistency
|
||||
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"] # Ziglang archiver for unified toolchain
|
||||
c_compiler = ["uv", "run", "python", "-m", "ziglang", "cc"]
|
||||
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
|
||||
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
|
||||
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
|
||||
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
|
||||
|
||||
[all]
|
||||
# Universal compilation flags (compiler command handled separately)
|
||||
compiler_flags = [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-Wno-deprecated-register", # Suppress 'register' keyword warnings for C++17 compatibility
|
||||
"-Wno-backslash-newline-escape", # Suppress backslash newline escape warnings
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
]
|
||||
|
||||
defines = [
|
||||
"STUB_PLATFORM",
|
||||
# NOTE: Examples do NOT get FASTLED_FORCE_NAMESPACE=1 - they use global namespace
|
||||
]
|
||||
|
||||
include_flags = [
|
||||
"-I.",
|
||||
"-Isrc",
|
||||
]
|
||||
|
||||
[windows]
|
||||
# ================================================================================================
|
||||
# WINDOWS-SPECIFIC BUILD FLAGS
|
||||
# ================================================================================================
|
||||
# Comprehensive Windows build configuration using Zig's bundled Clang with GNU target
|
||||
|
||||
# C compiler flags for Windows
|
||||
cc_flags = [
|
||||
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
|
||||
"-fuse-ld=lld", # Use Zig's bundled LLD linker
|
||||
"-std=c17", # C17 standard
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
]
|
||||
|
||||
# C++ compiler flags for Windows
|
||||
cpp_flags = [
|
||||
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
|
||||
"-fuse-ld=lld-link", # Use lld-link (Windows MSVC-compatible linker)
|
||||
"-std=c++17", # C++17 standard
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
|
||||
# Fix Arduino INPUT macro conflict with Windows headers
|
||||
"-DNOMINMAX", # Prevent Windows min/max macros
|
||||
"-DWIN32_LEAN_AND_MEAN", # Reduce Windows header conflicts
|
||||
]
|
||||
|
||||
# Linker flags for Windows
|
||||
link_flags = [
|
||||
"-mconsole", # Console application
|
||||
|
||||
# Prevent automatic library inclusion to avoid CRT collisions
|
||||
"-nodefaultlibs", # Don't automatically include default libraries
|
||||
|
||||
# Fix CRT collision between Zig/Clang and MSYS2/MinGW runtimes
|
||||
# This prevents "atexit was replaced" errors from conflicting CRT implementations
|
||||
# "-rtlib=compiler-rt", # Removed - was causing conflicts
|
||||
"-unwindlib=libunwind", # Use Clang's libunwind (avoid MinGW unwind)
|
||||
"-nostdlib++", # Don't auto-link standard C++ library
|
||||
"-lc++", # Manually link libc++ (Clang's C++ standard library)
|
||||
|
||||
"-lkernel32", # Windows kernel functions
|
||||
"-luser32", # Windows user interface
|
||||
"-lgdi32", # Graphics device interface
|
||||
"-ladvapi32", # Advanced Windows API
|
||||
]
|
||||
|
||||
[linking.base]
|
||||
# Base linking flags (Unix-like systems)
|
||||
flags = [
|
||||
"-pthread", # POSIX threads (not used on Windows)
|
||||
]
|
||||
|
||||
[build_modes.quick]
|
||||
flags = [
|
||||
"-O1",
|
||||
"-g0",
|
||||
"-fno-inline-functions",
|
||||
"-fno-vectorize",
|
||||
"-fno-unroll-loops",
|
||||
"-fno-strict-aliasing",
|
||||
]
|
||||
|
||||
link_flags = []
|
||||
|
||||
[strict_mode]
|
||||
flags = [
|
||||
"-Werror",
|
||||
"-Wextra",
|
||||
"-Wconversion",
|
||||
"-Wsign-conversion",
|
||||
"-Werror=unused-variable", # Force error on unused variables
|
||||
"-Werror=unused-parameter", # Force error on unused parameters
|
||||
"-Werror=unused-function", # Force error on unused functions
|
||||
"-Werror=unused-but-set-variable", # Force error on variables that are set but never used
|
||||
"-Wuninitialized",
|
||||
"-Wdouble-promotion",
|
||||
"-Wformat=2",
|
||||
"-Wcast-align",
|
||||
"-Wcast-qual",
|
||||
"-Werror=return-type",
|
||||
]
|
||||
|
||||
[stub_platform]
|
||||
# ================================================================================================
|
||||
# STUB PLATFORM DEFINES FOR EXAMPLE COMPILATION
|
||||
# ================================================================================================
|
||||
# These defines are used for compiling examples on the STUB platform
|
||||
# which provides Arduino-compatible functionality without hardware dependencies.
|
||||
|
||||
defines = [
|
||||
"STUB_PLATFORM", # Enable STUB platform
|
||||
"ARDUINO=10808", # Arduino version compatibility
|
||||
"FASTLED_USE_STUB_ARDUINO", # Use stub Arduino implementation
|
||||
"FASTLED_STUB_IMPL", # Enable STUB platform Arduino function implementations
|
||||
"SKETCH_HAS_LOTS_OF_MEMORY=1", # Enable memory-intensive features for STUB platform
|
||||
"FASTLED_HAS_ENGINE_EVENTS=1", # Enable EngineEvents for UI components
|
||||
]
|
||||
|
||||
# Include paths required for stub platform compilation
|
||||
include_paths = [
|
||||
"src/platforms/stub", # Arduino.h stub for examples
|
||||
]
|
||||
|
||||
# Additional compiler arguments for stub platform
|
||||
compiler_args = [
|
||||
# No additional args currently needed - all flags come from [all] and [build_modes.quick] sections
|
||||
]
|
||||
|
||||
[archive]
|
||||
# ================================================================================================
|
||||
# ARCHIVE CREATION FLAGS
|
||||
# ================================================================================================
|
||||
# Flags for creating static libraries (.a files) with deterministic builds.
|
||||
# Using Zig's unified toolchain (ziglang ar) which supports deterministic builds on all platforms.
|
||||
|
||||
flags = "rcsD" # r=insert, c=create if needed, s=write symbol table, D=deterministic (zero timestamps)
|
||||
|
||||
[linking.unix]
|
||||
# ================================================================================================
|
||||
# UNIX-SPECIFIC LINKING FLAGS
|
||||
# ================================================================================================
|
||||
# Linking flags specific to Unix-like systems (Linux, macOS).
|
||||
|
||||
flags = [
|
||||
# Core Libraries
|
||||
"-pthread", # POSIX threads support
|
||||
"-lm", # Math library
|
||||
"-ldl", # Dynamic loading library
|
||||
|
||||
# Platform-specific libraries (added as needed)
|
||||
# Note: Additional platform-specific flags can be added here
|
||||
]
|
||||
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Unified Build API - Example Usage
|
||||
|
||||
This script demonstrates how to use the new unified build API to replace
|
||||
the old scattered build logic for both unit tests and examples.
|
||||
|
||||
Before (scattered logic):
|
||||
- PCH building logic in multiple places
|
||||
- libfastled.a building in callers
|
||||
- Build flags mixed between unit tests and examples
|
||||
- Manual directory management
|
||||
|
||||
After (unified API):
|
||||
- Single API for both unit tests and examples
|
||||
- Automatic PCH and library building
|
||||
- Clean separation of build configurations
|
||||
- Organized build directory structure
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from ci.build_api import BuildType, create_example_builder, create_unit_test_builder
|
||||
|
||||
|
||||
def demo_unit_test_building():
|
||||
"""Demonstrate unit test building with the unified API"""
|
||||
print("=" * 60)
|
||||
print("UNIT TEST BUILDING DEMO")
|
||||
print("=" * 60)
|
||||
|
||||
# Create unit test builder - automatically uses ci/build_unit.toml
|
||||
unit_builder = create_unit_test_builder(
|
||||
build_dir=".build/unit", use_pch=True, parallel=True, clean=False
|
||||
)
|
||||
|
||||
# Get build info
|
||||
info = unit_builder.get_build_info()
|
||||
print(f"Build configuration: {info}")
|
||||
|
||||
# Build multiple unit test targets efficiently
|
||||
# The API automatically handles:
|
||||
# - PCH building (once, reused by all targets)
|
||||
# - libfastled.a building (once, reused by all targets)
|
||||
# - Individual target compilation and linking
|
||||
test_files = [
|
||||
Path("tests/test_json.cpp"),
|
||||
Path("tests/test_color.cpp"),
|
||||
Path("tests/test_math.cpp"),
|
||||
]
|
||||
|
||||
print(f"\nBuilding {len(test_files)} unit test targets...")
|
||||
results = unit_builder.build_targets(test_files)
|
||||
|
||||
# Report results
|
||||
for result in results:
|
||||
status = "SUCCESS" if result.success else "FAILED"
|
||||
print(f" {result.target.name}: {status} ({result.build_time:.2f}s)")
|
||||
if not result.success:
|
||||
print(f" Error: {result.message}")
|
||||
|
||||
print(f"\nBuild artifacts organized in: {unit_builder.build_dir}")
|
||||
print(f" - PCH file: {unit_builder.artifacts_dir}")
|
||||
print(f" - Library: {unit_builder.artifacts_dir}")
|
||||
print(f" - Targets: {unit_builder.targets_dir}")
|
||||
print(f" - Cache: {unit_builder.cache_dir}")
|
||||
|
||||
|
||||
def demo_example_building():
|
||||
"""Demonstrate example building with the unified API"""
|
||||
print("\n" + "=" * 60)
|
||||
print("EXAMPLE BUILDING DEMO")
|
||||
print("=" * 60)
|
||||
|
||||
# Create example builder - automatically uses ci/build_example.toml
|
||||
example_builder = create_example_builder(
|
||||
build_dir=".build/examples", use_pch=True, parallel=True, clean=False
|
||||
)
|
||||
|
||||
# Get build info
|
||||
info = example_builder.get_build_info()
|
||||
print(f"Build configuration: {info}")
|
||||
|
||||
# Build multiple example targets efficiently
|
||||
# The API automatically handles:
|
||||
# - PCH building (once, reused by all targets)
|
||||
# - libfastled.a building (once, with example-specific flags)
|
||||
# - Individual target compilation and linking
|
||||
example_files = [
|
||||
Path("examples/Blink/Blink.ino"),
|
||||
Path("examples/DemoReel100/DemoReel100.ino"),
|
||||
Path("examples/ColorPalette/ColorPalette.ino"),
|
||||
]
|
||||
|
||||
print(f"\nBuilding {len(example_files)} example targets...")
|
||||
results = example_builder.build_targets(example_files)
|
||||
|
||||
# Report results
|
||||
for result in results:
|
||||
status = "SUCCESS" if result.success else "FAILED"
|
||||
print(f" {result.target.name}: {status} ({result.build_time:.2f}s)")
|
||||
if not result.success:
|
||||
print(f" Error: {result.message}")
|
||||
|
||||
print(f"\nBuild artifacts organized in: {example_builder.build_dir}")
|
||||
print(f" - PCH file: {example_builder.artifacts_dir}")
|
||||
print(f" - Library: {example_builder.artifacts_dir}")
|
||||
print(f" - Targets: {example_builder.targets_dir}")
|
||||
print(f" - Cache: {example_builder.cache_dir}")
|
||||
|
||||
|
||||
def demo_build_configuration_differences():
|
||||
"""Demonstrate the differences between unit test and example configurations"""
|
||||
print("\n" + "=" * 60)
|
||||
print("BUILD CONFIGURATION DIFFERENCES")
|
||||
print("=" * 60)
|
||||
|
||||
unit_builder = create_unit_test_builder()
|
||||
example_builder = create_example_builder()
|
||||
|
||||
unit_info = unit_builder.get_build_info()
|
||||
example_info = example_builder.get_build_info()
|
||||
|
||||
print("Unit Test Configuration:")
|
||||
print(f" - Build flags TOML: {unit_info['build_flags_toml']}")
|
||||
print(f" - Build directory: {unit_info['build_dir']}")
|
||||
print(f" - Build type: {unit_info['build_type']}")
|
||||
print(f" - Includes FASTLED_FORCE_NAMESPACE=1")
|
||||
print(f" - Includes FASTLED_TESTING=1")
|
||||
|
||||
print("\nExample Configuration:")
|
||||
print(f" - Build flags TOML: {example_info['build_flags_toml']}")
|
||||
print(f" - Build directory: {example_info['build_dir']}")
|
||||
print(f" - Build type: {example_info['build_type']}")
|
||||
print(f" - Uses global namespace (no FASTLED_FORCE_NAMESPACE)")
|
||||
print(f" - Arduino compatibility defines")
|
||||
|
||||
|
||||
def demo_clean_builds():
|
||||
"""Demonstrate clean build functionality"""
|
||||
print("\n" + "=" * 60)
|
||||
print("CLEAN BUILD DEMO")
|
||||
print("=" * 60)
|
||||
|
||||
# Create builders with clean=True to force rebuilding of all artifacts
|
||||
unit_builder = create_unit_test_builder(clean=True)
|
||||
example_builder = create_example_builder(clean=True)
|
||||
|
||||
print("Unit test builder:")
|
||||
print(f" - Will clean and rebuild all artifacts in {unit_builder.build_dir}")
|
||||
|
||||
print("Example builder:")
|
||||
print(f" - Will clean and rebuild all artifacts in {example_builder.build_dir}")
|
||||
|
||||
# Demonstrate manual cleaning
|
||||
print("\nManual cleaning:")
|
||||
unit_builder.clean_build_artifacts()
|
||||
print(f" - Cleaned unit test artifacts")
|
||||
|
||||
example_builder.clean_build_artifacts()
|
||||
print(f" - Cleaned example artifacts")
|
||||
|
||||
|
||||
def demo_advanced_usage():
|
||||
"""Demonstrate advanced usage patterns"""
|
||||
print("\n" + "=" * 60)
|
||||
print("ADVANCED USAGE DEMO")
|
||||
print("=" * 60)
|
||||
|
||||
# Custom build directory structure
|
||||
unit_builder = create_unit_test_builder(
|
||||
build_dir=".build/my_custom_unit_tests",
|
||||
use_pch=False, # Disable PCH for debugging
|
||||
parallel=False, # Disable parallel for single-threaded debugging
|
||||
)
|
||||
|
||||
print("Custom configuration:")
|
||||
print(f" - Custom build directory: {unit_builder.build_dir}")
|
||||
print(f" - PCH disabled for easier debugging")
|
||||
print(f" - Parallel compilation disabled for sequential output")
|
||||
|
||||
# Show the organized directory structure
|
||||
print(f"\nOrganized directory structure:")
|
||||
print(f" {unit_builder.build_dir}/")
|
||||
print(f" ├── artifacts/ # PCH files and libfastled.a")
|
||||
print(f" ├── targets/ # Individual target builds")
|
||||
print(f" │ ├── test_json/")
|
||||
print(f" │ │ ├── test_json.o")
|
||||
print(f" │ │ └── test_json.exe")
|
||||
print(f" │ └── test_color/")
|
||||
print(f" │ ├── test_color.o")
|
||||
print(f" │ └── test_color.exe")
|
||||
print(f" └── cache/ # Fingerprint caches")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("FastLED Unified Build API - Example Usage")
|
||||
print("This demonstrates the new unified build system")
|
||||
|
||||
# Run all demos (commented out to avoid actual compilation during demo)
|
||||
|
||||
# Uncomment these to run actual builds:
|
||||
# demo_unit_test_building()
|
||||
# demo_example_building()
|
||||
|
||||
# These can run without actual building:
|
||||
demo_build_configuration_differences()
|
||||
demo_clean_builds()
|
||||
demo_advanced_usage()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("DEMO COMPLETE")
|
||||
print("=" * 60)
|
||||
print("The unified build API provides:")
|
||||
print("✓ Single API for both unit tests and examples")
|
||||
print("✓ Automatic PCH and library building")
|
||||
print("✓ Clean separation of build configurations")
|
||||
print("✓ Organized build directory structure")
|
||||
print("✓ Efficient multiple target building")
|
||||
print("✓ Simple, elegant interface")
|
||||
@@ -0,0 +1,276 @@
|
||||
# ================================================================================================
|
||||
# FastLED Unit Test Build Flags Configuration
|
||||
# ================================================================================================
|
||||
#
|
||||
# This file centralizes ALL compilation flags used by unit test compilation to ensure
|
||||
# they remain synchronized and compatible across the entire build system.
|
||||
#
|
||||
# Unit tests require special flags like FASTLED_FORCE_NAMESPACE=1 to put all symbols
|
||||
# in the fl:: namespace that the tests expect.
|
||||
|
||||
[tools]
|
||||
# Build tool configuration - Full compiler command with arguments
|
||||
# CRITICAL: Must use 'uv run python' for subprocess execution (subprocess doesn't inherit uv environment)
|
||||
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
linker = ["uv", "run", "python", "-m", "ziglang", "c++"] # Separate linker config for consistency
|
||||
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"] # Ziglang archiver for unified toolchain
|
||||
c_compiler = ["uv", "run", "python", "-m", "ziglang", "cc"]
|
||||
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
|
||||
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
|
||||
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
|
||||
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
|
||||
|
||||
[all]
|
||||
# Universal compilation flags (compiler command handled separately)
|
||||
compiler_flags = [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-Wno-deprecated-register", # Suppress 'register' keyword warnings for C++17 compatibility
|
||||
"-Wno-backslash-newline-escape", # Suppress backslash newline escape warnings
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
]
|
||||
|
||||
defines = [
|
||||
"STUB_PLATFORM",
|
||||
"FASTLED_UNIT_TEST=1",
|
||||
"FASTLED_FORCE_NAMESPACE=1", # CRITICAL: Required for unit tests to access fl:: namespace
|
||||
"FASTLED_TESTING=1", # Enable test-specific functionality like MockTimeProvider
|
||||
]
|
||||
|
||||
include_flags = [
|
||||
"-I.",
|
||||
"-Isrc",
|
||||
"-Itests",
|
||||
]
|
||||
|
||||
[windows]
|
||||
# ================================================================================================
|
||||
# WINDOWS-SPECIFIC BUILD FLAGS
|
||||
# ================================================================================================
|
||||
# Comprehensive Windows build configuration using Zig's bundled Clang with GNU target
|
||||
|
||||
# C compiler flags for Windows
|
||||
cc_flags = [
|
||||
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
|
||||
"-fuse-ld=lld", # Use Zig's bundled LLD linker
|
||||
"-std=c17", # C17 standard
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
]
|
||||
|
||||
# C++ compiler flags for Windows
|
||||
cpp_flags = [
|
||||
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
|
||||
"-fuse-ld=lld-link", # Use lld-link (Windows MSVC-compatible linker)
|
||||
"-std=c++17", # C++17 standard
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
|
||||
# Fix Arduino INPUT macro conflict with Windows headers
|
||||
"-DNOMINMAX", # Prevent Windows min/max macros
|
||||
"-DWIN32_LEAN_AND_MEAN", # Reduce Windows header conflicts
|
||||
]
|
||||
|
||||
# Linker flags for Windows
|
||||
link_flags = [
|
||||
"-mconsole", # Console application
|
||||
|
||||
# Prevent automatic library inclusion to avoid CRT collisions
|
||||
"-nodefaultlibs", # Don't automatically include default libraries
|
||||
|
||||
# Fix CRT collision between Zig/Clang and MSYS2/MinGW runtimes
|
||||
# This prevents "atexit was replaced" errors from conflicting CRT implementations
|
||||
# "-rtlib=compiler-rt", # Removed - was causing conflicts
|
||||
"-unwindlib=libunwind", # Use Clang's libunwind (avoid MinGW unwind)
|
||||
"-nostdlib++", # Don't auto-link standard C++ library
|
||||
"-lc++", # Manually link libc++ (Clang's C++ standard library)
|
||||
|
||||
"-lkernel32", # Windows kernel functions
|
||||
"-luser32", # Windows user interface
|
||||
"-lgdi32", # Graphics device interface
|
||||
"-ladvapi32", # Advanced Windows API
|
||||
]
|
||||
|
||||
[linking.base]
|
||||
# Base linking flags (Unix-like systems)
|
||||
flags = [
|
||||
"-pthread", # POSIX threads (not used on Windows)
|
||||
]
|
||||
|
||||
[linux]
|
||||
# Linux-specific build configuration – libunwind disabled (fallback to execinfo)
|
||||
link_flags = [
|
||||
]
|
||||
|
||||
defines = [
|
||||
]
|
||||
|
||||
[build_modes.quick]
|
||||
flags = [
|
||||
"-O1",
|
||||
"-fno-inline-functions",
|
||||
"-fno-vectorize",
|
||||
"-fno-unroll-loops",
|
||||
"-fno-strict-aliasing",
|
||||
"-fno-omit-frame-pointer", # Preserve stack frames for debugging
|
||||
]
|
||||
|
||||
link_flags = []
|
||||
|
||||
[build_modes.debug_asan]
|
||||
# AddressSanitizer and UndefinedBehaviorSanitizer for debugging memory issues
|
||||
flags = [
|
||||
"-fsanitize=address", # AddressSanitizer for memory error detection
|
||||
"-fsanitize=undefined", # UndefinedBehaviorSanitizer for undefined behavior
|
||||
"-O1", # Light optimization for performance
|
||||
"-fno-omit-frame-pointer", # Preserve stack frames for better debugging
|
||||
"-fno-optimize-sibling-calls", # Better stack traces
|
||||
]
|
||||
|
||||
link_flags = [
|
||||
"-fsanitize=address",
|
||||
"-fsanitize=undefined",
|
||||
]
|
||||
|
||||
[strict_mode]
|
||||
flags = [
|
||||
"-Werror",
|
||||
"-Wextra",
|
||||
"-Wconversion",
|
||||
"-Wsign-conversion",
|
||||
"-Werror=unused-variable", # Force error on unused variables
|
||||
"-Werror=unused-parameter", # Force error on unused parameters
|
||||
"-Werror=unused-function", # Force error on unused functions
|
||||
"-Werror=unused-but-set-variable", # Force error on variables that are set but never used
|
||||
"-Wuninitialized",
|
||||
"-Wdouble-promotion",
|
||||
"-Wformat=2",
|
||||
"-Wcast-align",
|
||||
"-Wcast-qual",
|
||||
"-Werror=return-type",
|
||||
]
|
||||
|
||||
[test]
|
||||
# Test-specific preprocessor defines
|
||||
defines = [
|
||||
# FastLED Test Configuration
|
||||
"-DFASTLED_UNIT_TEST=1", # Identify as unit test build
|
||||
"-DFASTLED_FORCE_NAMESPACE=1", # Force namespace usage to avoid conflicts
|
||||
"-DFASTLED_USE_PROGMEM=0", # Disable PROGMEM usage in tests
|
||||
"-DSTUB_PLATFORM", # Use stub platform for testing
|
||||
"-DARDUINO=10808", # Arduino version compatibility
|
||||
"-DFASTLED_USE_STUB_ARDUINO", # Use stub Arduino implementation
|
||||
"-DSKETCH_HAS_LOTS_OF_MEMORY=1", # Assume lots of memory available
|
||||
"-DFASTLED_STUB_IMPL", # Use stub implementations
|
||||
"-DFASTLED_USE_JSON_UI=1", # Enable JSON UI features
|
||||
"-DFASTLED_TESTING", # Enable testing features
|
||||
"-DFASTLED_NO_AUTO_NAMESPACE", # Disable automatic namespace
|
||||
"-DFASTLED_NO_PINMAP", # Disable pin mapping
|
||||
"-DHAS_HARDWARE_PIN_SUPPORT", # Enable hardware pin support
|
||||
"-DFASTLED_DEBUG_LEVEL=1", # Set debug level
|
||||
"-DFASTLED_NO_ATEXIT=1", # Disable atexit to fix Windows lld-link errors
|
||||
"-DDOCTEST_CONFIG_NO_EXCEPTIONS_BUT_WITH_ALL_ASSERTS", # Doctest configuration
|
||||
"-DENABLE_CRASH_HANDLER", # Enable global crash handler in tests
|
||||
]
|
||||
|
||||
# Test-specific compiler flags
|
||||
compiler_flags = [
|
||||
# Language Standard and Compatibility
|
||||
"-std=c++17", # Base C++17 standard
|
||||
"-std=gnu++17", # Use GNU C++17 standard with extensions
|
||||
"-fpermissive", # Allow some non-conforming code patterns
|
||||
|
||||
# Warning Configuration
|
||||
"-Wall", # Enable all warnings
|
||||
"-Wextra", # Enable extra warnings
|
||||
"-Werror=return-type", # Error on missing return statements
|
||||
|
||||
# Debug and Crash Handling
|
||||
"-fno-omit-frame-pointer", # Preserve stack frames for debugging
|
||||
|
||||
# AddressSanitizer and Memory Debugging
|
||||
"-fsanitize=address", # AddressSanitizer for memory error detection
|
||||
"-fsanitize=undefined", # UndefinedBehaviorSanitizer for undefined behavior
|
||||
|
||||
# Thread Safety and Runtime Behavior
|
||||
"-fno-threadsafe-statics", # Disable thread-safe static initialization
|
||||
"-fno-exceptions", # Disable C++ exceptions
|
||||
"-fno-rtti", # Disable RTTI
|
||||
|
||||
# PCH Support
|
||||
"-fpch-instantiate-templates", # Required for proper PCH template handling
|
||||
]
|
||||
|
||||
# Test-specific include flags
|
||||
include_flags = [
|
||||
"-I.", # Current directory
|
||||
"-Isrc", # FastLED source directory
|
||||
"-Itests", # Test directory
|
||||
]
|
||||
|
||||
# Test-specific linking flags for AddressSanitizer and libunwind
|
||||
link_flags = [
|
||||
"-fsanitize=address", # Link AddressSanitizer runtime
|
||||
"-fsanitize=undefined", # Link UndefinedBehaviorSanitizer runtime
|
||||
"-lunwind", # Link libunwind for stack traces
|
||||
"-lunwind-x86_64", # Platform-specific libunwind functions
|
||||
]
|
||||
|
||||
[archive]
|
||||
# ================================================================================================
|
||||
# ARCHIVE CREATION FLAGS
|
||||
# ================================================================================================
|
||||
# Flags for creating static libraries (.a files) with deterministic builds.
|
||||
# Using Zig's unified toolchain (ziglang ar) which supports deterministic builds on all platforms.
|
||||
|
||||
flags = "rcsD" # r=insert, c=create if needed, s=write symbol table, D=deterministic (zero timestamps)
|
||||
|
||||
[stub_platform]
|
||||
# ================================================================================================
|
||||
# STUB PLATFORM DEFINES FOR UNIT TEST COMPILATION
|
||||
# ================================================================================================
|
||||
# These defines are used when unit tests need to compile example code on the STUB platform
|
||||
# which provides Arduino-compatible functionality without hardware dependencies.
|
||||
|
||||
defines = [
|
||||
"STUB_PLATFORM", # Enable STUB platform
|
||||
"ARDUINO=10808", # Arduino version compatibility
|
||||
"FASTLED_USE_STUB_ARDUINO", # Use stub Arduino implementation
|
||||
"FASTLED_STUB_IMPL", # Enable STUB platform Arduino function implementations
|
||||
"SKETCH_HAS_LOTS_OF_MEMORY=1", # Enable memory-intensive features for STUB platform
|
||||
"FASTLED_HAS_ENGINE_EVENTS=1", # Enable EngineEvents for UI components
|
||||
"FASTLED_UNIT_TEST=1", # Identify as unit test build
|
||||
"FASTLED_FORCE_NAMESPACE=1", # Force namespace usage for unit tests
|
||||
"FASTLED_TESTING=1", # Enable testing features
|
||||
]
|
||||
|
||||
# Include paths required for stub platform compilation in unit tests
|
||||
include_paths = [
|
||||
"src/platforms/stub", # Arduino.h stub for examples
|
||||
]
|
||||
|
||||
# Additional compiler arguments for stub platform
|
||||
compiler_args = [
|
||||
# No additional args currently needed - all flags come from [all] and other sections
|
||||
]
|
||||
|
||||
[linking.unix]
|
||||
# ================================================================================================
|
||||
# UNIX-SPECIFIC LINKING FLAGS
|
||||
# ================================================================================================
|
||||
# Linking flags specific to Unix-like systems (Linux, macOS).
|
||||
|
||||
flags = [
|
||||
# Core Libraries
|
||||
"-pthread", # POSIX threads support
|
||||
"-lm", # Math library
|
||||
"-ldl", # Dynamic loading library
|
||||
|
||||
# Platform-specific libraries (added as needed)
|
||||
# Note: Additional platform-specific flags can be added here
|
||||
]
|
||||
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
TypeScript checking for FastLED JavaScript/WASM compiler modules.
|
||||
|
||||
This script runs TypeScript compiler with JSDoc checking enabled
|
||||
to catch type errors in JavaScript files that would be caught
|
||||
at compile time in TypeScript.
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def check_js_types():
|
||||
"""Run TypeScript checking on JavaScript files."""
|
||||
wasm_compiler_dir = (
|
||||
Path(__file__).parent.parent / "src" / "platforms" / "wasm" / "compiler"
|
||||
)
|
||||
|
||||
if not wasm_compiler_dir.exists():
|
||||
print(f"❌ WASM compiler directory not found: {wasm_compiler_dir}")
|
||||
return 1
|
||||
|
||||
jsconfig_path = wasm_compiler_dir / "jsconfig.json"
|
||||
if not jsconfig_path.exists():
|
||||
print(f"❌ jsconfig.json not found: {jsconfig_path}")
|
||||
return 1
|
||||
|
||||
print(f"🔍 Checking JavaScript types in: {wasm_compiler_dir}")
|
||||
|
||||
try:
|
||||
# Run TypeScript compiler with noEmit to only check types
|
||||
result = subprocess.run(
|
||||
["npx", "tsc", "--noEmit", "--project", str(jsconfig_path)],
|
||||
cwd=wasm_compiler_dir,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
print("✅ JavaScript type checking passed!")
|
||||
return 0
|
||||
else:
|
||||
print("❌ JavaScript type checking failed:")
|
||||
print(result.stdout)
|
||||
print(result.stderr)
|
||||
return 1
|
||||
|
||||
except FileNotFoundError:
|
||||
print(
|
||||
"❌ TypeScript (tsc) not found. Please install: npm install -g typescript"
|
||||
)
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(f"❌ Error running type check: {e}")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(check_js_types())
|
||||
@@ -0,0 +1,123 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, List, Match, Optional
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
PROJECT_ROOT = HERE.parent
|
||||
|
||||
IS_GITHUB = "GITHUB_ACTIONS" in os.environ
|
||||
|
||||
|
||||
def run_command(
|
||||
cmd_list: List[str],
|
||||
capture_output: bool = True,
|
||||
shell: bool = False,
|
||||
check: Optional[bool] = None,
|
||||
) -> Optional[str]:
|
||||
"""Run a command and return its output."""
|
||||
actual_check = check if check is not None else False
|
||||
cmd = cmd_list if not shell else subprocess.list2cmdline(cmd_list)
|
||||
|
||||
result: subprocess.CompletedProcess[str] = subprocess.run(
|
||||
cmd, capture_output=capture_output, text=True, shell=shell, check=actual_check
|
||||
)
|
||||
|
||||
if not capture_output:
|
||||
return None
|
||||
|
||||
stdout: str = result.stdout
|
||||
stdout = stdout.strip()
|
||||
return stdout
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check compiled program size for a board"
|
||||
)
|
||||
parser.add_argument("board", help="Board name")
|
||||
parser.add_argument(
|
||||
"--max-size", type=int, required=False, help="Maximum allowed size"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-build",
|
||||
action="store_true",
|
||||
help="Skip compilation and check existing build",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--example",
|
||||
default="Blink",
|
||||
help="Example to compile (default: Blink)",
|
||||
)
|
||||
|
||||
# Parse known args first
|
||||
args, unknown = parser.parse_known_args()
|
||||
|
||||
# Add remaining arguments as extra_args
|
||||
args.extra_args = unknown
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def main():
|
||||
os.chdir(str(PROJECT_ROOT))
|
||||
args = parse_args()
|
||||
|
||||
if not args.no_build:
|
||||
cmd_list = [
|
||||
"uv",
|
||||
"run",
|
||||
"python",
|
||||
"-m",
|
||||
"ci.ci-compile",
|
||||
args.board,
|
||||
"--examples",
|
||||
args.example,
|
||||
] + args.extra_args
|
||||
try:
|
||||
run_command(cmd_list, shell=True, capture_output=IS_GITHUB, check=True)
|
||||
except subprocess.CalledProcessError:
|
||||
run_command(cmd_list, shell=True, capture_output=False, check=True)
|
||||
|
||||
output = run_command(
|
||||
["uv", "run", "-m", "ci.compiled_size", "--board", args.board],
|
||||
capture_output=True,
|
||||
)
|
||||
size_match = re.search(r": *(\d+)", output) # type: ignore
|
||||
|
||||
if not size_match:
|
||||
print("Error: Unable to extract size from output")
|
||||
print(f"Output: {output}")
|
||||
sys.exit(1)
|
||||
|
||||
size_str = size_match.group(1) # type: ignore
|
||||
if not size_str:
|
||||
print("Error: Size group is empty")
|
||||
sys.exit(1)
|
||||
size = int(size_str) # type: ignore
|
||||
|
||||
if args.max_size is not None and args.max_size > 0:
|
||||
max_size = args.max_size
|
||||
if size > max_size:
|
||||
print(f"{args.board} size {size} is greater than max size {max_size}")
|
||||
print("::error::Compiled size exceeds maximum allowed size")
|
||||
sys.exit(1)
|
||||
else:
|
||||
print(f"{args.board} size {size} is within the limit of {max_size}")
|
||||
else:
|
||||
if not args.max_size:
|
||||
print("Warning: No max size specified")
|
||||
elif args.max_size <= 0:
|
||||
print("Warning: max size was <= 0 so no check was performed")
|
||||
print(f"{args.board} size: {size}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,5 @@
|
||||
#!/bin/bash
|
||||
|
||||
# cd to the directory of the script
|
||||
cd "$(dirname "$0")"
|
||||
uv run python ci-compile.py "$@"
|
||||
@@ -0,0 +1,424 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
BACKUP: Original concurrent build system - REPLACED by pio ci approach
|
||||
|
||||
This file is a backup of the original ci-compile.py that used concurrent builds.
|
||||
It has been replaced by a simpler pio ci approach in the main ci-compile.py.
|
||||
|
||||
The original system used ThreadPoolExecutor to run multiple board compilations
|
||||
in parallel with build directory creation and artifact recycling.
|
||||
|
||||
The new system uses pio ci command directly for each board/example combination,
|
||||
which is simpler and more straightforward.
|
||||
|
||||
This backup is kept for reference and potential future needs.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import io
|
||||
import multiprocessing
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
from typing import List, Set
|
||||
|
||||
from ci.boards import Board, create_board # type: ignore
|
||||
from ci.util.concurrent_run import ConcurrentRunArgs, concurrent_run
|
||||
from ci.util.locked_print import locked_print
|
||||
|
||||
|
||||
HERE = Path(__file__).parent.resolve()
|
||||
|
||||
LIBS = ["src"]
|
||||
EXTRA_LIBS = [
|
||||
"https://github.com/me-no-dev/ESPAsyncWebServer.git",
|
||||
"ArduinoOTA",
|
||||
"SD",
|
||||
"FS",
|
||||
"ESPmDNS",
|
||||
"WiFi",
|
||||
"WebSockets",
|
||||
]
|
||||
BUILD_FLAGS = ["-Wl,-Map,firmware.map", "-fopt-info-all=optimization_report.txt"]
|
||||
|
||||
# Default boards to compile for. You can use boards not defined here but
|
||||
# if the board isn't part of the officially supported platformio boards then
|
||||
# you will need to add the board to the ~/.platformio/platforms directory.
|
||||
# prior to running this script. This happens automatically as of 2024-08-20
|
||||
# with the github workflow scripts.
|
||||
DEFAULT_BOARDS_NAMES = [
|
||||
"apollo3_red",
|
||||
"apollo3_thing_explorable",
|
||||
"web", # work in progress
|
||||
"uno", # Build is faster if this is first, because it's used for global init.
|
||||
"esp32dev",
|
||||
"esp01", # ESP8266
|
||||
"esp32c3",
|
||||
"attiny85",
|
||||
"ATtiny1616",
|
||||
"esp32c6",
|
||||
"esp32s3",
|
||||
"esp32p4",
|
||||
"yun",
|
||||
"digix",
|
||||
"teensylc",
|
||||
"teensy30",
|
||||
"teensy31",
|
||||
"teensy41",
|
||||
"adafruit_feather_nrf52840_sense",
|
||||
"xiaoblesense_adafruit",
|
||||
"rpipico",
|
||||
"rpipico2",
|
||||
"uno_r4_wifi",
|
||||
"esp32rmt_51",
|
||||
"esp32dev_idf44",
|
||||
"bluepill",
|
||||
"esp32rmt_51",
|
||||
"giga_r1",
|
||||
"sparkfun_xrp_controller",
|
||||
]
|
||||
|
||||
OTHER_BOARDS_NAMES = [
|
||||
"nano_every",
|
||||
"esp32-c2-devkitm-1",
|
||||
]
|
||||
|
||||
# Examples to compile.
|
||||
DEFAULT_EXAMPLES = [
|
||||
"Animartrix",
|
||||
"Apa102",
|
||||
"Apa102HD",
|
||||
"Apa102HDOverride",
|
||||
"Audio",
|
||||
"Blink",
|
||||
"Blur",
|
||||
"Chromancer",
|
||||
"ColorPalette",
|
||||
"ColorTemperature",
|
||||
"Corkscrew",
|
||||
"Cylon",
|
||||
"DemoReel100",
|
||||
# "Downscale",
|
||||
"FestivalStick",
|
||||
"FirstLight",
|
||||
"Fire2012",
|
||||
"Multiple/MultipleStripsInOneArray",
|
||||
"Multiple/ArrayOfLedArrays",
|
||||
"Noise",
|
||||
"NoisePlayground",
|
||||
"NoisePlusPalette",
|
||||
# "LuminescentGrand",
|
||||
"Pacifica",
|
||||
"Pride2015",
|
||||
"RGBCalibrate",
|
||||
"RGBSetDemo",
|
||||
"RGBW",
|
||||
"Overclock",
|
||||
"RGBWEmulated",
|
||||
"TwinkleFox",
|
||||
"XYMatrix",
|
||||
"FireMatrix",
|
||||
"FireCylinder",
|
||||
"FxGfx2Video",
|
||||
"FxSdCard",
|
||||
"FxCylon",
|
||||
"FxDemoReel100",
|
||||
"FxTwinkleFox",
|
||||
"FxFire2012",
|
||||
"FxNoisePlusPalette",
|
||||
"FxPacifica",
|
||||
"FxEngine",
|
||||
"WS2816",
|
||||
]
|
||||
|
||||
EXTRA_EXAMPLES: dict[Board, list[str]] = {
|
||||
# ESP32DEV: ["EspI2SDemo"],
|
||||
# ESP32_S3_DEVKITC_1: ["EspS3I2SDemo"],
|
||||
}
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compile FastLED examples for various boards."
|
||||
)
|
||||
# parser.add_argument(
|
||||
# "--boards", type=str, help="Comma-separated list of boards to compile for"
|
||||
# )
|
||||
# needs to be a positional argument instead
|
||||
parser.add_argument(
|
||||
"boards",
|
||||
type=str,
|
||||
help="Comma-separated list of boards to compile for",
|
||||
nargs="?",
|
||||
)
|
||||
parser.add_argument(
|
||||
"positional_examples",
|
||||
type=str,
|
||||
help="Examples to compile (positional arguments after board name)",
|
||||
nargs="*",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--examples", type=str, help="Comma-separated list of examples to compile"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--exclude-examples", type=str, help="Examples that should be excluded"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-init", action="store_true", help="Skip the initialization step"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--defines", type=str, help="Comma-separated list of compiler definitions"
|
||||
)
|
||||
parser.add_argument("--customsdk", type=str, help="custom_sdkconfig project option")
|
||||
parser.add_argument(
|
||||
"--extra-packages",
|
||||
type=str,
|
||||
help="Comma-separated list of extra packages to install",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--add-extra-esp32-libs",
|
||||
action="store_true",
|
||||
help="Add extra libraries to the libraries list to check against compiler errors.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--build-dir", type=str, help="Override the default build directory"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-project-options",
|
||||
action="store_true",
|
||||
help="Don't use custom project options",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--interactive",
|
||||
action="store_true",
|
||||
help="Enable interactive mode to choose a board",
|
||||
)
|
||||
# Passed by the github action to disable interactive mode.
|
||||
parser.add_argument(
|
||||
"--no-interactive", action="store_true", help="Disable interactive mode"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v", "--verbose", action="store_true", help="Enable verbose output"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--supported-boards",
|
||||
action="store_true",
|
||||
help="Print the list of supported boards and exit",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--symbols",
|
||||
action="store_true",
|
||||
help="Run symbol analysis on compiled output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allsrc",
|
||||
action="store_true",
|
||||
help="Enable all-source build (adds FASTLED_ALL_SRC=1 define)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-allsrc",
|
||||
action="store_true",
|
||||
help="Disable all-source build (adds FASTLED_ALL_SRC=0 define)",
|
||||
)
|
||||
try:
|
||||
args = parser.parse_intermixed_args()
|
||||
unknown = []
|
||||
except SystemExit:
|
||||
# If parse_intermixed_args fails, fall back to parse_known_args
|
||||
args, unknown = parser.parse_known_args()
|
||||
|
||||
# Handle unknown arguments intelligently - treat non-flag arguments as examples
|
||||
unknown_examples = []
|
||||
unknown_flags = []
|
||||
for arg in unknown:
|
||||
if arg.startswith("-"):
|
||||
unknown_flags.append(arg)
|
||||
else:
|
||||
unknown_examples.append(arg)
|
||||
|
||||
# Add unknown examples to positional_examples
|
||||
if unknown_examples:
|
||||
if not hasattr(args, "positional_examples") or args.positional_examples is None:
|
||||
args.positional_examples = []
|
||||
args.positional_examples.extend(unknown_examples)
|
||||
|
||||
# Only warn about actual unknown flags, not examples
|
||||
if unknown_flags:
|
||||
warnings.warn(f"Unknown arguments: {unknown_flags}")
|
||||
|
||||
# Check for FASTLED_CI_NO_INTERACTIVE environment variable
|
||||
# This allows test.py and other scripts to force non-interactive mode
|
||||
if os.environ.get("FASTLED_CI_NO_INTERACTIVE") == "true":
|
||||
args.interactive = False
|
||||
args.no_interactive = True
|
||||
|
||||
# if --interactive and --no-interative are both passed, --no-interactive takes precedence.
|
||||
if args.interactive and args.no_interactive:
|
||||
warnings.warn(
|
||||
"Both --interactive and --no-interactive were passed, --no-interactive takes precedence."
|
||||
)
|
||||
args.interactive = False
|
||||
|
||||
# Validate that --allsrc and --no-allsrc are not both specified
|
||||
if args.allsrc and args.no_allsrc:
|
||||
warnings.warn(
|
||||
"Both --allsrc and --no-allsrc were passed, this is contradictory. Please specify only one."
|
||||
)
|
||||
sys.exit(1) # Exit with error
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def remove_duplicates(items: List[str]) -> List[str]:
|
||||
seen: Set[str] = set()
|
||||
out: List[str] = []
|
||||
for item in items:
|
||||
if item not in seen:
|
||||
seen.add(item)
|
||||
out.append(item)
|
||||
return out
|
||||
|
||||
|
||||
def choose_board_interactively(boards: List[str]) -> List[str]:
|
||||
print("Available boards:")
|
||||
boards = remove_duplicates(sorted(boards))
|
||||
for i, board in enumerate(boards):
|
||||
print(f"[{i}]: {board}")
|
||||
print("[all]: All boards")
|
||||
out: List[str] = []
|
||||
while True:
|
||||
try:
|
||||
# choice = int(input("Enter the number of the board(s) you want to compile to: "))
|
||||
input_str = input(
|
||||
"Enter the number of the board(s) you want to compile to, or it's name(s): "
|
||||
)
|
||||
if "all" in input_str:
|
||||
return boards
|
||||
for board in input_str.split(","):
|
||||
if board == "":
|
||||
continue
|
||||
if not board.isdigit():
|
||||
out.append(board) # Assume it's a board name.
|
||||
else:
|
||||
index = int(board) # Find the board from the index.
|
||||
if 0 <= index < len(boards):
|
||||
out.append(boards[index])
|
||||
else:
|
||||
warnings.warn(f"invalid board index: {index}, skipping")
|
||||
if not out:
|
||||
print("Please try again.")
|
||||
continue
|
||||
return out
|
||||
except ValueError:
|
||||
print("Invalid input. Please enter a number.")
|
||||
|
||||
|
||||
def resolve_example_path(example: str) -> Path:
|
||||
example_path = HERE.parent / "examples" / example
|
||||
if not example_path.exists():
|
||||
raise FileNotFoundError(f"Example '{example}' not found at '{example_path}'")
|
||||
return example_path
|
||||
|
||||
|
||||
def create_concurrent_run_args(args: argparse.Namespace) -> ConcurrentRunArgs:
|
||||
skip_init = args.skip_init
|
||||
if args.interactive:
|
||||
boards = choose_board_interactively(DEFAULT_BOARDS_NAMES + OTHER_BOARDS_NAMES)
|
||||
else:
|
||||
boards = args.boards.split(",") if args.boards else DEFAULT_BOARDS_NAMES
|
||||
projects: list[Board] = []
|
||||
for board in boards:
|
||||
projects.append(create_board(board, no_project_options=args.no_project_options))
|
||||
extra_examples: dict[Board, list[Path]] = {}
|
||||
# Handle both positional and named examples
|
||||
if args.positional_examples:
|
||||
# Convert positional examples, handling both "examples/Blink" and "Blink" formats
|
||||
examples = []
|
||||
for example in args.positional_examples:
|
||||
# Remove "examples/" prefix if present
|
||||
if example.startswith("examples/"):
|
||||
example = example[len("examples/") :]
|
||||
examples.append(example)
|
||||
elif args.examples:
|
||||
examples = args.examples.split(",")
|
||||
else:
|
||||
examples = DEFAULT_EXAMPLES
|
||||
# Only add extra examples when using defaults
|
||||
for b, _examples in EXTRA_EXAMPLES.items():
|
||||
resolved_examples = [resolve_example_path(example) for example in _examples]
|
||||
extra_examples[b] = resolved_examples
|
||||
examples_paths = [resolve_example_path(example) for example in examples]
|
||||
# now process example exclusions.
|
||||
if args.exclude_examples:
|
||||
exclude_examples = args.exclude_examples.split(",")
|
||||
examples_paths = [
|
||||
example
|
||||
for example in examples_paths
|
||||
if example.name not in exclude_examples
|
||||
]
|
||||
for exclude in exclude_examples:
|
||||
examples.remove(exclude)
|
||||
customsdk = args.customsdk
|
||||
defines: list[str] = []
|
||||
if args.defines:
|
||||
defines.extend(args.defines.split(","))
|
||||
# Add FASTLED_ALL_SRC define when --allsrc or --no-allsrc flag is specified
|
||||
if args.allsrc:
|
||||
defines.append("FASTLED_ALL_SRC=1")
|
||||
elif args.no_allsrc:
|
||||
defines.append("FASTLED_ALL_SRC=0")
|
||||
extra_packages: list[str] = []
|
||||
if args.extra_packages:
|
||||
extra_packages.extend(args.extra_packages.split(","))
|
||||
build_dir = args.build_dir
|
||||
extra_scripts = "pre:lib/ci/util-flags.py"
|
||||
verbose = args.verbose
|
||||
|
||||
out: ConcurrentRunArgs = ConcurrentRunArgs(
|
||||
projects=projects,
|
||||
examples=examples_paths,
|
||||
skip_init=skip_init,
|
||||
defines=defines,
|
||||
customsdk=customsdk,
|
||||
extra_packages=extra_packages,
|
||||
libs=LIBS,
|
||||
build_dir=build_dir,
|
||||
extra_scripts=extra_scripts,
|
||||
cwd=str(HERE.parent),
|
||||
board_dir=(HERE / "boards").absolute().as_posix(),
|
||||
build_flags=BUILD_FLAGS,
|
||||
verbose=verbose,
|
||||
extra_examples=extra_examples,
|
||||
symbols=args.symbols,
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main function."""
|
||||
args = parse_args()
|
||||
if args.supported_boards:
|
||||
print(",".join(DEFAULT_BOARDS_NAMES))
|
||||
return 0
|
||||
if args.add_extra_esp32_libs:
|
||||
LIBS.extend(EXTRA_LIBS)
|
||||
|
||||
# Set the working directory to the script's parent directory.
|
||||
run_args = create_concurrent_run_args(args)
|
||||
start_time = time.time()
|
||||
rtn = concurrent_run(args=run_args)
|
||||
time_taken = time.strftime("%Mm:%Ss", time.gmtime(time.time() - start_time))
|
||||
locked_print(f"Compilation finished in {time_taken}.")
|
||||
return rtn
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,21 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def main():
|
||||
# Change to the directory of the script
|
||||
os.chdir(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# Change to the 'native' directory and run 'pio run'
|
||||
os.chdir("native")
|
||||
result = subprocess.run(["pio", "run"], check=True)
|
||||
|
||||
# Exit with the same status as the pio run command
|
||||
sys.exit(result.returncode)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,596 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
Runs the compilation process for examples on boards using pio ci command.
|
||||
This replaces the previous concurrent build system with a simpler pio ci approach.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
from typing import List, Set
|
||||
|
||||
from ci.boards import Board, create_board # type: ignore
|
||||
from ci.util.locked_print import locked_print
|
||||
|
||||
|
||||
HERE = Path(__file__).parent.resolve()
|
||||
|
||||
# Default boards to compile for
|
||||
DEFAULT_BOARDS_NAMES = [
|
||||
"apollo3_red",
|
||||
"apollo3_thing_explorable",
|
||||
"web", # work in progress
|
||||
"uno", # Build is faster if this is first, because it's used for global init.
|
||||
"esp32dev",
|
||||
"esp01", # ESP8266
|
||||
"esp32c3",
|
||||
"attiny85",
|
||||
"ATtiny1616",
|
||||
"esp32c6",
|
||||
"esp32s3",
|
||||
"esp32p4",
|
||||
"yun",
|
||||
"digix",
|
||||
"teensylc",
|
||||
"teensy30",
|
||||
"teensy31",
|
||||
"teensy41",
|
||||
"adafruit_feather_nrf52840_sense",
|
||||
"xiaoblesense_adafruit",
|
||||
"rpipico",
|
||||
"rpipico2",
|
||||
"uno_r4_wifi",
|
||||
"esp32rmt_51",
|
||||
"esp32dev_idf44",
|
||||
"bluepill",
|
||||
"esp32rmt_51",
|
||||
"giga_r1",
|
||||
"sparkfun_xrp_controller",
|
||||
]
|
||||
|
||||
OTHER_BOARDS_NAMES = [
|
||||
"nano_every",
|
||||
"esp32-c2-devkitm-1",
|
||||
]
|
||||
|
||||
# Examples to compile.
|
||||
DEFAULT_EXAMPLES = [
|
||||
"Animartrix",
|
||||
"Apa102",
|
||||
"Apa102HD",
|
||||
"Apa102HDOverride",
|
||||
"Audio",
|
||||
"Blink",
|
||||
"Blur",
|
||||
"Chromancer",
|
||||
"ColorPalette",
|
||||
"ColorTemperature",
|
||||
"Corkscrew",
|
||||
"Cylon",
|
||||
"DemoReel100",
|
||||
# "Downscale",
|
||||
"FestivalStick",
|
||||
"FirstLight",
|
||||
"Fire2012",
|
||||
"Multiple/MultipleStripsInOneArray",
|
||||
"Multiple/ArrayOfLedArrays",
|
||||
"Noise",
|
||||
"NoisePlayground",
|
||||
"NoisePlusPalette",
|
||||
# "LuminescentGrand",
|
||||
"Pacifica",
|
||||
"Pride2015",
|
||||
"RGBCalibrate",
|
||||
"RGBSetDemo",
|
||||
"RGBW",
|
||||
"Overclock",
|
||||
"RGBWEmulated",
|
||||
"TwinkleFox",
|
||||
"XYMatrix",
|
||||
"FireMatrix",
|
||||
"FireCylinder",
|
||||
"FxGfx2Video",
|
||||
"FxSdCard",
|
||||
"FxCylon",
|
||||
"FxDemoReel100",
|
||||
"FxTwinkleFox",
|
||||
"FxFire2012",
|
||||
"FxNoisePlusPalette",
|
||||
"FxPacifica",
|
||||
"FxEngine",
|
||||
"WS2816",
|
||||
]
|
||||
|
||||
EXTRA_EXAMPLES: dict[Board, list[str]] = {
|
||||
# ESP32DEV: ["EspI2SDemo"],
|
||||
# ESP32_S3_DEVKITC_1: ["EspS3I2SDemo"],
|
||||
}
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compile FastLED examples for various boards using pio ci.util."
|
||||
)
|
||||
parser.add_argument(
|
||||
"boards",
|
||||
type=str,
|
||||
help="Comma-separated list of boards to compile for",
|
||||
nargs="?",
|
||||
)
|
||||
parser.add_argument(
|
||||
"positional_examples",
|
||||
type=str,
|
||||
help="Examples to compile (positional arguments after board name)",
|
||||
nargs="*",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--examples", type=str, help="Comma-separated list of examples to compile"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--exclude-examples", type=str, help="Examples that should be excluded"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--defines", type=str, help="Comma-separated list of compiler definitions"
|
||||
)
|
||||
parser.add_argument("--customsdk", type=str, help="custom_sdkconfig project option")
|
||||
parser.add_argument(
|
||||
"--extra-packages",
|
||||
type=str,
|
||||
help="Comma-separated list of extra packages to install",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--build-dir", type=str, help="Override the default build directory"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--interactive",
|
||||
action="store_true",
|
||||
help="Enable interactive mode to choose a board",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-interactive", action="store_true", help="Disable interactive mode"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v", "--verbose", action="store_true", help="Enable verbose output"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--supported-boards",
|
||||
action="store_true",
|
||||
help="Print the list of supported boards and exit",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--symbols",
|
||||
action="store_true",
|
||||
help="Run symbol analysis on compiled output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allsrc",
|
||||
action="store_true",
|
||||
help="Enable all-source build (adds FASTLED_ALL_SRC=1 define)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-allsrc",
|
||||
action="store_true",
|
||||
help="Disable all-source build (adds FASTLED_ALL_SRC=0 define)",
|
||||
)
|
||||
try:
|
||||
args = parser.parse_intermixed_args()
|
||||
unknown = []
|
||||
except SystemExit:
|
||||
# If parse_intermixed_args fails, fall back to parse_known_args
|
||||
args, unknown = parser.parse_known_args()
|
||||
|
||||
# Handle unknown arguments intelligently - treat non-flag arguments as examples
|
||||
unknown_examples = []
|
||||
unknown_flags = []
|
||||
for arg in unknown:
|
||||
if arg.startswith("-"):
|
||||
unknown_flags.append(arg)
|
||||
else:
|
||||
unknown_examples.append(arg)
|
||||
|
||||
# Add unknown examples to positional_examples
|
||||
if unknown_examples:
|
||||
if not hasattr(args, "positional_examples") or args.positional_examples is None:
|
||||
args.positional_examples = []
|
||||
args.positional_examples.extend(unknown_examples)
|
||||
|
||||
# Only warn about actual unknown flags, not examples
|
||||
if unknown_flags:
|
||||
warnings.warn(f"Unknown arguments: {unknown_flags}")
|
||||
|
||||
# Check for FASTLED_CI_NO_INTERACTIVE environment variable
|
||||
if os.environ.get("FASTLED_CI_NO_INTERACTIVE") == "true":
|
||||
args.interactive = False
|
||||
args.no_interactive = True
|
||||
|
||||
# if --interactive and --no-interative are both passed, --no-interactive takes precedence.
|
||||
if args.interactive and args.no_interactive:
|
||||
warnings.warn(
|
||||
"Both --interactive and --no-interactive were passed, --no-interactive takes precedence."
|
||||
)
|
||||
args.interactive = False
|
||||
|
||||
# Validate that --allsrc and --no-allsrc are not both specified
|
||||
if args.allsrc and args.no_allsrc:
|
||||
warnings.warn(
|
||||
"Both --allsrc and --no-allsrc were passed, this is contradictory. Please specify only one."
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def remove_duplicates(items: List[str]) -> List[str]:
|
||||
seen: Set[str] = set()
|
||||
out: List[str] = []
|
||||
for item in items:
|
||||
if item not in seen:
|
||||
seen.add(item)
|
||||
out.append(item)
|
||||
return out
|
||||
|
||||
|
||||
def choose_board_interactively(boards: List[str]) -> List[str]:
|
||||
print("Available boards:")
|
||||
boards = remove_duplicates(sorted(boards))
|
||||
for i, board in enumerate(boards):
|
||||
print(f"[{i}]: {board}")
|
||||
print("[all]: All boards")
|
||||
out: list[str] = []
|
||||
while True:
|
||||
try:
|
||||
input_str = input(
|
||||
"Enter the number of the board(s) you want to compile to, or it's name(s): "
|
||||
)
|
||||
if "all" in input_str:
|
||||
return boards
|
||||
for board in input_str.split(","):
|
||||
if board == "":
|
||||
continue
|
||||
if not board.isdigit():
|
||||
out.append(board) # Assume it's a board name.
|
||||
else:
|
||||
index = int(board) # Find the board from the index.
|
||||
if 0 <= index < len(boards):
|
||||
out.append(boards[index])
|
||||
else:
|
||||
warnings.warn(f"invalid board index: {index}, skipping")
|
||||
if not out:
|
||||
print("Please try again.")
|
||||
continue
|
||||
return out
|
||||
except ValueError:
|
||||
print("Invalid input. Please enter a number.")
|
||||
|
||||
|
||||
def resolve_example_path(example: str) -> Path:
|
||||
example_path = HERE.parent / "examples" / example
|
||||
if not example_path.exists():
|
||||
raise FileNotFoundError(f"Example '{example}' not found at '{example_path}'")
|
||||
return example_path
|
||||
|
||||
|
||||
def compile_with_pio_ci(
|
||||
board: Board,
|
||||
example_paths: list[Path],
|
||||
build_dir: str | None,
|
||||
defines: list[str],
|
||||
verbose: bool,
|
||||
) -> tuple[bool, str]:
|
||||
"""Compile examples for a board using pio ci command."""
|
||||
|
||||
# Skip web boards
|
||||
if board.board_name == "web":
|
||||
locked_print(f"Skipping web target for board {board.board_name}")
|
||||
return True, ""
|
||||
|
||||
board_name = board.board_name
|
||||
real_board_name = board.get_real_board_name()
|
||||
|
||||
# Set up build directory
|
||||
if build_dir:
|
||||
board_build_dir = Path(build_dir) / board_name
|
||||
else:
|
||||
board_build_dir = Path(".build") / board_name
|
||||
|
||||
board_build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
locked_print(f"*** Compiling examples for board {board_name} using pio ci ***")
|
||||
|
||||
errors: list[str] = []
|
||||
|
||||
for example_path in example_paths:
|
||||
locked_print(
|
||||
f"*** Building example {example_path.name} for board {board_name} ***"
|
||||
)
|
||||
|
||||
# Find the .ino file in the example directory
|
||||
ino_files = list(example_path.glob("*.ino"))
|
||||
if not ino_files:
|
||||
error_msg = f"No .ino file found in {example_path}"
|
||||
locked_print(f"ERROR: {error_msg}")
|
||||
errors.append(error_msg)
|
||||
continue
|
||||
|
||||
# Use the first .ino file found
|
||||
ino_file = ino_files[0]
|
||||
|
||||
# Build pio ci command
|
||||
cmd_list = [
|
||||
"pio",
|
||||
"ci",
|
||||
str(ino_file),
|
||||
"--board",
|
||||
real_board_name,
|
||||
"--lib",
|
||||
"src", # FastLED source directory
|
||||
"--keep-build-dir",
|
||||
"--build-dir",
|
||||
str(board_build_dir / example_path.name),
|
||||
]
|
||||
|
||||
# Add platform-specific options
|
||||
if board.platform:
|
||||
cmd_list.extend(["--project-option", f"platform={board.platform}"])
|
||||
|
||||
if board.platform_packages:
|
||||
cmd_list.extend(
|
||||
["--project-option", f"platform_packages={board.platform_packages}"]
|
||||
)
|
||||
|
||||
if board.framework:
|
||||
cmd_list.extend(["--project-option", f"framework={board.framework}"])
|
||||
|
||||
if board.board_build_core:
|
||||
cmd_list.extend(
|
||||
["--project-option", f"board_build.core={board.board_build_core}"]
|
||||
)
|
||||
|
||||
if board.board_build_filesystem_size:
|
||||
cmd_list.extend(
|
||||
[
|
||||
"--project-option",
|
||||
f"board_build.filesystem_size={board.board_build_filesystem_size}",
|
||||
]
|
||||
)
|
||||
|
||||
# Add defines
|
||||
all_defines = defines.copy()
|
||||
if board.defines:
|
||||
all_defines.extend(board.defines)
|
||||
|
||||
if all_defines:
|
||||
build_flags = " ".join(f"-D{define}" for define in all_defines)
|
||||
cmd_list.extend(["--project-option", f"build_flags={build_flags}"])
|
||||
|
||||
# Add custom SDK config if specified
|
||||
if board.customsdk:
|
||||
cmd_list.extend(["--project-option", f"custom_sdkconfig={board.customsdk}"])
|
||||
|
||||
# Add verbose flag if requested
|
||||
if verbose:
|
||||
cmd_list.append("--verbose")
|
||||
|
||||
# Execute the command
|
||||
cmd_str = subprocess.list2cmdline(cmd_list)
|
||||
locked_print(f"Running command: {cmd_str}")
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd_list,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(HERE.parent),
|
||||
timeout=300, # 5 minute timeout per example
|
||||
)
|
||||
|
||||
elapsed_time = time.time() - start_time
|
||||
|
||||
if result.returncode == 0:
|
||||
locked_print(
|
||||
f"*** Successfully built {example_path.name} for {board_name} in {elapsed_time:.2f}s ***"
|
||||
)
|
||||
if verbose and result.stdout:
|
||||
locked_print(f"Build output:\n{result.stdout}")
|
||||
else:
|
||||
error_msg = f"Failed to build {example_path.name} for {board_name}"
|
||||
locked_print(f"ERROR: {error_msg}")
|
||||
locked_print(f"Command: {cmd_str}")
|
||||
if result.stdout:
|
||||
locked_print(f"STDOUT:\n{result.stdout}")
|
||||
if result.stderr:
|
||||
locked_print(f"STDERR:\n{result.stderr}")
|
||||
errors.append(f"{error_msg}: {result.stderr}")
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
error_msg = f"Timeout building {example_path.name} for {board_name}"
|
||||
locked_print(f"ERROR: {error_msg}")
|
||||
errors.append(error_msg)
|
||||
except Exception as e:
|
||||
error_msg = f"Exception building {example_path.name} for {board_name}: {e}"
|
||||
locked_print(f"ERROR: {error_msg}")
|
||||
errors.append(error_msg)
|
||||
|
||||
if errors:
|
||||
return False, "\n".join(errors)
|
||||
|
||||
return True, f"Successfully compiled all examples for {board_name}"
|
||||
|
||||
|
||||
def run_symbol_analysis(boards: list[Board]) -> None:
|
||||
"""Run symbol analysis on compiled outputs if requested."""
|
||||
locked_print("\nRunning symbol analysis on compiled outputs...")
|
||||
|
||||
for board in boards:
|
||||
if board.board_name == "web":
|
||||
continue
|
||||
|
||||
try:
|
||||
locked_print(f"Running symbol analysis for board: {board.board_name}")
|
||||
|
||||
cmd = [
|
||||
"uv",
|
||||
"run",
|
||||
"ci/util/symbol_analysis.py",
|
||||
"--board",
|
||||
board.board_name,
|
||||
]
|
||||
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, cwd=str(HERE.parent)
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
locked_print(
|
||||
f"ERROR: Symbol analysis failed for board {board.board_name}: {result.stderr}"
|
||||
)
|
||||
else:
|
||||
locked_print(f"Symbol analysis completed for board: {board.board_name}")
|
||||
if result.stdout:
|
||||
print(result.stdout)
|
||||
|
||||
except Exception as e:
|
||||
locked_print(
|
||||
f"ERROR: Exception during symbol analysis for board {board.board_name}: {e}"
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main function."""
|
||||
args = parse_args()
|
||||
|
||||
if args.supported_boards:
|
||||
print(",".join(DEFAULT_BOARDS_NAMES))
|
||||
return 0
|
||||
|
||||
# Determine which boards to compile for
|
||||
if args.interactive:
|
||||
boards_names = choose_board_interactively(
|
||||
DEFAULT_BOARDS_NAMES + OTHER_BOARDS_NAMES
|
||||
)
|
||||
else:
|
||||
boards_names = args.boards.split(",") if args.boards else DEFAULT_BOARDS_NAMES
|
||||
|
||||
# Get board objects
|
||||
boards: list[Board] = []
|
||||
for board_name in boards_names:
|
||||
try:
|
||||
board = create_board(board_name, no_project_options=False)
|
||||
boards.append(board)
|
||||
except Exception as e:
|
||||
locked_print(f"ERROR: Failed to get board '{board_name}': {e}")
|
||||
return 1
|
||||
|
||||
# Determine which examples to compile
|
||||
if args.positional_examples:
|
||||
# Convert positional examples, handling both "examples/Blink" and "Blink" formats
|
||||
examples = []
|
||||
for example in args.positional_examples:
|
||||
# Remove "examples/" prefix if present
|
||||
if example.startswith("examples/"):
|
||||
example = example[len("examples/") :]
|
||||
examples.append(example)
|
||||
elif args.examples:
|
||||
examples = args.examples.split(",")
|
||||
else:
|
||||
examples = DEFAULT_EXAMPLES
|
||||
|
||||
# Process example exclusions
|
||||
if args.exclude_examples:
|
||||
exclude_examples = args.exclude_examples.split(",")
|
||||
examples = [ex for ex in examples if ex not in exclude_examples]
|
||||
|
||||
# Resolve example paths
|
||||
example_paths: list[Path] = []
|
||||
for example in examples:
|
||||
try:
|
||||
example_path = resolve_example_path(example)
|
||||
example_paths.append(example_path)
|
||||
except FileNotFoundError as e:
|
||||
locked_print(f"ERROR: {e}")
|
||||
return 1
|
||||
|
||||
# Add extra examples for specific boards
|
||||
extra_examples: dict[Board, list[Path]] = {}
|
||||
for board in boards:
|
||||
if board in EXTRA_EXAMPLES:
|
||||
board_examples = []
|
||||
for example in EXTRA_EXAMPLES[board]:
|
||||
try:
|
||||
board_examples.append(resolve_example_path(example))
|
||||
except FileNotFoundError as e:
|
||||
locked_print(f"WARNING: {e}")
|
||||
if board_examples:
|
||||
extra_examples[board] = board_examples
|
||||
|
||||
# Set up defines
|
||||
defines: list[str] = []
|
||||
if args.defines:
|
||||
defines.extend(args.defines.split(","))
|
||||
|
||||
# Add FASTLED_ALL_SRC define when --allsrc or --no-allsrc flag is specified
|
||||
if args.allsrc:
|
||||
defines.append("FASTLED_ALL_SRC=1")
|
||||
elif args.no_allsrc:
|
||||
defines.append("FASTLED_ALL_SRC=0")
|
||||
|
||||
# Start compilation
|
||||
start_time = time.time()
|
||||
locked_print(
|
||||
f"Starting compilation for {len(boards)} boards with {len(example_paths)} examples"
|
||||
)
|
||||
|
||||
compilation_errors: List[str] = []
|
||||
|
||||
# Compile for each board
|
||||
for board in boards:
|
||||
board_examples = example_paths.copy()
|
||||
if board in extra_examples:
|
||||
board_examples.extend(extra_examples[board])
|
||||
|
||||
success, message = compile_with_pio_ci(
|
||||
board=board,
|
||||
example_paths=board_examples,
|
||||
build_dir=args.build_dir,
|
||||
defines=defines,
|
||||
verbose=args.verbose,
|
||||
)
|
||||
|
||||
if not success:
|
||||
compilation_errors.append(f"Board {board.board_name}: {message}")
|
||||
locked_print(f"ERROR: Compilation failed for board {board.board_name}")
|
||||
# Continue with other boards instead of stopping
|
||||
|
||||
# Run symbol analysis if requested
|
||||
if args.symbols:
|
||||
run_symbol_analysis(boards)
|
||||
|
||||
# Report results
|
||||
elapsed_time = time.time() - start_time
|
||||
time_str = time.strftime("%Mm:%Ss", time.gmtime(elapsed_time))
|
||||
|
||||
if compilation_errors:
|
||||
locked_print(
|
||||
f"\nCompilation finished in {time_str} with {len(compilation_errors)} error(s):"
|
||||
)
|
||||
for error in compilation_errors:
|
||||
locked_print(f" - {error}")
|
||||
return 1
|
||||
else:
|
||||
locked_print(f"\nAll compilations completed successfully in {time_str}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
locked_print("\nInterrupted by user")
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,657 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Example Compiler
|
||||
|
||||
Streamlined compiler that uses the PioCompiler to build FastLED examples for various boards.
|
||||
This replaces the previous complex compilation system with a simpler approach using the Pio compiler.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import Future, as_completed
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import List, Optional, cast
|
||||
|
||||
from typeguard import typechecked
|
||||
|
||||
from ci.boards import ALL, Board, create_board
|
||||
from ci.compiler.compiler import CacheType, SketchResult
|
||||
from ci.compiler.pio import PioCompiler
|
||||
|
||||
|
||||
def green_text(text: str) -> str:
|
||||
"""Return text in green color."""
|
||||
return f"\033[32m{text}\033[0m"
|
||||
|
||||
|
||||
def red_text(text: str) -> str:
|
||||
"""Return text in red color."""
|
||||
return f"\033[31m{text}\033[0m"
|
||||
|
||||
|
||||
def get_default_boards() -> List[str]:
|
||||
"""Get all board names from the ALL boards list, with preferred boards first."""
|
||||
# These are the boards we want to compile first (preferred order)
|
||||
# Order matters: UNO first because it's used for global init and builds faster
|
||||
preferred_board_names = [
|
||||
"uno", # Build is faster if this is first, because it's used for global init.
|
||||
"esp32dev",
|
||||
"esp01", # ESP8266
|
||||
"esp32c3",
|
||||
"esp32c6",
|
||||
"esp32s3",
|
||||
"teensylc",
|
||||
"teensy31",
|
||||
"teensy41",
|
||||
"digix",
|
||||
"rpipico",
|
||||
"rpipico2",
|
||||
]
|
||||
|
||||
# Get all available board names from the ALL list
|
||||
available_board_names = {board.board_name for board in ALL}
|
||||
|
||||
# Start with preferred boards that exist, warn about missing ones
|
||||
default_boards: List[str] = []
|
||||
for board_name in preferred_board_names:
|
||||
if board_name in available_board_names:
|
||||
default_boards.append(board_name)
|
||||
else:
|
||||
print(
|
||||
f"WARNING: Preferred board '{board_name}' not found in available boards"
|
||||
)
|
||||
|
||||
# Add all remaining boards (sorted for consistency)
|
||||
remaining_boards = sorted(available_board_names - set(default_boards))
|
||||
default_boards.extend(remaining_boards)
|
||||
|
||||
return default_boards
|
||||
|
||||
|
||||
def get_all_examples() -> List[str]:
|
||||
"""Get all available example names from the examples directory."""
|
||||
project_root = Path(__file__).parent.parent.resolve()
|
||||
examples_dir = project_root / "examples"
|
||||
|
||||
if not examples_dir.exists():
|
||||
return []
|
||||
|
||||
examples: List[str] = []
|
||||
for item in examples_dir.iterdir():
|
||||
if item.is_dir():
|
||||
# Check if it contains a .ino file with the same name
|
||||
ino_file = item / f"{item.name}.ino"
|
||||
if ino_file.exists():
|
||||
examples.append(item.name)
|
||||
|
||||
# Sort for consistent ordering
|
||||
examples.sort()
|
||||
return examples
|
||||
|
||||
|
||||
def parse_args(args: Optional[list[str]] = None) -> argparse.Namespace:
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compile FastLED examples for various boards using PioCompiler"
|
||||
)
|
||||
parser.add_argument(
|
||||
"boards",
|
||||
type=str,
|
||||
help="Comma-separated list of boards to compile for",
|
||||
nargs="?",
|
||||
)
|
||||
parser.add_argument(
|
||||
"positional_examples",
|
||||
type=str,
|
||||
help="Examples to compile (positional arguments after board name)",
|
||||
nargs="*",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--examples", type=str, help="Comma-separated list of examples to compile"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--exclude-examples", type=str, help="Examples that should be excluded"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--defines", type=str, help="Comma-separated list of compiler definitions"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v", "--verbose", action="store_true", help="Enable verbose output"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-cache",
|
||||
action="store_true",
|
||||
help="Disable sccache for faster compilation (default is already disabled)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable-cache",
|
||||
action="store_true",
|
||||
help="Enable sccache for faster compilation",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cache",
|
||||
action="store_true",
|
||||
help="(Deprecated) Enable sccache for faster compilation - use --enable-cache instead",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--supported-boards",
|
||||
action="store_true",
|
||||
help="Print the list of supported boards and exit",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-interactive",
|
||||
action="store_true",
|
||||
help="Disables interactive mode (deprecated)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--log-failures",
|
||||
type=str,
|
||||
help="Directory to write per-example failure logs (created on first failure)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--global-cache",
|
||||
type=str,
|
||||
help="Override global PlatformIO cache directory path (for testing)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-o",
|
||||
"--out",
|
||||
type=str,
|
||||
help="Output path for build artifact. Requires exactly one sketch. "
|
||||
"If path ends with '/', treated as directory. If has suffix, treated as file. "
|
||||
"Use '-o .' to save in current directory with sketch name.",
|
||||
)
|
||||
|
||||
try:
|
||||
parsed_args = parser.parse_intermixed_args(args)
|
||||
unknown = []
|
||||
except SystemExit:
|
||||
# If parse_intermixed_args fails, fall back to parse_known_args
|
||||
parsed_args, unknown = parser.parse_known_args(args)
|
||||
|
||||
# Handle unknown arguments intelligently - treat non-flag arguments as examples
|
||||
unknown_examples: List[str] = []
|
||||
for arg in unknown:
|
||||
if not arg.startswith("-"):
|
||||
unknown_examples.append(arg)
|
||||
|
||||
# Add unknown examples to positional_examples
|
||||
if unknown_examples:
|
||||
if (
|
||||
not hasattr(parsed_args, "positional_examples")
|
||||
or parsed_args.positional_examples is None
|
||||
):
|
||||
parsed_args.positional_examples = []
|
||||
# Type assertion to help the type checker - argparse returns Any but we know it's List[str]
|
||||
positional_examples: List[str] = cast(
|
||||
List[str], getattr(parsed_args, "positional_examples", [])
|
||||
)
|
||||
positional_examples.extend(unknown_examples)
|
||||
parsed_args.positional_examples = positional_examples
|
||||
|
||||
return parsed_args
|
||||
|
||||
|
||||
def resolve_example_path(example: str) -> str:
|
||||
"""Resolve example name to path, ensuring it exists."""
|
||||
project_root = Path(__file__).parent.parent.resolve()
|
||||
examples_dir = project_root / "examples"
|
||||
|
||||
# Handle both "Blink" and "examples/Blink" formats
|
||||
if example.startswith("examples/"):
|
||||
example = example[len("examples/") :]
|
||||
|
||||
example_path = examples_dir / example
|
||||
if not example_path.exists():
|
||||
raise FileNotFoundError(f"Example not found: {example_path}")
|
||||
|
||||
return example
|
||||
|
||||
|
||||
@typechecked
|
||||
@dataclass
|
||||
class BoardCompilationResult:
|
||||
"""Aggregated result for compiling a set of examples on a single board."""
|
||||
|
||||
ok: bool
|
||||
sketch_results: List[SketchResult]
|
||||
|
||||
|
||||
def compile_board_examples(
|
||||
board: Board,
|
||||
examples: List[str],
|
||||
defines: List[str],
|
||||
verbose: bool,
|
||||
enable_cache: bool,
|
||||
global_cache_dir: Optional[Path] = None,
|
||||
) -> BoardCompilationResult:
|
||||
"""Compile examples for a single board using PioCompiler."""
|
||||
|
||||
# Resolve global cache directory immediately for display
|
||||
resolved_cache_dir = None
|
||||
if global_cache_dir is not None:
|
||||
# User specified a path - use it exactly as provided
|
||||
resolved_cache_dir = global_cache_dir.resolve()
|
||||
else:
|
||||
# Default path ends with 'global_cache'
|
||||
resolved_cache_dir = Path.home() / ".fastled" / "global_cache"
|
||||
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"COMPILING BOARD: {board.board_name}")
|
||||
print(f"EXAMPLES: {', '.join(examples)}")
|
||||
print(f"GLOBAL CACHE: {resolved_cache_dir}")
|
||||
|
||||
# Show other cache directories
|
||||
from ci.compiler.pio import FastLEDPaths
|
||||
|
||||
paths = FastLEDPaths(board.board_name)
|
||||
|
||||
print(f"BUILD CACHE: {paths.build_cache_dir}")
|
||||
print(f"CORE DIR: {paths.core_dir}")
|
||||
print(f"PACKAGES DIR: {paths.packages_dir}")
|
||||
|
||||
print(f"{'=' * 60}")
|
||||
|
||||
try:
|
||||
# Determine cache type based on flag and board frameworks
|
||||
frameworks = [f.strip().lower() for f in (board.framework or "").split(",")]
|
||||
mixed_frameworks = "arduino" in frameworks and "espidf" in frameworks
|
||||
cache_type = (
|
||||
CacheType.SCCACHE
|
||||
if enable_cache and not mixed_frameworks
|
||||
else CacheType.NO_CACHE
|
||||
)
|
||||
|
||||
# Create PioCompiler instance
|
||||
compiler = PioCompiler(
|
||||
board=board,
|
||||
verbose=verbose,
|
||||
global_cache_dir=resolved_cache_dir,
|
||||
additional_defines=defines,
|
||||
cache_type=cache_type,
|
||||
)
|
||||
|
||||
# Build all examples
|
||||
futures: List[Future[SketchResult]] = compiler.build(examples)
|
||||
|
||||
# Wait for completion and collect results
|
||||
results: List[SketchResult] = []
|
||||
|
||||
for future in futures:
|
||||
try:
|
||||
result = future.result()
|
||||
results.append(result)
|
||||
# SUCCESS/FAILED messages are printed by worker threads
|
||||
except KeyboardInterrupt:
|
||||
print("Keyboard interrupt detected, cancelling builds")
|
||||
compiler.cancel_all()
|
||||
import _thread
|
||||
|
||||
for future in futures:
|
||||
future.cancel()
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
# Represent unexpected exception as a failed SketchResult for consistency
|
||||
from pathlib import Path as _Path
|
||||
|
||||
results.append(
|
||||
SketchResult(
|
||||
success=False,
|
||||
output=f"Build exception: {str(e)}",
|
||||
build_dir=_Path("."),
|
||||
example="<exception>",
|
||||
)
|
||||
)
|
||||
print(f"EXCEPTION during build: {e}")
|
||||
# Cleanup
|
||||
compiler.cancel_all()
|
||||
|
||||
# Show compiler statistics after all builds complete
|
||||
try:
|
||||
stats = compiler.get_cache_stats()
|
||||
if stats:
|
||||
print("\n" + "=" * 60)
|
||||
print("COMPILER STATISTICS")
|
||||
print("=" * 60)
|
||||
print(stats)
|
||||
print("=" * 60)
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not retrieve compiler statistics: {e}")
|
||||
|
||||
any_failures = False
|
||||
for r in results:
|
||||
if not r.success:
|
||||
any_failures = True
|
||||
break
|
||||
return BoardCompilationResult(ok=not any_failures, sketch_results=results)
|
||||
except KeyboardInterrupt:
|
||||
print("Keyboard interrupt detected, cancelling builds")
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
# Compiler could not be set up; return a single failed result to carry message
|
||||
from pathlib import Path as _Path
|
||||
|
||||
return BoardCompilationResult(
|
||||
ok=False,
|
||||
sketch_results=[
|
||||
SketchResult(
|
||||
success=False,
|
||||
output=f"Compiler setup failed: {str(e)}",
|
||||
build_dir=_Path("."),
|
||||
example="<setup>",
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def get_board_artifact_extension(board: Board) -> str:
|
||||
"""Get the primary artifact extension for a board."""
|
||||
# ESP32/ESP8266 boards always produce .bin files
|
||||
if board.board_name.startswith("esp"):
|
||||
return ".bin"
|
||||
|
||||
# Most Arduino-based boards produce .hex files
|
||||
if board.framework and "arduino" in board.framework.lower():
|
||||
return ".hex"
|
||||
|
||||
# Default to .hex for most microcontroller boards
|
||||
return ".hex"
|
||||
|
||||
|
||||
def validate_output_path(
|
||||
output_path: str, sketch_name: str, board: Board
|
||||
) -> tuple[bool, str, str]:
|
||||
"""Validate output path and return (is_valid, resolved_path, error_message).
|
||||
|
||||
Args:
|
||||
output_path: The user-specified output path
|
||||
sketch_name: Name of the sketch being built
|
||||
board: Board configuration
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, resolved_output_path, error_message)
|
||||
"""
|
||||
import os
|
||||
|
||||
expected_ext = get_board_artifact_extension(board)
|
||||
|
||||
# Handle special case: -o .
|
||||
if output_path == ".":
|
||||
resolved_path = f"{sketch_name}{expected_ext}"
|
||||
return True, resolved_path, ""
|
||||
|
||||
# If path ends with /, it's a directory
|
||||
if output_path.endswith("/") or output_path.endswith("\\"):
|
||||
resolved_path = os.path.join(output_path, f"{sketch_name}{expected_ext}")
|
||||
return True, resolved_path, ""
|
||||
|
||||
# If path has an extension, it's a file - validate the extension
|
||||
if "." in os.path.basename(output_path):
|
||||
_, ext = os.path.splitext(output_path)
|
||||
if ext != expected_ext:
|
||||
return (
|
||||
False,
|
||||
"",
|
||||
f"Output file extension '{ext}' doesn't match expected '{expected_ext}' for board '{board.board_name}'",
|
||||
)
|
||||
return True, output_path, ""
|
||||
|
||||
# Path doesn't end with / and has no extension - treat as directory
|
||||
resolved_path = os.path.join(output_path, f"{sketch_name}{expected_ext}")
|
||||
return True, resolved_path, ""
|
||||
|
||||
|
||||
def copy_build_artifact(
|
||||
build_dir: Path, board: Board, sketch_name: str, output_path: str
|
||||
) -> bool:
|
||||
"""Copy the build artifact to the specified output path.
|
||||
|
||||
Args:
|
||||
build_dir: Build directory path
|
||||
board: Board configuration
|
||||
sketch_name: Name of the sketch
|
||||
output_path: Target output path
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
|
||||
expected_ext = get_board_artifact_extension(board)
|
||||
|
||||
# Find the source artifact
|
||||
# PlatformIO builds are in .build/pio/{board}/.pio/build/{board}/firmware.{ext}
|
||||
artifact_dir = build_dir / ".pio" / "build" / board.board_name
|
||||
source_artifact = artifact_dir / f"firmware{expected_ext}"
|
||||
|
||||
if not source_artifact.exists():
|
||||
print(f"ERROR: Build artifact not found: {source_artifact}")
|
||||
return False
|
||||
|
||||
# Ensure output directory exists
|
||||
output_path_obj = Path(output_path)
|
||||
output_path_obj.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
try:
|
||||
print(f"Copying {source_artifact} to {output_path}")
|
||||
shutil.copy2(source_artifact, output_path)
|
||||
print(f"✅ Build artifact saved to: {output_path}")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"ERROR: Failed to copy build artifact: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main function."""
|
||||
args = parse_args()
|
||||
if args.verbose:
|
||||
os.environ["VERBOSE"] = "1"
|
||||
|
||||
if args.supported_boards:
|
||||
print(",".join(get_default_boards()))
|
||||
return 0
|
||||
|
||||
# Determine which boards to compile for
|
||||
if args.boards:
|
||||
boards_names = args.boards.split(",")
|
||||
else:
|
||||
boards_names = get_default_boards()
|
||||
|
||||
# Get board objects
|
||||
boards: List[Board] = []
|
||||
for board_name in boards_names:
|
||||
try:
|
||||
board = create_board(board_name, no_project_options=False)
|
||||
boards.append(board)
|
||||
except Exception as e:
|
||||
print(f"ERROR: Failed to get board '{board_name}': {e}")
|
||||
return 1
|
||||
|
||||
# Determine which examples to compile
|
||||
if args.positional_examples:
|
||||
# Convert positional examples, handling both "examples/Blink" and "Blink" formats
|
||||
examples: List[str] = []
|
||||
for example in args.positional_examples:
|
||||
# Remove "examples/" prefix if present
|
||||
if example.startswith("examples/"):
|
||||
example = example[len("examples/") :]
|
||||
examples.append(example)
|
||||
elif args.examples:
|
||||
examples = args.examples.split(",")
|
||||
else:
|
||||
# Compile all available examples since builds are fast now!
|
||||
examples = get_all_examples()
|
||||
|
||||
# Process example exclusions
|
||||
if args.exclude_examples:
|
||||
exclude_examples = args.exclude_examples.split(",")
|
||||
examples = [ex for ex in examples if ex not in exclude_examples]
|
||||
|
||||
# Validate examples exist
|
||||
for example in examples:
|
||||
try:
|
||||
resolve_example_path(example)
|
||||
except FileNotFoundError as e:
|
||||
print(f"ERROR: {e}")
|
||||
return 1
|
||||
|
||||
# Validate -o/--out option requirements
|
||||
if args.out:
|
||||
if len(examples) != 1:
|
||||
print(
|
||||
f"ERROR: -o/--out option requires exactly one sketch, got {len(examples)}: {examples}"
|
||||
)
|
||||
return 1
|
||||
|
||||
if len(boards) != 1:
|
||||
print(
|
||||
f"ERROR: -o/--out option requires exactly one board, got {len(boards)}: {[b.board_name for b in boards]}"
|
||||
)
|
||||
return 1
|
||||
|
||||
# Validate the output path
|
||||
sketch_name = examples[0]
|
||||
board = boards[0]
|
||||
is_valid, resolved_output_path, error_msg = validate_output_path(
|
||||
args.out, sketch_name, board
|
||||
)
|
||||
if not is_valid:
|
||||
print(f"ERROR: {error_msg}")
|
||||
return 1
|
||||
|
||||
# Set up defines
|
||||
defines: List[str] = []
|
||||
if args.defines:
|
||||
defines.extend(args.defines.split(","))
|
||||
|
||||
# Start compilation
|
||||
start_time = time.time()
|
||||
print(
|
||||
f"Starting compilation for {len(boards)} boards with {len(examples)} examples"
|
||||
)
|
||||
|
||||
compilation_errors: List[str] = []
|
||||
failed_example_names: List[str] = []
|
||||
failure_logs_dir: Optional[Path] = (
|
||||
Path(args.log_failures) if getattr(args, "log_failures", None) else None
|
||||
)
|
||||
|
||||
# Compile for each board
|
||||
for board in boards:
|
||||
# Parse global cache directory if provided
|
||||
global_cache_dir = None
|
||||
if args.global_cache:
|
||||
global_cache_dir = Path(args.global_cache)
|
||||
|
||||
result = compile_board_examples(
|
||||
board=board,
|
||||
examples=examples,
|
||||
defines=defines,
|
||||
verbose=args.verbose,
|
||||
enable_cache=(args.enable_cache or args.cache) and not args.no_cache,
|
||||
global_cache_dir=global_cache_dir,
|
||||
)
|
||||
|
||||
if not result.ok:
|
||||
# Record board-level error
|
||||
compilation_errors.append(f"Board {board.board_name} failed")
|
||||
print(red_text(f"ERROR: Compilation failed for board {board.board_name}"))
|
||||
# Print each failing sketch's stdout and collect names for summary
|
||||
for sketch in result.sketch_results:
|
||||
if not sketch.success:
|
||||
if sketch.example and sketch.example not in failed_example_names:
|
||||
failed_example_names.append(sketch.example)
|
||||
# Write per-example failure logs when requested
|
||||
if failure_logs_dir is not None:
|
||||
try:
|
||||
failure_logs_dir.mkdir(parents=True, exist_ok=True)
|
||||
safe_name = f"{sketch.example}.log"
|
||||
log_path = failure_logs_dir / safe_name
|
||||
with log_path.open(
|
||||
"a", encoding="utf-8", errors="ignore"
|
||||
) as f:
|
||||
f.write(sketch.output)
|
||||
f.write("\n")
|
||||
except KeyboardInterrupt:
|
||||
print("Keyboard interrupt detected, cancelling builds")
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
print(
|
||||
f"Warning: Could not write failure log for {sketch.example}: {e}"
|
||||
)
|
||||
print(f"\n{'-' * 60}")
|
||||
print(f"Sketch: {sketch.example}")
|
||||
print(f"{'-' * 60}")
|
||||
# Print the collected output for this sketch
|
||||
print(sketch.output)
|
||||
# Continue with other boards instead of stopping
|
||||
else:
|
||||
# Compilation succeeded - handle -o/--out option if specified
|
||||
if args.out:
|
||||
sketch_name = examples[0] # We already validated there's exactly one
|
||||
is_valid, resolved_output_path, _ = validate_output_path(
|
||||
args.out, sketch_name, board
|
||||
)
|
||||
if is_valid:
|
||||
# Find the build directory for this board
|
||||
project_root = Path(__file__).parent.parent.resolve()
|
||||
build_dir = project_root / ".build" / "pio" / board.board_name
|
||||
|
||||
if not copy_build_artifact(
|
||||
build_dir, board, sketch_name, resolved_output_path
|
||||
):
|
||||
compilation_errors.append(
|
||||
f"Failed to copy artifact for {board.board_name}"
|
||||
)
|
||||
print(
|
||||
red_text(
|
||||
f"ERROR: Failed to copy build artifact for {board.board_name}"
|
||||
)
|
||||
)
|
||||
return 1
|
||||
|
||||
# Report results
|
||||
elapsed_time = time.time() - start_time
|
||||
time_str = time.strftime("%Mm:%Ss", time.gmtime(elapsed_time))
|
||||
|
||||
if compilation_errors:
|
||||
print(
|
||||
f"\nCompilation finished in {time_str} with {len(compilation_errors)} error(s):"
|
||||
)
|
||||
for error in compilation_errors:
|
||||
print(f" - {error}")
|
||||
if failed_example_names:
|
||||
print("")
|
||||
print(red_text(f"ERROR! There were {len(failed_example_names)} failures:"))
|
||||
# Sort for stable output
|
||||
for name in sorted(failed_example_names):
|
||||
# print(f" {name}")
|
||||
# same but in red
|
||||
print(red_text(f" {name}"))
|
||||
print("")
|
||||
return 1
|
||||
else:
|
||||
print(f"\nAll compilations completed successfully in {time_str}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
print("\nInterrupted by user")
|
||||
sys.exit(1)
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,134 @@
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
MINIMUM_REPORT_SEVERTIY = "medium"
|
||||
MINIMUM_FAIL_SEVERTIY = "high"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Run cppcheck on the project")
|
||||
parser.add_argument("board", nargs="?", help="Board to check, optional")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def find_platformio_project_dir(build_root: Path, board_name: str) -> Path | None:
|
||||
"""Find a directory containing platformio.ini file for the specified board.
|
||||
|
||||
Handles both old and new build directory structures:
|
||||
Old: .build/uno/Blink/platformio.ini
|
||||
New: .build/fled/examples/uno/Blink/platformio.ini
|
||||
|
||||
Returns the directory containing platformio.ini, or None if not found.
|
||||
"""
|
||||
if not build_root.exists():
|
||||
return None
|
||||
|
||||
# Search patterns for different build structures
|
||||
search_patterns = [
|
||||
# New structure: .build/fled/examples/{board}/**/*.ini
|
||||
build_root / "fled" / "examples" / board_name,
|
||||
# Old structure: .build/{board}/**/*.ini
|
||||
build_root / board_name,
|
||||
# Fallback: search all subdirectories for board name
|
||||
]
|
||||
|
||||
# Try each search pattern
|
||||
for base_dir in search_patterns:
|
||||
if not base_dir.exists():
|
||||
continue
|
||||
|
||||
# Look for platformio.ini in this directory and its subdirectories
|
||||
project_dir = find_platformio_ini_in_tree(base_dir)
|
||||
if project_dir:
|
||||
return project_dir
|
||||
|
||||
# Last resort: search the entire build directory for board-related platformio.ini
|
||||
for platformio_ini in build_root.rglob("platformio.ini"):
|
||||
# Check if this platformio.ini is for our board by looking at the path
|
||||
if board_name in str(platformio_ini):
|
||||
print(f"Found platformio.ini in {platformio_ini.parent}")
|
||||
return platformio_ini.parent
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def find_platformio_ini_in_tree(base_dir: Path) -> Path | None:
|
||||
"""Recursively search for platformio.ini in a directory tree."""
|
||||
if not base_dir.exists():
|
||||
return None
|
||||
|
||||
# Check direct subdirectories first (most common case)
|
||||
for subdir in base_dir.iterdir():
|
||||
if subdir.is_dir():
|
||||
platformio_ini = subdir / "platformio.ini"
|
||||
if platformio_ini.exists():
|
||||
print(f"Found platformio.ini in {subdir}")
|
||||
return subdir
|
||||
|
||||
# Check if there's a platformio.ini directly in the base directory
|
||||
if (base_dir / "platformio.ini").exists():
|
||||
print(f"Found platformio.ini directly in {base_dir}")
|
||||
return base_dir
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
here = Path(__file__).parent
|
||||
project_root = here.parent
|
||||
build = project_root / ".build"
|
||||
|
||||
if not build.exists():
|
||||
print(f"Build directory {build} not found")
|
||||
return 1
|
||||
|
||||
if args.board:
|
||||
# Search for the specified board
|
||||
project_dir = find_platformio_project_dir(build, args.board)
|
||||
if not project_dir:
|
||||
print(f"No platformio.ini found for board '{args.board}'")
|
||||
print("This usually means the board hasn't been compiled yet.")
|
||||
print(f"Try running: uv run ci/ci-compile.py {args.board} --examples Blink")
|
||||
return 1
|
||||
else:
|
||||
# Auto-detect available board by searching for any platformio.ini
|
||||
project_dir = None
|
||||
|
||||
# Try to find any platformio.ini file in the build directory
|
||||
for platformio_ini in build.rglob("platformio.ini"):
|
||||
project_dir = platformio_ini.parent
|
||||
print(f"Auto-detected project directory: {project_dir}")
|
||||
break
|
||||
|
||||
if not project_dir:
|
||||
print(f"No platformio.ini files found in {build}")
|
||||
print("This usually means no boards have been compiled yet.")
|
||||
print("Try running: uv run ci/ci-compile.py uno --examples Blink")
|
||||
return 1
|
||||
|
||||
print(f"Running pio check in {project_dir}")
|
||||
os.chdir(str(project_dir))
|
||||
|
||||
# Run pio check command
|
||||
cp = subprocess.run(
|
||||
[
|
||||
"pio",
|
||||
"check",
|
||||
"--skip-packages",
|
||||
"--src-filters=+<src/>",
|
||||
f"--severity={MINIMUM_REPORT_SEVERTIY}",
|
||||
f"--fail-on-defect={MINIMUM_FAIL_SEVERTIY}",
|
||||
"--flags",
|
||||
"--inline-suppr",
|
||||
],
|
||||
)
|
||||
return cp.returncode
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,6 @@
|
||||
# flake8: noqa
|
||||
# ruff: skip
|
||||
|
||||
Import("env") # type: ignore
|
||||
|
||||
env.Append(CXXFLAGS=["-Wno-register"]) # type: ignore
|
||||
@@ -0,0 +1,271 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run include-what-you-use on the project"
|
||||
)
|
||||
parser.add_argument("board", nargs="?", help="Board to check, optional")
|
||||
parser.add_argument(
|
||||
"--fix",
|
||||
action="store_true",
|
||||
help="Automatically apply suggested fixes using fix_includes",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-file",
|
||||
action="append",
|
||||
help="Additional mapping file to use (can be specified multiple times)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-line-length",
|
||||
type=int,
|
||||
default=100,
|
||||
help="Maximum line length for suggestions (default: 100)",
|
||||
)
|
||||
parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def find_platformio_project_dir(board_dir: Path) -> Path | None:
|
||||
"""Find a directory containing platformio.ini file in the board's build directory.
|
||||
|
||||
With the new pio ci build system, the structure is:
|
||||
.build/uno/Blink/platformio.ini
|
||||
.build/uno/SomeExample/platformio.ini
|
||||
|
||||
We need to find one of these example directories that has a platformio.ini file.
|
||||
"""
|
||||
if not board_dir.exists():
|
||||
return None
|
||||
|
||||
# Look for subdirectories containing platformio.ini
|
||||
for subdir in board_dir.iterdir():
|
||||
if subdir.is_dir():
|
||||
platformio_ini = subdir / "platformio.ini"
|
||||
if platformio_ini.exists():
|
||||
print(f"Found platformio.ini in {subdir}")
|
||||
return subdir
|
||||
|
||||
# Fallback: check if there's a platformio.ini directly in the board directory (old system)
|
||||
if (board_dir / "platformio.ini").exists():
|
||||
print(f"Found platformio.ini directly in {board_dir}")
|
||||
return board_dir
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def check_iwyu_available() -> bool:
|
||||
"""Check if include-what-you-use is available in the system"""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["include-what-you-use", "--version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (
|
||||
subprocess.CalledProcessError,
|
||||
FileNotFoundError,
|
||||
subprocess.TimeoutExpired,
|
||||
):
|
||||
return False
|
||||
|
||||
|
||||
def run_iwyu_on_cpp_tests(args: argparse.Namespace) -> int:
|
||||
"""Run IWYU on the C++ test suite using CMake integration"""
|
||||
here = Path(__file__).parent
|
||||
project_root = here.parent
|
||||
|
||||
print("Running include-what-you-use on C++ test suite...")
|
||||
|
||||
# Use the existing test infrastructure with --check flag
|
||||
cmd = [
|
||||
"uv",
|
||||
"run",
|
||||
"test.py",
|
||||
"--cpp",
|
||||
"--check", # This enables IWYU via CMake
|
||||
"--clang",
|
||||
"--no-interactive",
|
||||
]
|
||||
|
||||
if args.verbose:
|
||||
cmd.append("--verbose")
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd, cwd=project_root)
|
||||
return result.returncode
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"IWYU analysis failed with return code {e.returncode}")
|
||||
return e.returncode
|
||||
|
||||
|
||||
def run_iwyu_on_platformio_project(project_dir: Path, args: argparse.Namespace) -> int:
|
||||
"""Run IWYU on a PlatformIO project"""
|
||||
print(f"Running include-what-you-use in {project_dir}")
|
||||
os.chdir(str(project_dir))
|
||||
|
||||
# Build mapping file arguments
|
||||
mapping_args: List[str] = []
|
||||
|
||||
# Add FastLED mapping files if they exist
|
||||
project_root = project_dir
|
||||
while project_root.parent != project_root: # Find project root
|
||||
if (project_root / "ci" / "iwyu").exists():
|
||||
break
|
||||
project_root = project_root.parent
|
||||
|
||||
fastled_mapping = project_root / "ci" / "iwyu" / "fastled.imp"
|
||||
stdlib_mapping = project_root / "ci" / "iwyu" / "stdlib.imp"
|
||||
|
||||
if fastled_mapping.exists():
|
||||
mapping_args.extend(["--mapping_file", str(fastled_mapping)])
|
||||
|
||||
if stdlib_mapping.exists():
|
||||
mapping_args.extend(["--mapping_file", str(stdlib_mapping)])
|
||||
|
||||
# Add user-specified mapping files
|
||||
if args.mapping_file:
|
||||
for mapping in args.mapping_file:
|
||||
mapping_args.extend(["--mapping_file", mapping])
|
||||
|
||||
# Build IWYU command
|
||||
iwyu_cmd = [
|
||||
"include-what-you-use",
|
||||
f"--max_line_length={args.max_line_length}",
|
||||
"--quoted_includes_first",
|
||||
"--no_comments",
|
||||
]
|
||||
|
||||
if args.verbose:
|
||||
iwyu_cmd.append("--verbose=3")
|
||||
else:
|
||||
iwyu_cmd.append("--verbose=1")
|
||||
|
||||
iwyu_cmd.extend(mapping_args)
|
||||
|
||||
# Run through PlatformIO's check system with IWYU
|
||||
pio_cmd = [
|
||||
"pio",
|
||||
"check",
|
||||
"--skip-packages",
|
||||
"--src-filters=+<src/>",
|
||||
"--tool=include-what-you-use",
|
||||
"--flags",
|
||||
] + iwyu_cmd[1:] # Skip the include-what-you-use binary name
|
||||
|
||||
try:
|
||||
result = subprocess.run(pio_cmd)
|
||||
return result.returncode
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"PlatformIO IWYU check failed with return code {e.returncode}")
|
||||
return e.returncode
|
||||
|
||||
|
||||
def apply_iwyu_fixes(source_dir: Path) -> int:
|
||||
"""Apply IWYU fixes using fix_includes tool"""
|
||||
try:
|
||||
# Check if fix_includes is available
|
||||
subprocess.run(["fix_includes", "--help"], capture_output=True, check=True)
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
print(
|
||||
"Warning: fix_includes tool not found. Install IWYU tools to use --fix option."
|
||||
)
|
||||
return 1
|
||||
|
||||
# Find all .h and .cpp files
|
||||
cpp_files: List[Path] = []
|
||||
for pattern in ["**/*.cpp", "**/*.h", "**/*.hpp"]:
|
||||
cpp_files.extend(source_dir.glob(pattern))
|
||||
|
||||
if not cpp_files:
|
||||
print("No C++ files found to fix")
|
||||
return 0
|
||||
|
||||
print(f"Applying IWYU fixes to {len(cpp_files)} files...")
|
||||
|
||||
# Run fix_includes on the source directory
|
||||
cmd = ["fix_includes", "--update_comments", str(source_dir)]
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd)
|
||||
if result.returncode == 0:
|
||||
print("IWYU fixes applied successfully")
|
||||
else:
|
||||
print(f"fix_includes failed with return code {result.returncode}")
|
||||
return result.returncode
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"Failed to apply IWYU fixes: {e}")
|
||||
return e.returncode
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
here = Path(__file__).parent
|
||||
project_root = here.parent
|
||||
|
||||
# Check if IWYU is available
|
||||
if not check_iwyu_available():
|
||||
print("Error: include-what-you-use not found in PATH")
|
||||
print("Install it with:")
|
||||
print(" Ubuntu/Debian: sudo apt install iwyu")
|
||||
print(" macOS: brew install include-what-you-use")
|
||||
print(" Or build from source: https://include-what-you-use.org/")
|
||||
return 1
|
||||
|
||||
print("Found include-what-you-use")
|
||||
|
||||
# If no board specified, run on C++ test suite
|
||||
if not args.board:
|
||||
print("No board specified, running IWYU on C++ test suite")
|
||||
return run_iwyu_on_cpp_tests(args)
|
||||
|
||||
# Run on specific board
|
||||
build = project_root / ".build"
|
||||
|
||||
if not build.exists():
|
||||
print(f"Build directory {build} not found")
|
||||
print("Run a compilation first: ./compile [board] --examples [example]")
|
||||
return 1
|
||||
|
||||
board_dir = build / args.board
|
||||
if not board_dir.exists():
|
||||
print(f"Board {args.board} not found in {build}")
|
||||
print("Available boards:")
|
||||
for d in build.iterdir():
|
||||
if d.is_dir():
|
||||
print(f" {d.name}")
|
||||
return 1
|
||||
|
||||
project_dir = find_platformio_project_dir(board_dir)
|
||||
if not project_dir:
|
||||
print(f"No platformio.ini found in {board_dir} or its subdirectories")
|
||||
print("This usually means the board hasn't been compiled yet.")
|
||||
print(f"Try running: ./compile {args.board} --examples Blink")
|
||||
return 1
|
||||
|
||||
# Run IWYU on the PlatformIO project
|
||||
result = run_iwyu_on_platformio_project(project_dir, args)
|
||||
|
||||
# Apply fixes if requested and analysis succeeded
|
||||
if args.fix and result == 0:
|
||||
src_dir = project_dir / "src"
|
||||
if src_dir.exists():
|
||||
fix_result = apply_iwyu_fixes(src_dir)
|
||||
if fix_result != 0:
|
||||
result = fix_result
|
||||
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,238 @@
|
||||
"""
|
||||
Fingerprint Cache Feature
|
||||
|
||||
A two-layer file change detection system that efficiently determines if source files
|
||||
have been modified by combining fast modification time checks with slower but accurate
|
||||
MD5 hash verification.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class CacheEntry:
|
||||
"""Cache entry storing file modification time and content hash."""
|
||||
|
||||
modification_time: float
|
||||
md5_hash: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class FingerprintCacheConfig:
|
||||
"""Configuration for fingerprint cache behavior."""
|
||||
|
||||
cache_file: Path
|
||||
hash_algorithm: str
|
||||
ignore_patterns: list[str]
|
||||
max_cache_size: int
|
||||
|
||||
|
||||
class FingerprintCache:
|
||||
"""
|
||||
Two-layer file change detection using modification time and content hashing.
|
||||
|
||||
Provides efficient change detection by:
|
||||
1. Fast modification time comparison (microsecond performance)
|
||||
2. Accurate content verification via MD5 hashing when needed
|
||||
"""
|
||||
|
||||
def __init__(self, cache_file: Path, modtime_only: bool = False):
|
||||
"""
|
||||
Initialize fingerprint cache.
|
||||
|
||||
Args:
|
||||
cache_file: Path to JSON cache file
|
||||
modtime_only: When True, disable hashing and rely solely on modtime.
|
||||
"""
|
||||
self.cache_file = cache_file
|
||||
self._modtime_only: bool = modtime_only
|
||||
self.cache = self._load_cache()
|
||||
|
||||
def _load_cache(self) -> Dict[str, CacheEntry]:
|
||||
"""
|
||||
Load cache from JSON file, return empty dict if file doesn't exist.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping file paths to cache entries
|
||||
"""
|
||||
if not self.cache_file.exists():
|
||||
return {}
|
||||
|
||||
try:
|
||||
with open(self.cache_file, "r") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Convert JSON dict to CacheEntry objects
|
||||
cache: Dict[str, CacheEntry] = {}
|
||||
for file_path, entry_data in data.items():
|
||||
cache[file_path] = CacheEntry(
|
||||
modification_time=entry_data["modification_time"],
|
||||
md5_hash=entry_data["md5_hash"],
|
||||
)
|
||||
return cache
|
||||
except (json.JSONDecodeError, KeyError, TypeError):
|
||||
# Cache corrupted - start fresh
|
||||
return {}
|
||||
|
||||
def _save_cache(self) -> None:
|
||||
"""Save current cache state to JSON file."""
|
||||
# Ensure cache directory exists
|
||||
self.cache_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Convert CacheEntry objects to JSON-serializable dict
|
||||
data = {}
|
||||
for file_path, entry in self.cache.items():
|
||||
data[file_path] = {
|
||||
"modification_time": entry.modification_time,
|
||||
"md5_hash": entry.md5_hash,
|
||||
}
|
||||
|
||||
with open(self.cache_file, "w") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
|
||||
def _compute_md5(self, file_path: Path) -> str:
|
||||
"""
|
||||
Compute MD5 hash of file content.
|
||||
|
||||
Args:
|
||||
file_path: Path to file to hash
|
||||
|
||||
Returns:
|
||||
MD5 hash as hexadecimal string
|
||||
|
||||
Raises:
|
||||
IOError: If file cannot be read
|
||||
"""
|
||||
try:
|
||||
hasher = hashlib.md5()
|
||||
with open(file_path, "rb") as f:
|
||||
# Read in chunks to handle large files efficiently
|
||||
for chunk in iter(lambda: f.read(8192), b""):
|
||||
hasher.update(chunk)
|
||||
return hasher.hexdigest()
|
||||
except IOError as e:
|
||||
raise IOError(f"Cannot read file {file_path}: {e}")
|
||||
|
||||
def has_changed(self, src_path: Path, previous_modtime: float) -> bool:
|
||||
"""
|
||||
Determine if a source file has changed since the last known modification time.
|
||||
|
||||
Two-layer verification process:
|
||||
1. Fast modification time check - if times match, return False immediately
|
||||
2. Content verification - compute/use cached MD5 hash for accurate comparison
|
||||
|
||||
Args:
|
||||
src_path: Path to the source file to check
|
||||
previous_modtime: Previously known modification time (Unix timestamp)
|
||||
|
||||
Returns:
|
||||
True if the file has changed, False if unchanged
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If source file doesn't exist
|
||||
"""
|
||||
if not src_path.exists():
|
||||
raise FileNotFoundError(f"Source file not found: {src_path}")
|
||||
|
||||
current_modtime = os.path.getmtime(src_path)
|
||||
|
||||
# Optional mode: strictly use modification time only (no hashing)
|
||||
# This is required for toolchains that invalidate PCH on any newer
|
||||
# dependency regardless of content changes (e.g., Clang).
|
||||
if self._modtime_only:
|
||||
# Treat as changed only when file is newer than the reference time
|
||||
# (keeps behavior stable when reference is an external artifact's mtime)
|
||||
return current_modtime > previous_modtime
|
||||
|
||||
# Layer 1: Quick modification time check
|
||||
if current_modtime == previous_modtime:
|
||||
return False # No change detected
|
||||
|
||||
file_key = str(src_path.resolve()) # Use absolute path as key
|
||||
|
||||
# Layer 2: Content verification via hash comparison
|
||||
if file_key in self.cache:
|
||||
cached_entry = self.cache[file_key]
|
||||
previous_hash = (
|
||||
cached_entry.md5_hash
|
||||
) # Store previous hash before potential update
|
||||
|
||||
if current_modtime == cached_entry.modification_time:
|
||||
# File is cached with current modtime - use cached hash
|
||||
current_hash = cached_entry.md5_hash
|
||||
else:
|
||||
# Cache is stale - compute new hash
|
||||
current_hash = self._compute_md5(src_path)
|
||||
self._update_cache_entry(src_path, current_modtime, current_hash)
|
||||
|
||||
# Compare current hash with previous cached hash
|
||||
# If they match, content hasn't actually changed despite modtime difference
|
||||
return current_hash != previous_hash
|
||||
|
||||
else:
|
||||
# File not in cache - compute hash and cache it
|
||||
current_hash = self._compute_md5(src_path)
|
||||
self._update_cache_entry(src_path, current_modtime, current_hash)
|
||||
return True # Assume changed since we have no previous state to compare
|
||||
|
||||
def _update_cache_entry(
|
||||
self, file_path: Path, modification_time: float, md5_hash: str
|
||||
) -> None:
|
||||
"""
|
||||
Update cache with new entry and save to disk.
|
||||
|
||||
Args:
|
||||
file_path: Path to file being cached
|
||||
modification_time: File modification time
|
||||
md5_hash: File content hash
|
||||
"""
|
||||
file_key = str(file_path.resolve())
|
||||
self.cache[file_key] = CacheEntry(
|
||||
modification_time=modification_time, md5_hash=md5_hash
|
||||
)
|
||||
self._save_cache()
|
||||
|
||||
def get_cache_stats(self) -> dict[str, int]:
|
||||
"""
|
||||
Get cache statistics.
|
||||
|
||||
Returns:
|
||||
Dictionary with cache size and other metrics
|
||||
"""
|
||||
return {
|
||||
"total_entries": len(self.cache),
|
||||
"cache_file_exists": self.cache_file.exists(),
|
||||
"cache_file_size_bytes": self.cache_file.stat().st_size
|
||||
if self.cache_file.exists()
|
||||
else 0,
|
||||
}
|
||||
|
||||
def clear_cache(self) -> None:
|
||||
"""Clear all cache entries and remove cache file."""
|
||||
self.cache.clear()
|
||||
if self.cache_file.exists():
|
||||
self.cache_file.unlink()
|
||||
|
||||
|
||||
def has_changed(src_path: Path, previous_modtime: float, cache_file: Path) -> bool:
|
||||
"""
|
||||
Convenience function implementing the main API.
|
||||
|
||||
Args:
|
||||
src_path: Path to the source file to check
|
||||
previous_modtime: Previously known modification time (Unix timestamp)
|
||||
cache_file: Path to the fingerprint cache file
|
||||
|
||||
Returns:
|
||||
True if the file has changed, False if unchanged
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If source file doesn't exist
|
||||
"""
|
||||
cache = FingerprintCache(cache_file)
|
||||
return cache.has_changed(src_path, previous_modtime)
|
||||
@@ -0,0 +1,290 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
Compile Commands Generator
|
||||
|
||||
Generates a compile_commands.json for clangd/clang-tidy based on explicit
|
||||
tools/flags configuration. See FEATURE.md for usage and rationale.
|
||||
|
||||
Key behaviors:
|
||||
- Reads configuration from ci/build_commands.toml
|
||||
- Enumerates library source files under src/ (respecting optional filters.exclude_dirs)
|
||||
- Emits one entry per translation unit with accurate arguments/command fields
|
||||
|
||||
Notes:
|
||||
- This script intentionally avoids project-unit-test driven flags to keep
|
||||
IntelliSense consistent with the core FastLED library build.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import _thread
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
# Require Python 3.11+ environment for TOML parsing
|
||||
import tomllib
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional, cast
|
||||
|
||||
# Require typeguard; no try-import fallbacks
|
||||
from typeguard import typechecked
|
||||
|
||||
# Use the canonical BuildFlags/Compiler tooling
|
||||
from ci.compiler.clang_compiler import commands_json
|
||||
|
||||
|
||||
PROJECT_ROOT: Path = Path(__file__).resolve().parent.parent
|
||||
# Default to bespoke config for compile-commands generation
|
||||
DEFAULT_CONFIG_PATH: Path = PROJECT_ROOT / "ci" / "build_commands.toml"
|
||||
|
||||
|
||||
@typechecked
|
||||
@dataclass
|
||||
class CliConfig:
|
||||
build_flags_toml: Path
|
||||
exclude_dirs: list[str]
|
||||
|
||||
|
||||
@typechecked
|
||||
@dataclass
|
||||
class Args:
|
||||
output: Path
|
||||
clean: bool
|
||||
verbose: bool
|
||||
config: Path
|
||||
exclude: list[str]
|
||||
|
||||
|
||||
def parse_args(args: Optional[list[str]] = None) -> Args:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Generate compile_commands.json for clangd/clang-tidy based on explicit build configuration."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
dest="output",
|
||||
metavar="PATH",
|
||||
help="Output path for compile_commands.json (default: project root)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--config",
|
||||
dest="config",
|
||||
metavar="PATH",
|
||||
help=(
|
||||
"Path to BuildFlags-style TOML (default: ci/build_example.toml). "
|
||||
"Must be parseable by BuildFlags.parse()."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--clean",
|
||||
dest="clean",
|
||||
action="store_true",
|
||||
help="Rebuild command set from scratch (ignored currently; reserved)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbose",
|
||||
dest="verbose",
|
||||
action="store_true",
|
||||
help="Print collected flags/files",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--exclude-dir",
|
||||
dest="exclude",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Relative directory under project root to exclude (can be used multiple times)",
|
||||
)
|
||||
|
||||
parsed = parser.parse_args(args)
|
||||
|
||||
output_path_str: Optional[str] = None
|
||||
if hasattr(parsed, "output"):
|
||||
output_path_str = parsed.output
|
||||
|
||||
if output_path_str is None or len(output_path_str) == 0:
|
||||
# Default to project root compile_commands.json
|
||||
output_path = PROJECT_ROOT / "compile_commands.json"
|
||||
else:
|
||||
output_path = Path(output_path_str).resolve()
|
||||
|
||||
clean_value: bool = False
|
||||
if hasattr(parsed, "clean"):
|
||||
clean_value = bool(parsed.clean)
|
||||
|
||||
verbose_value: bool = False
|
||||
if hasattr(parsed, "verbose"):
|
||||
verbose_value = bool(parsed.verbose)
|
||||
|
||||
# Config path
|
||||
config_path: Path
|
||||
if getattr(parsed, "config", None):
|
||||
config_path = Path(parsed.config).resolve()
|
||||
else:
|
||||
config_path = DEFAULT_CONFIG_PATH
|
||||
|
||||
# Excluded directories
|
||||
exclude_list: list[str] = list(getattr(parsed, "exclude", []) or [])
|
||||
|
||||
return Args(
|
||||
output=output_path,
|
||||
clean=clean_value,
|
||||
verbose=verbose_value,
|
||||
config=config_path,
|
||||
exclude=exclude_list,
|
||||
)
|
||||
|
||||
|
||||
def _assert_config_exists(config_path: Path) -> None:
|
||||
if not config_path.exists():
|
||||
raise RuntimeError(
|
||||
f"CRITICAL: Required BuildFlags TOML not found at {config_path}. "
|
||||
f"Please provide a valid configuration parseable by BuildFlags.parse()."
|
||||
)
|
||||
|
||||
|
||||
def _create_cli_config(build_flags_toml: Path, exclude: list[str]) -> CliConfig:
|
||||
return CliConfig(build_flags_toml=build_flags_toml, exclude_dirs=exclude)
|
||||
|
||||
|
||||
def _collect_source_files(root: Path, exclude: list[str], verbose: bool) -> list[Path]:
|
||||
source_files: list[Path] = []
|
||||
|
||||
# Compute excluded directories as absolute paths for quick prefix checks
|
||||
excluded_dirs_abs: list[Path] = []
|
||||
for ex in exclude:
|
||||
p = (PROJECT_ROOT / ex).resolve()
|
||||
excluded_dirs_abs.append(p)
|
||||
|
||||
src_root: Path = PROJECT_ROOT / "src"
|
||||
if not src_root.exists():
|
||||
raise RuntimeError(
|
||||
f"CRITICAL: Expected source root at {src_root} not found. "
|
||||
f"This path is required to enumerate translation units."
|
||||
)
|
||||
|
||||
for dirpath, dirnames, filenames in os.walk(src_root):
|
||||
current_dir = Path(dirpath).resolve()
|
||||
|
||||
# Apply directory exclusions
|
||||
skip_dir: bool = False
|
||||
for ex in excluded_dirs_abs:
|
||||
try:
|
||||
current_dir.relative_to(ex)
|
||||
skip_dir = True
|
||||
break
|
||||
except ValueError:
|
||||
# Not under this excluded path
|
||||
pass
|
||||
|
||||
if skip_dir:
|
||||
# Prune search by clearing dirnames to avoid descending further
|
||||
del dirnames[:]
|
||||
continue
|
||||
|
||||
for fname in filenames:
|
||||
if fname.endswith(".cpp") or fname.endswith(".c"):
|
||||
source_files.append(current_dir / fname)
|
||||
|
||||
if verbose:
|
||||
print(f"Discovered {len(source_files)} source files under {src_root}")
|
||||
|
||||
if len(source_files) == 0:
|
||||
raise RuntimeError(
|
||||
"CRITICAL: No source files discovered under 'src/'. "
|
||||
"Please verify the repository layout and filters.exclude_dirs configuration."
|
||||
)
|
||||
|
||||
return source_files
|
||||
|
||||
# No per-file argument composition here; delegated to commands_json()
|
||||
# This function intentionally removed to rely on canonical toolchain
|
||||
|
||||
# Emission handled by commands_json()
|
||||
|
||||
|
||||
def main(entry_args: Optional[list[str]] = None) -> int:
|
||||
args = parse_args(entry_args)
|
||||
|
||||
# Log startup context
|
||||
print("[compile-commands] Starting generation")
|
||||
print(f"[compile-commands] Project root: {PROJECT_ROOT}")
|
||||
print(f"[compile-commands] Config (BuildFlags TOML): {args.config}")
|
||||
print(f"[compile-commands] Output path: {args.output}")
|
||||
if args.exclude:
|
||||
print(f"[compile-commands] Excluding directories: {args.exclude}")
|
||||
else:
|
||||
print("[compile-commands] No excluded directories configured")
|
||||
|
||||
# Determine BuildFlags TOML path
|
||||
_assert_config_exists(args.config)
|
||||
|
||||
# Collect source files
|
||||
source_root = PROJECT_ROOT / "src"
|
||||
print(f"[compile-commands] Source root: {source_root}")
|
||||
sources = _collect_source_files(
|
||||
source_root,
|
||||
exclude=args.exclude,
|
||||
verbose=args.verbose,
|
||||
)
|
||||
print(f"[compile-commands] Discovered {len(sources)} source files")
|
||||
|
||||
# Delegate to canonical commands_json() using BuildFlags toolchain
|
||||
print("[compile-commands] Generating compile_commands.json entries...")
|
||||
commands_json(
|
||||
config_path=args.config,
|
||||
include_root=source_root,
|
||||
sources=sources,
|
||||
output_json=args.output,
|
||||
quick_build=True,
|
||||
strict_mode=False,
|
||||
)
|
||||
|
||||
# Post-generation verification
|
||||
if not args.output.exists():
|
||||
print(
|
||||
f"CRITICAL: Generation completed but output file was not found at {args.output}"
|
||||
)
|
||||
return 2
|
||||
|
||||
try:
|
||||
text = args.output.read_text(encoding="utf-8")
|
||||
# Best-effort count of entries to report status
|
||||
import json as _json
|
||||
from typing import cast
|
||||
|
||||
entries = cast(list[dict[str, Any]], _json.loads(text))
|
||||
num_entries = len(entries) if isinstance(entries, list) else 0
|
||||
print(
|
||||
f"[compile-commands] Wrote {num_entries} entries to {args.output} (size: {args.output.stat().st_size} bytes)"
|
||||
)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(
|
||||
f"WARNING: Output written to {args.output} but could not parse JSON for summary: {e}"
|
||||
)
|
||||
|
||||
if args.verbose:
|
||||
print("[compile-commands] Done.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
print("Operation interrupted by user")
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except FileNotFoundError as e:
|
||||
print(f"CRITICAL: Required file not found: {e}")
|
||||
sys.exit(1)
|
||||
except ValueError as e:
|
||||
print(f"CRITICAL: Invalid configuration value: {e}")
|
||||
sys.exit(1)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"CRITICAL: Unexpected error: {e}")
|
||||
raise
|
||||
@@ -0,0 +1,115 @@
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
def _create_board_info(path: Path) -> Dict[str, Any]:
|
||||
build_info = json.loads(path.read_text())
|
||||
assert build_info.keys(), f"No boards found in {build_info}"
|
||||
assert len(build_info.keys()) == 1, (
|
||||
f"Multiple boards found in {build_info}, so correct board should be specified"
|
||||
)
|
||||
return build_info[next(iter(build_info))]
|
||||
|
||||
|
||||
def _find_build_info(board: str) -> Path:
|
||||
# Prefer PIO layout
|
||||
candidate = Path(".build") / "pio" / board / "build_info.json"
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
# Fallback legacy layout
|
||||
candidate = Path(".build") / board / "build_info.json"
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
raise FileNotFoundError(
|
||||
f"build_info.json not found for board '{board}' in .build/pio/{board} or .build/{board}"
|
||||
)
|
||||
|
||||
|
||||
def _run_pio_size(build_dir: Path) -> int | None:
|
||||
try:
|
||||
# Try to compute size without building first
|
||||
result = subprocess.run(
|
||||
["pio", "run", "-d", str(build_dir), "-t", "size"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
output = (result.stdout or "") + "\n" + (result.stderr or "")
|
||||
m = re.search(r"Program:\s*(\d+)\s*bytes", output)
|
||||
if m:
|
||||
return int(m.group(1))
|
||||
# If size target did not yield, try a full build then retry size
|
||||
subprocess.run(
|
||||
["pio", "run", "-d", str(build_dir)], capture_output=False, check=False
|
||||
)
|
||||
result = subprocess.run(
|
||||
["pio", "run", "-d", str(build_dir), "-t", "size"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
output = (result.stdout or "") + "\n" + (result.stderr or "")
|
||||
m = re.search(r"Program:\s*(\d+)\s*bytes", output)
|
||||
if m:
|
||||
return int(m.group(1))
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def check_firmware_size(board: str) -> int:
|
||||
build_info_json = _find_build_info(board)
|
||||
board_info = _create_board_info(build_info_json)
|
||||
assert board_info, f"Board {board} not found in {build_info_json}"
|
||||
prog_path = Path(board_info["prog_path"]) # absolute or relative path
|
||||
base_path = prog_path.parent
|
||||
suffixes = [".bin", ".hex", ".uf2"]
|
||||
firmware: Path | None = None
|
||||
for suffix in suffixes:
|
||||
candidate = base_path / f"firmware{suffix}"
|
||||
if candidate.exists():
|
||||
firmware = candidate
|
||||
break
|
||||
if firmware is None:
|
||||
# As a fallback, if firmware.* doesn't exist, use ELF size (filesize) if present
|
||||
if prog_path.exists():
|
||||
return prog_path.stat().st_size
|
||||
# As a last resort, run PlatformIO size target in the build directory
|
||||
build_dir = build_info_json.parent
|
||||
size = _run_pio_size(build_dir)
|
||||
if size is not None:
|
||||
return size
|
||||
msg = (
|
||||
", ".join([f"firmware{suffix}" for suffix in suffixes])
|
||||
+ f" not found in {base_path}"
|
||||
)
|
||||
raise FileNotFoundError(msg)
|
||||
return firmware.stat().st_size
|
||||
|
||||
|
||||
def main(board: str):
|
||||
try:
|
||||
size = check_firmware_size(board)
|
||||
print(f"Firmware size for {board}: {size} bytes")
|
||||
except FileNotFoundError as e:
|
||||
print(f"Error: {e}")
|
||||
except json.JSONDecodeError:
|
||||
print(f"Error: Unable to parse build_info.json for {board}")
|
||||
except Exception as e:
|
||||
print(f"An unexpected error occurred: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check FastLED firmware size for the specified board."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--board", type=str, required=True, help="Board to check firmware size for"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
main(args.board)
|
||||
@@ -0,0 +1,237 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
import dateutil.parser # type: ignore
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def run_command(command: str):
|
||||
process = subprocess.Popen(
|
||||
command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True
|
||||
)
|
||||
output, error = process.communicate()
|
||||
return output.decode("utf-8"), error.decode("utf-8")
|
||||
|
||||
|
||||
def step_back_commits(steps: int) -> bool:
|
||||
step_back_command = f"git reset --hard HEAD~{steps}"
|
||||
output, error = run_command(step_back_command)
|
||||
if error:
|
||||
print(f"Error stepping back {steps} commit(s): {error}")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def check_firmware_size(board: str) -> int:
|
||||
root_build_dir = Path(".build") / board
|
||||
build_info_json = root_build_dir / "build_info.json"
|
||||
build_info = json.loads(build_info_json.read_text())
|
||||
board_info = build_info.get(board)
|
||||
assert board_info, f"Board {board} not found in {build_info_json}"
|
||||
prog_path = Path(board_info["prog_path"])
|
||||
base_path = prog_path.parent
|
||||
suffixes = [".bin", ".hex", ".uf2"]
|
||||
firmware: Path
|
||||
for suffix in suffixes:
|
||||
firmware = base_path / f"firmware{suffix}"
|
||||
if firmware.exists():
|
||||
break
|
||||
else:
|
||||
msg = (
|
||||
", ".join([f"firmware{suffix}" for suffix in suffixes])
|
||||
+ f" not found in {base_path}"
|
||||
)
|
||||
raise FileNotFoundError(msg)
|
||||
size_command = f"du -b {firmware}"
|
||||
output, error = run_command(size_command)
|
||||
if error:
|
||||
print(f"Error checking firmware size: {error}")
|
||||
return -1
|
||||
size_in_bytes = output.strip().split()[0]
|
||||
return int(size_in_bytes)
|
||||
|
||||
|
||||
def get_commit_hash() -> str | None:
|
||||
hash_command = "git rev-parse HEAD"
|
||||
output, error = run_command(hash_command)
|
||||
if error:
|
||||
print(f"Error getting commit hash: {error}")
|
||||
return None
|
||||
return output.strip()
|
||||
|
||||
|
||||
def get_commit_date(commit_hash: str) -> str | None:
|
||||
date_command = f"git show -s --format=%ci {commit_hash}"
|
||||
output, error = run_command(date_command)
|
||||
if error:
|
||||
print(f"Error getting commit date: {error}")
|
||||
return None
|
||||
return dateutil.parser.parse(output.strip()).isoformat()
|
||||
|
||||
|
||||
def main(
|
||||
board: str,
|
||||
num_commits: int,
|
||||
skip_step: int,
|
||||
start_commit: str | None = None,
|
||||
end_commit: str | None = None,
|
||||
):
|
||||
# change to the script dir
|
||||
os.chdir(str(HERE))
|
||||
# Create tmp directory if it doesn't exist
|
||||
|
||||
if os.path.exists("tmp"):
|
||||
shutil.rmtree("tmp")
|
||||
os.makedirs("tmp", exist_ok=True)
|
||||
|
||||
# Change to the tmp directory
|
||||
os.chdir("tmp")
|
||||
|
||||
# 1. Git clone FastLED repository
|
||||
print("Cloning FastLED repository...")
|
||||
clone_command = "git clone https://github.com/FastLED/FastLED.git"
|
||||
output, error = run_command(clone_command)
|
||||
# if error:
|
||||
# print(f"Error cloning repository: {error}")
|
||||
# os.chdir("..")
|
||||
# return
|
||||
|
||||
# Change to the FastLED directory
|
||||
os.chdir("FastLED")
|
||||
|
||||
# Checkout the latest commit
|
||||
run_command("git checkout master")
|
||||
|
||||
# If end_commit is specified, checkout that commit
|
||||
# if end_commit:
|
||||
# print(f"Checking out end commit: {end_commit}")
|
||||
# checkout_command = f"git checkout {end_commit}"
|
||||
# output, error = run_command(checkout_command)
|
||||
# #if error:
|
||||
# # print(f"Error checking out end commit: {error}")
|
||||
# # return
|
||||
|
||||
# Prepare CSV file
|
||||
csv_filename = "../../firmware_sizes.csv"
|
||||
with open(csv_filename, "w", newline="") as csvfile:
|
||||
csvwriter = csv.writer(csvfile)
|
||||
csvwriter.writerow(["datetime", "commit_hash", "binary_size"])
|
||||
|
||||
commits_checked = 0
|
||||
first_iteration = True
|
||||
while True:
|
||||
current_commit = get_commit_hash()
|
||||
|
||||
if first_iteration and start_commit:
|
||||
first_iteration = False
|
||||
while True:
|
||||
if current_commit == start_commit:
|
||||
break
|
||||
if not step_back_commits(1):
|
||||
break
|
||||
current_commit = get_commit_hash()
|
||||
|
||||
if num_commits and commits_checked >= num_commits:
|
||||
print(f"Checked {num_commits} commits")
|
||||
break
|
||||
|
||||
if end_commit and current_commit == end_commit:
|
||||
print(f"Checked until end commit: {end_commit}")
|
||||
break
|
||||
|
||||
# 2. Run ci-compile.py for current commit
|
||||
print(f"\nChecking commit {commits_checked + 1}")
|
||||
|
||||
# remove .build/esp32dev/pio/build/esp32dev/ directory
|
||||
board_files = Path(".build") / board / ".pio" / "build" / board
|
||||
if board_files.exists():
|
||||
shutil.rmtree(str(board_files), ignore_errors=True)
|
||||
compile_command = f"uv run -m ci.ci-compile {board} --examples Blink"
|
||||
output, error = run_command(compile_command)
|
||||
if error:
|
||||
print(f"Error running ci-compile.py: {error}")
|
||||
if not step_back_commits(skip_step):
|
||||
break
|
||||
continue
|
||||
|
||||
# 3. Check firmware size and get commit hash
|
||||
print("Checking firmware size...")
|
||||
try:
|
||||
size = check_firmware_size(board)
|
||||
except FileNotFoundError as e:
|
||||
print(f"Error checking firmware size: {e}")
|
||||
if not step_back_commits(skip_step):
|
||||
break
|
||||
continue
|
||||
except AssertionError as e:
|
||||
print(f"Error: {e}")
|
||||
if not step_back_commits(skip_step):
|
||||
break
|
||||
continue
|
||||
commit_hash = get_commit_hash()
|
||||
if size and commit_hash:
|
||||
commit_date = get_commit_date(commit_hash)
|
||||
print(f"Firmware size: {size} bytes")
|
||||
|
||||
# Write to CSV incrementally
|
||||
with open(csv_filename, "a", newline="") as csvfile:
|
||||
csvwriter = csv.writer(csvfile)
|
||||
csvwriter.writerow([commit_date, commit_hash, size])
|
||||
|
||||
print(f"Result appended to {csv_filename}")
|
||||
|
||||
commits_checked += 1
|
||||
|
||||
# 4. Step back one commit
|
||||
print("Stepping back 1 commit...")
|
||||
if not step_back_commits(1):
|
||||
break
|
||||
|
||||
# Don't remove the tmp directory
|
||||
print("\nTemporary directory 'tmp' has been left intact for inspection.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check FastLED firmware size for multiple commits."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-commits", type=int, default=1, help="Number of commits to check"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-step",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Number of commits to skip between checks",
|
||||
)
|
||||
parser.add_argument("--start-commit", type=str, help="Starting commit hash")
|
||||
parser.add_argument("--end-commit", type=str, help="Ending commit hash")
|
||||
parser.add_argument(
|
||||
"--board", type=str, required=True, help="Board to check firmware size for"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.start_commit or args.end_commit:
|
||||
if not (args.start_commit and not args.end_commit):
|
||||
print("Both start commit and end commit must be specified.")
|
||||
exit(1)
|
||||
# if start_commit is specified, end_commit must be specified
|
||||
|
||||
num_commits = args.num_commits
|
||||
if args.start_commit and args.end_commit:
|
||||
if args.start_commit == args.end_commit:
|
||||
print("Start commit and end commit are the same.")
|
||||
exit(1)
|
||||
num_commits = 999999
|
||||
|
||||
main(args.board, num_commits, args.skip_step, args.start_commit, args.end_commit)
|
||||
@@ -0,0 +1,5 @@
|
||||
"""
|
||||
FastLED Compiler Module
|
||||
|
||||
High-performance compilation system built on proven Python compiler API.
|
||||
"""
|
||||
@@ -0,0 +1,196 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Dynamic Library Builder
|
||||
Builds FastLED as a shared/dynamic library for unit testing
|
||||
"""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ci.compiler.clang_compiler import (
|
||||
BuildFlags,
|
||||
Compiler,
|
||||
CompilerOptions,
|
||||
Result,
|
||||
)
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
|
||||
|
||||
def build_fastled_dynamic_library(build_dir: Path) -> Path:
|
||||
"""Build FastLED as a dynamic library"""
|
||||
print("Building FastLED dynamic library...")
|
||||
|
||||
# Define library path with appropriate extension
|
||||
if sys.platform == "win32":
|
||||
fastled_lib_path = build_dir / "fastled.dll"
|
||||
else:
|
||||
fastled_lib_path = build_dir / "libfastled.so"
|
||||
|
||||
# Configure compiler with shared library flags
|
||||
settings = CompilerOptions(
|
||||
include_path=str(Path(PROJECT_ROOT) / "src"),
|
||||
defines=[
|
||||
"STUB_PLATFORM",
|
||||
"ARDUINO=10808",
|
||||
"FASTLED_USE_STUB_ARDUINO",
|
||||
"SKETCH_HAS_LOTS_OF_MEMORY=1",
|
||||
"FASTLED_STUB_IMPL",
|
||||
"FASTLED_FORCE_NAMESPACE=1",
|
||||
"FASTLED_TESTING",
|
||||
"FASTLED_NO_AUTO_NAMESPACE",
|
||||
"FASTLED_NO_PINMAP",
|
||||
"HAS_HARDWARE_PIN_SUPPORT",
|
||||
# "FASTLED_DEBUG_LEVEL=1",
|
||||
# "BUILDING_FASTLED_DLL", # New define for DLL exports
|
||||
# "FASTLED_EXPORT=__declspec(dllexport)" if sys.platform == "win32" else "FASTLED_EXPORT=", # DLL exports for Windows
|
||||
],
|
||||
std_version="c++17",
|
||||
compiler="clang++",
|
||||
compiler_args=[
|
||||
# NOTE: All compiler flags should come from build_unit.toml
|
||||
# Keep only platform-specific include paths and shared library flags
|
||||
"-I" + str(Path(PROJECT_ROOT) / "src/platforms/stub"),
|
||||
"-I" + str(Path(PROJECT_ROOT) / "tests"),
|
||||
# Add shared library flags - platform specific
|
||||
"-shared" if sys.platform != "win32" else "/DLL",
|
||||
"-fPIC" if sys.platform != "win32" else "",
|
||||
],
|
||||
use_pch=True,
|
||||
parallel=True,
|
||||
)
|
||||
|
||||
# Load build flags from TOML
|
||||
build_flags_path = Path(PROJECT_ROOT) / "ci" / "build_unit.toml"
|
||||
build_flags = BuildFlags.parse(
|
||||
build_flags_path, quick_build=False, strict_mode=False
|
||||
)
|
||||
|
||||
compiler = Compiler(settings, build_flags)
|
||||
|
||||
# Compile all FastLED source files
|
||||
fastled_src_dir = Path(PROJECT_ROOT) / "src"
|
||||
all_cpp_files = list(fastled_src_dir.rglob("*.cpp"))
|
||||
|
||||
print(f"Found {len(all_cpp_files)} FastLED source files")
|
||||
|
||||
# Compile to object files
|
||||
object_files: List[Path] = []
|
||||
for src_file in all_cpp_files:
|
||||
relative_path = src_file.relative_to(fastled_src_dir)
|
||||
safe_name = (
|
||||
str(relative_path.with_suffix("")).replace("/", "_").replace("\\", "_")
|
||||
)
|
||||
obj_path = build_dir / f"{safe_name}_fastled.o"
|
||||
|
||||
future = compiler.compile_cpp_file(
|
||||
src_file,
|
||||
output_path=obj_path,
|
||||
additional_flags=[
|
||||
"-c",
|
||||
"-DFASTLED_STUB_IMPL",
|
||||
"-DFASTLED_FORCE_NAMESPACE=1",
|
||||
"-DFASTLED_NO_AUTO_NAMESPACE",
|
||||
"-DFASTLED_NO_PINMAP",
|
||||
"-DPROGMEM=",
|
||||
"-DHAS_HARDWARE_PIN_SUPPORT",
|
||||
"-DFASTLED_ENABLE_JSON=1",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
"-fPIC", # Position Independent Code for shared library
|
||||
],
|
||||
)
|
||||
result: Result = future.result()
|
||||
if not result.ok:
|
||||
print(f"ERROR: Failed to compile {src_file}: {result.stderr}")
|
||||
continue
|
||||
|
||||
object_files.append(obj_path)
|
||||
|
||||
print(f"Linking {len(object_files)} object files into dynamic library...")
|
||||
|
||||
# Link as shared library
|
||||
if sys.platform == "win32":
|
||||
# Use lld-link for Windows DLL creation
|
||||
link_cmd = [
|
||||
"lld-link",
|
||||
"/DLL",
|
||||
"/NOLOGO",
|
||||
"/OUT:" + str(fastled_lib_path),
|
||||
"/LIBPATH:C:/Program Files (x86)/Windows Kits/10/Lib/10.0.19041.0/um/x64",
|
||||
"/LIBPATH:C:/Program Files (x86)/Windows Kits/10/Lib/10.0.19041.0/ucrt/x64",
|
||||
"/LIBPATH:C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/lib/x64",
|
||||
*[str(obj) for obj in object_files],
|
||||
"msvcrt.lib",
|
||||
"legacy_stdio_definitions.lib",
|
||||
"kernel32.lib",
|
||||
"user32.lib",
|
||||
]
|
||||
else:
|
||||
link_cmd = [
|
||||
"clang++",
|
||||
"-shared",
|
||||
"-o",
|
||||
str(fastled_lib_path),
|
||||
*[str(obj) for obj in object_files],
|
||||
]
|
||||
|
||||
try:
|
||||
# Use streaming to prevent buffer overflow
|
||||
process = subprocess.Popen(
|
||||
link_cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
bufsize=1,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
|
||||
stdout_lines: list[str] = []
|
||||
stderr_lines: list[str] = []
|
||||
|
||||
while True:
|
||||
stdout_line = process.stdout.readline() if process.stdout else ""
|
||||
stderr_line = process.stderr.readline() if process.stderr else ""
|
||||
|
||||
if stdout_line:
|
||||
stdout_lines.append(stdout_line.rstrip())
|
||||
if stderr_line:
|
||||
stderr_lines.append(stderr_line.rstrip())
|
||||
|
||||
if process.poll() is not None:
|
||||
remaining_stdout = process.stdout.read() if process.stdout else ""
|
||||
remaining_stderr = process.stderr.read() if process.stderr else ""
|
||||
|
||||
if remaining_stdout:
|
||||
for line in remaining_stdout.splitlines():
|
||||
stdout_lines.append(line.rstrip())
|
||||
if remaining_stderr:
|
||||
for line in remaining_stderr.splitlines():
|
||||
stderr_lines.append(line.rstrip())
|
||||
break
|
||||
|
||||
stderr_result = "\n".join(stderr_lines)
|
||||
if process.returncode != 0:
|
||||
raise Exception(f"Failed to create dynamic library: {stderr_result}")
|
||||
|
||||
print(f"Successfully created dynamic library: {fastled_lib_path}")
|
||||
return fastled_lib_path
|
||||
|
||||
except Exception as e:
|
||||
print(f"ERROR: Exception during library creation: {e}")
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Create build directory in temp
|
||||
import tempfile
|
||||
|
||||
build_dir = Path(tempfile.gettempdir()) / "fastled_test_build"
|
||||
build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Build the dynamic library
|
||||
build_fastled_dynamic_library(build_dir)
|
||||
@@ -0,0 +1,273 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Clean Cache-Enhanced Compilation
|
||||
|
||||
Simple wrapper that adds fingerprint cache to FastLED compilation.
|
||||
Dramatically speeds up incremental builds by skipping unchanged files.
|
||||
"""
|
||||
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Optional, Set
|
||||
|
||||
from ci.ci.fingerprint_cache import FingerprintCache
|
||||
from ci.compiler.clang_compiler import Compiler
|
||||
from ci.compiler.test_example_compilation import CompilationResult
|
||||
|
||||
|
||||
class CacheAwareCompiler:
|
||||
"""Simple wrapper that adds cache checking to compilation."""
|
||||
|
||||
def __init__(self, compiler: Compiler, cache_file: Path, verbose: bool = False):
|
||||
self.compiler = compiler
|
||||
self.cache = FingerprintCache(cache_file)
|
||||
self.verbose = verbose
|
||||
self.stats = {
|
||||
"files_checked": 0,
|
||||
"files_skipped": 0,
|
||||
"files_compiled": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0,
|
||||
}
|
||||
|
||||
def should_compile(self, file_path: Path, baseline_time: float) -> bool:
|
||||
"""Check if file needs compilation using cache."""
|
||||
self.stats["files_checked"] += 1
|
||||
|
||||
try:
|
||||
needs_compile = self.cache.has_changed(file_path, baseline_time)
|
||||
|
||||
if needs_compile:
|
||||
self.stats["cache_misses"] += 1
|
||||
self.stats["files_compiled"] += 1
|
||||
return True
|
||||
else:
|
||||
self.stats["cache_hits"] += 1
|
||||
self.stats["files_skipped"] += 1
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Skipping unchanged: {file_path.name}")
|
||||
return False
|
||||
except KeyboardInterrupt:
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Error checking {file_path.name}: {e}")
|
||||
self.stats["cache_misses"] += 1
|
||||
self.stats["files_compiled"] += 1
|
||||
return True
|
||||
|
||||
def _headers_changed(self, baseline_time: float) -> bool:
|
||||
"""Detect if any relevant header dependency has changed since last run.
|
||||
|
||||
Uses the compiler's PCH dependency discovery to collect a conservative
|
||||
set of headers that impact example compilation, then queries the
|
||||
fingerprint cache to see if any actually changed content.
|
||||
"""
|
||||
try:
|
||||
# Reuse the compiler's dependency discovery (covers src/** and platforms/**)
|
||||
dependencies: List[Path] = []
|
||||
if hasattr(self.compiler, "_get_pch_dependencies"):
|
||||
dependencies = self.compiler._get_pch_dependencies() # type: ignore[attr-defined]
|
||||
else:
|
||||
# Fallback: hash all headers under the compiler's include path
|
||||
include_root = Path(self.compiler.settings.include_path)
|
||||
for pattern in ("**/*.h", "**/*.hpp"):
|
||||
dependencies.extend(include_root.glob(pattern))
|
||||
|
||||
changed_any = False
|
||||
for dep in dependencies:
|
||||
try:
|
||||
if self.cache.has_changed(dep, baseline_time):
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Header changed: {dep}")
|
||||
changed_any = True
|
||||
break
|
||||
except FileNotFoundError:
|
||||
# Missing dependency means we must rebuild conservatively
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Header missing (forces rebuild): {dep}")
|
||||
changed_any = True
|
||||
break
|
||||
return changed_any
|
||||
except KeyboardInterrupt:
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Header scan failed, forcing rebuild: {e}")
|
||||
return True
|
||||
|
||||
def compile_with_cache(
|
||||
self,
|
||||
ino_files: List[Path],
|
||||
pch_compatible_files: Set[Path],
|
||||
log_fn: Callable[[str], None],
|
||||
full_compilation: bool,
|
||||
verbose: bool = False,
|
||||
baseline_time: Optional[float] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Compile only files that have changed."""
|
||||
|
||||
if baseline_time is None:
|
||||
baseline_time = time.time() - 3600 # Default: 1 hour ago
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# If any header dependency changed, conservatively rebuild all example files
|
||||
force_recompile_due_to_headers = self._headers_changed(baseline_time)
|
||||
if force_recompile_due_to_headers and self.verbose:
|
||||
print(
|
||||
"[CACHE] Header dependency changes detected - forcing recompilation of example sources"
|
||||
)
|
||||
|
||||
# Check which files need compilation
|
||||
files_to_compile: List[Path] = []
|
||||
for ino_file in ino_files:
|
||||
if force_recompile_due_to_headers or self.should_compile(
|
||||
ino_file, baseline_time
|
||||
):
|
||||
files_to_compile.append(ino_file)
|
||||
|
||||
# Check .cpp files too
|
||||
cpp_files_to_compile: List[Path] = []
|
||||
for ino_file in ino_files:
|
||||
cpp_files = self.compiler.find_cpp_files_for_example(ino_file)
|
||||
for cpp_file in cpp_files:
|
||||
if force_recompile_due_to_headers or self.should_compile(
|
||||
cpp_file, baseline_time
|
||||
):
|
||||
cpp_files_to_compile.append(cpp_file)
|
||||
|
||||
# Log cache results
|
||||
total_files = len(ino_files) + sum(
|
||||
len(self.compiler.find_cpp_files_for_example(f)) for f in ino_files
|
||||
)
|
||||
log_fn(
|
||||
f"[CACHE] {self.stats['files_skipped']}/{total_files} files unchanged, {self.stats['files_compiled']}/{total_files} need compilation"
|
||||
)
|
||||
|
||||
if self.stats["files_skipped"] > 0:
|
||||
time_saved = self.stats["files_skipped"] * 0.5 # Estimate
|
||||
log_fn(f"[CACHE] Estimated time saved: {time_saved:.1f}s")
|
||||
|
||||
# Compile files that changed
|
||||
if files_to_compile or cpp_files_to_compile:
|
||||
result = self._run_actual_compilation(
|
||||
files_to_compile,
|
||||
cpp_files_to_compile,
|
||||
pch_compatible_files,
|
||||
log_fn,
|
||||
full_compilation,
|
||||
verbose,
|
||||
)
|
||||
else:
|
||||
log_fn("[CACHE] All files unchanged - no compilation needed!")
|
||||
result = self._create_success_result(len(ino_files), ino_files)
|
||||
|
||||
# Log final stats
|
||||
if total_files > 0:
|
||||
hit_rate = (
|
||||
(
|
||||
self.stats["cache_hits"]
|
||||
/ (self.stats["cache_hits"] + self.stats["cache_misses"])
|
||||
* 100
|
||||
)
|
||||
if (self.stats["cache_hits"] + self.stats["cache_misses"]) > 0
|
||||
else 0
|
||||
)
|
||||
skip_rate = self.stats["files_skipped"] / total_files * 100
|
||||
log_fn(
|
||||
f"[CACHE] {hit_rate:.1f}% cache hit rate, {skip_rate:.1f}% files skipped"
|
||||
)
|
||||
|
||||
return {
|
||||
"compilation_result": result,
|
||||
"cache_stats": self.stats,
|
||||
"total_time": time.time() - start_time,
|
||||
}
|
||||
|
||||
def _run_actual_compilation(
|
||||
self,
|
||||
ino_files: List[Path],
|
||||
cpp_files_to_compile: List[Path],
|
||||
pch_compatible_files: Set[Path],
|
||||
log_fn: Callable[[str], None],
|
||||
full_compilation: bool,
|
||||
verbose: bool,
|
||||
):
|
||||
"""Run the actual compilation for changed files."""
|
||||
from ci.compiler.test_example_compilation import compile_examples_simple
|
||||
|
||||
# Temporarily filter .cpp files to only compile changed ones
|
||||
original_method = self.compiler.find_cpp_files_for_example
|
||||
|
||||
def filtered_cpp_files(ino_file: Path) -> List[Path]:
|
||||
all_cpp = original_method(ino_file)
|
||||
return [cpp for cpp in all_cpp if cpp in cpp_files_to_compile]
|
||||
|
||||
self.compiler.find_cpp_files_for_example = filtered_cpp_files
|
||||
|
||||
try:
|
||||
return compile_examples_simple(
|
||||
self.compiler,
|
||||
ino_files,
|
||||
pch_compatible_files,
|
||||
log_fn,
|
||||
full_compilation,
|
||||
verbose,
|
||||
)
|
||||
finally:
|
||||
self.compiler.find_cpp_files_for_example = original_method
|
||||
|
||||
def _create_success_result(
|
||||
self, file_count: int, ino_files: Optional[List[Path]] = None
|
||||
) -> CompilationResult:
|
||||
"""Create a successful compilation result for cached files."""
|
||||
|
||||
# For cached files, we need to populate object_file_map with existing object files
|
||||
# so that linking can proceed properly
|
||||
object_file_map: Dict[Path, List[Path]] = {}
|
||||
if ino_files:
|
||||
for ino_file in ino_files:
|
||||
# Find existing object files for this example
|
||||
example_name = ino_file.parent.name
|
||||
build_dir = Path(".build/examples") / example_name
|
||||
|
||||
obj_files: List[Path] = []
|
||||
if build_dir.exists():
|
||||
# Look for the main .ino object file
|
||||
ino_obj = build_dir / f"{ino_file.stem}.o"
|
||||
if ino_obj.exists():
|
||||
obj_files.append(ino_obj)
|
||||
|
||||
# Look for additional .cpp files in the same directory as the .ino
|
||||
cpp_files = self.compiler.find_cpp_files_for_example(ino_file)
|
||||
for cpp_file in cpp_files:
|
||||
cpp_obj = build_dir / f"{cpp_file.stem}.o"
|
||||
if cpp_obj.exists():
|
||||
obj_files.append(cpp_obj)
|
||||
|
||||
if obj_files:
|
||||
object_file_map[ino_file] = obj_files
|
||||
|
||||
return CompilationResult(
|
||||
successful_count=file_count,
|
||||
failed_count=0,
|
||||
compile_time=0.0,
|
||||
failed_examples=[],
|
||||
object_file_map=object_file_map,
|
||||
)
|
||||
|
||||
|
||||
def create_cache_compiler(
|
||||
compiler: Compiler, cache_file: Optional[Path] = None, verbose: bool = False
|
||||
) -> CacheAwareCompiler:
|
||||
"""Create a cache-aware compiler. Simple factory function."""
|
||||
if cache_file is None:
|
||||
cache_file = Path(".build/fingerprint_cache.json")
|
||||
return CacheAwareCompiler(compiler, cache_file, verbose)
|
||||
@@ -0,0 +1,514 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Standalone cache setup script for PlatformIO builds with Python fake compilers.
|
||||
This script is executed by PlatformIO as a post-build extra_script.
|
||||
|
||||
Configuration is passed through environment variables to avoid template string issues.
|
||||
"""
|
||||
|
||||
# ruff: noqa: F405, F821 # Suppress SCons-specific import and undefined name warnings
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
|
||||
# Import env and try to import projenv (SCons-specific imports)
|
||||
# These are dynamically available in PlatformIO/SCons environment
|
||||
try:
|
||||
Import("env") # type: ignore[name-defined] # SCons-specific import
|
||||
env: Any # SCons environment object
|
||||
except NameError:
|
||||
env = None # For type checking when not in SCons context
|
||||
|
||||
_VERBOSE = os.environ.get("VERBOSE", "0") in ("1", "true", "True", "yes", "y")
|
||||
|
||||
# Try to import projenv if it exists
|
||||
has_projenv: bool = False
|
||||
projenv: Optional[Any] = None
|
||||
try:
|
||||
Import("projenv") # type: ignore[name-defined] # SCons-specific import
|
||||
# projenv is now available in scope from Import
|
||||
has_projenv = True
|
||||
except (NameError, Exception):
|
||||
has_projenv = False
|
||||
projenv = None
|
||||
|
||||
# Import cached compiler functions with type stubs
|
||||
create_cached_toolchain: Optional[Any] = None
|
||||
get_platform_packages_paths: Optional[Any] = None
|
||||
cached_compiler_available: bool = False
|
||||
|
||||
try:
|
||||
from ci.util.cached_compiler import (
|
||||
create_cached_toolchain,
|
||||
get_platform_packages_paths,
|
||||
)
|
||||
|
||||
cached_compiler_available = True
|
||||
except ImportError as e:
|
||||
print("WARNING: Could not import cached compiler module: " + str(e))
|
||||
cached_compiler_available = False
|
||||
|
||||
# Debug: Dump the environment state to disk for inspection
|
||||
env_dump: Dict[str, str] = {}
|
||||
try:
|
||||
if env is not None and hasattr(env, "Dictionary"): # type: ignore[has-type]
|
||||
for key in env.Dictionary(): # type: ignore[union-attr]
|
||||
try:
|
||||
value = env[key] # type: ignore[index]
|
||||
# Convert to string to avoid JSON serialization issues
|
||||
env_dump[key] = str(value) # type: ignore[arg-type]
|
||||
except Exception:
|
||||
env_dump[key] = "<error getting value>"
|
||||
else:
|
||||
env_dump["error"] = "env not available or missing Dictionary method"
|
||||
except Exception as e:
|
||||
env_dump["error"] = f"Failed to access env: {e}"
|
||||
|
||||
# Write environment dump to disk
|
||||
env_dump_path = "env_dump.json"
|
||||
with open(env_dump_path, "w") as f:
|
||||
json.dump(env_dump, f, indent=2)
|
||||
print("Environment state dumped to: " + env_dump_path)
|
||||
|
||||
# Also dump projenv if available
|
||||
if has_projenv and projenv is not None:
|
||||
projenv_dump: Dict[str, str] = {}
|
||||
for key in projenv.Dictionary(): # type: ignore
|
||||
try:
|
||||
value = projenv[key] # type: ignore
|
||||
projenv_dump[key] = str(value)
|
||||
except Exception:
|
||||
projenv_dump[key] = "<error getting value>"
|
||||
|
||||
projenv_dump_path = "projenv_dump.json"
|
||||
with open(projenv_dump_path, "w") as f:
|
||||
json.dump(projenv_dump, f, indent=2)
|
||||
print("Projenv state dumped to: " + projenv_dump_path)
|
||||
|
||||
# Read cache configuration from environment variables
|
||||
cache_type = os.environ.get("FASTLED_CACHE_TYPE", "no_cache")
|
||||
cache_executable = os.environ.get("FASTLED_CACHE_EXECUTABLE", "")
|
||||
sccache_path = os.environ.get("FASTLED_SCCACHE_PATH", "")
|
||||
sccache_dir = os.environ.get("FASTLED_SCCACHE_DIR", "")
|
||||
sccache_cache_size = os.environ.get("FASTLED_SCCACHE_CACHE_SIZE", "2G")
|
||||
xcache_path = os.environ.get("FASTLED_XCACHE_PATH", "")
|
||||
debug_enabled = os.environ.get("FASTLED_CACHE_DEBUG", "0") == "1"
|
||||
|
||||
print("Cache configuration from environment:")
|
||||
print(" Cache type: " + cache_type)
|
||||
print(" Cache executable: " + cache_executable)
|
||||
print(" SCCACHE path: " + sccache_path)
|
||||
print(" SCCACHE dir: " + sccache_dir)
|
||||
print(" Debug enabled: " + str(debug_enabled))
|
||||
|
||||
# Set up cache environment variables for subprocess execution
|
||||
try:
|
||||
if env is not None and hasattr(env, "Append"): # type: ignore[has-type]
|
||||
if sccache_dir:
|
||||
env.Append(ENV={"SCCACHE_DIR": sccache_dir}) # type: ignore[union-attr]
|
||||
os.environ["SCCACHE_DIR"] = sccache_dir
|
||||
|
||||
if sccache_cache_size:
|
||||
env.Append(ENV={"SCCACHE_CACHE_SIZE": sccache_cache_size}) # type: ignore[union-attr]
|
||||
os.environ["SCCACHE_CACHE_SIZE"] = sccache_cache_size
|
||||
|
||||
# Ensure sccache binary directory is on PATH so xcache can find it even
|
||||
# inside ESP-IDF's virtual environments where PATH is heavily modified.
|
||||
if sccache_path:
|
||||
sccache_dir_path = str(Path(sccache_path).parent)
|
||||
env.PrependENVPath("PATH", sccache_dir_path) # type: ignore[union-attr]
|
||||
if has_projenv and projenv is not None and hasattr(projenv, "PrependENVPath"):
|
||||
projenv.PrependENVPath("PATH", sccache_dir_path) # type: ignore[union-attr]
|
||||
os.environ["PATH"] = sccache_dir_path + os.pathsep + os.environ.get("PATH", "")
|
||||
|
||||
if debug_enabled:
|
||||
env.Append(ENV={"XCACHE_DEBUG": "1", "SCCACHE_DEBUG": "1"}) # type: ignore[union-attr]
|
||||
if has_projenv and projenv is not None and hasattr(projenv, "Append"):
|
||||
projenv.Append(ENV={"XCACHE_DEBUG": "1", "SCCACHE_DEBUG": "1"}) # type: ignore[union-attr]
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to set up cache environment: {e}")
|
||||
|
||||
# Check if cache is available and cached compiler system can be used
|
||||
USE_CACHE = False
|
||||
|
||||
if cached_compiler_available and cache_executable:
|
||||
if cache_type == "xcache":
|
||||
# For xcache, check if the Python script exists and sccache is available
|
||||
USE_CACHE = (
|
||||
cache_executable
|
||||
and xcache_path
|
||||
and Path(xcache_path).exists()
|
||||
and sccache_path
|
||||
and shutil.which(sccache_path)
|
||||
)
|
||||
|
||||
if USE_CACHE:
|
||||
print("xcache wrapper detected and configured for Python fake compilers")
|
||||
print(" xcache path: " + str(xcache_path))
|
||||
print(" cache executable: " + str(cache_executable))
|
||||
else:
|
||||
# For sccache/ccache, check if executable is in PATH
|
||||
USE_CACHE = shutil.which(cache_executable) is not None
|
||||
|
||||
if USE_CACHE:
|
||||
print(
|
||||
str(cache_type) + " detected and configured for Python cached compilers"
|
||||
)
|
||||
print(" cache executable: " + str(cache_executable))
|
||||
elif not cached_compiler_available:
|
||||
print(
|
||||
"WARNING: Python cached compiler system not available, cache will be disabled"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
"Cache executable not found: "
|
||||
+ str(cache_executable)
|
||||
+ ", cache will be disabled"
|
||||
)
|
||||
|
||||
if USE_CACHE and env is not None and hasattr(env, "get"): # type: ignore[has-type]
|
||||
# Get current compilers from environment
|
||||
original_cc = env.get("CC") # type: ignore[union-attr]
|
||||
original_cxx = env.get("CXX") # type: ignore[union-attr]
|
||||
|
||||
print("DEBUG: Found compilers in env:")
|
||||
# Use repr for safer type conversion
|
||||
cc_str = repr(original_cc)
|
||||
cxx_str = repr(original_cxx)
|
||||
cc_type_str = type(original_cc).__name__
|
||||
cxx_type_str = type(original_cxx).__name__
|
||||
print(" CC: " + cc_str + " (type: " + cc_type_str + ")")
|
||||
print(" CXX: " + cxx_str + " (type: " + cxx_type_str + ")")
|
||||
|
||||
# Extract compiler information for fake compiler generation
|
||||
def extract_compiler_info(compiler_env_var: Any) -> Optional[str]:
|
||||
"""Extract compiler name from environment variable value."""
|
||||
if not compiler_env_var:
|
||||
return None
|
||||
|
||||
if isinstance(compiler_env_var, list):
|
||||
return str(compiler_env_var[0]) if compiler_env_var else None # type: ignore[arg-type]
|
||||
else:
|
||||
# Handle string values like "arm-none-eabi-gcc" or "gcc"
|
||||
return str(compiler_env_var).split()[0] # type: ignore[arg-type]
|
||||
|
||||
cc_name = extract_compiler_info(original_cc) or "gcc"
|
||||
cxx_name = extract_compiler_info(original_cxx) or "g++"
|
||||
|
||||
print("Extracted compiler names:")
|
||||
print(" CC name: " + str(cc_name))
|
||||
print(" CXX name: " + str(cxx_name))
|
||||
|
||||
# Create toolchain info for fake compiler generation
|
||||
toolchain_info: Dict[str, str] = {
|
||||
"CC": cc_name,
|
||||
"CXX": cxx_name,
|
||||
}
|
||||
|
||||
# Create cache config for fake compiler system
|
||||
cache_config: Dict[str, str] = {
|
||||
"CACHE_TYPE": cache_type,
|
||||
"CACHE_EXECUTABLE": cache_executable,
|
||||
"SCCACHE_PATH": sccache_path,
|
||||
"SCCACHE_DIR": sccache_dir,
|
||||
"XCACHE_PATH": xcache_path,
|
||||
}
|
||||
|
||||
# Check if toolset exists and is valid
|
||||
# Create a cache key based on the configuration to invalidate cache when config changes
|
||||
cache_key = f"{cc_name}_{cxx_name}_{cache_type}_{cache_executable}"
|
||||
|
||||
# Cache in the local build directory (e.g., .build/pio/uno/)
|
||||
cache_file = Path(current_dir) / "compiler_cache.json"
|
||||
|
||||
cached_tools: Optional[Dict[str, str]] = None
|
||||
|
||||
# Try to load from persistent cache file
|
||||
if cache_file.exists():
|
||||
try:
|
||||
import json
|
||||
|
||||
with open(cache_file, "r") as f:
|
||||
cache_data = json.load(f)
|
||||
|
||||
cached_real_cc = cache_data.get("real_cc")
|
||||
cached_real_cxx = cache_data.get("real_cxx")
|
||||
cached_fake_cc = cache_data.get("fake_cc")
|
||||
cached_fake_cxx = cache_data.get("fake_cxx")
|
||||
|
||||
if (
|
||||
cached_real_cc
|
||||
and cached_real_cxx
|
||||
and cached_fake_cc
|
||||
and cached_fake_cxx
|
||||
):
|
||||
print("Found local compiler cache:")
|
||||
print(f" Cache file: {cache_file}")
|
||||
print(f" Real CC: {cached_real_cc}")
|
||||
print(f" Real CXX: {cached_real_cxx}")
|
||||
|
||||
# Check if fake compiler scripts still exist, recreate if needed
|
||||
fake_cc_path = Path(cached_fake_cc.replace("python ", ""))
|
||||
fake_cxx_path = Path(cached_fake_cxx.replace("python ", ""))
|
||||
|
||||
if fake_cc_path.exists() and fake_cxx_path.exists():
|
||||
# Fast path: use existing cached compilers
|
||||
# On Windows, ensure we have .cmd shims to avoid 'python script.py @file'
|
||||
# causing Python to treat @file as the script name.
|
||||
import sys as _sys
|
||||
import os as _os
|
||||
|
||||
is_win = (
|
||||
_os.name == "nt"
|
||||
or _sys.platform.startswith("win")
|
||||
or _sys.platform.startswith("cygwin")
|
||||
or _sys.platform.startswith("msys")
|
||||
)
|
||||
|
||||
def _ensure_cmd_wrapper(fake_str: str, real_path: str) -> str:
|
||||
p = Path(fake_str.replace("python ", ""))
|
||||
if is_win and p.suffix.lower() == ".py":
|
||||
cmd = p.with_suffix(".cmd")
|
||||
if not cmd.exists():
|
||||
# Create missing .cmd wrapper now
|
||||
from ci.util.cached_compiler import create_cached_compiler_script
|
||||
out_dir = p.parent
|
||||
created = create_cached_compiler_script(
|
||||
compiler_name=p.stem.split("cached_")[-1],
|
||||
cache_executable=cache_config.get("CACHE_EXECUTABLE", "sccache"),
|
||||
real_compiler_path=real_path,
|
||||
output_dir=out_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
cmd = Path(str(created).replace("python ", ""))
|
||||
if cmd.exists():
|
||||
return str(cmd)
|
||||
return fake_str
|
||||
|
||||
# Create/choose proper wrapper paths
|
||||
new_cc = _ensure_cmd_wrapper(cached_fake_cc, cached_real_cc)
|
||||
new_cxx = _ensure_cmd_wrapper(cached_fake_cxx, cached_real_cxx)
|
||||
|
||||
cached_tools = {"CC": new_cc, "CXX": new_cxx}
|
||||
|
||||
# Update cache file if we switched to .cmd
|
||||
try:
|
||||
if new_cc != cached_fake_cc or new_cxx != cached_fake_cxx:
|
||||
cache_data["fake_cc"] = new_cc
|
||||
cache_data["fake_cxx"] = new_cxx
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump(cache_data, f, indent=2)
|
||||
except KeyboardInterrupt as ke:
|
||||
import _thread
|
||||
_thread.interrupt_main()
|
||||
raise ke
|
||||
except Exception:
|
||||
pass
|
||||
print(
|
||||
"SUCCESS: Using cached compilers (instant, no platform search needed):"
|
||||
)
|
||||
print(f" CC: {cached_tools['CC']}")
|
||||
print(f" CXX: {cached_tools['CXX']}")
|
||||
print(" Platform search skipped - using cached toolset")
|
||||
else:
|
||||
print(
|
||||
"Cached compiler scripts missing, recreating with cached real paths..."
|
||||
)
|
||||
# Recreate cached compilers using cached real paths (fast)
|
||||
cached_compilers_dir = Path(current_dir) / "cached_compilers"
|
||||
cached_compilers_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
from ci.util.cached_compiler import create_cached_compiler_script
|
||||
|
||||
cache_executable = cache_config.get("CACHE_EXECUTABLE", "sccache")
|
||||
|
||||
# Create cached CC script using cached real path
|
||||
cached_cc_script = create_cached_compiler_script(
|
||||
compiler_name="CC",
|
||||
cache_executable=cache_executable,
|
||||
real_compiler_path=cached_real_cc,
|
||||
output_dir=cached_compilers_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
|
||||
# Create cached CXX script using cached real path
|
||||
cached_cxx_script = create_cached_compiler_script(
|
||||
compiler_name="CXX",
|
||||
cache_executable=cache_executable,
|
||||
real_compiler_path=cached_real_cxx,
|
||||
output_dir=cached_compilers_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
|
||||
cached_tools = {
|
||||
"CC": str(cached_cc_script),
|
||||
"CXX": str(cached_cxx_script),
|
||||
}
|
||||
|
||||
# Update cache file with new script paths
|
||||
cache_data["fake_cc"] = cached_tools["CC"]
|
||||
cache_data["fake_cxx"] = cached_tools["CXX"]
|
||||
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump(cache_data, f, indent=2)
|
||||
|
||||
print("Recreated cached compilers using cached real paths:")
|
||||
print(f" CC: {cached_tools['CC']}")
|
||||
print(f" CXX: {cached_tools['CXX']}")
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to load cache file {cache_file}: {e}")
|
||||
# Fall through to full recreation
|
||||
|
||||
# If no valid cache found, create the toolset from scratch
|
||||
if cached_tools is None:
|
||||
print("No valid cache found, creating compiler toolset from scratch...")
|
||||
print(" This is the first compile or configuration changed")
|
||||
|
||||
# Get platform packages paths for toolchain resolution (expensive operation)
|
||||
platform_packages: List[str] = []
|
||||
if get_platform_packages_paths is not None:
|
||||
print("Searching platform packages (this may take ~10 seconds)...")
|
||||
platform_packages = get_platform_packages_paths()
|
||||
print(f"Found {len(platform_packages)} platform package directories")
|
||||
|
||||
if create_cached_toolchain is not None:
|
||||
try:
|
||||
# Find the real compiler paths (expensive operation, done only once)
|
||||
from ci.util.cached_compiler import find_toolchain_compiler
|
||||
|
||||
print("Resolving real compiler paths...")
|
||||
|
||||
real_cc_path = find_toolchain_compiler(cc_name, platform_packages)
|
||||
real_cxx_path = find_toolchain_compiler(cxx_name, platform_packages)
|
||||
|
||||
if not real_cc_path or not real_cxx_path:
|
||||
print(f"ERROR: Could not find real compilers:")
|
||||
print(f" CC '{cc_name}': {real_cc_path}")
|
||||
print(f" CXX '{cxx_name}': {real_cxx_path}")
|
||||
cached_tools = None
|
||||
else:
|
||||
print(f"Found real compilers:")
|
||||
print(f" Real CC: {real_cc_path}")
|
||||
print(f" Real CXX: {real_cxx_path}")
|
||||
|
||||
# Create cached compiler scripts
|
||||
cached_compilers_dir = Path(current_dir) / "cached_compilers"
|
||||
cached_compilers_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
from ci.util.cached_compiler import create_cached_compiler_script
|
||||
|
||||
cache_executable = cache_config.get("CACHE_EXECUTABLE", "sccache")
|
||||
|
||||
# Create cached CC script
|
||||
cached_cc_script = create_cached_compiler_script(
|
||||
compiler_name="CC",
|
||||
cache_executable=cache_executable,
|
||||
real_compiler_path=real_cc_path,
|
||||
output_dir=cached_compilers_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
|
||||
# Create cached CXX script
|
||||
cached_cxx_script = create_cached_compiler_script(
|
||||
compiler_name="CXX",
|
||||
cache_executable=cache_executable,
|
||||
real_compiler_path=real_cxx_path,
|
||||
output_dir=cached_compilers_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
|
||||
cached_tools = {
|
||||
"CC": str(cached_cc_script),
|
||||
"CXX": str(cached_cxx_script),
|
||||
}
|
||||
|
||||
print("Created new compiler toolset:")
|
||||
print(f" CC: {cached_tools['CC']}")
|
||||
print(f" CXX: {cached_tools['CXX']}")
|
||||
|
||||
# Save to local build directory cache file
|
||||
cache_data = {
|
||||
"cache_key": cache_key,
|
||||
"real_cc": real_cc_path,
|
||||
"real_cxx": real_cxx_path,
|
||||
"fake_cc": cached_tools["CC"],
|
||||
"fake_cxx": cached_tools["CXX"],
|
||||
"build_dir": str(current_dir),
|
||||
"platform_packages_count": len(platform_packages),
|
||||
}
|
||||
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump(cache_data, f, indent=2)
|
||||
|
||||
print(f"Saved compiler toolset to local cache: {cache_file}")
|
||||
print(" This cache will persist across builds for this platform")
|
||||
|
||||
except Exception as e:
|
||||
print("ERROR: Toolset creation failed with exception: " + str(e))
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
cached_tools = None
|
||||
else:
|
||||
print("ERROR: create_cached_toolchain function is None")
|
||||
|
||||
if cached_tools:
|
||||
# Use Python cached compilers instead of batch scripts
|
||||
new_cc = cached_tools.get("CC")
|
||||
new_cxx = cached_tools.get("CXX")
|
||||
|
||||
if new_cc and new_cxx:
|
||||
print("Created Python cached compilers:")
|
||||
print(" CC: " + str(new_cc))
|
||||
print(" CXX: " + str(new_cxx))
|
||||
|
||||
# Apply to both environments
|
||||
env.Replace(CC=new_cc, CXX=new_cxx) # type: ignore
|
||||
if has_projenv and projenv is not None:
|
||||
projenv.Replace(CC=new_cc, CXX=new_cxx) # type: ignore
|
||||
print("Applied Python fake compilers to both env and projenv")
|
||||
else:
|
||||
print("Applied Python fake compilers to env (projenv not available)")
|
||||
|
||||
# Apply to library builders (critical for framework caching)
|
||||
try:
|
||||
for lib_builder in env.GetLibBuilders(): # type: ignore
|
||||
lib_builder.env.Replace(CC=new_cc, CXX=new_cxx) # type: ignore
|
||||
if _VERBOSE:
|
||||
print(
|
||||
"Applied Python fake compilers to library builder: "
|
||||
+ str(getattr(lib_builder, "name", "unnamed"))
|
||||
)
|
||||
except KeyboardInterrupt:
|
||||
import _thread
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
print("WARNING: Could not apply to library builders: " + str(e))
|
||||
|
||||
print("Python fake compiler cache enabled: " + str(cache_type))
|
||||
print(" Original CC: " + str(original_cc))
|
||||
print(" Original CXX: " + str(original_cxx))
|
||||
print(" Fake CC: " + str(new_cc))
|
||||
print(" Fake CXX: " + str(new_cxx))
|
||||
else:
|
||||
print(
|
||||
"ERROR: Failed to create Python fake compilers, falling back to no cache"
|
||||
)
|
||||
USE_CACHE = False
|
||||
else:
|
||||
print("ERROR: Python fake compiler creation failed, falling back to no cache")
|
||||
USE_CACHE = False
|
||||
|
||||
if not USE_CACHE:
|
||||
if cache_executable:
|
||||
print("Warning: " + str(cache_type) + " setup failed; using default compilers")
|
||||
else:
|
||||
print("No cache executable configured; using default compilers")
|
||||
|
||||
print("Python fake compiler cache environment configured successfully")
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,261 @@
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
Compilation support for individual boards.
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
from threading import Lock
|
||||
from typing import List
|
||||
|
||||
from ci.boards import Board # type: ignore
|
||||
from ci.util.locked_print import locked_print
|
||||
|
||||
|
||||
ERROR_HAPPENED = False
|
||||
|
||||
|
||||
IS_GITHUB = "GITHUB_ACTIONS" in os.environ
|
||||
FIRST_BUILD_LOCK = Lock()
|
||||
USE_FIRST_BUILD_LOCK = IS_GITHUB
|
||||
|
||||
|
||||
def errors_happened() -> bool:
|
||||
"""Return whether any errors happened during the build."""
|
||||
return ERROR_HAPPENED
|
||||
|
||||
|
||||
def _fastled_js_is_parent_directory(p: Path) -> bool:
|
||||
"""Check if fastled_js is a parent directory of the given path."""
|
||||
# Check if fastled_js is a parent directory of p
|
||||
return "fastled_js" in str(p.absolute())
|
||||
|
||||
|
||||
def compile_for_board_and_example(
|
||||
board: Board,
|
||||
example: Path,
|
||||
build_dir: str | None,
|
||||
verbose_on_failure: bool,
|
||||
libs: list[str] | None,
|
||||
) -> tuple[bool, str]:
|
||||
"""Compile the given example for the given board."""
|
||||
global ERROR_HAPPENED # pylint: disable=global-statement
|
||||
if board.board_name == "web":
|
||||
locked_print(f"Skipping web target for example {example}")
|
||||
return True, ""
|
||||
board_name = board.board_name
|
||||
use_pio_run = board.use_pio_run
|
||||
real_board_name = board.get_real_board_name()
|
||||
libs = libs or []
|
||||
builddir = (
|
||||
Path(build_dir) / board_name if build_dir else Path(".build") / board_name
|
||||
)
|
||||
builddir.mkdir(parents=True, exist_ok=True)
|
||||
srcdir = builddir / "src"
|
||||
# Remove the previous *.ino file if it exists, everything else is recycled
|
||||
# to speed up the next build.
|
||||
if srcdir.exists():
|
||||
shutil.rmtree(srcdir, ignore_errors=False)
|
||||
locked_print(f"*** Building example {example} for board {board_name} ***")
|
||||
cwd: str | None = None
|
||||
shell: bool = False
|
||||
# Copy all files from the example directory to the "src" directory
|
||||
for src_file in example.rglob("*"):
|
||||
if src_file.is_file():
|
||||
if _fastled_js_is_parent_directory(src_file):
|
||||
# Skip the fastled_js folder, it's not needed for the build.
|
||||
continue
|
||||
src_dir = src_file.parent
|
||||
path = src_dir.relative_to(example)
|
||||
dst_dir = srcdir / path
|
||||
os.makedirs(dst_dir, exist_ok=True)
|
||||
locked_print(f"Copying {src_file} to {dst_dir / src_file.name}")
|
||||
os.makedirs(srcdir, exist_ok=True)
|
||||
shutil.copy(src_file, dst_dir / src_file.name)
|
||||
# libs = ["src", "ci"]
|
||||
if use_pio_run:
|
||||
# we have to copy a few folders of pio ci in order to get this to work.
|
||||
for lib in libs:
|
||||
project_libdir = Path(lib)
|
||||
assert project_libdir.exists()
|
||||
build_lib = builddir / "lib" / lib
|
||||
shutil.rmtree(build_lib, ignore_errors=True)
|
||||
shutil.copytree(project_libdir, build_lib)
|
||||
|
||||
cwd = str(builddir)
|
||||
cmd_list = [
|
||||
"pio",
|
||||
"run",
|
||||
]
|
||||
# in this case we need to manually copy the example to the src directory
|
||||
# because platformio doesn't support building a single file.
|
||||
# ino_file = example / f"{example.name}.ino"
|
||||
else:
|
||||
cmd_list = [
|
||||
"pio",
|
||||
"ci",
|
||||
"--board",
|
||||
real_board_name,
|
||||
*[f"--lib={lib}" for lib in libs],
|
||||
"--keep-build-dir",
|
||||
f"--build-dir={builddir.as_posix()}",
|
||||
]
|
||||
cmd_list.append(f"{example.as_posix()}/*ino")
|
||||
cmd_str = subprocess.list2cmdline(cmd_list)
|
||||
msg_lsit = [
|
||||
"\n\n******************************",
|
||||
f"* Running command in cwd: {cwd if cwd else os.getcwd()}",
|
||||
f"* {cmd_str}",
|
||||
"******************************\n",
|
||||
]
|
||||
msg = "\n".join(msg_lsit)
|
||||
locked_print(msg)
|
||||
|
||||
# Start timing for the process
|
||||
start_time = time.time()
|
||||
|
||||
# Run the process with real-time output capture and timing
|
||||
result = subprocess.Popen(
|
||||
cmd_list,
|
||||
cwd=cwd,
|
||||
shell=shell,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
)
|
||||
|
||||
# Capture output lines in real-time with timing
|
||||
stdout_lines: List[str] = []
|
||||
if result.stdout:
|
||||
for line in iter(result.stdout.readline, ""):
|
||||
if line:
|
||||
elapsed = time.time() - start_time
|
||||
# Format timing as seconds with 2 decimal places
|
||||
timing_prefix = f"{elapsed:5.2f} "
|
||||
timed_line = timing_prefix + line.rstrip()
|
||||
stdout_lines.append(
|
||||
line.rstrip()
|
||||
) # Store original line for return value
|
||||
locked_print(timed_line)
|
||||
|
||||
# Wait for process to complete
|
||||
result.wait()
|
||||
|
||||
# Join all stdout lines for the return value
|
||||
stdout = "\n".join(stdout_lines)
|
||||
|
||||
# replace all instances of "lib/src" => "src" so intellisense can find the files
|
||||
# with one click.
|
||||
stdout = stdout.replace("lib/src", "src").replace("lib\\src", "src")
|
||||
if result.returncode != 0:
|
||||
if not verbose_on_failure:
|
||||
ERROR_HAPPENED = True
|
||||
return False, stdout
|
||||
if ERROR_HAPPENED:
|
||||
return False, ""
|
||||
ERROR_HAPPENED = True
|
||||
locked_print(
|
||||
f"*** Error compiling example {example} for board {board_name} ***"
|
||||
)
|
||||
# re-running command with verbose output to see what the defines are.
|
||||
cmd_list.append("-v")
|
||||
cmd_str = subprocess.list2cmdline(cmd_list)
|
||||
msg_lsit = [
|
||||
"\n\n******************************",
|
||||
"* Re-running failed command but with verbose output:",
|
||||
f"* {cmd_str}",
|
||||
"******************************\n",
|
||||
]
|
||||
msg = "\n".join(msg_lsit)
|
||||
locked_print(msg)
|
||||
|
||||
# Start timing for the verbose re-run
|
||||
start_time_verbose = time.time()
|
||||
|
||||
# Run the verbose process with real-time output capture and timing
|
||||
result = subprocess.Popen(
|
||||
cmd_list,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
)
|
||||
|
||||
# Capture output lines in real-time with timing for verbose run
|
||||
stdout_lines_verbose: List[str] = []
|
||||
if result.stdout:
|
||||
for line in iter(result.stdout.readline, ""):
|
||||
if line:
|
||||
elapsed = time.time() - start_time_verbose
|
||||
# Format timing as seconds with 2 decimal places
|
||||
timing_prefix = f"{elapsed:5.2f} "
|
||||
timed_line = timing_prefix + line.rstrip()
|
||||
stdout_lines_verbose.append(
|
||||
line.rstrip()
|
||||
) # Store original line for return value
|
||||
locked_print(timed_line)
|
||||
|
||||
# Wait for verbose process to complete
|
||||
result.wait()
|
||||
|
||||
# Join all stdout lines for the return value
|
||||
stdout = "\n".join(stdout_lines_verbose)
|
||||
stdout = (
|
||||
stdout
|
||||
+ "\n\nThis is a second attempt, but with verbose output, look above for compiler errors.\n"
|
||||
)
|
||||
return False, stdout
|
||||
locked_print(f"*** Finished building example {example} for board {board_name} ***")
|
||||
return True, stdout
|
||||
|
||||
|
||||
# Function to process task queues for each board
|
||||
def compile_examples(
|
||||
board: Board,
|
||||
examples: list[Path],
|
||||
build_dir: str | None,
|
||||
verbose_on_failure: bool,
|
||||
libs: list[str] | None,
|
||||
) -> tuple[bool, str]:
|
||||
"""Process the task queue for the given board."""
|
||||
global ERROR_HAPPENED # pylint: disable=global-statement
|
||||
board_name = board.board_name
|
||||
is_first = True
|
||||
for example in examples:
|
||||
example = example.relative_to(Path(".").resolve())
|
||||
if ERROR_HAPPENED:
|
||||
return True, ""
|
||||
locked_print(f"\n*** Building {example} for board {board_name} ***")
|
||||
if is_first:
|
||||
locked_print(
|
||||
f"*** Building for first example {example} board {board_name} ***"
|
||||
)
|
||||
if is_first and USE_FIRST_BUILD_LOCK:
|
||||
with FIRST_BUILD_LOCK:
|
||||
# Github runners are memory limited and the first job is the most
|
||||
# memory intensive since all the artifacts are being generated in parallel.
|
||||
success, message = compile_for_board_and_example(
|
||||
board=board,
|
||||
example=example,
|
||||
build_dir=build_dir,
|
||||
verbose_on_failure=verbose_on_failure,
|
||||
libs=libs,
|
||||
)
|
||||
else:
|
||||
success, message = compile_for_board_and_example(
|
||||
board=board,
|
||||
example=example,
|
||||
build_dir=build_dir,
|
||||
verbose_on_failure=verbose_on_failure,
|
||||
libs=libs,
|
||||
)
|
||||
is_first = False
|
||||
if not success:
|
||||
ERROR_HAPPENED = True
|
||||
return (
|
||||
False,
|
||||
f"Error building {example} for board {board_name}. stdout:\n{message}",
|
||||
)
|
||||
return True, ""
|
||||
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Abstract Base Class for FastLED Compilers
|
||||
|
||||
Defines the interface that all FastLED compiler implementations must follow.
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from concurrent.futures import Future
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
class CacheType(Enum):
|
||||
"""Compiler cache type options."""
|
||||
|
||||
NO_CACHE = "no_cache"
|
||||
SCCACHE = "sccache"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompilerResult:
|
||||
"""Base result class for compiler operations."""
|
||||
|
||||
success: bool
|
||||
output: str
|
||||
build_dir: Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class InitResult(CompilerResult):
|
||||
"""Result from compiler initialization."""
|
||||
|
||||
@property
|
||||
def platformio_ini(self) -> Path:
|
||||
return self.build_dir / "platformio.ini"
|
||||
|
||||
|
||||
@dataclass
|
||||
class SketchResult(CompilerResult):
|
||||
"""Result from sketch compilation."""
|
||||
|
||||
example: str
|
||||
|
||||
|
||||
class Compiler(ABC):
|
||||
"""Abstract base class defining the interface for FastLED compilers."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
"""Initialize the compiler."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def build(self, examples: list[str]) -> list[Future[SketchResult]]:
|
||||
"""Build a list of examples with proper resource management.
|
||||
|
||||
Args:
|
||||
examples: List of example names or paths to compile
|
||||
|
||||
Returns:
|
||||
List of Future objects containing SketchResult for each example
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def clean(self) -> None:
|
||||
"""Clean build artifacts for this platform."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def clean_all(self) -> None:
|
||||
"""Clean all build artifacts (local and global) for this platform."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def deploy(
|
||||
self, example: str, upload_port: str | None = None, monitor: bool = False
|
||||
) -> SketchResult:
|
||||
"""Deploy (upload) a specific example to the target device.
|
||||
|
||||
Args:
|
||||
example: Name of the example to deploy
|
||||
upload_port: Optional specific port for upload
|
||||
monitor: If True, attach to device monitor after successful upload
|
||||
|
||||
Returns:
|
||||
SketchResult indicating success/failure of deployment
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def cancel_all(self) -> None:
|
||||
"""Cancel all currently running builds."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def check_usb_permissions(self) -> tuple[bool, str]:
|
||||
"""Check if USB device access is properly configured.
|
||||
|
||||
Returns:
|
||||
Tuple of (has_access, status_message)
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def install_usb_permissions(self) -> bool:
|
||||
"""Install platform-specific USB permissions or equivalent.
|
||||
|
||||
Returns:
|
||||
True if installation succeeded, False otherwise
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_cache_stats(self) -> str:
|
||||
"""Get compiler statistics as a formatted string.
|
||||
|
||||
This can include cache statistics, build metrics, performance data,
|
||||
or any other relevant compiler statistics.
|
||||
|
||||
Returns:
|
||||
Formatted string containing compiler statistics, or empty string if none available
|
||||
"""
|
||||
pass
|
||||
@@ -0,0 +1,993 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false, reportReturnType=false, reportMissingParameterType=false
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from concurrent.futures import Future
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple, Union
|
||||
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
from ci.util.running_process import RunningProcess
|
||||
from ci.util.test_args import parse_args as parse_global_test_args
|
||||
|
||||
from .clang_compiler import (
|
||||
BuildFlags,
|
||||
Compiler,
|
||||
CompilerOptions,
|
||||
LinkOptions,
|
||||
test_clang_accessibility,
|
||||
)
|
||||
from .test_example_compilation import create_fastled_compiler
|
||||
|
||||
|
||||
# Configure logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BUILD_DIR = PROJECT_ROOT / "tests" / ".build"
|
||||
BUILD_DIR.mkdir(parents=True, exist_ok=True)
|
||||
CACHE_DIR = PROJECT_ROOT / ".cache"
|
||||
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
TEST_FILES_LIST = CACHE_DIR / "test_files_list.txt"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# HASH-BASED LINKING CACHE (same optimization as examples)
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def calculate_file_hash(file_path: Path) -> str:
|
||||
"""Calculate SHA256 hash of a file (same as FastLEDTestCompiler)"""
|
||||
if not file_path.exists():
|
||||
return "no_file"
|
||||
|
||||
hash_sha256 = hashlib.sha256()
|
||||
with open(file_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
hash_sha256.update(chunk)
|
||||
return hash_sha256.hexdigest()
|
||||
|
||||
|
||||
def calculate_linker_args_hash(linker_args: list[str]) -> str:
|
||||
"""Calculate SHA256 hash of linker arguments (same as FastLEDTestCompiler)"""
|
||||
args_str = "|".join(sorted(linker_args))
|
||||
hash_sha256 = hashlib.sha256()
|
||||
hash_sha256.update(args_str.encode("utf-8"))
|
||||
return hash_sha256.hexdigest()
|
||||
|
||||
|
||||
def calculate_link_cache_key(
|
||||
object_files: list[str | Path], fastled_lib_path: Path, linker_args: list[str]
|
||||
) -> str:
|
||||
"""Calculate comprehensive cache key for linking (same as examples)"""
|
||||
# Calculate hash for each object file
|
||||
obj_hashes: list[str] = []
|
||||
for obj_file in object_files:
|
||||
obj_hashes.append(calculate_file_hash(Path(obj_file)))
|
||||
|
||||
# Combine object file hashes in a stable way (sorted by path for consistency)
|
||||
sorted_obj_paths = sorted(str(obj) for obj in object_files)
|
||||
sorted_obj_hashes: list[str] = []
|
||||
for obj_path in sorted_obj_paths:
|
||||
obj_file = Path(obj_path)
|
||||
sorted_obj_hashes.append(calculate_file_hash(obj_file))
|
||||
|
||||
combined_obj_hash = hashlib.sha256(
|
||||
"|".join(sorted_obj_hashes).encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
# Calculate other hashes
|
||||
fastled_hash = calculate_file_hash(fastled_lib_path)
|
||||
linker_hash = calculate_linker_args_hash(linker_args)
|
||||
|
||||
# Combine all components (same format as FastLEDTestCompiler)
|
||||
combined = f"fastled:{fastled_hash}|objects:{combined_obj_hash}|flags:{linker_hash}"
|
||||
final_hash = hashlib.sha256(combined.encode("utf-8")).hexdigest()
|
||||
|
||||
return final_hash[:16] # Use first 16 chars for readability
|
||||
|
||||
|
||||
def get_link_cache_dir() -> Path:
|
||||
"""Get the link cache directory (same as examples)"""
|
||||
cache_dir = Path(".build/link_cache")
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_dir
|
||||
|
||||
|
||||
def get_cached_executable(test_name: str, cache_key: str) -> Optional[Path]:
|
||||
"""Check if cached executable exists (same as examples)"""
|
||||
cache_dir = get_link_cache_dir()
|
||||
cached_exe = cache_dir / f"{test_name}_{cache_key}.exe"
|
||||
return cached_exe if cached_exe.exists() else None
|
||||
|
||||
|
||||
def cache_executable(test_name: str, cache_key: str, exe_path: Path) -> None:
|
||||
"""Cache an executable for future use (same as examples)"""
|
||||
cache_dir = get_link_cache_dir()
|
||||
cached_exe = cache_dir / f"{test_name}_{cache_key}.exe"
|
||||
try:
|
||||
shutil.copy2(exe_path, cached_exe)
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to cache {test_name}: {e}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# END HASH-BASED LINKING CACHE
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def get_test_files() -> Set[str]:
|
||||
"""Get a set of all test files"""
|
||||
test_files: Set[str] = set()
|
||||
tests_dir = PROJECT_ROOT / "tests"
|
||||
if tests_dir.exists():
|
||||
for file_path in tests_dir.rglob("test_*.cpp"):
|
||||
# Store relative paths for consistency
|
||||
test_files.add(str(file_path.relative_to(PROJECT_ROOT)))
|
||||
return test_files
|
||||
|
||||
|
||||
def check_test_files_changed() -> bool:
|
||||
"""Check if test files have changed since last run"""
|
||||
try:
|
||||
current_files = get_test_files()
|
||||
|
||||
if TEST_FILES_LIST.exists():
|
||||
# Read previous file list
|
||||
with open(TEST_FILES_LIST, "r") as f:
|
||||
previous_files = set(line.strip() for line in f if line.strip())
|
||||
|
||||
# Compare file sets
|
||||
if current_files == previous_files:
|
||||
return False # No changes
|
||||
else:
|
||||
print("Test files have changed, cleaning build directory...")
|
||||
return True # Files changed
|
||||
else:
|
||||
# No previous file list, need to clean
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Warning: Error checking test file changes: {e}")
|
||||
return True # Default to cleaning on error
|
||||
|
||||
|
||||
def save_test_files_list() -> None:
|
||||
"""Save current test file list"""
|
||||
try:
|
||||
current_files = get_test_files()
|
||||
with open(TEST_FILES_LIST, "w") as f:
|
||||
for file_path in sorted(current_files):
|
||||
f.write(f"{file_path}\n")
|
||||
except KeyboardInterrupt:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to save test file list: {e}")
|
||||
|
||||
|
||||
def clean_build_directory():
|
||||
print("Cleaning build directory...")
|
||||
shutil.rmtree(BUILD_DIR, ignore_errors=True)
|
||||
BUILD_DIR.mkdir(parents=True, exist_ok=True)
|
||||
print("Build directory cleaned.")
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
WASM_BUILD = False
|
||||
USE_ZIG = False
|
||||
USE_CLANG = False
|
||||
|
||||
|
||||
# Legacy CMake helper functions removed - using optimized Python API
|
||||
|
||||
|
||||
def get_unit_test_fastled_sources() -> list[Path]:
|
||||
"""Get essential FastLED .cpp files for unit test library creation (optimized paradigm)."""
|
||||
# Always work from project root
|
||||
project_root = Path(
|
||||
__file__
|
||||
).parent.parent.parent # Go up from ci/compiler/ to project root
|
||||
src_dir = project_root / "src"
|
||||
|
||||
# Core FastLED files that must be included for unit tests
|
||||
core_files: list[Path] = [
|
||||
src_dir / "FastLED.cpp",
|
||||
src_dir / "colorutils.cpp",
|
||||
src_dir / "hsv2rgb.cpp",
|
||||
]
|
||||
|
||||
# Find all .cpp files in key directories
|
||||
additional_sources: list[Path] = []
|
||||
for pattern in ["*.cpp", "lib8tion/*.cpp", "platforms/stub/*.cpp"]:
|
||||
additional_sources.extend(list(src_dir.glob(pattern)))
|
||||
|
||||
# Include essential .cpp files from nested directories
|
||||
additional_sources.extend(list(src_dir.rglob("*.cpp")))
|
||||
|
||||
# Filter out duplicates and ensure files exist
|
||||
all_sources: list[Path] = []
|
||||
seen_files: set[Path] = set()
|
||||
|
||||
for cpp_file in core_files + additional_sources:
|
||||
# Skip stub_main.cpp since unit tests have their own main
|
||||
if cpp_file.name == "stub_main.cpp":
|
||||
continue
|
||||
|
||||
# Skip platform-specific files that aren't needed for unit tests
|
||||
rel_path_str = str(cpp_file)
|
||||
if any(
|
||||
skip in rel_path_str for skip in ["wasm", "esp", "avr", "arm", "teensy"]
|
||||
):
|
||||
continue
|
||||
|
||||
if cpp_file.exists() and cpp_file not in seen_files:
|
||||
all_sources.append(cpp_file)
|
||||
seen_files.add(cpp_file)
|
||||
|
||||
return all_sources
|
||||
|
||||
|
||||
def create_unit_test_fastled_library(
|
||||
clean: bool = False, use_pch: bool = True
|
||||
) -> Path | None:
|
||||
"""Create libfastled.a static library specifically for unit tests with FASTLED_FORCE_NAMESPACE=1.
|
||||
|
||||
CRITICAL: Unit tests need their own separate library from examples because they use
|
||||
different compilation flags. Unit tests require FASTLED_FORCE_NAMESPACE=1 to put
|
||||
all symbols in the fl:: namespace that the tests expect.
|
||||
"""
|
||||
|
||||
# Unit tests get their own separate library directory under .build/fastled/unit/
|
||||
fastled_build_dir = BUILD_DIR.parent / ".build" / "fastled" / "unit"
|
||||
fastled_build_dir.mkdir(parents=True, exist_ok=True)
|
||||
lib_file = fastled_build_dir / "libfastled.a"
|
||||
|
||||
if lib_file.exists() and not clean:
|
||||
print(f"[LIBRARY] Using existing FastLED library: {lib_file}")
|
||||
return lib_file
|
||||
|
||||
print("[LIBRARY] Creating FastLED static library with proper compiler flags...")
|
||||
|
||||
# Create a proper FastLED compiler (NOT unit test compiler) for the library
|
||||
# This ensures the library is compiled without FASTLED_FORCE_NAMESPACE=1
|
||||
print("[LIBRARY] Creating FastLED library compiler (without unit test flags)...")
|
||||
|
||||
# Save current directory and change to project root for create_fastled_compiler
|
||||
import os
|
||||
|
||||
project_root = Path(__file__).parent.parent.parent
|
||||
original_cwd = os.getcwd()
|
||||
os.chdir(str(project_root))
|
||||
|
||||
try:
|
||||
library_compiler = create_fastled_compiler(
|
||||
use_pch=use_pch,
|
||||
parallel=True,
|
||||
)
|
||||
|
||||
# CRITICAL: Add required defines for unit test library compilation
|
||||
if library_compiler.settings.defines is None:
|
||||
library_compiler.settings.defines = []
|
||||
|
||||
# Add FASTLED_FORCE_NAMESPACE=1 to export symbols in fl:: namespace
|
||||
library_compiler.settings.defines.append("FASTLED_FORCE_NAMESPACE=1")
|
||||
print("[LIBRARY] Added FASTLED_FORCE_NAMESPACE=1 to library compiler")
|
||||
|
||||
# Add FASTLED_TESTING=1 to include MockTimeProvider and test utility functions
|
||||
library_compiler.settings.defines.append("FASTLED_TESTING=1")
|
||||
print("[LIBRARY] Added FASTLED_TESTING=1 to library compiler")
|
||||
|
||||
finally:
|
||||
# Restore original working directory
|
||||
os.chdir(original_cwd)
|
||||
|
||||
# Get FastLED sources using optimized selection
|
||||
fastled_sources = get_unit_test_fastled_sources()
|
||||
fastled_objects: list[Path] = []
|
||||
|
||||
obj_dir = fastled_build_dir / "obj"
|
||||
obj_dir.mkdir(exist_ok=True)
|
||||
|
||||
print(f"[LIBRARY] Compiling {len(fastled_sources)} FastLED source files...")
|
||||
|
||||
# Compile each source file with optimized naming (same as examples)
|
||||
futures: List[Tuple[Any, ...]] = []
|
||||
project_root = Path(__file__).parent.parent.parent
|
||||
src_dir = project_root / "src"
|
||||
|
||||
for cpp_file in fastled_sources:
|
||||
# Create unique object file name by including relative path to prevent collisions
|
||||
# Convert path separators to underscores to create valid filename
|
||||
if cpp_file.is_relative_to(src_dir):
|
||||
rel_path = cpp_file.relative_to(src_dir)
|
||||
else:
|
||||
rel_path = cpp_file
|
||||
|
||||
# Replace path separators with underscores for unique object file names
|
||||
obj_name = str(rel_path.with_suffix(".o")).replace("/", "_").replace("\\", "_")
|
||||
obj_file = obj_dir / obj_name
|
||||
future = library_compiler.compile_cpp_file(cpp_file, obj_file)
|
||||
futures.append((future, obj_file, cpp_file))
|
||||
|
||||
# Wait for compilation to complete
|
||||
compiled_count = 0
|
||||
for future, obj_file, cpp_file in futures:
|
||||
try:
|
||||
result = future.result()
|
||||
if result.ok:
|
||||
fastled_objects.append(obj_file)
|
||||
compiled_count += 1
|
||||
else:
|
||||
print(
|
||||
f"[LIBRARY] WARNING: Failed to compile {cpp_file.relative_to(src_dir)}: {result.stderr[:100]}..."
|
||||
)
|
||||
except Exception as e:
|
||||
print(
|
||||
f"[LIBRARY] WARNING: Exception compiling {cpp_file.relative_to(src_dir)}: {e}"
|
||||
)
|
||||
|
||||
print(
|
||||
f"[LIBRARY] Successfully compiled {compiled_count}/{len(fastled_sources)} FastLED sources"
|
||||
)
|
||||
|
||||
if not fastled_objects:
|
||||
print("[LIBRARY] ERROR: No FastLED source files compiled successfully")
|
||||
return None
|
||||
|
||||
# Create static library using the same approach as examples
|
||||
print(f"[LIBRARY] Creating static library: {lib_file}")
|
||||
|
||||
archive_future = library_compiler.create_archive(fastled_objects, lib_file)
|
||||
archive_result = archive_future.result()
|
||||
|
||||
if not archive_result.ok:
|
||||
print(f"[LIBRARY] ERROR: Library creation failed: {archive_result.stderr}")
|
||||
return None
|
||||
|
||||
print(f"[LIBRARY] SUCCESS: FastLED library created: {lib_file}")
|
||||
return lib_file
|
||||
|
||||
|
||||
def create_unit_test_compiler(
|
||||
use_pch: bool = True, enable_static_analysis: bool = False, debug: bool = False
|
||||
) -> Compiler:
|
||||
"""Create compiler optimized for unit test compilation with PCH support."""
|
||||
|
||||
# Always work from the project root, not from ci/compiler
|
||||
project_root = Path(
|
||||
__file__
|
||||
).parent.parent.parent # Go up from ci/compiler/ to project root
|
||||
current_dir = project_root
|
||||
src_path = current_dir / "src"
|
||||
|
||||
# Load build flags configuration
|
||||
build_flags_path = current_dir / "ci" / "build_unit.toml"
|
||||
build_flags = BuildFlags.parse(
|
||||
build_flags_path, quick_build=True, strict_mode=False
|
||||
)
|
||||
|
||||
# Unit test specific defines
|
||||
unit_test_defines = [
|
||||
"FASTLED_UNIT_TEST=1",
|
||||
"FASTLED_FORCE_NAMESPACE=1",
|
||||
"FASTLED_USE_PROGMEM=0",
|
||||
"STUB_PLATFORM",
|
||||
"ARDUINO=10808",
|
||||
"FASTLED_USE_STUB_ARDUINO",
|
||||
"SKETCH_HAS_LOTS_OF_MEMORY=1",
|
||||
"FASTLED_STUB_IMPL",
|
||||
"FASTLED_USE_JSON_UI=1",
|
||||
"FASTLED_TESTING",
|
||||
"FASTLED_NO_AUTO_NAMESPACE",
|
||||
"FASTLED_NO_PINMAP",
|
||||
"HAS_HARDWARE_PIN_SUPPORT",
|
||||
"FASTLED_DEBUG_LEVEL=1",
|
||||
"FASTLED_NO_ATEXIT=1",
|
||||
"DOCTEST_CONFIG_NO_EXCEPTIONS_BUT_WITH_ALL_ASSERTS",
|
||||
"ENABLE_CRASH_HANDLER",
|
||||
"RELEASE=1", # Disable FASTLED_FORCE_DBG to avoid fl::println dependency
|
||||
]
|
||||
|
||||
# Unit test specific compiler args
|
||||
unit_test_args = [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-Wno-deprecated-register",
|
||||
"-Wno-backslash-newline-escape",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
# Optimization/debug controls set below based on debug flag
|
||||
"-fno-omit-frame-pointer",
|
||||
"-fno-inline-functions",
|
||||
"-fno-vectorize",
|
||||
"-fno-unroll-loops",
|
||||
"-fno-strict-aliasing",
|
||||
f"-I{current_dir}",
|
||||
f"-I{src_path}",
|
||||
f"-I{current_dir / 'tests'}",
|
||||
f"-I{src_path / 'platforms' / 'stub'}",
|
||||
]
|
||||
|
||||
# Apply quick vs debug modes
|
||||
if debug:
|
||||
unit_test_args.extend(
|
||||
[
|
||||
"-O0",
|
||||
"-g3",
|
||||
"-fstandalone-debug",
|
||||
]
|
||||
)
|
||||
if os.name == "nt":
|
||||
unit_test_args.extend(
|
||||
["-gdwarf-4"]
|
||||
) # GNU debug info for Windows GNU toolchain
|
||||
else:
|
||||
unit_test_args.extend(
|
||||
[
|
||||
"-O0",
|
||||
"-g0",
|
||||
]
|
||||
)
|
||||
|
||||
# Note: DWARF flags are added above conditionally when debug=True
|
||||
|
||||
# PCH configuration with unit test specific headers
|
||||
pch_output_path = None
|
||||
pch_header_content = None
|
||||
if use_pch:
|
||||
cache_dir = current_dir / ".build" / "cache"
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
pch_output_path = str(cache_dir / "fastled_unit_test_pch.hpp.pch")
|
||||
|
||||
# Unit test specific PCH header content
|
||||
pch_header_content = """// FastLED Unit Test PCH - Common headers for faster test compilation
|
||||
#pragma once
|
||||
|
||||
// Core test framework
|
||||
#include "test.h"
|
||||
|
||||
// Core FastLED headers that are used in nearly all unit tests
|
||||
#include "FastLED.h"
|
||||
|
||||
// Common C++ standard library headers used in tests
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include <stdio.h>
|
||||
#include <cstdint>
|
||||
#include <cmath>
|
||||
#include <cassert>
|
||||
#include <iostream>
|
||||
#include <memory>
|
||||
|
||||
// Platform headers for stub environment
|
||||
#include "platforms/stub/fastled_stub.h"
|
||||
|
||||
// Commonly tested FastLED components
|
||||
#include "lib8tion.h"
|
||||
#include "colorutils.h"
|
||||
#include "hsv2rgb.h"
|
||||
#include "fl/math.h"
|
||||
#include "fl/vector.h"
|
||||
|
||||
// Using namespace to match test files
|
||||
using namespace fl;
|
||||
"""
|
||||
print(f"[PCH] Unit tests will use precompiled headers: {pch_output_path}")
|
||||
print(
|
||||
f"[PCH] PCH includes: test.h, FastLED.h, lib8tion.h, colorutils.h, and more"
|
||||
)
|
||||
else:
|
||||
print("[PCH] Precompiled headers disabled for unit tests")
|
||||
|
||||
# Determine compiler
|
||||
compiler_cmd = "python -m ziglang c++"
|
||||
if USE_CLANG:
|
||||
compiler_cmd = "clang++"
|
||||
print("USING CLANG COMPILER FOR UNIT TESTS")
|
||||
elif USE_ZIG:
|
||||
print("USING ZIG COMPILER FOR UNIT TESTS")
|
||||
else:
|
||||
print("USING DEFAULT COMPILER FOR UNIT TESTS")
|
||||
|
||||
settings = CompilerOptions(
|
||||
include_path=str(src_path),
|
||||
defines=unit_test_defines,
|
||||
std_version="c++17",
|
||||
compiler=compiler_cmd,
|
||||
compiler_args=unit_test_args,
|
||||
use_pch=use_pch,
|
||||
pch_output_path=pch_output_path,
|
||||
pch_header_content=pch_header_content,
|
||||
parallel=True,
|
||||
)
|
||||
|
||||
return Compiler(settings, build_flags)
|
||||
|
||||
|
||||
def compile_unit_tests_python_api(
|
||||
specific_test: str | None = None,
|
||||
enable_static_analysis: bool = False,
|
||||
use_pch: bool = True,
|
||||
clean: bool = False,
|
||||
debug: bool = False,
|
||||
) -> bool:
|
||||
"""Compile unit tests using the fast Python API instead of CMake.
|
||||
|
||||
Returns:
|
||||
bool: True if all tests compiled and linked successfully, False otherwise
|
||||
"""
|
||||
from .clang_compiler import Compiler, LinkOptions
|
||||
|
||||
print("=" * 60)
|
||||
print("COMPILING UNIT TESTS WITH PYTHON API")
|
||||
print("=" * 60)
|
||||
|
||||
if clean:
|
||||
print("Cleaning build directory...")
|
||||
shutil.rmtree(BUILD_DIR, ignore_errors=True)
|
||||
BUILD_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create optimized compiler for unit tests
|
||||
compiler = create_unit_test_compiler(
|
||||
use_pch=use_pch, enable_static_analysis=enable_static_analysis, debug=debug
|
||||
)
|
||||
|
||||
# Find all test files - work from project root
|
||||
project_root = Path(__file__).parent.parent.parent
|
||||
tests_dir = project_root / "tests"
|
||||
test_files = []
|
||||
|
||||
if specific_test:
|
||||
# Handle specific test with case-insensitive matching
|
||||
test_name = (
|
||||
specific_test
|
||||
if specific_test.startswith("test_")
|
||||
else f"test_{specific_test}"
|
||||
)
|
||||
test_file = tests_dir / f"{test_name}.cpp"
|
||||
|
||||
# First try exact case match
|
||||
if test_file.exists():
|
||||
test_files = [test_file]
|
||||
print(f"Compiling specific test: {test_file.name}")
|
||||
else:
|
||||
# Try case-insensitive matching for all test files
|
||||
found_match = False
|
||||
for existing_file in tests_dir.glob("test_*.cpp"):
|
||||
# Check if the file matches case-insensitively
|
||||
existing_stem = existing_file.stem
|
||||
existing_name = existing_stem.replace("test_", "")
|
||||
|
||||
if (
|
||||
existing_stem.lower() == test_name.lower()
|
||||
or existing_name.lower() == specific_test.lower()
|
||||
):
|
||||
test_files = [existing_file]
|
||||
print(
|
||||
f"Compiling specific test (case-insensitive match): {existing_file.name}"
|
||||
)
|
||||
found_match = True
|
||||
break
|
||||
|
||||
if not found_match:
|
||||
raise RuntimeError(f"Test file not found: {test_file}")
|
||||
else:
|
||||
# Find all test files
|
||||
test_files = list(tests_dir.glob("test_*.cpp"))
|
||||
print(f"Found {len(test_files)} unit test files")
|
||||
|
||||
if not test_files:
|
||||
print("No test files found")
|
||||
return
|
||||
|
||||
# Ensure output directory exists
|
||||
bin_dir = BUILD_DIR / "bin"
|
||||
bin_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create clean execution directory as specified in requirements
|
||||
clean_bin_dir = PROJECT_ROOT / "tests" / "bin"
|
||||
clean_bin_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Step 1: Compile doctest main once
|
||||
print("Compiling doctest main...")
|
||||
doctest_main_path = tests_dir / "doctest_main.cpp"
|
||||
doctest_main_obj = bin_dir / "doctest_main.o"
|
||||
|
||||
if not doctest_main_obj.exists() or clean:
|
||||
doctest_compile_future = compiler.compile_cpp_file(
|
||||
cpp_path=str(doctest_main_path), output_path=str(doctest_main_obj)
|
||||
)
|
||||
doctest_result = doctest_compile_future.result()
|
||||
|
||||
if doctest_result.return_code != 0:
|
||||
raise RuntimeError(
|
||||
f"Failed to compile doctest main: {doctest_result.stderr}"
|
||||
)
|
||||
|
||||
# Step 2: Build FastLED library using optimized examples paradigm
|
||||
print("Building FastLED library...")
|
||||
|
||||
fastled_lib_path = create_unit_test_fastled_library(clean, use_pch=use_pch)
|
||||
|
||||
# Step 3: Compile and link each test (PARALLEL OPTIMIZATION)
|
||||
print(f"Compiling {len(test_files)} tests...")
|
||||
start_time = time.time()
|
||||
|
||||
# Phase 1: Start all compilations in parallel (NON-BLOCKING)
|
||||
print("🚀 Starting parallel compilation...")
|
||||
compile_start = time.time()
|
||||
compile_futures: Dict[str, Future[Any]] = {}
|
||||
test_info: Dict[str, Any] = {}
|
||||
|
||||
for test_file in test_files:
|
||||
test_name = test_file.stem
|
||||
executable_path = bin_dir / test_name
|
||||
object_path = bin_dir / f"{test_name}.o"
|
||||
|
||||
test_info[test_name] = {
|
||||
"test_file": test_file,
|
||||
"executable_path": executable_path,
|
||||
"object_path": object_path,
|
||||
}
|
||||
|
||||
print(f" Compiling {test_name}...")
|
||||
|
||||
# Start compilation (NON-BLOCKING)
|
||||
compile_future = compiler.compile_cpp_file(
|
||||
cpp_path=str(test_file), output_path=str(object_path)
|
||||
)
|
||||
compile_futures[test_name] = compile_future
|
||||
|
||||
# Phase 2: Wait for all compilations to complete and check cache before linking
|
||||
compile_dispatch_time = time.time() - compile_start
|
||||
print(
|
||||
f"⏳ Waiting for compilations to complete... (dispatch took {compile_dispatch_time:.2f}s)"
|
||||
)
|
||||
compile_wait_start = time.time()
|
||||
link_futures: Dict[str, Dict[str, Any]] = {}
|
||||
cache_hits = 0
|
||||
success_count = 0
|
||||
|
||||
for test_name, compile_future in compile_futures.items():
|
||||
try:
|
||||
compile_result = (
|
||||
compile_future.result()
|
||||
) # BLOCKING WAIT for this specific test
|
||||
|
||||
if compile_result.return_code != 0:
|
||||
print(f"❌ Compilation failed for {test_name}: {compile_result.stderr}")
|
||||
continue
|
||||
|
||||
# Prepare linking info (same logic as before)
|
||||
info = test_info[test_name]
|
||||
executable_path = info["executable_path"]
|
||||
object_path = info["object_path"]
|
||||
|
||||
tests_with_own_main = ["test_example_compilation"]
|
||||
if test_name in tests_with_own_main:
|
||||
object_files: list[str | Path] = [object_path]
|
||||
else:
|
||||
object_files: list[str | Path] = [object_path, doctest_main_obj]
|
||||
|
||||
# Platform-specific linker arguments for crash handler support
|
||||
if os.name == "nt": # Windows
|
||||
linker_args = ["-ldbghelp", "-lpsapi"]
|
||||
else: # Linux/macOS
|
||||
linker_args = ["-pthread"]
|
||||
|
||||
# HASH-BASED CACHE CHECK (same as examples)
|
||||
if not fastled_lib_path:
|
||||
print(f"⚠️ No FastLED library found, skipping cache for {test_name}")
|
||||
cache_key = "no_fastled_lib"
|
||||
cached_exe = None
|
||||
else:
|
||||
cache_key = calculate_link_cache_key(
|
||||
object_files, fastled_lib_path, linker_args
|
||||
)
|
||||
cached_exe = get_cached_executable(test_name, cache_key)
|
||||
|
||||
if cached_exe:
|
||||
# Cache hit! Copy cached executable to target location
|
||||
try:
|
||||
shutil.copy2(cached_exe, executable_path)
|
||||
# Also copy to clean execution directory
|
||||
clean_executable_path = clean_bin_dir / f"{test_name}.exe"
|
||||
shutil.copy2(cached_exe, clean_executable_path)
|
||||
cache_hits += 1
|
||||
success_count += 1
|
||||
print(f" ⚡ {test_name}: Using cached executable (cache hit)")
|
||||
continue # Skip linking entirely
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Failed to copy cached {test_name}, will relink: {e}")
|
||||
# Fall through to actual linking
|
||||
|
||||
# Cache miss - proceed with actual linking
|
||||
static_libraries: List[Union[str, Path]] = []
|
||||
if fastled_lib_path and fastled_lib_path.exists():
|
||||
static_libraries.append(fastled_lib_path)
|
||||
|
||||
link_options = LinkOptions(
|
||||
output_executable=str(executable_path),
|
||||
object_files=object_files,
|
||||
static_libraries=static_libraries,
|
||||
linker_args=linker_args,
|
||||
)
|
||||
|
||||
# Start linking (NON-BLOCKING) and store cache info for later
|
||||
link_future = compiler.link_program(link_options)
|
||||
link_futures[test_name] = {
|
||||
"future": link_future,
|
||||
"cache_key": cache_key,
|
||||
"executable_path": executable_path,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ ERROR compiling {test_name}: {e}")
|
||||
continue
|
||||
|
||||
# Phase 3: Wait for all linking to complete and cache successful results
|
||||
compile_wait_time = time.time() - compile_wait_start
|
||||
print(
|
||||
f"🔗 Waiting for linking to complete... (compilation took {compile_wait_time:.2f}s)"
|
||||
)
|
||||
link_start = time.time()
|
||||
cache_misses = 0
|
||||
|
||||
for test_name, link_info in link_futures.items():
|
||||
try:
|
||||
link_future = link_info["future"]
|
||||
cache_key = link_info["cache_key"]
|
||||
executable_path = link_info["executable_path"]
|
||||
|
||||
link_result = link_future.result() # BLOCKING WAIT for this specific test
|
||||
|
||||
if link_result.return_code != 0:
|
||||
print(f"⚠️ Linking failed for {test_name}: {link_result.stderr}")
|
||||
cache_misses += 1
|
||||
continue
|
||||
else:
|
||||
success_count += 1
|
||||
cache_misses += 1 # This was a fresh link, not from cache
|
||||
|
||||
# Cache the successful executable for future use (same as examples)
|
||||
cache_executable(test_name, cache_key, executable_path)
|
||||
|
||||
# Also copy to clean execution directory
|
||||
clean_executable_path = clean_bin_dir / f"{test_name}.exe"
|
||||
shutil.copy2(executable_path, clean_executable_path)
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ ERROR linking {test_name}: {e}")
|
||||
cache_misses += 1
|
||||
continue
|
||||
|
||||
link_time = time.time() - link_start
|
||||
compilation_time = time.time() - start_time
|
||||
|
||||
print(f"✅ Unit test compilation completed in {compilation_time:.2f}s")
|
||||
print(f" 📊 Time breakdown:")
|
||||
print(f" • Dispatch: {compile_dispatch_time:.2f}s")
|
||||
print(f" • Compilation: {compile_wait_time:.2f}s")
|
||||
print(f" • Linking: {link_time:.2f}s")
|
||||
print(f" 🎯 Cache statistics (archive + hash linking optimization):")
|
||||
print(f" • Cache hits: {cache_hits} (skipped linking)")
|
||||
print(f" • Cache misses: {cache_misses} (fresh linking)")
|
||||
print(
|
||||
f" • Cache hit ratio: {cache_hits / max(1, cache_hits + cache_misses) * 100:.1f}%"
|
||||
)
|
||||
print(f" Successfully compiled: {success_count}/{len(test_files)} tests")
|
||||
print(f" Average: {compilation_time / len(test_files):.2f}s per test")
|
||||
print(f" Output directory: {bin_dir}")
|
||||
print(f" Cache directory: {get_link_cache_dir()}")
|
||||
|
||||
# Return success only if ALL tests compiled and linked successfully
|
||||
all_tests_successful = success_count == len(test_files)
|
||||
if not all_tests_successful:
|
||||
failed_count = len(test_files) - success_count
|
||||
print(f"❌ {failed_count} test(s) failed to compile or link")
|
||||
|
||||
return all_tests_successful
|
||||
|
||||
|
||||
def parse_arguments():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compile FastLED library with different compiler options."
|
||||
)
|
||||
parser.add_argument("--use-zig", action="store_true", help="Use Zig compiler")
|
||||
parser.add_argument("--use-clang", action="store_true", help="Use Clang compiler")
|
||||
parser.add_argument("--wasm", action="store_true", help="Build for WebAssembly")
|
||||
parser.add_argument(
|
||||
"--clean",
|
||||
action="store_true",
|
||||
help="Clean the build directory before compiling",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--test",
|
||||
help="Specific test to compile (without test_ prefix)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--check",
|
||||
action="store_true",
|
||||
help="Enable static analysis (IWYU, clang-tidy)",
|
||||
)
|
||||
parser.add_argument("--no-unity", action="store_true", help="Disable unity build")
|
||||
parser.add_argument(
|
||||
"--no-pch", action="store_true", help="Disable precompiled headers (PCH)"
|
||||
)
|
||||
parser.add_argument("--debug", action="store_true", help="Enable debug symbols")
|
||||
|
||||
parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def get_build_info(args: argparse.Namespace) -> dict[str, str | dict[str, str]]:
|
||||
return {
|
||||
"USE_ZIG": str(USE_ZIG),
|
||||
"USE_CLANG": str(USE_CLANG),
|
||||
"WASM_BUILD": str(WASM_BUILD),
|
||||
"CC": os.environ.get("CC", ""),
|
||||
"CXX": os.environ.get("CXX", ""),
|
||||
"AR": os.environ.get("AR", ""),
|
||||
"CFLAGS": os.environ.get("CFLAGS", ""),
|
||||
"CXXFLAGS": os.environ.get("CXXFLAGS", ""),
|
||||
"ARGS": {
|
||||
"use_zig": str(args.use_zig),
|
||||
"use_clang": str(args.use_clang),
|
||||
"wasm": str(args.wasm),
|
||||
"specific_test": str(args.test) if args.test else "all",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def should_clean_build(build_info: dict[str, str | dict[str, str]]) -> bool:
|
||||
build_info_file = BUILD_DIR / "build_info.json"
|
||||
if not build_info_file.exists():
|
||||
return True
|
||||
|
||||
try:
|
||||
with open(build_info_file, "r") as f:
|
||||
old_build_info = json.load(f)
|
||||
except (json.JSONDecodeError, ValueError) as e:
|
||||
# If JSON is corrupted or empty, remove it and force clean build
|
||||
logger.warning(
|
||||
f"Corrupted build_info.json detected ({e}), removing and forcing clean build"
|
||||
)
|
||||
try:
|
||||
build_info_file.unlink()
|
||||
except OSError:
|
||||
pass # Ignore if we can't remove it
|
||||
return True
|
||||
|
||||
# If build parameters have changed, we need to rebuild
|
||||
if old_build_info != build_info:
|
||||
# Check if this is just a change in specific test target
|
||||
old_args_raw = old_build_info.get("ARGS", {})
|
||||
new_args_raw = build_info.get("ARGS", {})
|
||||
|
||||
# Ensure ARGS is a dictionary
|
||||
if not isinstance(old_args_raw, dict) or not isinstance(new_args_raw, dict):
|
||||
return True
|
||||
|
||||
old_args: Dict[str, Any] = old_args_raw # type: ignore
|
||||
new_args: Dict[str, Any] = new_args_raw # type: ignore
|
||||
|
||||
# If only the specific test changed and everything else is the same,
|
||||
# we don't need to clean the build directory
|
||||
old_test = old_args.get("specific_test", "all")
|
||||
new_test = new_args.get("specific_test", "all")
|
||||
|
||||
# Create copies without the specific_test field for comparison
|
||||
old_args_no_test: Dict[str, Any] = {
|
||||
k: v for k, v in old_args.items() if k != "specific_test"
|
||||
}
|
||||
new_args_no_test: Dict[str, Any] = {
|
||||
k: v for k, v in new_args.items() if k != "specific_test"
|
||||
}
|
||||
|
||||
# If only the test target changed, don't clean
|
||||
if (
|
||||
old_args_no_test == new_args_no_test
|
||||
and old_build_info.get("USE_ZIG") == build_info.get("USE_ZIG")
|
||||
and old_build_info.get("USE_CLANG") == build_info.get("USE_CLANG")
|
||||
and old_build_info.get("WASM_BUILD") == build_info.get("WASM_BUILD")
|
||||
and old_build_info.get("CC") == build_info.get("CC")
|
||||
and old_build_info.get("CXX") == build_info.get("CXX")
|
||||
and old_build_info.get("AR") == build_info.get("AR")
|
||||
and old_build_info.get("CFLAGS") == build_info.get("CFLAGS")
|
||||
and old_build_info.get("CXXFLAGS") == build_info.get("CXXFLAGS")
|
||||
):
|
||||
print(
|
||||
f"Build parameters unchanged, only test target changed: {old_test} -> {new_test}"
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def update_build_info(build_info: dict[str, str | dict[str, str]]):
|
||||
build_info_file = BUILD_DIR / "build_info.json"
|
||||
with open(build_info_file, "w") as f:
|
||||
json.dump(build_info, f, indent=2)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
global USE_ZIG, USE_CLANG, WASM_BUILD
|
||||
|
||||
args = parse_arguments()
|
||||
USE_ZIG = args.use_zig # use Zig's clang compiler
|
||||
USE_CLANG = args.use_clang # Use pure Clang for WASM builds
|
||||
WASM_BUILD = args.wasm
|
||||
|
||||
using_gcc = not USE_ZIG and not USE_CLANG and not WASM_BUILD
|
||||
if using_gcc:
|
||||
if not shutil.which("g++"):
|
||||
print(
|
||||
"gcc compiler not found in PATH, falling back zig's built in clang compiler"
|
||||
)
|
||||
USE_ZIG = True
|
||||
USE_CLANG = False
|
||||
|
||||
if USE_CLANG:
|
||||
if not test_clang_accessibility():
|
||||
print(
|
||||
"Clang compiler not found in PATH, falling back to Zig-clang compiler"
|
||||
)
|
||||
USE_ZIG = True
|
||||
USE_CLANG = False
|
||||
|
||||
os.chdir(str(HERE))
|
||||
print(f"Current directory: {Path('.').absolute()}")
|
||||
|
||||
# Auto-detection for --clean based on test file changes
|
||||
need_clean = args.clean
|
||||
if not need_clean:
|
||||
# Only check for changes if --clean wasn't explicitly specified
|
||||
need_clean = check_test_files_changed()
|
||||
|
||||
build_info = get_build_info(args)
|
||||
if need_clean or should_clean_build(build_info):
|
||||
clean_build_directory()
|
||||
# Save the file list after cleaning
|
||||
save_test_files_list()
|
||||
elif args.clean:
|
||||
# If --clean was explicitly specified but not needed according to build info,
|
||||
# still clean and save file list
|
||||
clean_build_directory()
|
||||
save_test_files_list()
|
||||
|
||||
# Unit tests now use optimized Python API by default (same as examples)
|
||||
use_pch = not getattr(
|
||||
args, "no_pch", False
|
||||
) # Default to PCH enabled unless --no-pch specified
|
||||
|
||||
# Use the optimized Python API with PCH optimization (now default for unit tests)
|
||||
compilation_successful = compile_unit_tests_python_api(
|
||||
specific_test=args.test,
|
||||
enable_static_analysis=args.check,
|
||||
use_pch=use_pch,
|
||||
clean=need_clean,
|
||||
debug=args.debug,
|
||||
)
|
||||
|
||||
if compilation_successful:
|
||||
update_build_info(build_info)
|
||||
print("FastLED library compiled successfully.")
|
||||
else:
|
||||
print("❌ FastLED unit test compilation failed!")
|
||||
import sys
|
||||
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,906 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false, reportMissingParameterType=false
|
||||
import argparse
|
||||
import multiprocessing
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time # Added for timing test execution
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from queue import Empty, PriorityQueue
|
||||
from threading import Event, Lock, Thread
|
||||
from typing import List
|
||||
|
||||
import psutil
|
||||
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
|
||||
|
||||
def optimize_python_command(cmd: list[str]) -> list[str]:
|
||||
"""
|
||||
Optimize command list for subprocess execution in uv environment.
|
||||
|
||||
For python commands, we need to use 'uv run python' to ensure access to
|
||||
installed packages like ziglang. Direct sys.executable bypasses uv environment.
|
||||
|
||||
Args:
|
||||
cmd: Command list that may contain 'python' as first element
|
||||
|
||||
Returns:
|
||||
list[str]: Optimized command with 'python' prefixed by 'uv run'
|
||||
"""
|
||||
if cmd and (cmd[0] == "python" or cmd[0] == "python3"):
|
||||
# Use uv run python to ensure access to uv-managed packages
|
||||
optimized_cmd = ["uv", "run", "python"] + cmd[1:]
|
||||
return optimized_cmd
|
||||
return cmd
|
||||
|
||||
|
||||
from ci.util.test_exceptions import (
|
||||
CompilationFailedException,
|
||||
TestExecutionFailedException,
|
||||
TestFailureInfo,
|
||||
TestTimeoutException,
|
||||
)
|
||||
|
||||
|
||||
class OutputBuffer:
|
||||
"""Thread-safe output buffer with ordered output display"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.output_queue: PriorityQueue[tuple[int, int, str]] = PriorityQueue()
|
||||
self.next_sequence: int = 0
|
||||
self.sequence_lock: Lock = Lock()
|
||||
self.stop_event: Event = Event()
|
||||
self.output_thread: Thread = Thread(target=self._output_worker, daemon=True)
|
||||
self.output_thread.start()
|
||||
|
||||
def write(self, test_index: int, message: str) -> None:
|
||||
"""Write a message to the buffer with test index for ordering"""
|
||||
with self.sequence_lock:
|
||||
sequence = self.next_sequence
|
||||
self.next_sequence += 1
|
||||
self.output_queue.put((test_index, sequence, message))
|
||||
|
||||
def _output_worker(self) -> None:
|
||||
"""Worker thread that processes output in order"""
|
||||
while not self.stop_event.is_set() or not self.output_queue.empty():
|
||||
try:
|
||||
item: tuple[int, int, str] = self.output_queue.get(timeout=0.1)
|
||||
_, _, message = item
|
||||
print(message, flush=True)
|
||||
self.output_queue.task_done()
|
||||
except Empty:
|
||||
continue
|
||||
except Exception as e:
|
||||
print(f"Error in output worker: {e}")
|
||||
continue
|
||||
|
||||
def stop(self) -> None:
|
||||
"""Stop the output worker thread"""
|
||||
self.stop_event.set()
|
||||
if self.output_thread.is_alive():
|
||||
self.output_thread.join()
|
||||
|
||||
|
||||
# Configure console for UTF-8 output on Windows
|
||||
if os.name == "nt": # Windows
|
||||
# Try to set console to UTF-8 mode
|
||||
try:
|
||||
# Set stdout and stderr to UTF-8 encoding
|
||||
# Note: reconfigure() was added in Python 3.7
|
||||
if hasattr(sys.stdout, "reconfigure") and callable(
|
||||
getattr(sys.stdout, "reconfigure", None)
|
||||
):
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
|
||||
if hasattr(sys.stderr, "reconfigure") and callable(
|
||||
getattr(sys.stderr, "reconfigure", None)
|
||||
):
|
||||
sys.stderr.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
|
||||
except (AttributeError, OSError):
|
||||
# Fallback for older Python versions or if reconfigure fails
|
||||
pass
|
||||
|
||||
|
||||
# Environment flags for backward compatibility
|
||||
_SHOW_COMPILE = os.environ.get("FASTLED_TEST_SHOW_COMPILE", "").lower() in (
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
)
|
||||
_SHOW_LINK = os.environ.get("FASTLED_TEST_SHOW_LINK", "").lower() in (
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class FailedTest:
|
||||
name: str
|
||||
return_code: int
|
||||
stdout: str
|
||||
|
||||
|
||||
def check_iwyu_available() -> bool:
|
||||
"""Check if include-what-you-use is available in the system"""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["include-what-you-use", "--version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (
|
||||
subprocess.CalledProcessError,
|
||||
FileNotFoundError,
|
||||
subprocess.TimeoutExpired,
|
||||
):
|
||||
return False
|
||||
|
||||
|
||||
def run_command(
|
||||
command: str | list[str],
|
||||
use_gdb: bool = False,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> tuple[int, str]:
|
||||
captured_lines: list[str] = []
|
||||
|
||||
# Determine command type
|
||||
is_test_execution = False
|
||||
is_compile = False
|
||||
is_link = False
|
||||
if isinstance(command, str):
|
||||
cmd_lower = command.replace("\\", "/").lower()
|
||||
# Check if running test executable
|
||||
is_test_execution = (
|
||||
"/test_" in cmd_lower
|
||||
or ".build/bin/test_" in cmd_lower
|
||||
or cmd_lower.endswith(".exe")
|
||||
)
|
||||
# Check if compiling
|
||||
is_compile = "-c" in cmd_lower and (".cpp" in cmd_lower or ".c" in cmd_lower)
|
||||
# Check if linking
|
||||
is_link = (
|
||||
not is_compile
|
||||
and not is_test_execution
|
||||
and ("-o" in cmd_lower or "lib" in cmd_lower)
|
||||
)
|
||||
|
||||
if use_gdb:
|
||||
with tempfile.NamedTemporaryFile(mode="w+", delete=False) as gdb_script:
|
||||
gdb_script.write("set pagination off\n")
|
||||
gdb_script.write("run\n")
|
||||
gdb_script.write("bt full\n")
|
||||
gdb_script.write("info registers\n")
|
||||
gdb_script.write("x/16i $pc\n")
|
||||
gdb_script.write("thread apply all bt full\n")
|
||||
gdb_script.write("quit\n")
|
||||
|
||||
gdb_command = (
|
||||
f"gdb -return-child-result -batch -x {gdb_script.name} --args {command}"
|
||||
)
|
||||
process = subprocess.Popen(
|
||||
gdb_command,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, # Merge stderr into stdout
|
||||
shell=True,
|
||||
text=False,
|
||||
)
|
||||
assert process.stdout is not None
|
||||
# Stream and capture output
|
||||
while True:
|
||||
line_bytes = process.stdout.readline()
|
||||
line = line_bytes.decode("utf-8", errors="ignore")
|
||||
if not line and process.poll() is not None:
|
||||
break
|
||||
if line:
|
||||
captured_lines.append(line.rstrip())
|
||||
# Always print GDB output (it's only used for crashes anyway)
|
||||
try:
|
||||
print(line, end="", flush=True)
|
||||
except UnicodeEncodeError:
|
||||
# Fallback: replace problematic characters
|
||||
print(
|
||||
line.encode("utf-8", errors="replace").decode(
|
||||
"utf-8", errors="replace"
|
||||
),
|
||||
end="",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
os.unlink(gdb_script.name)
|
||||
output = "\n".join(captured_lines)
|
||||
return process.returncode, output
|
||||
else:
|
||||
# Optimize list commands to avoid shell overhead
|
||||
if isinstance(command, list):
|
||||
# Optimize python commands and use shell=False for better performance
|
||||
python_exe = optimize_python_command(command)
|
||||
process = subprocess.Popen(
|
||||
python_exe,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, # Merge stderr into stdout
|
||||
shell=False, # Use shell=False for better performance with list commands
|
||||
text=False,
|
||||
)
|
||||
else:
|
||||
# String commands still need shell=True
|
||||
process = subprocess.Popen(
|
||||
command,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, # Merge stderr into stdout
|
||||
shell=True,
|
||||
text=False,
|
||||
)
|
||||
assert process.stdout is not None
|
||||
# Stream and capture output
|
||||
while True:
|
||||
line_bytes = process.stdout.readline()
|
||||
line = line_bytes.decode("utf-8", errors="ignore")
|
||||
if not line and process.poll() is not None:
|
||||
break
|
||||
if line:
|
||||
captured_lines.append(line.rstrip())
|
||||
# Determine if we should print this line
|
||||
should_print = (
|
||||
verbose # Always print in verbose mode
|
||||
or (is_compile and show_compile) # Print compilation if enabled
|
||||
or (is_link and show_link) # Print linking if enabled
|
||||
or (not is_test_execution) # Print non-test output
|
||||
or (
|
||||
is_test_execution and process.returncode != 0
|
||||
) # Print failed test output
|
||||
or (
|
||||
is_test_execution
|
||||
and any(
|
||||
marker in line
|
||||
for marker in [
|
||||
"Running test:",
|
||||
"Test passed",
|
||||
"Test FAILED",
|
||||
"passed with return code",
|
||||
"Test output:",
|
||||
]
|
||||
)
|
||||
) # Print test status
|
||||
)
|
||||
if should_print:
|
||||
try:
|
||||
# Add prefix for compile/link commands
|
||||
if is_compile and show_compile:
|
||||
print("[COMPILE] ", end="", flush=True)
|
||||
elif is_link and show_link:
|
||||
print("[LINK] ", end="", flush=True)
|
||||
print(line, end="", flush=True)
|
||||
except UnicodeEncodeError:
|
||||
# Fallback: replace problematic characters
|
||||
print(
|
||||
line.encode("utf-8", errors="replace").decode(
|
||||
"utf-8", errors="replace"
|
||||
),
|
||||
end="",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
output = "\n".join(captured_lines)
|
||||
return process.returncode, output
|
||||
|
||||
|
||||
def compile_tests(
|
||||
clean: bool = False,
|
||||
unknown_args: list[str] = [],
|
||||
specific_test: str | None = None,
|
||||
quick_build: bool = True,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Compile C++ tests using the Python build system.
|
||||
"""
|
||||
os.chdir(str(PROJECT_ROOT))
|
||||
print("🔧 Compiling tests using Python build system")
|
||||
|
||||
_compile_tests_python(
|
||||
clean,
|
||||
unknown_args,
|
||||
specific_test,
|
||||
quick_build=quick_build,
|
||||
verbose=verbose,
|
||||
show_compile=show_compile,
|
||||
show_link=show_link,
|
||||
)
|
||||
|
||||
|
||||
def _compile_tests_python(
|
||||
clean: bool = False,
|
||||
unknown_args: list[str] = [],
|
||||
specific_test: str | None = None,
|
||||
quick_build: bool = True,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""Python build system with PCH optimization"""
|
||||
# Use the optimized cpp_test_compile system directly
|
||||
import subprocess
|
||||
|
||||
cmd = ["uv", "run", "python", "-m", "ci.compiler.cpp_test_compile"]
|
||||
|
||||
if specific_test:
|
||||
cmd.extend(["--test", specific_test])
|
||||
if clean:
|
||||
cmd.append("--clean")
|
||||
if verbose:
|
||||
cmd.append("--verbose")
|
||||
if "--check" in unknown_args:
|
||||
cmd.append("--check")
|
||||
if "--no-pch" in unknown_args:
|
||||
cmd.append("--no-pch")
|
||||
# Forward debug mode to the compiler when quick_build is disabled
|
||||
if not quick_build:
|
||||
cmd.append("--debug")
|
||||
|
||||
print("🚀 Using Python build system with PCH optimization")
|
||||
result = subprocess.run(cmd)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"Unit test compilation failed with return code {result.returncode}"
|
||||
)
|
||||
|
||||
|
||||
def run_tests(
|
||||
specific_test: str | None = None,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Run compiled tests with GDB crash analysis support.
|
||||
"""
|
||||
_run_tests_python(specific_test)
|
||||
|
||||
|
||||
def _run_tests_python(
|
||||
specific_test: str | None = None,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""Run tests from Python build system"""
|
||||
# Import the test compiler system
|
||||
from ci.compiler.test_compiler import FastLEDTestCompiler
|
||||
|
||||
# Get test executables from Python build system
|
||||
test_compiler = FastLEDTestCompiler.get_existing_instance()
|
||||
if not test_compiler:
|
||||
print("No compiled tests found. Run compilation first.")
|
||||
sys.exit(1)
|
||||
|
||||
test_executables = test_compiler.get_test_executables(specific_test)
|
||||
if not test_executables:
|
||||
test_name = specific_test or "any tests"
|
||||
print(f"No test executables found for: {test_name}")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"Running {len(test_executables)} tests from Python build...")
|
||||
|
||||
# Print list of tests that will be executed
|
||||
print("Tests to execute:")
|
||||
for i, test_exec in enumerate(test_executables, 1):
|
||||
# Convert absolute path to relative for display
|
||||
rel_path = os.path.relpath(test_exec.executable_path)
|
||||
print(f" {i}. {test_exec.name} ({rel_path})")
|
||||
print("")
|
||||
|
||||
failed_tests: list[FailedTest] = []
|
||||
|
||||
# Convert to file list format for compatibility with existing logic
|
||||
files: list[str] = []
|
||||
test_paths: dict[str, str] = {}
|
||||
for test_exec in test_executables:
|
||||
file_name = test_exec.name
|
||||
if os.name == "nt" and not file_name.endswith(".exe"):
|
||||
file_name += ".exe"
|
||||
files.append(file_name)
|
||||
test_paths[file_name] = str(test_exec.executable_path)
|
||||
|
||||
print(f"Starting test execution for {len(files)} test files...")
|
||||
_execute_test_files(files, "", failed_tests, specific_test, test_paths)
|
||||
_handle_test_results(failed_tests)
|
||||
|
||||
|
||||
def _execute_test_files(
|
||||
files: list[str],
|
||||
test_dir: str,
|
||||
failed_tests: list[FailedTest],
|
||||
specific_test: str | None,
|
||||
test_paths: dict[str, str] | None = None,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Execute test files in parallel with full GDB crash analysis.
|
||||
|
||||
Args:
|
||||
files: List of test file names
|
||||
test_dir: Directory containing tests (for CMake) or empty string (for Python API)
|
||||
failed_tests: List to collect failed tests
|
||||
specific_test: Specific test name if filtering
|
||||
test_paths: Dict mapping file names to full paths (for Python API)
|
||||
"""
|
||||
total_tests = len(files)
|
||||
successful_tests = 0
|
||||
completed_tests = 0
|
||||
|
||||
# Initialize output buffer for ordered output
|
||||
output_buffer = OutputBuffer()
|
||||
output_buffer.write(0, f"Executing {total_tests} test files in parallel...")
|
||||
|
||||
# Determine number of workers based on configuration
|
||||
|
||||
# Get configuration from args
|
||||
args = parse_args()
|
||||
|
||||
# Force sequential execution if NO_PARALLEL is set
|
||||
if os.environ.get("NO_PARALLEL"):
|
||||
max_workers = 1
|
||||
output_buffer.write(
|
||||
0, "NO_PARALLEL environment variable set - forcing sequential execution"
|
||||
)
|
||||
elif args.sequential:
|
||||
max_workers = 1
|
||||
elif args.parallel:
|
||||
max_workers = args.parallel
|
||||
else:
|
||||
max_workers = max(1, multiprocessing.cpu_count() - 1) # Leave one core free
|
||||
|
||||
# Check memory limit
|
||||
if args.max_memory:
|
||||
memory_limit = args.max_memory * 1024 * 1024 # Convert MB to bytes
|
||||
available_memory = psutil.virtual_memory().available
|
||||
if memory_limit > available_memory:
|
||||
output_buffer.write(
|
||||
0,
|
||||
f"Warning: Requested memory limit {args.max_memory}MB exceeds available memory {available_memory / (1024 * 1024):.0f}MB",
|
||||
)
|
||||
output_buffer.write(
|
||||
0, "Reducing number of parallel workers to stay within memory limits"
|
||||
)
|
||||
# Estimate memory per test based on previous runs or default to 100MB
|
||||
memory_per_test = 100 * 1024 * 1024 # 100MB per test
|
||||
max_parallel_by_memory = max(1, memory_limit // memory_per_test)
|
||||
max_workers = min(max_workers, max_parallel_by_memory)
|
||||
|
||||
output_buffer.write(0, f"Using {max_workers} parallel workers")
|
||||
|
||||
# Thread-safe counter
|
||||
counter_lock = Lock()
|
||||
|
||||
def run_single_test(
|
||||
test_file: str, test_index: int
|
||||
) -> tuple[bool, float, str, int]:
|
||||
"""Run a single test and return its results"""
|
||||
nonlocal completed_tests
|
||||
|
||||
if test_paths:
|
||||
test_path = test_paths[test_file]
|
||||
else:
|
||||
test_path = os.path.join(test_dir, test_file)
|
||||
|
||||
# For .cpp files, compile them first
|
||||
if test_path.endswith(".cpp"):
|
||||
# Create a temporary directory for compilation
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Compile the test file
|
||||
output_buffer.write(
|
||||
test_index,
|
||||
f"[{test_index}/{total_tests}] Compiling test: {test_file}",
|
||||
)
|
||||
compile_cmd = [
|
||||
"python",
|
||||
"-m",
|
||||
"ziglang",
|
||||
"c++",
|
||||
"-o",
|
||||
os.path.join(temp_dir, "test.exe"),
|
||||
test_path,
|
||||
"-I",
|
||||
os.path.join(PROJECT_ROOT, "src"),
|
||||
"-I",
|
||||
os.path.join(PROJECT_ROOT, "tests"),
|
||||
# NOTE: Compiler flags now come from build configuration TOML
|
||||
]
|
||||
return_code, stdout = run_command(compile_cmd)
|
||||
if return_code != 0:
|
||||
output_buffer.write(
|
||||
test_index,
|
||||
f"[{test_index}/{total_tests}] ERROR: Failed to compile test: {test_file}",
|
||||
)
|
||||
return False, 0.0, f"Failed to compile test: {stdout}", return_code
|
||||
|
||||
# Update test_path to point to the compiled executable
|
||||
test_path = os.path.join(temp_dir, "test.exe")
|
||||
|
||||
if not (os.path.isfile(test_path) and os.access(test_path, os.X_OK)):
|
||||
output_buffer.write(
|
||||
test_index,
|
||||
f"[{test_index}/{total_tests}] ERROR: Test file not found or not executable: {test_path}",
|
||||
)
|
||||
return False, 0.0, f"Test file not found or not executable: {test_path}", 1
|
||||
|
||||
output_buffer.write(
|
||||
test_index, f"[{test_index}/{total_tests}] Running test: {test_file}"
|
||||
)
|
||||
if verbose:
|
||||
output_buffer.write(test_index, f" Command: {test_path}")
|
||||
|
||||
start_time = time.time()
|
||||
# Pass --minimal flag to doctest when not in verbose mode to suppress output unless tests fail
|
||||
cmd = [test_path]
|
||||
if not verbose:
|
||||
cmd.append("--minimal")
|
||||
return_code, stdout = run_command(cmd)
|
||||
elapsed_time = time.time() - start_time
|
||||
|
||||
output = stdout
|
||||
failure_pattern = re.compile(r"Test .+ failed with return code (\d+)")
|
||||
failure_match = failure_pattern.search(output)
|
||||
is_crash = failure_match is not None
|
||||
|
||||
# Handle crashes with GDB (must be done synchronously)
|
||||
if is_crash:
|
||||
output_buffer.write(
|
||||
test_index, f"Test crashed. Re-running with GDB to get stack trace..."
|
||||
)
|
||||
_, gdb_stdout = run_command(test_path, use_gdb=True)
|
||||
stdout += "\n--- GDB Output ---\n" + gdb_stdout
|
||||
|
||||
# Extract crash information
|
||||
crash_info = extract_crash_info(gdb_stdout)
|
||||
output_buffer.write(
|
||||
test_index, f"Crash occurred at: {crash_info.file}:{crash_info.line}"
|
||||
)
|
||||
output_buffer.write(test_index, f"Cause: {crash_info.cause}")
|
||||
output_buffer.write(test_index, f"Stack: {crash_info.stack}")
|
||||
|
||||
# Print output based on verbosity and status
|
||||
if verbose or return_code != 0:
|
||||
output_buffer.write(test_index, "Test output:")
|
||||
output_buffer.write(test_index, stdout)
|
||||
|
||||
if return_code == 0:
|
||||
output_buffer.write(
|
||||
test_index, f" Test {test_file} passed in {elapsed_time:.2f}s"
|
||||
)
|
||||
else:
|
||||
output_buffer.write(
|
||||
test_index,
|
||||
f" Test {test_file} FAILED with return code {return_code} in {elapsed_time:.2f}s",
|
||||
)
|
||||
|
||||
with counter_lock:
|
||||
completed_tests += 1
|
||||
|
||||
return return_code == 0, elapsed_time, stdout, return_code
|
||||
|
||||
try:
|
||||
# Run tests in parallel using ThreadPoolExecutor
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
# Submit all tests
|
||||
future_to_test = {
|
||||
executor.submit(run_single_test, test_file, i + 1): test_file
|
||||
for i, test_file in enumerate(files)
|
||||
}
|
||||
|
||||
# Process results as they complete
|
||||
for future in as_completed(future_to_test):
|
||||
test_file = future_to_test[future]
|
||||
try:
|
||||
success, _, stdout, return_code = future.result()
|
||||
if success:
|
||||
with counter_lock:
|
||||
successful_tests += 1
|
||||
else:
|
||||
failed_tests.append(
|
||||
FailedTest(
|
||||
name=test_file, return_code=return_code, stdout=stdout
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
output_buffer.write(
|
||||
0, f"ERROR: Test {test_file} failed with exception: {e}"
|
||||
)
|
||||
failed_tests.append(
|
||||
FailedTest(name=test_file, return_code=1, stdout=str(e))
|
||||
)
|
||||
|
||||
# Print final summary
|
||||
output_buffer.write(
|
||||
0,
|
||||
f"Test execution complete: {successful_tests} passed, {len(failed_tests)} failed",
|
||||
)
|
||||
if successful_tests == total_tests:
|
||||
output_buffer.write(0, "All tests passed successfully!")
|
||||
else:
|
||||
output_buffer.write(
|
||||
0, f"Some tests failed ({len(failed_tests)} of {total_tests})"
|
||||
)
|
||||
|
||||
finally:
|
||||
# Ensure output buffer is stopped
|
||||
output_buffer.stop()
|
||||
|
||||
|
||||
def _handle_test_results(
|
||||
failed_tests: list[FailedTest], *, verbose: bool = False
|
||||
) -> None:
|
||||
"""Handle test results and exit appropriately (preserving existing logic)"""
|
||||
if failed_tests:
|
||||
print("Failed tests summary:")
|
||||
failures: List[TestFailureInfo] = []
|
||||
for failed_test in failed_tests:
|
||||
print(
|
||||
f"Test {failed_test.name} failed with return code {failed_test.return_code}"
|
||||
)
|
||||
# Always show output on failure
|
||||
print("Output:")
|
||||
# Show indented output for better readability
|
||||
for line in failed_test.stdout.splitlines():
|
||||
print(f" {line}")
|
||||
print() # Add spacing between failed tests
|
||||
|
||||
failures.append(
|
||||
TestFailureInfo(
|
||||
test_name=failed_test.name,
|
||||
command=f"test_{failed_test.name}",
|
||||
return_code=failed_test.return_code,
|
||||
output=failed_test.stdout,
|
||||
error_type="test_execution_failure",
|
||||
)
|
||||
)
|
||||
|
||||
tests_failed = len(failed_tests)
|
||||
failed_test_names = [test.name for test in failed_tests]
|
||||
print(
|
||||
f"{tests_failed} test{'s' if tests_failed != 1 else ''} failed: {', '.join(failed_test_names)}"
|
||||
)
|
||||
raise TestExecutionFailedException(f"{tests_failed} test(s) failed", failures)
|
||||
if verbose:
|
||||
print("All tests passed.")
|
||||
|
||||
|
||||
@dataclass
|
||||
class CrashInfo:
|
||||
cause: str = "Unknown"
|
||||
stack: str = "Unknown"
|
||||
file: str = "Unknown"
|
||||
line: str = "Unknown"
|
||||
|
||||
|
||||
def extract_crash_info(gdb_output: str) -> CrashInfo:
|
||||
lines = gdb_output.split("\n")
|
||||
crash_info = CrashInfo()
|
||||
|
||||
try:
|
||||
for i, line in enumerate(lines):
|
||||
if line.startswith("Program received signal"):
|
||||
try:
|
||||
crash_info.cause = line.split(":", 1)[1].strip()
|
||||
except IndexError:
|
||||
crash_info.cause = line.strip()
|
||||
elif line.startswith("#0"):
|
||||
crash_info.stack = line
|
||||
for j in range(i, len(lines)):
|
||||
if "at" in lines[j]:
|
||||
try:
|
||||
_, location = lines[j].split("at", 1)
|
||||
location = location.strip()
|
||||
if ":" in location:
|
||||
crash_info.file, crash_info.line = location.rsplit(
|
||||
":", 1
|
||||
)
|
||||
else:
|
||||
crash_info.file = location
|
||||
except ValueError:
|
||||
pass # If split fails, we keep the default values
|
||||
break
|
||||
break
|
||||
except Exception as e:
|
||||
print(f"Error parsing GDB output: {e}")
|
||||
|
||||
return crash_info
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Compile and run C++ tests")
|
||||
parser.add_argument(
|
||||
"--compile-only",
|
||||
action="store_true",
|
||||
help="Only compile the tests without running them",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--run-only",
|
||||
action="store_true",
|
||||
help="Only run the tests without compiling them",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--only-run-failed-test",
|
||||
action="store_true",
|
||||
help="Only run the tests that failed in the previous run",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--clean", action="store_true", help="Clean build before compiling"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--test",
|
||||
help="Specific test to run (without extension)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbose",
|
||||
action="store_true",
|
||||
help="Enable verbose output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--show-compile",
|
||||
action="store_true",
|
||||
help="Show compilation commands and output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--show-link",
|
||||
action="store_true",
|
||||
help="Show linking commands and output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--parallel",
|
||||
type=int,
|
||||
help="Number of parallel test processes to run (default: CPU count - 1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sequential",
|
||||
action="store_true",
|
||||
help="Run tests sequentially (disables parallel execution)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-memory",
|
||||
type=int,
|
||||
help="Maximum memory usage in MB for parallel test execution",
|
||||
)
|
||||
|
||||
# Create mutually exclusive group for compiler selection
|
||||
compiler_group = parser.add_mutually_exclusive_group()
|
||||
compiler_group.add_argument(
|
||||
"--clang",
|
||||
help="Use Clang compiler",
|
||||
action="store_true",
|
||||
)
|
||||
compiler_group.add_argument(
|
||||
"--gcc",
|
||||
help="Use GCC compiler (default on non-Windows)",
|
||||
action="store_true",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--check",
|
||||
action="store_true",
|
||||
help="Enable static analysis (IWYU, clang-tidy)",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--no-unity",
|
||||
action="store_true",
|
||||
help="Disable unity builds for cpp tests",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-pch",
|
||||
action="store_true",
|
||||
help="Disable precompiled headers (PCH) for unit tests",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--debug",
|
||||
action="store_true",
|
||||
help="Use debug build mode with full debug symbols (default is quick mode with -g0)",
|
||||
)
|
||||
|
||||
args, unknown = parser.parse_known_args()
|
||||
args.unknown = unknown
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def main() -> None:
|
||||
try:
|
||||
args = parse_args()
|
||||
|
||||
# Get verbosity flags from args
|
||||
|
||||
run_only = args.run_only
|
||||
compile_only = args.compile_only
|
||||
specific_test = args.test
|
||||
# only_run_failed_test feature to be implemented in future
|
||||
_ = args.only_run_failed_test
|
||||
use_clang = args.clang
|
||||
|
||||
no_unity = args.no_unity
|
||||
quick_build = (
|
||||
not args.debug
|
||||
) # Default to quick mode unless --debug is specified
|
||||
# use_gcc = args.gcc
|
||||
|
||||
if not run_only:
|
||||
passthrough_args = args.unknown
|
||||
if use_clang:
|
||||
passthrough_args.append("--use-clang")
|
||||
if args.check:
|
||||
passthrough_args.append("--check")
|
||||
if no_unity:
|
||||
passthrough_args.append("--no-unity")
|
||||
if args.no_pch:
|
||||
passthrough_args.append("--no-pch")
|
||||
# Note: --gcc is handled by not passing --use-clang (GCC is the default in compiler/cpp_test_compile.py)
|
||||
compile_tests(
|
||||
clean=args.clean,
|
||||
unknown_args=passthrough_args,
|
||||
specific_test=specific_test,
|
||||
quick_build=quick_build,
|
||||
verbose=args.verbose,
|
||||
show_compile=args.show_compile,
|
||||
show_link=args.show_link,
|
||||
)
|
||||
|
||||
if not compile_only:
|
||||
if specific_test:
|
||||
run_tests(
|
||||
specific_test,
|
||||
verbose=args.verbose,
|
||||
show_compile=args.show_compile,
|
||||
show_link=args.show_link,
|
||||
)
|
||||
else:
|
||||
# Use our own test runner instead of CTest since CTest integration is broken
|
||||
run_tests(
|
||||
None,
|
||||
verbose=args.verbose,
|
||||
show_compile=args.show_compile,
|
||||
show_link=args.show_link,
|
||||
)
|
||||
except (
|
||||
CompilationFailedException,
|
||||
TestExecutionFailedException,
|
||||
TestTimeoutException,
|
||||
) as e:
|
||||
# Print detailed failure information
|
||||
print("\n" + "=" * 60)
|
||||
print("FASTLED TEST FAILURE DETAILS")
|
||||
print("=" * 60)
|
||||
print(e.get_detailed_failure_info())
|
||||
print("=" * 60)
|
||||
|
||||
# Exit with appropriate code
|
||||
if e.failures:
|
||||
# Use the return code from the first failure, or 1 if none available
|
||||
exit_code = (
|
||||
e.failures[0].return_code if e.failures[0].return_code != 0 else 1
|
||||
)
|
||||
else:
|
||||
exit_code = 1
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,784 @@
|
||||
"""
|
||||
Enhanced Arduino Package Index Implementation with Pydantic
|
||||
|
||||
This module provides a robust, production-ready Arduino package management system
|
||||
that fully complies with the Arduino CLI Package Index JSON Specification.
|
||||
|
||||
Key Features:
|
||||
- Pydantic models with comprehensive validation
|
||||
- Multi-source package index support
|
||||
- Caching and persistence
|
||||
- Package installation and dependency resolution
|
||||
- Search and filtering capabilities
|
||||
- Checksum validation and error handling
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import sys
|
||||
import tarfile
|
||||
import zipfile
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Optional, Set, Union
|
||||
from urllib.parse import urlparse
|
||||
from urllib.request import urlopen
|
||||
|
||||
import httpx
|
||||
from pydantic import (
|
||||
BaseModel,
|
||||
EmailStr,
|
||||
Field,
|
||||
HttpUrl,
|
||||
ValidationError,
|
||||
field_validator,
|
||||
model_validator,
|
||||
)
|
||||
|
||||
|
||||
# Custom Exceptions
|
||||
class PackageParsingError(Exception):
|
||||
"""Error parsing package index data"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class PackageInstallationError(Exception):
|
||||
"""Error during package installation"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class PackageValidationError(Exception):
|
||||
"""Error validating package data"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
# Core Pydantic Models with Enhanced Validation
|
||||
|
||||
|
||||
class Help(BaseModel):
|
||||
"""Help information with online resources"""
|
||||
|
||||
online: HttpUrl = Field(description="URL to online help resources")
|
||||
|
||||
class Config:
|
||||
schema_extra = {
|
||||
"example": {"online": "https://github.com/espressif/arduino-esp32"}
|
||||
}
|
||||
|
||||
|
||||
class Board(BaseModel):
|
||||
"""Board information with complete properties"""
|
||||
|
||||
name: str = Field(min_length=1, max_length=200, description="Board display name")
|
||||
properties: Dict[str, Any] = Field(
|
||||
default_factory=dict, description="Board configuration properties"
|
||||
)
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def validate_board_name(cls, v: str) -> str:
|
||||
"""Validate board name format"""
|
||||
if not v.strip():
|
||||
raise ValueError("Board name cannot be empty or whitespace")
|
||||
return v.strip()
|
||||
|
||||
class Config:
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"name": "ESP32 Dev Module",
|
||||
"properties": {
|
||||
"upload.tool": "esptool_py",
|
||||
"upload.maximum_size": "1310720",
|
||||
"build.mcu": "esp32",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class ToolDependency(BaseModel):
|
||||
"""Tool dependency specification with validation"""
|
||||
|
||||
packager: str = Field(
|
||||
min_length=1, max_length=100, description="Tool packager name"
|
||||
)
|
||||
name: str = Field(pattern=r"^[a-zA-Z0-9_.-]+$", description="Tool name")
|
||||
version: str = Field(min_length=1, description="Required tool version")
|
||||
|
||||
@field_validator("version")
|
||||
@classmethod
|
||||
def validate_version_format(cls, v: str) -> str:
|
||||
"""Validate version format - supports semantic versioning and Arduino versioning"""
|
||||
if not v.strip():
|
||||
raise ValueError("Version cannot be empty")
|
||||
# Allow flexible versioning (semantic, date-based, etc.)
|
||||
return v.strip()
|
||||
|
||||
class Config:
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"packager": "esp32",
|
||||
"name": "xtensa-esp32-elf-gcc",
|
||||
"version": "esp-2021r2-patch5-8.4.0",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class Platform(BaseModel):
|
||||
"""Platform specification with comprehensive validation"""
|
||||
|
||||
name: str = Field(min_length=1, max_length=200, description="Platform display name")
|
||||
architecture: str = Field(
|
||||
pattern=r"^[a-zA-Z0-9_-]+$", description="Target architecture"
|
||||
)
|
||||
version: str = Field(description="Platform version")
|
||||
category: str = Field(min_length=1, description="Platform category")
|
||||
url: HttpUrl = Field(description="Download URL for platform archive")
|
||||
archive_filename: str = Field(
|
||||
alias="archiveFileName", description="Archive file name"
|
||||
)
|
||||
checksum: str = Field(
|
||||
pattern=r"^SHA-256:[a-fA-F0-9]{64}$", description="SHA-256 checksum"
|
||||
)
|
||||
size_mb: float = Field(gt=0, alias="size", description="Archive size in megabytes")
|
||||
boards: List[Board] = Field(
|
||||
default_factory=lambda: [], description="Supported boards"
|
||||
)
|
||||
tool_dependencies: List[ToolDependency] = Field(
|
||||
default_factory=lambda: [],
|
||||
alias="toolsDependencies",
|
||||
description="Required tool dependencies",
|
||||
)
|
||||
help: Help = Field(description="Help and documentation links")
|
||||
|
||||
@field_validator("version")
|
||||
@classmethod
|
||||
def validate_version_format(cls, v: str) -> str:
|
||||
"""Validate version format"""
|
||||
if not v.strip():
|
||||
raise ValueError("Version cannot be empty")
|
||||
return v.strip()
|
||||
|
||||
@field_validator("size_mb", mode="before")
|
||||
@classmethod
|
||||
def convert_size_from_bytes(cls, v: Union[str, int, float]) -> float:
|
||||
"""Convert size from bytes to megabytes if needed"""
|
||||
if isinstance(v, str):
|
||||
try:
|
||||
size_bytes = int(v)
|
||||
return size_bytes / (1024 * 1024)
|
||||
except ValueError:
|
||||
raise ValueError(f"Invalid size format: {v}")
|
||||
elif isinstance(v, (int, float)):
|
||||
if v > 1024 * 1024: # Assume bytes if > 1MB
|
||||
return v / (1024 * 1024)
|
||||
return v # Already in MB
|
||||
return v
|
||||
|
||||
@field_validator("archive_filename")
|
||||
@classmethod
|
||||
def validate_archive_filename(cls, v: str) -> str:
|
||||
"""Validate archive filename format"""
|
||||
valid_extensions = [".zip", ".tar.gz", ".tar.bz2", ".tar.xz", ".tar.zst"]
|
||||
if not any(v.lower().endswith(ext) for ext in valid_extensions):
|
||||
raise ValueError(
|
||||
f"Archive must have one of these extensions: {valid_extensions}"
|
||||
)
|
||||
return v
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
schema_extra: Dict[str, Any] = {
|
||||
"example": {
|
||||
"name": "ESP32 Arduino",
|
||||
"architecture": "esp32",
|
||||
"version": "2.0.5",
|
||||
"category": "ESP32",
|
||||
"url": "https://github.com/espressif/arduino-esp32/releases/download/2.0.5/esp32-2.0.5.zip",
|
||||
"archiveFileName": "esp32-2.0.5.zip",
|
||||
"checksum": "SHA-256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
|
||||
"size": "50000000",
|
||||
"boards": [],
|
||||
"toolsDependencies": [],
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class SystemDownload(BaseModel):
|
||||
"""System-specific download information for tools"""
|
||||
|
||||
host: str = Field(min_length=1, description="Target host system identifier")
|
||||
url: HttpUrl = Field(description="Download URL for this system")
|
||||
archive_filename: str = Field(
|
||||
alias="archiveFileName", description="Archive file name"
|
||||
)
|
||||
checksum: str = Field(
|
||||
pattern=r"^SHA-256:[a-fA-F0-9]{64}$", description="SHA-256 checksum"
|
||||
)
|
||||
size_mb: float = Field(gt=0, alias="size", description="Archive size in megabytes")
|
||||
|
||||
@field_validator("size_mb", mode="before")
|
||||
@classmethod
|
||||
def convert_size_from_bytes(cls, v: Union[str, int, float]) -> float:
|
||||
"""Convert size from bytes to megabytes if needed"""
|
||||
if isinstance(v, str):
|
||||
try:
|
||||
size_bytes = int(v)
|
||||
return size_bytes / (1024 * 1024)
|
||||
except ValueError:
|
||||
raise ValueError(f"Invalid size format: {v}")
|
||||
elif isinstance(v, (int, float)):
|
||||
if v > 1024 * 1024: # Assume bytes if > 1MB
|
||||
return v / (1024 * 1024)
|
||||
return v # Already in MB
|
||||
return v
|
||||
|
||||
@field_validator("host")
|
||||
@classmethod
|
||||
def validate_host_format(cls, v: str) -> str:
|
||||
"""Validate host system identifier"""
|
||||
# Common host patterns: i686-pc-linux-gnu, x86_64-apple-darwin, etc.
|
||||
if not v.strip():
|
||||
raise ValueError("Host identifier cannot be empty")
|
||||
return v.strip()
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"host": "x86_64-pc-linux-gnu",
|
||||
"url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
|
||||
"archiveFileName": "xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
|
||||
"checksum": "SHA-256:abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890",
|
||||
"size": "150000000",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class Tool(BaseModel):
|
||||
"""Tool with system-specific downloads and enhanced functionality"""
|
||||
|
||||
name: str = Field(pattern=r"^[a-zA-Z0-9_.-]+$", description="Tool name")
|
||||
version: str = Field(min_length=1, description="Tool version")
|
||||
systems: List[SystemDownload] = Field(
|
||||
min_length=1, description="System-specific downloads"
|
||||
)
|
||||
|
||||
@field_validator("systems")
|
||||
@classmethod
|
||||
def validate_unique_systems(cls, v: List[SystemDownload]) -> List[SystemDownload]:
|
||||
"""Ensure no duplicate host systems"""
|
||||
hosts = [system.host for system in v]
|
||||
if len(hosts) != len(set(hosts)):
|
||||
raise ValueError("Duplicate host systems found in tool downloads")
|
||||
return v
|
||||
|
||||
def get_system_download(self, host_pattern: str) -> Optional[SystemDownload]:
|
||||
"""Get system download matching host pattern"""
|
||||
for system in self.systems:
|
||||
if host_pattern in system.host:
|
||||
return system
|
||||
return None
|
||||
|
||||
def get_compatible_systems(self) -> List[str]:
|
||||
"""Get list of compatible host systems"""
|
||||
return [system.host for system in self.systems]
|
||||
|
||||
class Config:
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"name": "xtensa-esp32-elf-gcc",
|
||||
"version": "esp-2021r2-patch5-8.4.0",
|
||||
"systems": [
|
||||
{
|
||||
"host": "x86_64-pc-linux-gnu",
|
||||
"url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
|
||||
"archiveFileName": "xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
|
||||
"checksum": "SHA-256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
|
||||
"size": "150000000",
|
||||
}
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class Package(BaseModel):
|
||||
"""Package containing platforms and tools with comprehensive validation"""
|
||||
|
||||
name: str = Field(pattern=r"^[A-Za-z0-9_.-]+$", description="Package identifier")
|
||||
maintainer: str = Field(
|
||||
min_length=1, max_length=200, description="Package maintainer"
|
||||
)
|
||||
website_url: HttpUrl = Field(alias="websiteURL", description="Package website URL")
|
||||
email: EmailStr = Field(description="Maintainer contact email")
|
||||
help: Help = Field(description="Help and documentation links")
|
||||
platforms: List[Platform] = Field(
|
||||
default_factory=lambda: [], description="Available platforms"
|
||||
)
|
||||
tools: List[Tool] = Field(default_factory=lambda: [], description="Available tools")
|
||||
|
||||
@field_validator("platforms")
|
||||
@classmethod
|
||||
def validate_unique_platforms(cls, v: List[Platform]) -> List[Platform]:
|
||||
"""Ensure no duplicate platform architecture/version combinations"""
|
||||
seen: Set[tuple[str, str]] = set()
|
||||
for platform in v:
|
||||
key = (platform.architecture, platform.version)
|
||||
if key in seen:
|
||||
raise ValueError(
|
||||
f"Duplicate platform found: {platform.architecture} v{platform.version}"
|
||||
)
|
||||
seen.add(key)
|
||||
return v
|
||||
|
||||
@field_validator("tools")
|
||||
@classmethod
|
||||
def validate_unique_tools(cls, v: List[Tool]) -> List[Tool]:
|
||||
"""Ensure no duplicate tool name/version combinations"""
|
||||
seen: Set[tuple[str, str]] = set()
|
||||
for tool in v:
|
||||
key = (tool.name, tool.version)
|
||||
if key in seen:
|
||||
raise ValueError(f"Duplicate tool found: {tool.name} v{tool.version}")
|
||||
seen.add(key)
|
||||
return v
|
||||
|
||||
def find_platform(
|
||||
self, architecture: str, version: Optional[str] = None
|
||||
) -> Optional[Platform]:
|
||||
"""Find platform by architecture and optionally version"""
|
||||
for platform in self.platforms:
|
||||
if platform.architecture == architecture:
|
||||
if version is None or platform.version == version:
|
||||
return platform
|
||||
return None
|
||||
|
||||
def find_tool(self, name: str, version: Optional[str] = None) -> Optional[Tool]:
|
||||
"""Find tool by name and optionally version"""
|
||||
for tool in self.tools:
|
||||
if tool.name == name:
|
||||
if version is None or tool.version == version:
|
||||
return tool
|
||||
return None
|
||||
|
||||
def get_latest_platform_version(self, architecture: str) -> Optional[str]:
|
||||
"""Get the latest version for a given architecture"""
|
||||
versions = [p.version for p in self.platforms if p.architecture == architecture]
|
||||
if not versions:
|
||||
return None
|
||||
# Simple version sorting - can be enhanced with proper semver parsing
|
||||
return sorted(versions)[-1]
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
schema_extra: Dict[str, Any] = {
|
||||
"example": {
|
||||
"name": "esp32",
|
||||
"maintainer": "Espressif Systems",
|
||||
"websiteURL": "https://github.com/espressif/arduino-esp32",
|
||||
"email": "hr@espressif.com",
|
||||
"help": {"online": "https://github.com/espressif/arduino-esp32"},
|
||||
"platforms": [],
|
||||
"tools": [],
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class PackageIndex(BaseModel):
|
||||
"""Root package index containing multiple packages"""
|
||||
|
||||
packages: List[Package] = Field(min_length=1, description="Available packages")
|
||||
|
||||
@field_validator("packages")
|
||||
@classmethod
|
||||
def validate_unique_packages(cls, v: List[Package]) -> List[Package]:
|
||||
"""Ensure no duplicate package names"""
|
||||
names = [pkg.name for pkg in v]
|
||||
if len(names) != len(set(names)):
|
||||
raise ValueError("Duplicate package names found in index")
|
||||
return v
|
||||
|
||||
def find_package(self, name: str) -> Optional[Package]:
|
||||
"""Find package by name"""
|
||||
for package in self.packages:
|
||||
if package.name == name:
|
||||
return package
|
||||
return None
|
||||
|
||||
def get_all_platforms(self) -> List[Platform]:
|
||||
"""Get all platforms from all packages"""
|
||||
platforms: List[Platform] = []
|
||||
for package in self.packages:
|
||||
platforms.extend(package.platforms)
|
||||
return platforms
|
||||
|
||||
def get_all_tools(self) -> List[Tool]:
|
||||
"""Get all tools from all packages"""
|
||||
tools: List[Tool] = []
|
||||
for package in self.packages:
|
||||
tools.extend(package.tools)
|
||||
return tools
|
||||
|
||||
|
||||
# Enhanced Parser with Validation and Error Handling
|
||||
|
||||
|
||||
class PackageIndexParser:
|
||||
"""Enhanced parser with comprehensive validation and error handling"""
|
||||
|
||||
def __init__(self, timeout: int = 30):
|
||||
"""Initialize parser with timeout configuration"""
|
||||
self.timeout = timeout
|
||||
|
||||
def parse_package_index(self, json_str: str) -> PackageIndex:
|
||||
"""Parse and validate package index JSON"""
|
||||
try:
|
||||
raw_data = json.loads(json_str)
|
||||
return PackageIndex(**raw_data)
|
||||
except ValidationError as e:
|
||||
raise PackageParsingError(f"Invalid package index format: {e}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise PackageParsingError(f"Invalid JSON format: {e}")
|
||||
|
||||
def parse_from_url(self, url: str) -> PackageIndex:
|
||||
"""Fetch and parse package index from URL with validation"""
|
||||
try:
|
||||
print(f"Fetching package index from: {url}")
|
||||
with urlopen(url, timeout=self.timeout) as response:
|
||||
content = response.read()
|
||||
json_str = content.decode("utf-8")
|
||||
|
||||
return self.parse_package_index(json_str)
|
||||
|
||||
except Exception as e:
|
||||
raise PackageParsingError(f"Error fetching package index from {url}: {e}")
|
||||
|
||||
async def parse_from_url_async(self, url: str) -> PackageIndex:
|
||||
"""Async version of URL parsing"""
|
||||
try:
|
||||
print(f"Fetching package index from: {url}")
|
||||
async with httpx.AsyncClient(timeout=self.timeout) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
json_str = response.text
|
||||
|
||||
return self.parse_package_index(json_str)
|
||||
|
||||
except Exception as e:
|
||||
raise PackageParsingError(f"Error fetching package index from {url}: {e}")
|
||||
|
||||
|
||||
# Package Manager Configuration
|
||||
|
||||
|
||||
class PackageManagerConfig(BaseModel):
|
||||
"""Configuration for package manager with validation"""
|
||||
|
||||
cache_dir: Path = Field(default_factory=lambda: Path.home() / ".arduino_packages")
|
||||
sources: List[HttpUrl] = Field(
|
||||
default_factory=lambda: [], description="Package index URLs"
|
||||
)
|
||||
timeout: int = Field(
|
||||
default=30, gt=0, le=300, description="Request timeout in seconds"
|
||||
)
|
||||
max_retries: int = Field(
|
||||
default=3, ge=0, le=10, description="Maximum retry attempts"
|
||||
)
|
||||
verify_checksums: bool = Field(
|
||||
default=True, description="Verify download checksums"
|
||||
)
|
||||
allow_insecure: bool = Field(default=False, description="Allow insecure downloads")
|
||||
|
||||
@field_validator("cache_dir")
|
||||
@classmethod
|
||||
def validate_cache_dir(cls, v: Path) -> Path:
|
||||
"""Ensure cache directory is valid"""
|
||||
if v.exists() and not v.is_dir():
|
||||
raise ValueError(f"Cache path exists but is not a directory: {v}")
|
||||
return v
|
||||
|
||||
class Config:
|
||||
validate_assignment = True
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"cache_dir": "~/.arduino_packages",
|
||||
"sources": [
|
||||
"https://espressif.github.io/arduino-esp32/package_esp32_index.json"
|
||||
],
|
||||
"timeout": 30,
|
||||
"max_retries": 3,
|
||||
"verify_checksums": True,
|
||||
"allow_insecure": False,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
# Utility Functions for Enhanced Functionality
|
||||
|
||||
|
||||
def verify_checksum(file_path: Path, expected_checksum: str) -> bool:
|
||||
"""Verify file checksum against expected SHA-256 value"""
|
||||
if not expected_checksum.startswith("SHA-256:"):
|
||||
raise ValueError(f"Invalid checksum format: {expected_checksum}")
|
||||
|
||||
expected_hash = expected_checksum[8:] # Remove 'SHA-256:' prefix
|
||||
|
||||
sha256_hash = hashlib.sha256()
|
||||
with open(file_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
sha256_hash.update(chunk)
|
||||
|
||||
actual_hash = sha256_hash.hexdigest()
|
||||
return actual_hash.lower() == expected_hash.lower()
|
||||
|
||||
|
||||
def extract_archive(archive_path: Path, extract_to: Path) -> bool:
|
||||
"""Extract archive to specified directory with format detection"""
|
||||
try:
|
||||
archive_str = str(archive_path)
|
||||
|
||||
if archive_path.suffix == ".zip":
|
||||
with zipfile.ZipFile(archive_path, "r") as zip_ref:
|
||||
zip_ref.extractall(extract_to)
|
||||
elif archive_str.endswith((".tar.gz", ".tgz")):
|
||||
with tarfile.open(archive_path, "r:gz") as tar_ref:
|
||||
tar_ref.extractall(extract_to)
|
||||
elif archive_str.endswith(".tar.bz2"):
|
||||
with tarfile.open(archive_path, "r:bz2") as tar_ref:
|
||||
tar_ref.extractall(extract_to)
|
||||
elif archive_str.endswith(".tar.xz"):
|
||||
with tarfile.open(archive_path, "r:xz") as tar_ref:
|
||||
tar_ref.extractall(extract_to)
|
||||
else:
|
||||
raise ValueError(f"Unsupported archive format: {archive_path}")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error extracting archive {archive_path}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def format_size(size_mb: float) -> str:
|
||||
"""Format size in a human-readable way"""
|
||||
if size_mb < 1:
|
||||
return f"{size_mb * 1024:.1f} KB"
|
||||
elif size_mb < 1024:
|
||||
return f"{size_mb:.1f} MB"
|
||||
else:
|
||||
return f"{size_mb / 1024:.1f} GB"
|
||||
|
||||
|
||||
# Display Functions with Enhanced Formatting
|
||||
|
||||
|
||||
def display_package_info(package: Package) -> None:
|
||||
"""Display package information with enhanced formatting"""
|
||||
print(f"\n📦 Package: {package.name}")
|
||||
print(f"👤 Maintainer: {package.maintainer}")
|
||||
print(f"🌐 Website: {package.website_url}")
|
||||
print(f"📧 Email: {package.email}")
|
||||
print(f"📚 Help: {package.help.online}")
|
||||
print(f"🛠️ Platforms: {len(package.platforms)}")
|
||||
print(f"🔧 Tools: {len(package.tools)}")
|
||||
|
||||
# Show platform information
|
||||
for i, platform in enumerate(package.platforms[:3]): # Show first 3 platforms
|
||||
print(f"\n 📋 Platform {i + 1}: {platform.name} v{platform.version}")
|
||||
print(f" Architecture: {platform.architecture}")
|
||||
print(f" Category: {platform.category}")
|
||||
print(f" Size: {format_size(platform.size_mb)}")
|
||||
print(f" Archive: {platform.archive_filename}")
|
||||
print(f" Checksum: {platform.checksum[:24]}...")
|
||||
print(f" Help: {platform.help.online}")
|
||||
print(f" Boards: {len(platform.boards)}")
|
||||
|
||||
# Show first 5 boards
|
||||
for board in platform.boards[:5]:
|
||||
print(f" • {board.name}")
|
||||
if len(platform.boards) > 5:
|
||||
print(f" ... and {len(platform.boards) - 5} more")
|
||||
|
||||
print(f" Tool Dependencies: {len(platform.tool_dependencies)}")
|
||||
for dep in platform.tool_dependencies[:3]: # Show first 3 dependencies
|
||||
print(f" • {dep.name} v{dep.version} ({dep.packager})")
|
||||
if len(platform.tool_dependencies) > 3:
|
||||
print(f" ... and {len(platform.tool_dependencies) - 3} more")
|
||||
|
||||
if len(package.platforms) > 3:
|
||||
print(f"\n ... and {len(package.platforms) - 3} more platforms")
|
||||
|
||||
|
||||
def display_validation_summary(package_index: PackageIndex) -> None:
|
||||
"""Display validation summary for the package index"""
|
||||
print(f"\n✅ VALIDATION SUMMARY")
|
||||
print(f" 📦 Total packages: {len(package_index.packages)}")
|
||||
|
||||
total_platforms = sum(len(pkg.platforms) for pkg in package_index.packages)
|
||||
total_tools = sum(len(pkg.tools) for pkg in package_index.packages)
|
||||
total_boards = sum(
|
||||
len(platform.boards)
|
||||
for pkg in package_index.packages
|
||||
for platform in pkg.platforms
|
||||
)
|
||||
|
||||
print(f" 🛠️ Total platforms: {total_platforms}")
|
||||
print(f" 🔧 Total tools: {total_tools}")
|
||||
print(f" 💾 Total boards: {total_boards}")
|
||||
|
||||
# Show architectures
|
||||
architectures: Set[str] = set()
|
||||
for pkg in package_index.packages:
|
||||
for platform in pkg.platforms:
|
||||
architectures.add(platform.architecture)
|
||||
|
||||
print(f" 🏗️ Architectures: {', '.join(sorted(architectures))}")
|
||||
|
||||
|
||||
# Demonstration Functions
|
||||
|
||||
|
||||
def demo_esp32_parsing() -> Optional[PackageIndex]:
|
||||
"""Demonstrate parsing ESP32 package index with enhanced validation"""
|
||||
ESP32_URL = "https://espressif.github.io/arduino-esp32/package_esp32_index.json"
|
||||
|
||||
try:
|
||||
parser = PackageIndexParser(timeout=30)
|
||||
package_index = parser.parse_from_url(ESP32_URL)
|
||||
|
||||
print("🎉 Successfully parsed ESP32 package index with Pydantic validation!")
|
||||
display_validation_summary(package_index)
|
||||
|
||||
# Display first package
|
||||
if package_index.packages:
|
||||
display_package_info(package_index.packages[0])
|
||||
|
||||
return package_index
|
||||
|
||||
except PackageParsingError as e:
|
||||
print(f"❌ Package parsing error: {e}")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"❌ Unexpected error: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def demo_model_validation() -> None:
|
||||
"""Demonstrate Pydantic model validation capabilities"""
|
||||
print("\n🧪 TESTING PYDANTIC MODEL VALIDATION")
|
||||
|
||||
# Define valid platform data
|
||||
valid_platform_data: Dict[str, Any] = {
|
||||
"name": "ESP32 Arduino",
|
||||
"architecture": "esp32",
|
||||
"version": "2.0.5",
|
||||
"category": "ESP32",
|
||||
"url": "https://github.com/espressif/arduino-esp32/releases/download/2.0.5/esp32-2.0.5.zip",
|
||||
"archiveFileName": "esp32-2.0.5.zip",
|
||||
"checksum": "SHA-256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
|
||||
"size": "50000000", # Should convert to MB
|
||||
"boards": [],
|
||||
"toolsDependencies": [],
|
||||
"help": {"online": "https://github.com/espressif/arduino-esp32"},
|
||||
}
|
||||
|
||||
# Test valid platform
|
||||
try:
|
||||
platform = Platform(**valid_platform_data)
|
||||
print(f"✅ Valid platform created: {platform.name} v{platform.version}")
|
||||
print(f" Size converted: {platform.size_mb:.1f} MB")
|
||||
|
||||
except ValidationError as e:
|
||||
print(f"❌ Unexpected validation error: {e}")
|
||||
|
||||
# Test invalid platform
|
||||
try:
|
||||
invalid_platform_data = valid_platform_data.copy()
|
||||
invalid_platform_data["checksum"] = "invalid-checksum-format"
|
||||
|
||||
platform = Platform(**invalid_platform_data)
|
||||
print("❌ Should have failed validation!")
|
||||
|
||||
except ValidationError as e:
|
||||
print(f"✅ Correctly caught invalid checksum: {e.errors()[0]['msg']}")
|
||||
|
||||
# Test invalid URL
|
||||
try:
|
||||
invalid_url_data = valid_platform_data.copy()
|
||||
invalid_url_data["url"] = "not-a-valid-url"
|
||||
|
||||
platform = Platform(**invalid_url_data)
|
||||
print("❌ Should have failed URL validation!")
|
||||
|
||||
except ValidationError as e:
|
||||
print(f"✅ Correctly caught invalid URL: {e.errors()[0]['msg']}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Main function demonstrating enhanced package index functionality"""
|
||||
print("🚀 ENHANCED ARDUINO PACKAGE INDEX WITH PYDANTIC")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
# Demo model validation
|
||||
demo_model_validation()
|
||||
|
||||
# Demo ESP32 parsing
|
||||
package_index = demo_esp32_parsing()
|
||||
|
||||
# Interactive options
|
||||
try:
|
||||
print(f"\n📋 AVAILABLE OPTIONS:")
|
||||
print("1. Show detailed tools information")
|
||||
print("2. Search for specific architecture")
|
||||
print("3. Exit")
|
||||
|
||||
choice = input("\nEnter your choice (1-3): ").strip()
|
||||
|
||||
if choice == "1" and package_index and package_index.packages:
|
||||
# Simple tools info display (without importing original)
|
||||
pkg = package_index.packages[0]
|
||||
print(f"\n🔧 TOOLS INFORMATION for {pkg.name}")
|
||||
print(f"Total tools: {len(pkg.tools)}")
|
||||
for tool in pkg.tools[:3]: # Show first 3 tools
|
||||
print(
|
||||
f" • {tool.name} v{tool.version} ({len(tool.systems)} systems)"
|
||||
)
|
||||
if len(pkg.tools) > 3:
|
||||
print(f" ... and {len(pkg.tools) - 3} more tools")
|
||||
|
||||
elif choice == "2" and package_index:
|
||||
arch = input("Enter architecture to search for: ").strip()
|
||||
platforms = [
|
||||
p
|
||||
for pkg in package_index.packages
|
||||
for p in pkg.platforms
|
||||
if p.architecture == arch
|
||||
]
|
||||
if platforms:
|
||||
print(
|
||||
f"\n🔍 Found {len(platforms)} platforms for architecture '{arch}':"
|
||||
)
|
||||
for platform in platforms:
|
||||
print(
|
||||
f" • {platform.name} v{platform.version} ({format_size(platform.size_mb)})"
|
||||
)
|
||||
else:
|
||||
print(f"❌ No platforms found for architecture '{arch}'")
|
||||
else:
|
||||
print("👋 Goodbye!")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n👋 Interrupted by user")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n👋 Interrupted by user")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,937 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PlatformIO artifact cache implementation for speeding up CI builds.
|
||||
|
||||
This module implements the cache mechanism described in FEATURE_PIO_SPEEDUP.md,
|
||||
providing functionality to:
|
||||
1. Parse platformio.ini files with zip URLs
|
||||
2. Download and cache platform/framework artifacts
|
||||
3. Install artifacts via PlatformIO CLI
|
||||
4. Modify platformio.ini in-place with resolved local paths
|
||||
"""
|
||||
|
||||
import _thread
|
||||
import configparser
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import urllib.parse
|
||||
import zipfile
|
||||
from concurrent.futures import Future, ThreadPoolExecutor, as_completed
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from types import TracebackType
|
||||
from typing import Any, Dict, List, Optional, Tuple, Union, cast
|
||||
|
||||
import fasteners
|
||||
import httpx
|
||||
|
||||
from ci.compiler.platformio_ini import PlatformIOIni
|
||||
from ci.util.running_process import RunningProcess
|
||||
from ci.util.url_utils import sanitize_url_for_path
|
||||
|
||||
|
||||
@dataclass
|
||||
class DownloadResult:
|
||||
"""Result of a download operation."""
|
||||
|
||||
url: str
|
||||
temp_path: Path
|
||||
exception: Optional[BaseException] = None
|
||||
|
||||
@property
|
||||
def success(self) -> bool:
|
||||
"""True if download succeeded."""
|
||||
return self.exception is None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ArtifactProcessingResult:
|
||||
"""Result of processing an artifact."""
|
||||
|
||||
url: str
|
||||
is_framework: bool
|
||||
env_section: str
|
||||
resolved_path: Optional[str] = None
|
||||
exception: Optional[BaseException] = None
|
||||
|
||||
@property
|
||||
def success(self) -> bool:
|
||||
"""True if processing succeeded."""
|
||||
return self.exception is None and self.resolved_path is not None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ZipUrlInfo:
|
||||
"""Information about a zip URL found in platformio.ini."""
|
||||
|
||||
url: str
|
||||
section_name: str
|
||||
option_name: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class ManifestResult:
|
||||
"""Result of manifest validation and type detection."""
|
||||
|
||||
is_valid: bool
|
||||
manifest_path: Optional[Path]
|
||||
is_framework: bool
|
||||
|
||||
|
||||
def _get_remote_file_size(url: str) -> Optional[int]:
|
||||
"""Get file size from URL using HEAD request."""
|
||||
try:
|
||||
parsed_url = urllib.parse.urlparse(url)
|
||||
if parsed_url.scheme in ("http", "https"):
|
||||
with httpx.Client(follow_redirects=True) as client:
|
||||
response = client.head(
|
||||
url, headers={"User-Agent": "PlatformIO-Cache/1.0"}
|
||||
)
|
||||
response.raise_for_status()
|
||||
content_length = response.headers.get("Content-Length")
|
||||
if content_length:
|
||||
return int(content_length)
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to get file size for {url}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def _format_file_size(size_bytes: Optional[int]) -> str:
|
||||
"""Format file size in human-readable format."""
|
||||
if size_bytes is None:
|
||||
return "unknown size"
|
||||
|
||||
if size_bytes < 1024:
|
||||
return f"{size_bytes} B"
|
||||
elif size_bytes < 1024 * 1024:
|
||||
return f"{size_bytes / 1024:.1f} KB"
|
||||
elif size_bytes < 1024 * 1024 * 1024:
|
||||
return f"{size_bytes / (1024 * 1024):.1f} MB"
|
||||
else:
|
||||
return f"{size_bytes / (1024 * 1024 * 1024):.1f} GB"
|
||||
|
||||
|
||||
# Global cache to track PlatformIO installations in this session
|
||||
_session_installation_cache: set[str] = set()
|
||||
|
||||
# Global cancellation event for handling keyboard interrupts
|
||||
_global_cancel_event = threading.Event()
|
||||
|
||||
# Lock for thread-safe operations
|
||||
_download_lock = threading.Lock()
|
||||
|
||||
|
||||
def _get_status_file(artifact_dir: Path, cache_key: str) -> Path:
|
||||
"""Get the JSON status file path for an artifact."""
|
||||
# Use simple descriptive filename since artifacts are already in unique directories
|
||||
return artifact_dir / "info.json"
|
||||
|
||||
|
||||
def _read_status(status_file: Path) -> Optional[Dict[str, Any]]:
|
||||
"""Read status from JSON file."""
|
||||
if not status_file.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(status_file, "r") as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def _write_status(status_file: Path, status: Dict[str, Any]) -> None:
|
||||
"""Write status to JSON file."""
|
||||
try:
|
||||
with open(status_file, "w") as f:
|
||||
json.dump(status, f, indent=2)
|
||||
except OSError as e:
|
||||
logger.warning(f"Failed to write status file {status_file}: {e}")
|
||||
|
||||
|
||||
def _is_processing_complete(status_file: Path) -> bool:
|
||||
"""Check if processing is complete based on status file."""
|
||||
status = _read_status(status_file)
|
||||
return status is not None and status.get("status") == "complete"
|
||||
|
||||
|
||||
def _download_with_progress(
|
||||
url: str, temp_path: Path, cancel_event: threading.Event
|
||||
) -> DownloadResult:
|
||||
"""Download HTTP/HTTPS file with cancellation support."""
|
||||
try:
|
||||
# HTTP/HTTPS download using httpx with streaming
|
||||
with httpx.Client(follow_redirects=True, timeout=30.0) as client:
|
||||
with client.stream(
|
||||
"GET", url, headers={"User-Agent": "PlatformIO-Cache/1.0"}
|
||||
) as response:
|
||||
response.raise_for_status()
|
||||
|
||||
with open(temp_path, "wb") as f:
|
||||
for chunk in response.iter_bytes(chunk_size=8192):
|
||||
# Check for cancellation periodically
|
||||
if cancel_event.is_set():
|
||||
cancelled_error = RuntimeError(
|
||||
f"Download cancelled for {url}"
|
||||
)
|
||||
logger.warning(
|
||||
f"Download cancelled for {url}",
|
||||
exc_info=cancelled_error,
|
||||
)
|
||||
return DownloadResult(url, temp_path, cancelled_error)
|
||||
f.write(chunk)
|
||||
|
||||
return DownloadResult(url, temp_path) # Success
|
||||
|
||||
except KeyboardInterrupt as e:
|
||||
# Set cancel event and interrupt main thread
|
||||
cancel_event.set()
|
||||
_thread.interrupt_main()
|
||||
logger.warning(
|
||||
f"Download interrupted by KeyboardInterrupt for {url}", exc_info=e
|
||||
)
|
||||
return DownloadResult(url, temp_path, e)
|
||||
except Exception as e:
|
||||
if not cancel_event.is_set():
|
||||
logger.error(f"Download failed for {url}: {e}", exc_info=e)
|
||||
else:
|
||||
logger.warning(f"Download failed for {url} (cancelled): {e}", exc_info=e)
|
||||
return DownloadResult(url, temp_path, e)
|
||||
|
||||
|
||||
def _copy_file_with_progress(
|
||||
url: str, temp_path: Path, cancel_event: threading.Event
|
||||
) -> DownloadResult:
|
||||
"""Copy local file with cancellation support."""
|
||||
try:
|
||||
parsed_url = urllib.parse.urlparse(url)
|
||||
|
||||
# File URL - copy local file
|
||||
logger.debug(f"Parsing file URL: {url}")
|
||||
logger.debug(f"Parsed URL path: {parsed_url.path}")
|
||||
|
||||
# Handle both Unix and Windows file URLs
|
||||
if os.name == "nt": # Windows
|
||||
# On Windows, file:///C:/path becomes /C:/path, so remove leading slash
|
||||
if (
|
||||
parsed_url.path.startswith("/")
|
||||
and len(parsed_url.path) > 3
|
||||
and parsed_url.path[2] == ":"
|
||||
):
|
||||
source_path = Path(parsed_url.path[1:])
|
||||
else:
|
||||
source_path = Path(parsed_url.path)
|
||||
else: # Unix-like
|
||||
source_path = Path(parsed_url.path)
|
||||
|
||||
logger.debug(f"Resolved file path: {source_path}")
|
||||
|
||||
if not source_path.exists():
|
||||
raise FileNotFoundError(f"Source file not found: {source_path}")
|
||||
|
||||
# Check for cancellation before copy
|
||||
if cancel_event.is_set():
|
||||
cancelled_error = RuntimeError(f"Copy cancelled for {url}")
|
||||
logger.warning(f"Copy cancelled for {url}", exc_info=cancelled_error)
|
||||
return DownloadResult(url, temp_path, cancelled_error)
|
||||
|
||||
shutil.copy2(source_path, temp_path)
|
||||
return DownloadResult(url, temp_path) # Success
|
||||
|
||||
except KeyboardInterrupt as e:
|
||||
# Set cancel event and interrupt main thread
|
||||
cancel_event.set()
|
||||
_thread.interrupt_main()
|
||||
logger.warning(f"Copy interrupted by KeyboardInterrupt for {url}", exc_info=e)
|
||||
return DownloadResult(url, temp_path, e)
|
||||
except Exception as e:
|
||||
if not cancel_event.is_set():
|
||||
logger.error(f"Copy failed for {url}: {e}", exc_info=e)
|
||||
else:
|
||||
logger.warning(f"Copy failed for {url} (cancelled): {e}", exc_info=e)
|
||||
return DownloadResult(url, temp_path, e)
|
||||
|
||||
|
||||
def clear_session_cache() -> None:
|
||||
"""Clear the session installation cache. Useful for testing."""
|
||||
global _session_installation_cache
|
||||
_session_installation_cache.clear()
|
||||
logger.debug("Cleared session installation cache")
|
||||
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Reduce httpx verbosity to avoid log spew from HTTP requests
|
||||
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||
|
||||
|
||||
class PlatformIOCache:
|
||||
"""Enhanced cache manager for PlatformIO artifacts."""
|
||||
|
||||
def __init__(self, cache_dir: Path):
|
||||
"""Initialize cache manager with directory structure."""
|
||||
self.cache_dir = cache_dir
|
||||
# Simplified structure: each artifact gets its own directory directly in cache root
|
||||
# containing both the .zip and extracted/ folder
|
||||
|
||||
# Create directory structure
|
||||
self.cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _get_cache_key(self, url: str) -> str:
|
||||
"""Generate cache key from URL - sanitized for filesystem use."""
|
||||
return str(sanitize_url_for_path(url))
|
||||
|
||||
def download_artifact(self, url: str) -> str:
|
||||
"""
|
||||
Download and cache an artifact from the given URL.
|
||||
Returns the absolute path to the cached zip file.
|
||||
"""
|
||||
cache_key = self._get_cache_key(url)
|
||||
# Each artifact gets its own directory
|
||||
artifact_dir = self.cache_dir / cache_key
|
||||
artifact_dir.mkdir(parents=True, exist_ok=True)
|
||||
cached_path = artifact_dir / "artifact.zip"
|
||||
|
||||
# Use read-write locking for concurrent safety
|
||||
# Start with write lock for downloading, can upgrade to read if cache hit
|
||||
lock_path = str(artifact_dir / "artifact.lock")
|
||||
rw_lock = fasteners.InterProcessReaderWriterLock(lock_path)
|
||||
with rw_lock.write_lock():
|
||||
if cached_path.exists():
|
||||
# Check if processing was completed successfully
|
||||
status_file = _get_status_file(artifact_dir, cache_key)
|
||||
if not _is_processing_complete(status_file):
|
||||
logger.warning(
|
||||
f"Cache incomplete (no completion status), re-downloading: {url}"
|
||||
)
|
||||
# Remove incomplete zip file
|
||||
cached_path.unlink()
|
||||
# Also remove any partial extraction
|
||||
extracted_dir = artifact_dir / "extracted"
|
||||
if extracted_dir.exists():
|
||||
shutil.rmtree(extracted_dir)
|
||||
else:
|
||||
print(f"Using cached artifact: {cached_path}")
|
||||
# Cache hit - return the cached path (write lock will be released)
|
||||
return str(cached_path)
|
||||
|
||||
# Check URL scheme to determine action
|
||||
parsed_url = urllib.parse.urlparse(url)
|
||||
|
||||
if parsed_url.scheme in ("http", "https"):
|
||||
# Download to temporary file first (atomic operation)
|
||||
file_size = _get_remote_file_size(url)
|
||||
size_str = _format_file_size(file_size)
|
||||
print(f"Downloading: {url} ({size_str})")
|
||||
temp_file_handle = tempfile.NamedTemporaryFile(
|
||||
delete=False, suffix=".zip"
|
||||
)
|
||||
temp_path = Path(temp_file_handle.name)
|
||||
temp_file_handle.close() # Close immediately to avoid Windows file locking issues
|
||||
|
||||
# Create a thread-local cancel event
|
||||
thread_cancel_event = threading.Event()
|
||||
|
||||
# Download directly (we're already in a thread from the main pool)
|
||||
download_result = _download_with_progress(
|
||||
url, temp_path, thread_cancel_event
|
||||
)
|
||||
elif parsed_url.scheme == "file":
|
||||
# File URL - copy from local file
|
||||
print(f"Copying from local file: {url}")
|
||||
temp_file_handle = tempfile.NamedTemporaryFile(
|
||||
delete=False, suffix=".zip"
|
||||
)
|
||||
temp_path = Path(temp_file_handle.name)
|
||||
temp_file_handle.close() # Close immediately to avoid Windows file locking issues
|
||||
|
||||
# Create a thread-local cancel event
|
||||
thread_cancel_event = threading.Event()
|
||||
|
||||
# Copy directly (we're already in a thread from the main pool)
|
||||
download_result = _copy_file_with_progress(
|
||||
url, temp_path, thread_cancel_event
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unsupported URL scheme: {parsed_url.scheme}")
|
||||
|
||||
try:
|
||||
# Check if global cancellation was requested
|
||||
if _global_cancel_event.is_set():
|
||||
raise KeyboardInterrupt(
|
||||
"Download cancelled due to global interrupt"
|
||||
)
|
||||
|
||||
# Process the download result
|
||||
if not download_result.success:
|
||||
if download_result.exception is not None:
|
||||
raise download_result.exception
|
||||
else:
|
||||
raise RuntimeError(
|
||||
f"Download failed for unknown reason: {download_result.url}"
|
||||
)
|
||||
else:
|
||||
print(f"Download completed successfully: {download_result.url}")
|
||||
|
||||
# Atomic move to final location
|
||||
shutil.move(str(temp_path), str(cached_path))
|
||||
print(f"Successfully cached: {cached_path}")
|
||||
return str(cached_path)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Download failed: {e}")
|
||||
if temp_path.exists():
|
||||
temp_path.unlink()
|
||||
raise
|
||||
|
||||
|
||||
def _is_zip_web_url(value: str) -> bool:
|
||||
"""Enhanced URL detection for zip artifacts."""
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
|
||||
parsed = urllib.parse.urlparse(value)
|
||||
# Direct zip URLs
|
||||
if value.endswith(".zip"):
|
||||
return parsed.scheme in ("http", "https")
|
||||
return False
|
||||
|
||||
|
||||
def validate_and_detect_manifest(
|
||||
content_path: Path,
|
||||
) -> ManifestResult:
|
||||
"""
|
||||
Validate manifest files and auto-detect artifact type.
|
||||
"""
|
||||
# Try framework manifests first
|
||||
framework_manifests = ["framework.json", "package.json"]
|
||||
for manifest in framework_manifests:
|
||||
manifest_path = content_path / manifest
|
||||
if manifest_path.exists():
|
||||
try:
|
||||
with open(manifest_path, "r") as f:
|
||||
json.load(f) # Validate JSON syntax
|
||||
print(f"Found valid framework manifest: {manifest_path}")
|
||||
return ManifestResult(
|
||||
is_valid=True, manifest_path=manifest_path, is_framework=True
|
||||
)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.warning(f"Invalid JSON in {manifest_path}: {e}")
|
||||
|
||||
# Try platform manifest
|
||||
platform_manifest = content_path / "platform.json"
|
||||
if platform_manifest.exists():
|
||||
try:
|
||||
with open(platform_manifest, "r") as f:
|
||||
json.load(f) # Validate JSON syntax
|
||||
print(f"Found valid platform manifest: {platform_manifest}")
|
||||
return ManifestResult(
|
||||
is_valid=True, manifest_path=platform_manifest, is_framework=False
|
||||
)
|
||||
except json.JSONDecodeError as e:
|
||||
logger.warning(f"Invalid JSON in {platform_manifest}: {e}")
|
||||
|
||||
return ManifestResult(is_valid=False, manifest_path=None, is_framework=False)
|
||||
|
||||
|
||||
def get_platformio_command_path(path: Path) -> str:
|
||||
"""Get path format for PlatformIO command line usage."""
|
||||
import platform
|
||||
|
||||
resolved_path = path.resolve()
|
||||
|
||||
# On Windows, PlatformIO has issues with file:// URLs for local directories
|
||||
# Use native Windows paths instead
|
||||
if platform.system() == "Windows":
|
||||
return str(resolved_path)
|
||||
else:
|
||||
# On Unix systems, use proper file:// URLs
|
||||
posix_path = resolved_path.as_posix()
|
||||
return f"file://{posix_path}"
|
||||
|
||||
|
||||
def get_proper_file_url(path: Path) -> str:
|
||||
"""Convert a path to a proper file:// URL for platformio.ini files."""
|
||||
# For PlatformIO on Windows, use the Windows path directly (not file:// URL)
|
||||
# This avoids issues with path parsing in PlatformIO
|
||||
import platform
|
||||
|
||||
resolved_path = path.resolve()
|
||||
|
||||
if platform.system() == "Windows":
|
||||
# On Windows, PlatformIO handles Windows paths directly better than file:// URLs
|
||||
# Convert to Windows path format
|
||||
return str(resolved_path)
|
||||
else:
|
||||
# On Unix systems, use proper file:// URL format
|
||||
posix_path = resolved_path.as_posix()
|
||||
return f"file://{posix_path}"
|
||||
|
||||
|
||||
def unzip_and_install(
|
||||
cached_zip_path: Path,
|
||||
cache_manager: PlatformIOCache,
|
||||
is_framework: bool,
|
||||
env_section: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Enhanced unzip and install with manifest validation and cleanup.
|
||||
"""
|
||||
cached_zip_path_obj = cached_zip_path
|
||||
# Extract to a directory alongside the zip file
|
||||
artifact_dir = cached_zip_path_obj.parent
|
||||
extracted_dir = artifact_dir / "extracted"
|
||||
temp_unzip_dir = artifact_dir / "temp_extract"
|
||||
|
||||
try:
|
||||
# Clean extraction directory
|
||||
if temp_unzip_dir.exists():
|
||||
shutil.rmtree(temp_unzip_dir)
|
||||
temp_unzip_dir.mkdir(parents=True)
|
||||
|
||||
print(f"Extracting {cached_zip_path_obj} to {temp_unzip_dir}")
|
||||
|
||||
# Extract with better error handling
|
||||
try:
|
||||
with zipfile.ZipFile(cached_zip_path_obj, "r") as zip_ref:
|
||||
# Check for zip bombs (basic protection)
|
||||
total_size = sum(info.file_size for info in zip_ref.infolist())
|
||||
if total_size > 500 * 1024 * 1024: # 500MB limit
|
||||
raise ValueError("Archive too large, possible zip bomb")
|
||||
|
||||
zip_ref.extractall(temp_unzip_dir)
|
||||
print("Extraction completed successfully")
|
||||
|
||||
except zipfile.BadZipFile:
|
||||
logger.error(f"Invalid zip file: {cached_zip_path_obj}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Extraction failed: {e}")
|
||||
return False
|
||||
|
||||
# Find content directory (handle nested structures like GitHub archives)
|
||||
content_items = list(temp_unzip_dir.iterdir())
|
||||
if len(content_items) == 1 and content_items[0].is_dir():
|
||||
# Single root directory (common with GitHub/GitLab archives) - use it
|
||||
unzipped_content_path = content_items[0]
|
||||
print(f"Found nested directory structure: {content_items[0].name}")
|
||||
else:
|
||||
# Multiple items at root - use temp dir
|
||||
unzipped_content_path = temp_unzip_dir
|
||||
print("Using flat directory structure")
|
||||
|
||||
# Validate manifest files and auto-detect type
|
||||
manifest_result = validate_and_detect_manifest(unzipped_content_path)
|
||||
if not manifest_result.is_valid:
|
||||
logger.error(f"No valid manifest found in {unzipped_content_path}")
|
||||
return False
|
||||
|
||||
# Move to final location
|
||||
if extracted_dir.exists():
|
||||
shutil.rmtree(extracted_dir)
|
||||
shutil.move(unzipped_content_path, extracted_dir)
|
||||
|
||||
# Install via PlatformIO
|
||||
return install_with_platformio(
|
||||
extracted_dir, manifest_result.is_framework, env_section
|
||||
)
|
||||
|
||||
finally:
|
||||
# Cleanup temporary extraction
|
||||
if temp_unzip_dir.exists():
|
||||
print(f"Cleaning up extraction directory: {temp_unzip_dir}")
|
||||
shutil.rmtree(temp_unzip_dir)
|
||||
|
||||
|
||||
def install_with_platformio(
|
||||
content_path: Path, is_framework: bool, env_section: str
|
||||
) -> bool:
|
||||
"""Install extracted content using appropriate PlatformIO command."""
|
||||
content_path_obj = Path(content_path)
|
||||
command_path = get_platformio_command_path(content_path_obj)
|
||||
|
||||
# Create a cache key based on the installation type and path
|
||||
cache_key = f"{'framework' if is_framework else 'platform'}:{command_path}"
|
||||
|
||||
# Check if we've already installed this in this session
|
||||
if cache_key in _session_installation_cache:
|
||||
print(
|
||||
f"Skipping installation for {env_section}: already installed in this session ({command_path})"
|
||||
)
|
||||
return True
|
||||
|
||||
if is_framework:
|
||||
# Framework installation - use pkg install
|
||||
command = ["pio", "pkg", "install", "--global", command_path]
|
||||
else:
|
||||
# Platform installation - also use pkg install (new recommended way)
|
||||
command = ["pio", "pkg", "install", "--global", "--platform", command_path]
|
||||
|
||||
try:
|
||||
print(f"Installing for {env_section}: {' '.join(command)}")
|
||||
|
||||
# Use RunningProcess for streaming output
|
||||
process = RunningProcess(
|
||||
command,
|
||||
check=False, # We'll handle errors ourselves
|
||||
timeout=300, # 5 minute timeout
|
||||
)
|
||||
|
||||
# Stream output in real-time
|
||||
for line in process.line_iter(timeout=300):
|
||||
line_str = cast(str, line)
|
||||
print(f"PIO: {line_str}")
|
||||
|
||||
# Wait for completion and check result
|
||||
process.wait()
|
||||
|
||||
if process.returncode == 0:
|
||||
print("PlatformIO installation successful")
|
||||
# Add to session cache to avoid redundant installations
|
||||
_session_installation_cache.add(cache_key)
|
||||
return True
|
||||
else:
|
||||
logger.error(
|
||||
f"PlatformIO installation failed with return code: {process.returncode}"
|
||||
)
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
if "No such file or directory" in str(e) or "not found" in str(e).lower():
|
||||
logger.error("PlatformIO CLI not found. Is it installed and in PATH?")
|
||||
else:
|
||||
logger.error(f"PlatformIO installation failed: {e}")
|
||||
logger.error(f"Command: {' '.join(command)}")
|
||||
return False
|
||||
|
||||
|
||||
def handle_zip_artifact(
|
||||
zip_source: str,
|
||||
cache_manager: PlatformIOCache,
|
||||
env_section: str,
|
||||
) -> str:
|
||||
"""
|
||||
Enhanced artifact handler with validation and error recovery.
|
||||
Returns the resolved local path for the artifact.
|
||||
"""
|
||||
try:
|
||||
# Download and cache the artifact
|
||||
cached_zip_path = cache_manager.download_artifact(zip_source)
|
||||
|
||||
if not cached_zip_path or not Path(cached_zip_path).exists():
|
||||
raise FileNotFoundError(f"Download failed for {zip_source}")
|
||||
|
||||
# Extract and get the content path
|
||||
cache_key = cache_manager._get_cache_key(zip_source)
|
||||
cached_zip_path_obj = Path(cached_zip_path)
|
||||
|
||||
# Extract to a directory alongside the zip file
|
||||
artifact_dir = cached_zip_path_obj.parent
|
||||
extracted_dir = artifact_dir / "extracted"
|
||||
temp_unzip_dir = artifact_dir / "temp_extract"
|
||||
|
||||
# Check if already extracted
|
||||
if extracted_dir.exists():
|
||||
# Validate existing extraction and auto-detect type
|
||||
manifest_result = validate_and_detect_manifest(extracted_dir)
|
||||
if manifest_result.is_valid:
|
||||
print(f"Using existing extraction: {extracted_dir}")
|
||||
|
||||
# Check for completion status using URL-based cache key (consistent with status file creation)
|
||||
url_cache_key = cache_manager._get_cache_key(zip_source)
|
||||
status_file = _get_status_file(artifact_dir, url_cache_key)
|
||||
if _is_processing_complete(status_file):
|
||||
print(
|
||||
f"Skipping PlatformIO installation for {env_section}: found completion status"
|
||||
)
|
||||
return get_proper_file_url(extracted_dir)
|
||||
|
||||
# Create a session cache key for this artifact
|
||||
session_cache_key = f"{'framework' if manifest_result.is_framework else 'platform'}:{get_platformio_command_path(extracted_dir)}"
|
||||
|
||||
# If we've already handled this exact artifact in this session, skip PlatformIO entirely
|
||||
if session_cache_key in _session_installation_cache:
|
||||
print(
|
||||
f"Skipping PlatformIO installation for {env_section}: already processed in this session"
|
||||
)
|
||||
return get_proper_file_url(extracted_dir)
|
||||
|
||||
# Otherwise, install via PlatformIO
|
||||
install_success = install_with_platformio(
|
||||
extracted_dir, manifest_result.is_framework, env_section
|
||||
)
|
||||
# Status file will be created later using URL-based cache key
|
||||
|
||||
if install_success:
|
||||
print(f"Successfully installed {zip_source} for {env_section}")
|
||||
else:
|
||||
logger.warning(
|
||||
f"Installation completed with warnings for {zip_source}"
|
||||
)
|
||||
return get_proper_file_url(extracted_dir)
|
||||
|
||||
# Clean extraction directory for fresh extraction
|
||||
if temp_unzip_dir.exists():
|
||||
shutil.rmtree(temp_unzip_dir)
|
||||
temp_unzip_dir.mkdir(parents=True)
|
||||
|
||||
# Extract the zip
|
||||
with zipfile.ZipFile(cached_zip_path, "r") as zip_ref:
|
||||
zip_ref.extractall(temp_unzip_dir)
|
||||
|
||||
# Find content directory (handle nested structures)
|
||||
content_items = list(temp_unzip_dir.iterdir())
|
||||
if len(content_items) == 1 and content_items[0].is_dir():
|
||||
unzipped_content_path = content_items[0]
|
||||
else:
|
||||
unzipped_content_path = temp_unzip_dir
|
||||
|
||||
# Validate manifest files and auto-detect type
|
||||
manifest_result = validate_and_detect_manifest(unzipped_content_path)
|
||||
if not manifest_result.is_valid:
|
||||
raise ValueError(f"No valid manifest found in {unzipped_content_path}")
|
||||
|
||||
# Move to final location
|
||||
if extracted_dir.exists():
|
||||
shutil.rmtree(extracted_dir)
|
||||
shutil.move(unzipped_content_path, extracted_dir)
|
||||
|
||||
# Clean up temp directory
|
||||
if temp_unzip_dir.exists():
|
||||
shutil.rmtree(temp_unzip_dir)
|
||||
|
||||
# Install via PlatformIO
|
||||
install_success = install_with_platformio(
|
||||
extracted_dir, manifest_result.is_framework, env_section
|
||||
)
|
||||
|
||||
# Create status file with processing results
|
||||
cache_key = cache_manager._get_cache_key(zip_source)
|
||||
status_file = _get_status_file(artifact_dir, cache_key)
|
||||
|
||||
# Get zip file size
|
||||
zip_file = artifact_dir / "artifact.zip"
|
||||
zip_size = zip_file.stat().st_size if zip_file.exists() else 0
|
||||
|
||||
status = {
|
||||
"status": "complete" if install_success else "warning",
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"url": zip_source,
|
||||
"env_section": env_section,
|
||||
"extracted_dir": str(extracted_dir.relative_to(artifact_dir)),
|
||||
"zip_size_bytes": zip_size,
|
||||
}
|
||||
_write_status(status_file, status)
|
||||
|
||||
if install_success:
|
||||
print(f"Successfully installed {zip_source} for {env_section}")
|
||||
else:
|
||||
logger.warning(f"Installation completed with warnings for {zip_source}")
|
||||
|
||||
# Return the file URL for the extracted directory
|
||||
return get_proper_file_url(extracted_dir)
|
||||
|
||||
except Exception as e:
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
logger.error(f"Failed to handle artifact {zip_source}: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def _process_artifact(
|
||||
artifact_url: str,
|
||||
env_section: str,
|
||||
cache_manager: "PlatformIOCache",
|
||||
) -> ArtifactProcessingResult:
|
||||
"""Process a single artifact (download, extract, install)."""
|
||||
try:
|
||||
resolved_path = handle_zip_artifact(artifact_url, cache_manager, env_section)
|
||||
return ArtifactProcessingResult(
|
||||
url=artifact_url,
|
||||
is_framework=False, # This will be determined during processing
|
||||
env_section=env_section,
|
||||
resolved_path=resolved_path,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to process {artifact_url}: {e}", exc_info=e)
|
||||
return ArtifactProcessingResult(
|
||||
url=artifact_url,
|
||||
is_framework=False, # This will be determined during processing
|
||||
env_section=env_section,
|
||||
exception=e,
|
||||
)
|
||||
|
||||
|
||||
def _collect_all_zip_urls(pio_ini: PlatformIOIni) -> List[ZipUrlInfo]:
|
||||
"""
|
||||
Collect all zip URLs from platformio.ini in a single pass.
|
||||
"""
|
||||
all_urls: List[ZipUrlInfo] = []
|
||||
|
||||
# Collect platform URLs
|
||||
for section_name, option_name, url in pio_ini.get_platform_urls():
|
||||
if _is_zip_web_url(url):
|
||||
print(f"Found platform zip: {url} in {section_name}")
|
||||
all_urls.append(ZipUrlInfo(url, section_name, option_name))
|
||||
|
||||
# Collect framework URLs
|
||||
for section_name, option_name, url in pio_ini.get_framework_urls():
|
||||
if _is_zip_web_url(url):
|
||||
print(f"Found framework zip: {url} in {section_name}")
|
||||
all_urls.append(ZipUrlInfo(url, section_name, option_name))
|
||||
|
||||
return all_urls
|
||||
|
||||
|
||||
def _dedupe_urls(
|
||||
all_urls: List[ZipUrlInfo],
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Deduplicate URLs, keeping the first occurrence of each unique URL.
|
||||
Returns dict mapping url -> env_section.
|
||||
"""
|
||||
unique_urls: Dict[str, str] = {}
|
||||
for url_info in all_urls:
|
||||
if url_info.url not in unique_urls:
|
||||
unique_urls[url_info.url] = url_info.section_name
|
||||
return unique_urls
|
||||
|
||||
|
||||
def _download_and_process_urls(
|
||||
unique_urls: Dict[str, str], cache_manager: PlatformIOCache
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Download and process all URLs concurrently.
|
||||
Returns dict mapping original_url -> resolved_local_path.
|
||||
Raises exception immediately if any download fails.
|
||||
"""
|
||||
if not unique_urls:
|
||||
return {}
|
||||
|
||||
max_workers = min(4, len(unique_urls)) # Don't create more threads than needed
|
||||
with ThreadPoolExecutor(
|
||||
max_workers=max_workers, thread_name_prefix="download"
|
||||
) as executor:
|
||||
# Submit all download tasks
|
||||
futures: List[Future[ArtifactProcessingResult]] = []
|
||||
for url, env_section in unique_urls.items():
|
||||
future = executor.submit(_process_artifact, url, env_section, cache_manager)
|
||||
futures.append(future)
|
||||
|
||||
replacements: Dict[str, str] = {}
|
||||
|
||||
try:
|
||||
for future in as_completed(futures):
|
||||
try:
|
||||
result = future.result()
|
||||
if result.success and result.resolved_path is not None:
|
||||
replacements[result.url] = result.resolved_path
|
||||
print(f"✅ Resolved {result.url} -> {result.resolved_path}")
|
||||
else:
|
||||
logger.error(
|
||||
f"❌ Failed to process {result.url}: {result.exception}"
|
||||
)
|
||||
# Re-raise all exceptions - no downloads should fail silently
|
||||
if result.exception:
|
||||
raise result.exception
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Future failed with unexpected error: {e}", exc_info=e
|
||||
)
|
||||
raise # Re-raise unexpected exceptions
|
||||
except KeyboardInterrupt:
|
||||
logger.warning("Processing interrupted, cancelling remaining downloads...")
|
||||
_global_cancel_event.set()
|
||||
# The context manager will handle cleanup of the executor
|
||||
raise
|
||||
|
||||
return replacements
|
||||
|
||||
|
||||
def _replace_all_urls(
|
||||
pio_ini: PlatformIOIni,
|
||||
all_urls: List[ZipUrlInfo],
|
||||
replacements: Dict[str, str],
|
||||
) -> None:
|
||||
"""
|
||||
Replace all URLs in platformio.ini with their resolved local paths.
|
||||
"""
|
||||
for url_info in all_urls:
|
||||
if url_info.url in replacements:
|
||||
pio_ini.replace_url(
|
||||
url_info.section_name,
|
||||
url_info.option_name,
|
||||
url_info.url,
|
||||
replacements[url_info.url],
|
||||
)
|
||||
|
||||
|
||||
def _apply_board_specific_config(
|
||||
platformio_ini_path: Path, custom_zip_cache_dir: Path
|
||||
) -> None:
|
||||
"""
|
||||
Enhanced config parser with improved zip detection and error handling.
|
||||
Modifies platformio.ini in-place with resolved local paths.
|
||||
Uses concurrent downloads for better performance.
|
||||
"""
|
||||
cache_manager = PlatformIOCache(custom_zip_cache_dir)
|
||||
|
||||
try:
|
||||
pio_ini = PlatformIOIni.parseFile(platformio_ini_path)
|
||||
print(f"Parsed platformio.ini: {platformio_ini_path}")
|
||||
except (configparser.Error, FileNotFoundError) as e:
|
||||
logger.error(f"Error reading platformio.ini: {e}")
|
||||
return
|
||||
|
||||
# Step 1: Collect all URLs from platformio.ini
|
||||
all_urls = _collect_all_zip_urls(pio_ini)
|
||||
if not all_urls:
|
||||
print("No zip artifacts found to process")
|
||||
return
|
||||
|
||||
# Step 2: Deduplicate URLs
|
||||
unique_urls = _dedupe_urls(all_urls)
|
||||
print(f"Found {len(all_urls)} total URLs, {len(unique_urls)} unique")
|
||||
|
||||
# Step 3: Download and process all unique URLs
|
||||
replacements = _download_and_process_urls(unique_urls, cache_manager)
|
||||
|
||||
# Step 4: Replace all URLs in platformio.ini with resolved local paths
|
||||
if replacements:
|
||||
_replace_all_urls(pio_ini, all_urls, replacements)
|
||||
|
||||
# Write back atomically
|
||||
try:
|
||||
pio_ini.dump(platformio_ini_path)
|
||||
print(f"Updated platformio.ini with {len(replacements)} resolved artifacts")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update platformio.ini: {e}")
|
||||
raise
|
||||
|
||||
|
||||
# Public API function
|
||||
def resolve_and_cache_platform_artifacts(
|
||||
platformio_ini_path: Path, cache_dir: Path
|
||||
) -> None:
|
||||
"""
|
||||
Main entry point for resolving and caching PlatformIO platform artifacts.
|
||||
|
||||
Args:
|
||||
platformio_ini_path: Path to the platformio.ini file to process
|
||||
cache_dir: Cache directory for storing artifacts
|
||||
"""
|
||||
print(f"Starting platform artifact resolution for {platformio_ini_path}")
|
||||
print(f"Using cache directory: {cache_dir}")
|
||||
|
||||
_apply_board_specific_config(platformio_ini_path, cache_dir)
|
||||
|
||||
print("Platform artifact resolution completed")
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,983 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Test Build System - Python Compiler API Integration
|
||||
|
||||
This module provides a high-performance test compilation system built on the proven
|
||||
ci.util.ci.util.clang_compiler API that delivers 8x faster build times compared to CMake.
|
||||
|
||||
Key Features:
|
||||
- Parallel test compilation using ThreadPoolExecutor
|
||||
- 90%+ cache hit rates with proven compiler settings
|
||||
- Integration with build_unit.toml configuration
|
||||
- Support for STUB platform testing without hardware dependencies
|
||||
- GDB-compatible debug symbol generation
|
||||
- Consistent performance across all platforms
|
||||
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import time
|
||||
import tomllib
|
||||
from concurrent.futures import Future, as_completed
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from types import TracebackType
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
|
||||
|
||||
def optimize_python_command(cmd: list[str]) -> list[str]:
|
||||
"""
|
||||
Optimize command list for subprocess execution in uv environment.
|
||||
|
||||
For python commands, we need to use 'uv run python' to ensure access to
|
||||
installed packages like ziglang. Direct sys.executable bypasses uv environment.
|
||||
|
||||
Args:
|
||||
cmd: Command list that may contain 'python' as first element
|
||||
|
||||
Returns:
|
||||
list[str]: Optimized command with 'python' prefixed by 'uv run'
|
||||
"""
|
||||
if cmd and (cmd[0] == "python" or cmd[0] == "python3"):
|
||||
# Use uv run python to ensure access to uv-managed packages
|
||||
optimized_cmd = ["uv", "run", "python"] + cmd[1:]
|
||||
return optimized_cmd
|
||||
return cmd
|
||||
|
||||
|
||||
from ci.compiler.clang_compiler import (
|
||||
BuildFlags,
|
||||
BuildTools,
|
||||
Compiler,
|
||||
CompilerOptions,
|
||||
LinkOptions,
|
||||
Result,
|
||||
create_compiler_options_from_toml,
|
||||
link_program_sync,
|
||||
load_build_flags_from_toml,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestExecutable:
|
||||
"""Represents a compiled test executable"""
|
||||
|
||||
name: str
|
||||
executable_path: Path
|
||||
test_source_path: Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompileResult:
|
||||
"""Result of test compilation operation"""
|
||||
|
||||
success: bool
|
||||
compiled_count: int
|
||||
duration: float
|
||||
errors: List["CompileError"] = field(default_factory=lambda: list())
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompileError:
|
||||
"""Compilation error for a specific test"""
|
||||
|
||||
test_name: str
|
||||
message: str
|
||||
|
||||
|
||||
class FastLEDTestCompiler:
|
||||
"""
|
||||
Test compiler built on proven Compiler API for 8x faster builds.
|
||||
|
||||
This class leverages the proven ci.util.ci.util.clang_compiler.Compiler API that has already
|
||||
demonstrated 4x+ performance improvements in example compilation. By using the same
|
||||
proven patterns and configurations, we inherit the optimized build flags, caching,
|
||||
and parallel compilation capabilities.
|
||||
"""
|
||||
|
||||
# Class variable to store existing instance for get_existing_instance()
|
||||
_existing_instance: Optional["FastLEDTestCompiler"] = None
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
compiler: Compiler,
|
||||
build_dir: Path,
|
||||
project_root: Path,
|
||||
quick_build: bool = True,
|
||||
strict_mode: bool = False,
|
||||
no_unity: bool = False,
|
||||
):
|
||||
self.compiler = compiler
|
||||
self.build_dir = build_dir
|
||||
self.project_root = project_root
|
||||
self.quick_build = quick_build
|
||||
self.strict_mode = strict_mode
|
||||
self.no_unity = no_unity
|
||||
self.compiled_tests: List[TestExecutable] = []
|
||||
self.linking_failures: List[str] = []
|
||||
self.build_flags = self._load_build_flags()
|
||||
FastLEDTestCompiler._existing_instance = self
|
||||
|
||||
def _load_build_flags(self) -> BuildFlags:
|
||||
"""Load build flags from TOML configuration"""
|
||||
toml_path = self.project_root / "ci" / "build_unit.toml"
|
||||
return BuildFlags.parse(toml_path, self.quick_build, self.strict_mode)
|
||||
|
||||
@classmethod
|
||||
def create_for_unit_tests(
|
||||
cls,
|
||||
project_root: Path,
|
||||
clean_build: bool = False,
|
||||
enable_static_analysis: bool = False,
|
||||
specific_test: str | None = None,
|
||||
quick_build: bool = True,
|
||||
strict_mode: bool = False,
|
||||
no_unity: bool = False,
|
||||
) -> "FastLEDTestCompiler":
|
||||
"""
|
||||
Create compiler configured for FastLED unit tests using TOML build flags.
|
||||
|
||||
This configuration uses the new TOML-based build flag system:
|
||||
- ci/build_unit.toml for centralized flag configuration
|
||||
- Support for build modes (quick/debug) and strict mode
|
||||
- STUB platform for hardware-free testing
|
||||
- Precompiled headers for faster compilation
|
||||
- Parallel compilation with ThreadPoolExecutor
|
||||
"""
|
||||
|
||||
# Set up build directory in .build/fled/unit for consistency
|
||||
build_dir = project_root / ".build" / "fled" / "unit"
|
||||
if clean_build and build_dir.exists():
|
||||
print("###########################")
|
||||
print("# CLEANING UNIT TEST BUILD DIR #")
|
||||
print("###########################")
|
||||
import errno
|
||||
import os
|
||||
import shutil
|
||||
import stat
|
||||
import time
|
||||
|
||||
def handle_remove_readonly(
|
||||
func: Callable[[str], None],
|
||||
path: str,
|
||||
exc: tuple[type[BaseException], BaseException, TracebackType | None],
|
||||
) -> None:
|
||||
"""Error handler for Windows readonly files"""
|
||||
if hasattr(exc[1], "errno") and exc[1].errno == errno.EACCES: # type: ignore
|
||||
os.chmod(path, stat.S_IWRITE)
|
||||
func(path)
|
||||
else:
|
||||
raise
|
||||
|
||||
# Windows-compatible directory removal with retry
|
||||
max_retries = 3
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
# Use onerror parameter for compatibility across Python versions
|
||||
shutil.rmtree(build_dir, onerror=handle_remove_readonly)
|
||||
break
|
||||
except (OSError, PermissionError) as e:
|
||||
if attempt < max_retries - 1:
|
||||
print(
|
||||
f"Warning: Failed to remove build directory (attempt {attempt + 1}): {e}"
|
||||
)
|
||||
time.sleep(0.01) # Brief pause before retry
|
||||
continue
|
||||
else:
|
||||
print(
|
||||
f"Warning: Could not remove build directory after {max_retries} attempts: {e}"
|
||||
)
|
||||
print("Continuing with existing directory...")
|
||||
break
|
||||
build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Additional compiler args beyond TOML
|
||||
additional_compiler_args = [
|
||||
f"-I{project_root}/src/platforms/stub", # STUB platform headers
|
||||
]
|
||||
|
||||
# Load build flags using BuildFlags.parse() to get platform-specific configuration
|
||||
toml_path = project_root / "ci" / "build_unit.toml"
|
||||
build_flags = BuildFlags.parse(toml_path, quick_build, strict_mode)
|
||||
|
||||
# Load TOML config for test-specific flags that aren't in BuildFlags
|
||||
with open(toml_path, "rb") as f:
|
||||
config = tomllib.load(f)
|
||||
|
||||
# Get test-specific defines from TOML (not included in BuildFlags.parse)
|
||||
test_defines = config.get("test", {}).get("defines", [])
|
||||
|
||||
# Create compiler args from BuildFlags (includes platform-specific flags)
|
||||
compiler_args: List[str] = []
|
||||
compiler_args.extend(
|
||||
build_flags.compiler_flags
|
||||
) # Platform-aware compiler flags
|
||||
compiler_args.extend(build_flags.include_flags) # Platform-aware include flags
|
||||
|
||||
# Add strict mode flags if enabled (already handled by BuildFlags.parse)
|
||||
|
||||
# Add additional compiler args
|
||||
if additional_compiler_args:
|
||||
compiler_args.extend(additional_compiler_args)
|
||||
|
||||
# Extract defines without the "-D" prefix for CompilerOptions
|
||||
defines: List[str] = []
|
||||
for define in test_defines:
|
||||
if define.startswith("-D"):
|
||||
defines.append(define[2:]) # Remove "-D" prefix
|
||||
else:
|
||||
defines.append(define)
|
||||
|
||||
# Get tools configuration from BuildFlags (respects build_unit.toml)
|
||||
# Use modern command-based approach
|
||||
compiler_command = build_flags.tools.cpp_compiler
|
||||
|
||||
print(f"Using compiler from build_unit.toml: {compiler_command}")
|
||||
|
||||
# Use compiler command from TOML + flags as final compiler args
|
||||
final_compiler_args = compiler_command + compiler_args
|
||||
|
||||
# Create compiler options with TOML-loaded flags and tools
|
||||
settings = CompilerOptions(
|
||||
include_path=str(project_root / "src"),
|
||||
defines=defines,
|
||||
compiler_args=final_compiler_args, # Use cache-aware compiler args
|
||||
# Note: archiver and compiler fields are now handled via command-based approach
|
||||
)
|
||||
|
||||
compiler = Compiler(settings, build_flags)
|
||||
|
||||
# Create final instance with proper compiler and flags
|
||||
instance = cls(
|
||||
compiler, build_dir, project_root, quick_build, strict_mode, no_unity
|
||||
)
|
||||
return instance
|
||||
|
||||
def discover_test_files(self, specific_test: str | None = None) -> List[Path]:
|
||||
"""
|
||||
Discover test_*.cpp files using proven patterns.
|
||||
|
||||
Uses the same file discovery patterns as the existing CMake system
|
||||
to ensure 100% compatibility with current test discovery logic.
|
||||
"""
|
||||
tests_dir = Path(PROJECT_ROOT) / "tests"
|
||||
test_files: List[Path] = []
|
||||
|
||||
# Find all test files in tests directory and subdirectories
|
||||
for test_file in tests_dir.rglob("test_*.cpp"):
|
||||
# Skip the doctest_main.cpp file (not a test file)
|
||||
if test_file.name == "doctest_main.cpp":
|
||||
continue
|
||||
|
||||
# Handle both "test_name" and "name" formats for compatibility
|
||||
test_stem = test_file.stem
|
||||
test_name = test_stem.replace("test_", "")
|
||||
|
||||
# Check if we should do fuzzy matching (if there's a * in the name)
|
||||
if specific_test:
|
||||
if "*" in specific_test:
|
||||
# Convert glob pattern to regex pattern
|
||||
import re
|
||||
|
||||
pattern = specific_test.replace("*", ".*")
|
||||
if re.search(pattern, test_stem) or re.search(pattern, test_name):
|
||||
test_files.append(test_file)
|
||||
else:
|
||||
# Exact matching - either the full name or with test_ prefix (case-insensitive)
|
||||
if (
|
||||
test_stem.lower() == specific_test.lower()
|
||||
or test_stem.lower() == f"test_{specific_test}".lower()
|
||||
or test_name.lower() == specific_test.lower()
|
||||
):
|
||||
test_files.append(test_file)
|
||||
else:
|
||||
test_files.append(test_file)
|
||||
|
||||
return test_files
|
||||
|
||||
def compile_all_tests(self, specific_test: str | None = None) -> CompileResult:
|
||||
"""
|
||||
Compile all tests in parallel using proven API patterns.
|
||||
|
||||
This method uses the same parallel compilation patterns proven in
|
||||
example compilation that deliver 4x+ performance improvements.
|
||||
"""
|
||||
compile_start = time.time()
|
||||
|
||||
# If specific_test is not provided in the method call, use the one from discover_test_files
|
||||
test_files = self.discover_test_files(specific_test)
|
||||
|
||||
if not test_files:
|
||||
return CompileResult(success=True, compiled_count=0, duration=0.0)
|
||||
|
||||
print(f"Compiling {len(test_files)} test files using proven Python API...")
|
||||
# Convert absolute path to relative for display
|
||||
rel_build_dir = os.path.relpath(self.build_dir)
|
||||
print(f"Build directory: {rel_build_dir}")
|
||||
|
||||
# Always show build configuration in unit test mode
|
||||
print("\nFastLED Library Build Configuration:")
|
||||
if self.no_unity:
|
||||
print(" ❌ Unity build disabled - Compiling individual source files")
|
||||
else:
|
||||
print(" ✅ Unity build enabled - Using glob pattern:")
|
||||
print(" src/**/*.cpp")
|
||||
|
||||
print("\nPrecompiled header status:")
|
||||
if self.compiler.settings.use_pch:
|
||||
print(" ✅ PCH enabled - Using precompiled headers for faster compilation")
|
||||
print(" PCH content:")
|
||||
print(textwrap.indent(self.compiler.generate_pch_header(), " "))
|
||||
|
||||
# Create PCH for faster compilation
|
||||
print("\n🔧 Creating precompiled header...")
|
||||
pch_success = self.compiler.create_pch_file()
|
||||
if pch_success:
|
||||
print(" ✅ PCH created successfully")
|
||||
else:
|
||||
print(" ❌ PCH creation failed - continuing with direct compilation")
|
||||
else:
|
||||
print(" ❌ PCH disabled - Not using precompiled headers")
|
||||
|
||||
print("\nCompiler flags:")
|
||||
for flag in self.compiler.get_compiler_args():
|
||||
print(f" {flag}")
|
||||
print("\nLinker flags:")
|
||||
for flag in self._get_platform_linker_args(self.build_dir / "libfastled.lib"):
|
||||
print(f" {flag}")
|
||||
print("")
|
||||
|
||||
# Print list of test files being compiled
|
||||
print("Test files to compile:")
|
||||
for i, test_file in enumerate(test_files, 1):
|
||||
print(f" {i}. {test_file.name}")
|
||||
print("")
|
||||
|
||||
# Submit parallel compilation jobs (proven pattern from example compilation)
|
||||
future_to_test: Dict[Future[Result], Path] = {}
|
||||
|
||||
# Check if we're running in parallel or sequential mode
|
||||
if os.environ.get("NO_PARALLEL"):
|
||||
print("Starting sequential compilation of test files...")
|
||||
else:
|
||||
print("Starting parallel compilation of test files...")
|
||||
for test_file in test_files:
|
||||
# Compile to object file first (since compile_cpp_file uses -c flag)
|
||||
obj_path = self.build_dir / f"{test_file.stem}.o"
|
||||
# Convert absolute path to relative for display
|
||||
rel_obj_path = os.path.relpath(obj_path)
|
||||
print(f"Submitting compilation job for: {test_file.name} -> {rel_obj_path}")
|
||||
# Show compilation command if enabled
|
||||
if os.environ.get("FASTLED_TEST_SHOW_COMPILE", "").lower() in (
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
):
|
||||
# Convert absolute path to relative for display
|
||||
rel_obj_path = os.path.relpath(obj_path)
|
||||
print(f"[COMPILE] {test_file.name} -> {rel_obj_path}")
|
||||
|
||||
compile_future = self.compiler.compile_cpp_file(
|
||||
test_file,
|
||||
output_path=obj_path,
|
||||
additional_flags=[
|
||||
"-I",
|
||||
str(self.project_root / "src"),
|
||||
"-I",
|
||||
str(self.project_root / "tests"),
|
||||
# NOTE: Compiler flags now come from build_unit.toml
|
||||
],
|
||||
)
|
||||
future_to_test[compile_future] = test_file
|
||||
|
||||
# Collect compilation results with progress reporting
|
||||
compiled_objects: List[Path] = []
|
||||
errors: List[CompileError] = []
|
||||
completed = 0
|
||||
|
||||
print(f"Waiting for {len(future_to_test)} compilation jobs to complete...")
|
||||
|
||||
try:
|
||||
for future in as_completed(future_to_test.keys()):
|
||||
test_file = future_to_test[future]
|
||||
result: Result = future.result()
|
||||
completed += 1
|
||||
|
||||
if result.ok:
|
||||
obj_path = self.build_dir / f"{test_file.stem}.o"
|
||||
compiled_objects.append(obj_path)
|
||||
print(f"[{completed}/{len(test_files)}] Compiled {test_file.name}")
|
||||
else:
|
||||
errors.append(
|
||||
CompileError(
|
||||
test_name=test_file.stem,
|
||||
message=result.stderr
|
||||
or result.stdout
|
||||
or "Compilation failed",
|
||||
)
|
||||
)
|
||||
print(
|
||||
f"[{completed}/{len(test_files)}] FAILED {test_file.name}: {result.stderr}"
|
||||
)
|
||||
except KeyboardInterrupt:
|
||||
print("\nKeyboard interrupt detected during compilation")
|
||||
# Clean up any in-progress compilations
|
||||
for future in future_to_test.keys():
|
||||
future.cancel()
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
|
||||
if errors:
|
||||
duration = time.time() - compile_start
|
||||
print(f"Compilation failed with {len(errors)} errors in {duration:.2f}s")
|
||||
return CompileResult(
|
||||
success=False, compiled_count=0, duration=duration, errors=errors
|
||||
)
|
||||
|
||||
print(f"All {len(compiled_objects)} object files compiled successfully")
|
||||
|
||||
# Link each test to executable using proven linking API
|
||||
self.compiled_tests = self._link_tests(compiled_objects)
|
||||
|
||||
# Check for linking failures and add them to errors
|
||||
if hasattr(self, "linking_failures") and self.linking_failures:
|
||||
for failure in self.linking_failures:
|
||||
# Parse the failure string to extract test name and error
|
||||
if ":" in failure:
|
||||
test_name, error_msg = failure.split(":", 1)
|
||||
errors.append(
|
||||
CompileError(
|
||||
test_name=test_name.strip(),
|
||||
message=f"Linking failed: {error_msg.strip()}",
|
||||
)
|
||||
)
|
||||
|
||||
duration = time.time() - compile_start
|
||||
|
||||
# Success requires both compilation AND linking to succeed
|
||||
success = len(errors) == 0 and len(self.compiled_tests) > 0
|
||||
|
||||
if success:
|
||||
print(
|
||||
f"SUCCESS: Compiled and linked {len(self.compiled_tests)} tests in {duration:.2f}s"
|
||||
)
|
||||
# List all the successful test executables
|
||||
print("Test executables created:")
|
||||
for i, test in enumerate(self.compiled_tests, 1):
|
||||
# Convert absolute path to relative for display
|
||||
rel_exe_path = os.path.relpath(test.executable_path)
|
||||
print(f" {i}. {test.name} -> {rel_exe_path}")
|
||||
else:
|
||||
print(f"FAILED: {len(errors)} total failures (compilation + linking)")
|
||||
|
||||
return CompileResult(
|
||||
success=success,
|
||||
compiled_count=len(self.compiled_tests),
|
||||
duration=duration,
|
||||
errors=errors,
|
||||
)
|
||||
|
||||
def _link_tests(self, compiled_objects: List[Path]) -> List[TestExecutable]:
|
||||
"""Link each test to executable using proven linking API"""
|
||||
print(f"Linking {len(compiled_objects)} test executables...")
|
||||
|
||||
# First, build a complete FastLED library similar to CMake approach
|
||||
fastled_lib_path = self._build_fastled_library()
|
||||
|
||||
# Compile doctest_main.cpp once for all tests (provides main function and doctest implementation)
|
||||
doctest_main_path = self.project_root / "tests" / "doctest_main.cpp"
|
||||
doctest_obj_path = self.build_dir / "doctest_main.o"
|
||||
|
||||
if not doctest_obj_path.exists():
|
||||
print("Compiling doctest_main.cpp...")
|
||||
doctest_future = self.compiler.compile_cpp_file(
|
||||
doctest_main_path,
|
||||
output_path=doctest_obj_path,
|
||||
additional_flags=["-c"], # Compile only
|
||||
)
|
||||
doctest_result: Result = doctest_future.result()
|
||||
if not doctest_result.ok:
|
||||
print(
|
||||
f"ERROR: Failed to compile doctest_main.cpp: {doctest_result.stderr}"
|
||||
)
|
||||
return []
|
||||
|
||||
print(f"Compiled FastLED library and doctest_main")
|
||||
|
||||
compiled_tests: List[TestExecutable] = []
|
||||
linking_failures: List[str] = []
|
||||
cache_hits = 0
|
||||
cache_misses = 0
|
||||
|
||||
# Link each test with the FastLED library
|
||||
for obj_path in compiled_objects:
|
||||
# Derive test name from object file
|
||||
test_name = obj_path.stem.replace("test_", "")
|
||||
exe_path = self.build_dir / f"test_{test_name}.exe"
|
||||
|
||||
# Define clean execution path in tests/bin/ (this is where tests will actually run from)
|
||||
tests_bin_dir = self.project_root / "tests" / "bin"
|
||||
tests_bin_dir.mkdir(parents=True, exist_ok=True)
|
||||
clean_exe_path = tests_bin_dir / f"test_{test_name}.exe"
|
||||
|
||||
# Check if this test has its own main() function
|
||||
test_source = self.project_root / "tests" / f"test_{test_name}.cpp"
|
||||
has_own_main = self._test_has_own_main(test_source)
|
||||
|
||||
# Get linker args (needed for cache key calculation)
|
||||
linker_args = self._get_platform_linker_args(fastled_lib_path)
|
||||
|
||||
# Calculate comprehensive cache key based on all linking inputs
|
||||
if has_own_main:
|
||||
# For standalone tests, only include test object file
|
||||
cache_key = self._calculate_link_cache_key(
|
||||
obj_path, fastled_lib_path, linker_args
|
||||
)
|
||||
else:
|
||||
# For doctest tests, include both test object and doctest_main object
|
||||
# Calculate hashes for both object files and combine them
|
||||
test_obj_hash = self._calculate_file_hash(obj_path)
|
||||
doctest_obj_hash = self._calculate_file_hash(doctest_obj_path)
|
||||
combined_obj_hash = hashlib.sha256(
|
||||
f"{test_obj_hash}+{doctest_obj_hash}".encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
# Create a temporary cache key with combined object hash
|
||||
fastled_hash = self._calculate_file_hash(fastled_lib_path)
|
||||
linker_hash = self._calculate_linker_args_hash(linker_args)
|
||||
combined = f"fastled:{fastled_hash}|test:{combined_obj_hash}|flags:{linker_hash}"
|
||||
cache_key = hashlib.sha256(combined.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
# Check for cached executable based on comprehensive cache key
|
||||
cached_exe = self._get_cached_executable(test_name, cache_key)
|
||||
if cached_exe:
|
||||
print(f" {test_name}: Using cached executable (link cache hit)")
|
||||
try:
|
||||
shutil.copy2(cached_exe, exe_path)
|
||||
# Also copy to clean execution location
|
||||
shutil.copy2(cached_exe, clean_exe_path)
|
||||
cache_hits += 1
|
||||
|
||||
test_exe = TestExecutable(
|
||||
name=test_name,
|
||||
executable_path=clean_exe_path, # Point to clean location for execution
|
||||
test_source_path=test_source,
|
||||
)
|
||||
compiled_tests.append(test_exe)
|
||||
continue # Skip linking, we have cached result
|
||||
except Exception as e:
|
||||
print(
|
||||
f" {test_name}: Warning - failed to copy cached executable, will relink: {e}"
|
||||
)
|
||||
# Fall through to linking logic
|
||||
|
||||
if has_own_main:
|
||||
# Link standalone test without doctest_main.o
|
||||
link_options = LinkOptions(
|
||||
object_files=[obj_path], # Only the test object file
|
||||
output_executable=exe_path,
|
||||
linker=f'"{sys.executable}" -m ziglang c++', # Use optimized Python executable for linking
|
||||
linker_args=linker_args, # Use pre-calculated args for consistency
|
||||
)
|
||||
else:
|
||||
# Link with the FastLED library instead of individual objects
|
||||
link_options = LinkOptions(
|
||||
object_files=[
|
||||
obj_path,
|
||||
doctest_obj_path,
|
||||
], # Test object + doctest main
|
||||
output_executable=exe_path,
|
||||
linker=f'"{sys.executable}" -m ziglang c++', # Use optimized Python executable for linking
|
||||
linker_args=linker_args, # Use pre-calculated args for consistency
|
||||
)
|
||||
|
||||
# Show linking command if enabled
|
||||
if os.environ.get("FASTLED_TEST_SHOW_LINK", "").lower() in (
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
):
|
||||
# Convert absolute path to relative for display
|
||||
rel_exe_path = os.path.relpath(exe_path)
|
||||
print(f"[LINK] {test_name} -> {rel_exe_path}")
|
||||
|
||||
print(f"Linking test: {test_name}")
|
||||
# STRICT: Provide explicit BuildFlags - NO defaults allowed
|
||||
link_result: Result = link_program_sync(
|
||||
link_options, self.compiler.build_flags
|
||||
)
|
||||
|
||||
if not link_result.ok:
|
||||
# Create more detailed error message showing link command context
|
||||
error_msg = f" {test_name}: Linking failed - Linker: {link_options.linker}, Output: {link_options.output_executable}, Objects: {len(link_options.object_files)} files"
|
||||
print(error_msg)
|
||||
print(f" Error details: {link_result.stderr}")
|
||||
linking_failures.append(f"{test_name}: {link_result.stderr}")
|
||||
continue
|
||||
|
||||
print(f" {test_name}: Linking successful")
|
||||
cache_misses += 1
|
||||
|
||||
# Cache the newly linked executable for future use
|
||||
self._cache_executable(test_name, cache_key, exe_path)
|
||||
|
||||
# Copy to clean execution location
|
||||
shutil.copy2(exe_path, clean_exe_path)
|
||||
|
||||
test_exe = TestExecutable(
|
||||
name=test_name,
|
||||
executable_path=clean_exe_path,
|
||||
test_source_path=test_source, # Point to clean location for execution
|
||||
)
|
||||
compiled_tests.append(test_exe)
|
||||
|
||||
if linking_failures:
|
||||
print(f"ERROR: {len(linking_failures)} linking failures:")
|
||||
for failure in linking_failures:
|
||||
print(f" {failure}")
|
||||
# Store linking failures for later reporting
|
||||
self.linking_failures = linking_failures
|
||||
else:
|
||||
self.linking_failures = []
|
||||
|
||||
# Report link cache statistics
|
||||
total_tests = cache_hits + cache_misses
|
||||
if total_tests > 0:
|
||||
hit_rate = (cache_hits / total_tests) * 100
|
||||
print(
|
||||
f"Link cache: {cache_hits} hits, {cache_misses} misses ({hit_rate:.1f}% hit rate) - caching by lib+test+flags"
|
||||
)
|
||||
|
||||
print(f"Successfully linked {len(compiled_tests)} test executables")
|
||||
return compiled_tests
|
||||
|
||||
def _test_has_own_main(self, test_source_path: Path) -> bool:
|
||||
"""Check if a test file defines its own main() function"""
|
||||
try:
|
||||
if not test_source_path.exists():
|
||||
return False
|
||||
|
||||
content = test_source_path.read_text(encoding="utf-8")
|
||||
# Look for main function definition (simple pattern matching)
|
||||
import re
|
||||
|
||||
# Match "int main(" allowing for whitespace and various formats
|
||||
main_pattern = r"\bint\s+main\s*\("
|
||||
return bool(re.search(main_pattern, content))
|
||||
except Exception:
|
||||
# If we can't read the file, assume it doesn't have main
|
||||
return False
|
||||
|
||||
def _build_fastled_library(self) -> Path:
|
||||
"""Build a complete FastLED static library like CMake does"""
|
||||
print("Building FastLED static library...")
|
||||
|
||||
# Define library path
|
||||
fastled_lib_path = self.build_dir / "libfastled.lib"
|
||||
|
||||
# If library already exists, return it (for faster rebuilds)
|
||||
if fastled_lib_path.exists():
|
||||
# Convert absolute path to relative for display
|
||||
rel_lib_path = os.path.relpath(fastled_lib_path)
|
||||
print(f"Using existing FastLED library: {rel_lib_path}")
|
||||
return fastled_lib_path
|
||||
|
||||
# Compile essential FastLED source files for STUB platform
|
||||
# User directive: Just glob ALL .cpp files in src/** - no exclusions needed
|
||||
import glob
|
||||
|
||||
fastled_src_dir = self.project_root / "src"
|
||||
# Include ALL .cpp files recursively - no platform exclusions
|
||||
all_cpp_files: List[str] = glob.glob(
|
||||
str(fastled_src_dir / "**" / "*.cpp"), recursive=True
|
||||
)
|
||||
|
||||
fastled_sources: List[Path] = []
|
||||
for cpp_file in all_cpp_files:
|
||||
# Include ALL files - no filtering needed
|
||||
fastled_sources.append(Path(cpp_file))
|
||||
|
||||
if self.no_unity:
|
||||
print(
|
||||
f"Individual compilation: Found {len(fastled_sources)} FastLED .cpp files for compilation"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f"Unity build mode: Found {len(fastled_sources)} FastLED .cpp files for compilation"
|
||||
)
|
||||
|
||||
# Compile all FastLED source files to object files
|
||||
fastled_objects: List[Path] = []
|
||||
for src_file in fastled_sources:
|
||||
src_path = self.project_root / src_file
|
||||
# Create unique object file name including directory path to avoid collisions
|
||||
# Convert path separators to underscores to create valid filename
|
||||
relative_path = src_path.relative_to(self.project_root / "src")
|
||||
safe_name = (
|
||||
str(relative_path.with_suffix("")).replace("/", "_").replace("\\", "_")
|
||||
)
|
||||
obj_path = self.build_dir / f"{safe_name}_fastled.o"
|
||||
|
||||
if not obj_path.exists():
|
||||
future = self.compiler.compile_cpp_file(
|
||||
src_path,
|
||||
output_path=obj_path,
|
||||
additional_flags=[
|
||||
"-c", # Compile only
|
||||
"-DFASTLED_STUB_IMPL", # Essential for STUB platform
|
||||
# NOTE: All defines and compiler flags now come from build_unit.toml
|
||||
# Remove hardcoded defines - they should be in [test] section
|
||||
],
|
||||
)
|
||||
result: Result = future.result()
|
||||
if not result.ok:
|
||||
print(f"ERROR: Failed to compile {src_file}: {result.stderr}")
|
||||
# Continue with other files rather than failing completely
|
||||
continue
|
||||
|
||||
fastled_objects.append(obj_path)
|
||||
|
||||
print(f"Creating static library from {len(fastled_objects)} object files...")
|
||||
|
||||
# Create static library using ar (archiver)
|
||||
# First try llvm-ar since it's more likely to be available
|
||||
ar_cmd: List[str] = [
|
||||
"llvm-ar", # LLVM archiver tool (works with Clang)
|
||||
"rcs", # Create archive with symbol table
|
||||
str(fastled_lib_path),
|
||||
] + [str(obj) for obj in fastled_objects]
|
||||
|
||||
# Create static library using ziglang ar (archiver)
|
||||
print("Creating static library using ziglang ar...")
|
||||
lib_cmd: List[str] = [
|
||||
"python",
|
||||
"-m",
|
||||
"ziglang",
|
||||
"ar",
|
||||
"rcs", # Create archive with symbol table
|
||||
str(fastled_lib_path),
|
||||
] + [str(obj) for obj in fastled_objects]
|
||||
|
||||
try:
|
||||
# Optimize command to use sys.executable instead of shell 'python' resolution
|
||||
python_exe = optimize_python_command(lib_cmd)
|
||||
|
||||
# Use streaming to prevent buffer overflow
|
||||
process = subprocess.Popen(
|
||||
python_exe,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
bufsize=1,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
cwd=self.build_dir,
|
||||
)
|
||||
|
||||
stdout_lines: list[str] = []
|
||||
stderr_lines: list[str] = []
|
||||
|
||||
while True:
|
||||
stdout_line = process.stdout.readline() if process.stdout else ""
|
||||
stderr_line = process.stderr.readline() if process.stderr else ""
|
||||
|
||||
if stdout_line:
|
||||
stdout_lines.append(stdout_line.rstrip())
|
||||
if stderr_line:
|
||||
stderr_lines.append(stderr_line.rstrip())
|
||||
|
||||
if process.poll() is not None:
|
||||
remaining_stdout = process.stdout.read() if process.stdout else ""
|
||||
remaining_stderr = process.stderr.read() if process.stderr else ""
|
||||
|
||||
if remaining_stdout:
|
||||
for line in remaining_stdout.splitlines():
|
||||
stdout_lines.append(line.rstrip())
|
||||
if remaining_stderr:
|
||||
for line in remaining_stderr.splitlines():
|
||||
stderr_lines.append(line.rstrip())
|
||||
break
|
||||
|
||||
if process.returncode != 0:
|
||||
print(f"ERROR: Failed to create static library: {stderr_lines}")
|
||||
return fastled_lib_path # Return path even if creation failed
|
||||
|
||||
# Convert absolute path to relative for display
|
||||
rel_lib_path = os.path.relpath(fastled_lib_path)
|
||||
print(f"Successfully created FastLED library: {rel_lib_path}")
|
||||
return fastled_lib_path
|
||||
|
||||
except Exception as e:
|
||||
print(f"ERROR: Exception during library creation: {e}")
|
||||
raise
|
||||
|
||||
def get_test_executables(
|
||||
self, specific_test: str | None = None
|
||||
) -> List[TestExecutable]:
|
||||
"""Get compiled test executables with name filtering"""
|
||||
if specific_test:
|
||||
# Check if we should do fuzzy matching (if there's a * in the name)
|
||||
if "*" in specific_test:
|
||||
# Convert glob pattern to regex pattern
|
||||
import re
|
||||
|
||||
pattern = specific_test.replace("*", ".*")
|
||||
return [
|
||||
t
|
||||
for t in self.compiled_tests
|
||||
if re.search(pattern, t.name)
|
||||
or re.search(pattern, f"test_{t.name}")
|
||||
]
|
||||
else:
|
||||
# Exact matching - either the full name or with test_ prefix (case-insensitive)
|
||||
return [
|
||||
t
|
||||
for t in self.compiled_tests
|
||||
if (
|
||||
t.name.lower() == specific_test.lower()
|
||||
or t.name.lower() == f"test_{specific_test}".lower()
|
||||
or t.name.lower() == specific_test.replace("test_", "").lower()
|
||||
)
|
||||
]
|
||||
return self.compiled_tests
|
||||
|
||||
@property
|
||||
def link_cache_dir(self) -> Path:
|
||||
"""Get the link cache directory"""
|
||||
cache_dir = self.build_dir / "link_cache"
|
||||
cache_dir.mkdir(exist_ok=True)
|
||||
return cache_dir
|
||||
|
||||
def _calculate_file_hash(self, file_path: Path) -> str:
|
||||
"""Calculate SHA256 hash of a file"""
|
||||
if not file_path.exists():
|
||||
return "no_file"
|
||||
|
||||
hash_sha256 = hashlib.sha256()
|
||||
with open(file_path, "rb") as f:
|
||||
# Read in chunks to handle large files efficiently
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
hash_sha256.update(chunk)
|
||||
return hash_sha256.hexdigest()
|
||||
|
||||
def _calculate_linker_args_hash(self, linker_args: List[str]) -> str:
|
||||
"""Calculate SHA256 hash of linker arguments"""
|
||||
# Convert linker args to a stable string representation
|
||||
args_str = "|".join(sorted(linker_args))
|
||||
hash_sha256 = hashlib.sha256()
|
||||
hash_sha256.update(args_str.encode("utf-8"))
|
||||
return hash_sha256.hexdigest()
|
||||
|
||||
def _calculate_link_cache_key(
|
||||
self, test_obj_path: Path, fastled_lib_path: Path, linker_args: List[str]
|
||||
) -> str:
|
||||
"""Calculate comprehensive cache key for linking"""
|
||||
# Hash all the factors that affect linking output
|
||||
fastled_hash = self._calculate_file_hash(fastled_lib_path)
|
||||
test_obj_hash = self._calculate_file_hash(test_obj_path)
|
||||
linker_hash = self._calculate_linker_args_hash(linker_args)
|
||||
|
||||
# Combine all hashes into a single cache key
|
||||
combined = f"fastled:{fastled_hash}|test:{test_obj_hash}|flags:{linker_hash}"
|
||||
final_hash = hashlib.sha256(combined.encode("utf-8")).hexdigest()
|
||||
|
||||
return final_hash[:16] # Use first 16 chars for readability
|
||||
|
||||
def _get_cached_executable(self, test_name: str, cache_key: str) -> Optional[Path]:
|
||||
"""Check if a cached executable exists for this test and cache key"""
|
||||
cached_exe = self.link_cache_dir / f"{test_name}_{cache_key}.exe"
|
||||
return cached_exe if cached_exe.exists() else None
|
||||
|
||||
def _cache_executable(self, test_name: str, cache_key: str, exe_path: Path) -> None:
|
||||
"""Cache an executable with the given cache key"""
|
||||
if not exe_path.exists():
|
||||
return
|
||||
|
||||
cached_exe = self.link_cache_dir / f"{test_name}_{cache_key}.exe"
|
||||
try:
|
||||
shutil.copy2(exe_path, cached_exe)
|
||||
print(f" {test_name}: Cached executable for future use")
|
||||
except Exception as e:
|
||||
print(f" {test_name}: Warning - failed to cache executable: {e}")
|
||||
|
||||
def _get_platform_linker_args(self, fastled_lib_path: Path) -> List[str]:
|
||||
"""Get platform-specific linker arguments using BuildFlags"""
|
||||
# Use the build_flags that includes platform-specific configuration
|
||||
args = self.build_flags.link_flags.copy()
|
||||
|
||||
# Load TOML config for test-specific and build mode flags
|
||||
toml_path = self.project_root / "ci" / "build_unit.toml"
|
||||
with open(toml_path, "rb") as f:
|
||||
config = tomllib.load(f)
|
||||
|
||||
# Add test-specific linking flags if available
|
||||
test_link_flags = config.get("linking", {}).get("test", {}).get("flags", [])
|
||||
args.extend(test_link_flags)
|
||||
|
||||
# Add build mode specific flags (use test-specific debug mode)
|
||||
mode = "quick" if self.quick_build else "test_debug"
|
||||
mode_flags = config.get("build_modes", {}).get(mode, {}).get("link_flags", [])
|
||||
args.extend(mode_flags)
|
||||
|
||||
# Add FastLED library
|
||||
args.append(str(fastled_lib_path)) # Link against the FastLED static library
|
||||
|
||||
return args
|
||||
|
||||
@classmethod
|
||||
def get_existing_instance(cls) -> Optional["FastLEDTestCompiler"]:
|
||||
"""Get existing compiler instance for reuse"""
|
||||
return cls._existing_instance
|
||||
|
||||
|
||||
def check_iwyu_available() -> bool:
|
||||
"""Check if include-what-you-use is available (preserving existing logic)"""
|
||||
import shutil
|
||||
|
||||
return (
|
||||
shutil.which("include-what-you-use") is not None
|
||||
or shutil.which("iwyu") is not None
|
||||
)
|
||||
|
||||
|
||||
# For testing this module directly
|
||||
if __name__ == "__main__":
|
||||
print("Testing FastLEDTestCompiler...")
|
||||
|
||||
# Create test compiler
|
||||
test_compiler = FastLEDTestCompiler.create_for_unit_tests(
|
||||
project_root=Path(PROJECT_ROOT), clean_build=True
|
||||
)
|
||||
|
||||
# Discover test files
|
||||
test_files = test_compiler.discover_test_files()
|
||||
print(f"Discovered {len(test_files)} test files")
|
||||
|
||||
# Test compilation of a few files
|
||||
if test_files:
|
||||
limited_files = test_files[:3] # Test first 3 files
|
||||
print(f"Testing compilation of {len(limited_files)} test files...")
|
||||
|
||||
compile_result = test_compiler.compile_all_tests()
|
||||
|
||||
if compile_result.success:
|
||||
print(
|
||||
f"SUCCESS: Compiled {compile_result.compiled_count} tests in {compile_result.duration:.2f}s"
|
||||
)
|
||||
executables = test_compiler.get_test_executables()
|
||||
for exe in executables:
|
||||
print(f" Executable: {exe.name} -> {exe.executable_path}")
|
||||
else:
|
||||
print(f"FAILED: {len(compile_result.errors)} errors")
|
||||
for error in compile_result.errors:
|
||||
print(f" {error.test_name}: {error.message}")
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,171 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from concurrent.futures import Future
|
||||
from pathlib import Path
|
||||
from typing import List, Sequence
|
||||
|
||||
from ci.compiler.clang_compiler import Compiler, CompilerOptions, Result
|
||||
from ci.compiler.test_example_compilation import (
|
||||
create_fastled_compiler,
|
||||
get_fastled_core_sources,
|
||||
)
|
||||
|
||||
|
||||
def _sorted_src_files(files: Sequence[Path]) -> List[Path]:
|
||||
project_root = Path.cwd()
|
||||
|
||||
def key_fn(p: Path) -> str:
|
||||
try:
|
||||
rel = p.relative_to(project_root)
|
||||
except Exception:
|
||||
rel = p
|
||||
return rel.as_posix()
|
||||
|
||||
return sorted(files, key=key_fn)
|
||||
|
||||
|
||||
def _partition(files: List[Path], chunks: int) -> List[List[Path]]:
|
||||
total = len(files)
|
||||
if chunks <= 0:
|
||||
raise ValueError("chunks must be >= 1")
|
||||
chunks = min(chunks, total) if total > 0 else 1
|
||||
base = total // chunks
|
||||
rem = total % chunks
|
||||
partitions: List[List[Path]] = []
|
||||
start = 0
|
||||
for i in range(chunks):
|
||||
size = base + (1 if i < rem else 0)
|
||||
end = start + size
|
||||
if size > 0:
|
||||
partitions.append(files[start:end])
|
||||
start = end
|
||||
return partitions
|
||||
|
||||
|
||||
def build_unity_chunks_and_archive(
|
||||
compiler: Compiler,
|
||||
chunks: int,
|
||||
output_archive: Path,
|
||||
unity_dir: Path,
|
||||
no_parallel: bool,
|
||||
) -> Path:
|
||||
unity_dir.mkdir(parents=True, exist_ok=True)
|
||||
# Gather and sort source files consistently with existing library logic
|
||||
files = get_fastled_core_sources()
|
||||
# Exclude any main-like TU
|
||||
files = [p for p in files if p.name != "stub_main.cpp"]
|
||||
files = _sorted_src_files(files)
|
||||
if not files:
|
||||
raise RuntimeError("No source files found under src/** for unity build")
|
||||
|
||||
partitions = _partition(files, chunks)
|
||||
|
||||
# Prepare per-chunk unity paths
|
||||
unity_cpp_paths: List[Path] = [
|
||||
unity_dir / f"unity{i + 1}.cpp" for i in range(len(partitions))
|
||||
]
|
||||
unity_obj_paths: List[Path] = [p.with_suffix(".o") for p in unity_cpp_paths]
|
||||
|
||||
# Compile chunks
|
||||
futures: List[Future[Result]] = []
|
||||
compile_opts = CompilerOptions(
|
||||
include_path=compiler.settings.include_path,
|
||||
compiler=compiler.settings.compiler,
|
||||
defines=compiler.settings.defines,
|
||||
std_version=compiler.settings.std_version,
|
||||
compiler_args=compiler.settings.compiler_args,
|
||||
use_pch=False,
|
||||
additional_flags=["-c"],
|
||||
parallel=not no_parallel,
|
||||
)
|
||||
|
||||
def submit(idx: int) -> Future[Result]:
|
||||
chunk_files = partitions[idx]
|
||||
unity_cpp = unity_cpp_paths[idx]
|
||||
return compiler.compile_unity(compile_opts, chunk_files, unity_cpp)
|
||||
|
||||
if no_parallel:
|
||||
# Sequential execution
|
||||
for i in range(len(partitions)):
|
||||
result = submit(i).result()
|
||||
if not result.ok:
|
||||
raise RuntimeError(f"Unity chunk {i + 1} failed: {result.stderr}")
|
||||
else:
|
||||
futures = [submit(i) for i in range(len(partitions))]
|
||||
for i, fut in enumerate(futures):
|
||||
result = fut.result()
|
||||
if not result.ok:
|
||||
raise RuntimeError(f"Unity chunk {i + 1} failed: {result.stderr}")
|
||||
|
||||
# Create archive from chunk objects
|
||||
from ci.compiler.clang_compiler import LibarchiveOptions
|
||||
|
||||
output_archive.parent.mkdir(parents=True, exist_ok=True)
|
||||
archive_result = compiler.create_archive(
|
||||
unity_obj_paths, output_archive, LibarchiveOptions()
|
||||
).result()
|
||||
if not archive_result.ok:
|
||||
raise RuntimeError(f"Archive creation failed: {archive_result.stderr}")
|
||||
|
||||
return output_archive
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Chunked UNITY build and archive for FastLED src/**"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--chunks", type=int, default=1, help="Number of unity chunks (1-4 typical)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-parallel", action="store_true", help="Compile unity chunks sequentially"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=str,
|
||||
default=str(Path(".build/fastled/libfastled.a")),
|
||||
help="Path to output archive libfastled.a",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--unity-dir",
|
||||
type=str,
|
||||
default=str(Path(".build/fastled/unity")),
|
||||
help="Directory to place generated unity{N}.cpp/.o files",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--use-pch",
|
||||
action="store_true",
|
||||
help="Enable PCH (not recommended for unity builds)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Anchor to project root (two levels up from this file)
|
||||
here = Path(__file__).resolve()
|
||||
project_root = here.parent.parent.parent
|
||||
os.chdir(project_root)
|
||||
|
||||
try:
|
||||
compiler = create_fastled_compiler(
|
||||
use_pch=args.use_pch, parallel=not args.no_parallel
|
||||
)
|
||||
out = build_unity_chunks_and_archive(
|
||||
compiler=compiler,
|
||||
chunks=args.chunks,
|
||||
output_archive=Path(args.output),
|
||||
unity_dir=Path(args.unity_dir),
|
||||
no_parallel=args.no_parallel,
|
||||
)
|
||||
print(f"[UNITY] Created archive: {out}")
|
||||
return 0
|
||||
except KeyboardInterrupt:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"\x1b[31m[UNITY] FAILED: {e}\x1b[0m")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
Demo script showing how to use the new generic symbol analysis functionality
|
||||
"""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
|
||||
def run_symbol_analysis(board_name: str):
|
||||
"""Run symbol analysis for a specific board"""
|
||||
print(f"Running symbol analysis for {board_name}...")
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[sys.executable, "ci/util/symbol_analysis.py", "--board", board_name],
|
||||
cwd=".",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
print(f"✅ Symbol analysis completed successfully for {board_name}")
|
||||
return True
|
||||
else:
|
||||
print(f"❌ Symbol analysis failed for {board_name}")
|
||||
print(f"Error: {result.stderr}")
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f"❌ Error running symbol analysis for {board_name}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def load_and_summarize_results(board_name: str):
|
||||
"""Load and display a summary of symbol analysis results"""
|
||||
json_file = Path(f".build/{board_name}_symbol_analysis.json")
|
||||
|
||||
if not json_file.exists():
|
||||
print(f"❌ Results file not found: {json_file}")
|
||||
return
|
||||
|
||||
try:
|
||||
with open(json_file) as f:
|
||||
data = json.load(f)
|
||||
|
||||
summary = data["summary"]
|
||||
print(f"\n📊 SUMMARY FOR {board_name.upper()}:")
|
||||
print(f" Total symbols: {summary['total_symbols']}")
|
||||
print(
|
||||
f" Total size: {summary['total_size']} bytes ({summary['total_size'] / 1024:.1f} KB)"
|
||||
)
|
||||
|
||||
print("\n🔝 TOP 5 LARGEST SYMBOLS:")
|
||||
for i, symbol in enumerate(summary["largest_symbols"][:5], 1):
|
||||
name = symbol.get("demangled_name", symbol["name"])
|
||||
print(
|
||||
f" {i}. {symbol['size']:6d} bytes - {name[:80]}{'...' if len(name) > 80 else ''}"
|
||||
)
|
||||
|
||||
print("\n📁 SYMBOL TYPES:")
|
||||
for sym_type, stats in list(summary["type_breakdown"].items())[:5]:
|
||||
print(f" {sym_type}: {stats['count']} symbols, {stats['size']} bytes")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error loading results for {board_name}: {e}")
|
||||
|
||||
|
||||
def main():
|
||||
"""Demo the symbol analysis functionality"""
|
||||
print("=" * 80)
|
||||
print("FASTLED SYMBOL ANALYSIS DEMO")
|
||||
print("=" * 80)
|
||||
|
||||
# List of boards to analyze (you can add more)
|
||||
boards_to_analyze: List[str] = []
|
||||
|
||||
# Check which boards have build info available
|
||||
build_dir = Path(".build")
|
||||
if build_dir.exists():
|
||||
for item in build_dir.iterdir():
|
||||
if item.is_dir() and (item / "build_info.json").exists():
|
||||
boards_to_analyze.append(item.name)
|
||||
|
||||
if not boards_to_analyze:
|
||||
print("❌ No boards with build_info.json found in .build directory")
|
||||
print(
|
||||
" Please compile some examples first using: uv run -m ci.ci-compile <board> --examples Blink"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
print(
|
||||
f"📋 Found {len(boards_to_analyze)} boards with build info: {', '.join(boards_to_analyze)}"
|
||||
)
|
||||
|
||||
# Run symbol analysis for each board
|
||||
successful_boards: List[str] = []
|
||||
for board in boards_to_analyze:
|
||||
if run_symbol_analysis(board):
|
||||
successful_boards.append(board)
|
||||
|
||||
print(f"\n📈 RESULTS SUMMARY FOR {len(successful_boards)} BOARDS:")
|
||||
print("=" * 80)
|
||||
|
||||
# Load and summarize results
|
||||
for board in successful_boards:
|
||||
load_and_summarize_results(board)
|
||||
print()
|
||||
|
||||
print("🎉 Symbol analysis demo completed!")
|
||||
print("📁 Detailed JSON files saved in .build/ directory")
|
||||
print("💡 Use these files to identify optimization opportunities")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,339 @@
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
|
||||
class DockerManager:
|
||||
def __init__(self):
|
||||
# No direct client object, will use subprocess for docker CLI commands
|
||||
pass
|
||||
|
||||
def _run_docker_command(
|
||||
self,
|
||||
command: List[str],
|
||||
check: bool = True,
|
||||
stream_output: bool = False,
|
||||
timeout: Optional[int] = None,
|
||||
interrupt_pattern: Optional[str] = None,
|
||||
output_file: Optional[str] = None,
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
full_command = ["docker"] + command
|
||||
print(f"Executing Docker command: {' '.join(full_command)}")
|
||||
|
||||
# Set environment to prevent MSYS2/Git Bash path conversion on Windows
|
||||
env = os.environ.copy()
|
||||
# Set UTF-8 encoding environment variables for Windows
|
||||
env["PYTHONIOENCODING"] = "utf-8"
|
||||
env["PYTHONUTF8"] = "1"
|
||||
# Only set MSYS_NO_PATHCONV if we're in a Git Bash/MSYS2 environment
|
||||
if (
|
||||
"MSYSTEM" in os.environ
|
||||
or os.environ.get("TERM") == "xterm"
|
||||
or "bash.exe" in os.environ.get("SHELL", "")
|
||||
):
|
||||
env["MSYS_NO_PATHCONV"] = "1"
|
||||
|
||||
if stream_output:
|
||||
# Use RunningProcess for streaming output
|
||||
import re
|
||||
import time
|
||||
|
||||
from ci.util.running_process import RunningProcess
|
||||
|
||||
proc = RunningProcess(full_command, env=env, auto_run=True)
|
||||
pattern_found = False
|
||||
timeout_occurred = False
|
||||
start_time = time.time()
|
||||
|
||||
# Open output file if specified
|
||||
output_handle = None
|
||||
if output_file:
|
||||
try:
|
||||
output_handle = open(output_file, "w", encoding="utf-8")
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not open output file {output_file}: {e}")
|
||||
|
||||
try:
|
||||
with proc.line_iter(timeout=None) as it:
|
||||
for line in it:
|
||||
print(line)
|
||||
|
||||
# Write to output file if specified
|
||||
if output_handle:
|
||||
output_handle.write(line + "\n")
|
||||
output_handle.flush() # Ensure immediate write
|
||||
|
||||
# Check for interrupt pattern (for informational purposes only)
|
||||
if interrupt_pattern and re.search(interrupt_pattern, line):
|
||||
print(f"Pattern detected: {line}")
|
||||
pattern_found = True
|
||||
|
||||
# Check timeout
|
||||
if timeout and (time.time() - start_time) > timeout:
|
||||
print(
|
||||
f"Timeout reached ({timeout}s), terminating container..."
|
||||
)
|
||||
timeout_occurred = True
|
||||
break
|
||||
|
||||
# Wait for process to complete
|
||||
returncode = proc.wait()
|
||||
|
||||
# Handle timeout case: return success (0) for timeouts as per requirement
|
||||
if timeout_occurred:
|
||||
print("Process terminated due to timeout - treating as success")
|
||||
final_returncode = 0
|
||||
else:
|
||||
# Return the actual container exit code for all other cases
|
||||
final_returncode = returncode
|
||||
|
||||
return subprocess.CompletedProcess(
|
||||
args=full_command, returncode=final_returncode, stdout="", stderr=""
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error during streaming: {e}")
|
||||
returncode = proc.wait() if hasattr(proc, "wait") else 1
|
||||
return subprocess.CompletedProcess(
|
||||
args=full_command, returncode=returncode, stdout="", stderr=str(e)
|
||||
)
|
||||
finally:
|
||||
# Close output file if it was opened
|
||||
if output_handle:
|
||||
output_handle.close()
|
||||
else:
|
||||
# Use regular subprocess for non-streaming commands
|
||||
result = subprocess.run(
|
||||
full_command,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=check,
|
||||
env=env,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
if check and result.returncode != 0:
|
||||
print(
|
||||
f"Docker command failed with exit code {result.returncode}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
print(f"Stdout: {result.stdout}", file=sys.stderr)
|
||||
print(f"Stderr: {result.stderr}", file=sys.stderr)
|
||||
return result
|
||||
|
||||
def pull_image(self, image_name: str, tag: str = "latest"):
|
||||
"""Pulls the specified Docker image."""
|
||||
print(f"Pulling image: {image_name}:{tag}")
|
||||
self._run_docker_command(["pull", f"{image_name}:{tag}"])
|
||||
print(f"Image {image_name}:{tag} pulled successfully.")
|
||||
|
||||
def run_container(
|
||||
self,
|
||||
image_name: str,
|
||||
command: List[str],
|
||||
volumes: Optional[Dict[str, Dict[str, str]]] = None,
|
||||
environment: Optional[Dict[str, str]] = None,
|
||||
detach: bool = False,
|
||||
name: Optional[str] = None,
|
||||
) -> str: # Return container ID as string
|
||||
"""
|
||||
Runs a Docker container with specified command, volume mounts, and environment variables.
|
||||
Returns the container ID.
|
||||
"""
|
||||
print(
|
||||
f"Running container from image: {image_name} with command: {' '.join(command)}"
|
||||
)
|
||||
docker_cmd = ["run", "--rm"] # --rm to automatically remove container on exit
|
||||
if detach:
|
||||
docker_cmd.append("-d")
|
||||
if name:
|
||||
docker_cmd.extend(["--name", name])
|
||||
if volumes:
|
||||
for host_path, container_path_info in volumes.items():
|
||||
mode = container_path_info.get("mode", "rw")
|
||||
docker_cmd.extend(
|
||||
["-v", f"{host_path}:{container_path_info['bind']}:{mode}"]
|
||||
)
|
||||
if environment:
|
||||
for key, value in environment.items():
|
||||
docker_cmd.extend(["-e", f"{key}={value}"])
|
||||
|
||||
docker_cmd.append(image_name)
|
||||
docker_cmd.extend(command)
|
||||
|
||||
result = self._run_docker_command(docker_cmd)
|
||||
container_id = result.stdout.strip()
|
||||
print(f"Container {container_id} started.")
|
||||
return container_id
|
||||
|
||||
def run_container_streaming(
|
||||
self,
|
||||
image_name: str,
|
||||
command: List[str],
|
||||
volumes: Optional[Dict[str, Dict[str, str]]] = None,
|
||||
environment: Optional[Dict[str, str]] = None,
|
||||
name: Optional[str] = None,
|
||||
timeout: Optional[int] = None,
|
||||
interrupt_pattern: Optional[str] = None,
|
||||
output_file: Optional[str] = None,
|
||||
) -> int:
|
||||
"""
|
||||
Runs a Docker container with streaming output.
|
||||
Returns actual container exit code (0 for success, timeout is treated as success).
|
||||
|
||||
Args:
|
||||
image_name: Docker image to run
|
||||
command: Command to execute in container
|
||||
volumes: Volume mounts
|
||||
environment: Environment variables
|
||||
name: Container name
|
||||
timeout: Timeout in seconds (if exceeded, returns 0)
|
||||
interrupt_pattern: Pattern to detect in output (informational only)
|
||||
output_file: Optional file path to write output to
|
||||
"""
|
||||
print(
|
||||
f"Running container from image: {image_name} with command: {' '.join(command)}"
|
||||
)
|
||||
docker_cmd = ["run", "--rm"] # --rm to automatically remove container on exit
|
||||
if name:
|
||||
docker_cmd.extend(["--name", name])
|
||||
if volumes:
|
||||
for host_path, container_path_info in volumes.items():
|
||||
mode = container_path_info.get("mode", "rw")
|
||||
docker_cmd.extend(
|
||||
["-v", f"{host_path}:{container_path_info['bind']}:{mode}"]
|
||||
)
|
||||
if environment:
|
||||
for key, value in environment.items():
|
||||
docker_cmd.extend(["-e", f"{key}={value}"])
|
||||
|
||||
docker_cmd.append(image_name)
|
||||
docker_cmd.extend(command)
|
||||
|
||||
result = self._run_docker_command(
|
||||
docker_cmd,
|
||||
check=False,
|
||||
stream_output=True,
|
||||
timeout=timeout,
|
||||
interrupt_pattern=interrupt_pattern,
|
||||
output_file=output_file,
|
||||
)
|
||||
return result.returncode
|
||||
|
||||
def get_container_logs(self, container_id_or_name: str) -> str:
|
||||
"""Retrieves logs from a container."""
|
||||
print(f"Getting logs for container: {container_id_or_name}")
|
||||
try:
|
||||
result = self._run_docker_command(
|
||||
["logs", container_id_or_name], check=False
|
||||
)
|
||||
if result.returncode == 0:
|
||||
logs = result.stdout
|
||||
print(f"Logs retrieved for {container_id_or_name}.")
|
||||
return logs
|
||||
else:
|
||||
# Container may have exited, try to get logs anyway
|
||||
print(f"Warning: docker logs returned exit code {result.returncode}")
|
||||
logs = result.stdout if result.stdout else result.stderr
|
||||
return logs
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to get container logs: {e}")
|
||||
return ""
|
||||
|
||||
def stop_container(self, container_id_or_name: str):
|
||||
"""Stops a running container."""
|
||||
print(f"Stopping container: {container_id_or_name}")
|
||||
self._run_docker_command(["stop", container_id_or_name])
|
||||
print(f"Container {container_id_or_name} stopped.")
|
||||
|
||||
def remove_container(self, container_id_or_name: str):
|
||||
"""Removes a container."""
|
||||
print(f"Removing container: {container_id_or_name}")
|
||||
self._run_docker_command(["rm", container_id_or_name])
|
||||
print(f"Container {container_id_or_name} removed.")
|
||||
|
||||
def execute_command_in_container(
|
||||
self, container_id_or_name: str, command: List[str]
|
||||
) -> tuple[int, str, str]:
|
||||
"""Executes a command inside a running container and returns exit code, stdout, and stderr."""
|
||||
print(
|
||||
f"Executing command '{' '.join(command)}' in container: {container_id_or_name}"
|
||||
)
|
||||
result = self._run_docker_command(
|
||||
["exec", container_id_or_name] + command, check=False
|
||||
)
|
||||
stdout = result.stdout
|
||||
stderr = result.stderr
|
||||
print(
|
||||
f"Command executed in {container_id_or_name}. Exit code: {result.returncode}"
|
||||
)
|
||||
return result.returncode, stdout, stderr
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Example Usage:
|
||||
manager = DockerManager()
|
||||
image_to_use = "mluis/qemu-esp32"
|
||||
container_name = "test_esp32_qemu"
|
||||
|
||||
try:
|
||||
print(f"--- Starting DockerManager test with image: {image_to_use} ---")
|
||||
manager.pull_image(image_to_use)
|
||||
|
||||
# Example: Run a simple command in the container
|
||||
print("\n--- Running a simple command ---")
|
||||
container_id = manager.run_container(
|
||||
image_name=image_to_use,
|
||||
command=["echo", "Hello from Docker!"],
|
||||
detach=True,
|
||||
name=container_name,
|
||||
)
|
||||
|
||||
# Wait a bit for the command to execute and logs to be generated
|
||||
import time
|
||||
|
||||
time.sleep(2)
|
||||
|
||||
logs = manager.get_container_logs(container_id)
|
||||
print("Container Logs:\n", logs)
|
||||
|
||||
# Check if the expected output is in the logs
|
||||
if "Hello from Docker!" in logs:
|
||||
print("SUCCESS: Expected output found in container logs.")
|
||||
else:
|
||||
print("FAILURE: Expected output NOT found in container logs.")
|
||||
|
||||
# Example: Execute a command inside a running container
|
||||
print("\n--- Executing command inside container ---")
|
||||
exit_code, stdout, stderr = manager.execute_command_in_container(
|
||||
container_id, ["sh", "-c", "echo 'Executed inside!'; exit 42"]
|
||||
)
|
||||
print(f"Exec command stdout: {stdout}")
|
||||
print(f"Exec command stderr: {stderr}")
|
||||
print(f"Exec command exit code: {exit_code}")
|
||||
if exit_code == 42 and "Executed inside!" in stdout:
|
||||
print("SUCCESS: Command executed inside container as expected.")
|
||||
else:
|
||||
print("FAILURE: Command execution inside container failed.")
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"Error during Docker operation: {e}", file=sys.stderr)
|
||||
print(f"Stdout: {e.stdout}", file=sys.stderr)
|
||||
print(f"Stderr: {e.stderr}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"An unexpected error occurred: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
finally:
|
||||
# Ensure cleanup
|
||||
try:
|
||||
manager.stop_container(container_name)
|
||||
manager.remove_container(container_name)
|
||||
print(f"Container {container_name} stopped and removed.")
|
||||
except subprocess.CalledProcessError:
|
||||
print(
|
||||
f"Container {container_name} not found or already removed during cleanup."
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"Error during cleanup of {container_name}: {e}", file=sys.stderr)
|
||||
@@ -0,0 +1,106 @@
|
||||
# Dockerfile for ESP32 QEMU emulation environment
|
||||
# Based on ESP-IDF with QEMU support
|
||||
|
||||
FROM espressif/idf:v5.1.2
|
||||
|
||||
# Install additional dependencies for QEMU
|
||||
RUN apt-get update && apt-get install -y \
|
||||
qemu-system-misc \
|
||||
qemu-utils \
|
||||
python3-pip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Python dependencies for ESP tools
|
||||
RUN pip3 install --no-cache-dir \
|
||||
esptool \
|
||||
pyserial
|
||||
|
||||
# Create workspace directory
|
||||
WORKDIR /workspace
|
||||
|
||||
# Set up environment for QEMU ESP32
|
||||
ENV QEMU_ESP32_VERSION=esp-develop-8.2.0-20240122
|
||||
|
||||
# Download and install ESP32 QEMU if not using system QEMU
|
||||
RUN mkdir -p /opt/qemu-esp32 && \
|
||||
cd /opt/qemu-esp32 && \
|
||||
wget -q https://github.com/espressif/qemu/releases/download/${QEMU_ESP32_VERSION}/qemu-xtensa-softmmu-esp_develop_8.2.0_20240122-x86_64-linux-gnu.tar.xz && \
|
||||
tar -xf qemu-xtensa-softmmu-esp_develop_8.2.0_20240122-x86_64-linux-gnu.tar.xz && \
|
||||
rm qemu-xtensa-softmmu-esp_develop_8.2.0_20240122-x86_64-linux-gnu.tar.xz && \
|
||||
ln -sf /opt/qemu-esp32/qemu/bin/qemu-system-xtensa /usr/local/bin/qemu-system-xtensa
|
||||
|
||||
# Download ESP32 ROM files for QEMU
|
||||
RUN mkdir -p /opt/esp-rom && \
|
||||
cd /opt/esp-rom && \
|
||||
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32-v3-rom.bin && \
|
||||
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32-v3-rom-eco3.bin && \
|
||||
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32c3-rom.bin && \
|
||||
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32s2-rom.bin && \
|
||||
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32s3-rom.bin
|
||||
|
||||
# Set environment variables for ROM files
|
||||
ENV ESP_ROM_PATH=/opt/esp-rom
|
||||
ENV ESP32_ROM=${ESP_ROM_PATH}/esp32-v3-rom.bin
|
||||
ENV ESP32C3_ROM=${ESP_ROM_PATH}/esp32c3-rom.bin
|
||||
ENV ESP32S2_ROM=${ESP_ROM_PATH}/esp32s2-rom.bin
|
||||
ENV ESP32S3_ROM=${ESP_ROM_PATH}/esp32s3-rom.bin
|
||||
|
||||
# Create helper script for running QEMU with proper settings
|
||||
RUN cat > /usr/local/bin/run-qemu-esp32 << 'EOF'
|
||||
#!/bin/bash
|
||||
# Helper script to run QEMU with ESP32 firmware
|
||||
|
||||
FIRMWARE_PATH=${1:-/workspace/firmware.bin}
|
||||
MACHINE=${2:-esp32}
|
||||
FLASH_SIZE=${3:-4}
|
||||
|
||||
# Select ROM file based on machine type
|
||||
case "$MACHINE" in
|
||||
esp32)
|
||||
ROM_FILE=$ESP32_ROM
|
||||
;;
|
||||
esp32c3)
|
||||
ROM_FILE=$ESP32C3_ROM
|
||||
;;
|
||||
esp32s2)
|
||||
ROM_FILE=$ESP32S2_ROM
|
||||
;;
|
||||
esp32s3)
|
||||
ROM_FILE=$ESP32S3_ROM
|
||||
;;
|
||||
*)
|
||||
echo "Unknown machine type: $MACHINE"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# Create flash image if needed
|
||||
FLASH_IMAGE=/tmp/flash_${RANDOM}.bin
|
||||
if [ -f "$FIRMWARE_PATH" ]; then
|
||||
# Create empty flash image
|
||||
dd if=/dev/zero of=$FLASH_IMAGE bs=1M count=$FLASH_SIZE 2>/dev/null
|
||||
dd if=$FIRMWARE_PATH of=$FLASH_IMAGE bs=1 seek=$((0x10000)) conv=notrunc 2>/dev/null
|
||||
else
|
||||
echo "Firmware not found: $FIRMWARE_PATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Run QEMU
|
||||
exec qemu-system-xtensa \
|
||||
-nographic \
|
||||
-machine $MACHINE \
|
||||
-bios $ROM_FILE \
|
||||
-drive file=$FLASH_IMAGE,if=mtd,format=raw \
|
||||
-global driver=timer.esp32.timg,property=wdt_disable,value=true \
|
||||
"$@"
|
||||
EOF
|
||||
|
||||
RUN chmod +x /usr/local/bin/run-qemu-esp32
|
||||
|
||||
# Default command
|
||||
CMD ["/usr/local/bin/run-qemu-esp32"]
|
||||
|
||||
# Labels
|
||||
LABEL maintainer="FastLED Team"
|
||||
LABEL description="ESP32 QEMU emulation environment for FastLED testing"
|
||||
LABEL version="1.0.0"
|
||||
@@ -0,0 +1,92 @@
|
||||
# Lightweight Dockerfile for ESP32 QEMU emulation
|
||||
# Uses pre-built QEMU binaries without full ESP-IDF
|
||||
|
||||
FROM ubuntu:22.04
|
||||
|
||||
# Install minimal dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
wget \
|
||||
xz-utils \
|
||||
python3 \
|
||||
python3-pip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install esptool for flash image creation
|
||||
RUN pip3 install --no-cache-dir esptool
|
||||
|
||||
# Create workspace
|
||||
WORKDIR /workspace
|
||||
|
||||
# Download and install ESP32 QEMU binaries
|
||||
ENV QEMU_VERSION=esp-develop-8.2.0-20240122
|
||||
RUN mkdir -p /opt/qemu && \
|
||||
cd /opt/qemu && \
|
||||
wget -q https://github.com/espressif/qemu/releases/download/${QEMU_VERSION}/qemu-xtensa-softmmu-${QEMU_VERSION}-x86_64-linux-gnu.tar.xz && \
|
||||
tar -xf qemu-xtensa-softmmu-${QEMU_VERSION}-x86_64-linux-gnu.tar.xz && \
|
||||
rm qemu-xtensa-softmmu-${QEMU_VERSION}-x86_64-linux-gnu.tar.xz && \
|
||||
mv qemu /usr/local/
|
||||
|
||||
# Add QEMU to PATH
|
||||
ENV PATH=/usr/local/qemu/bin:$PATH
|
||||
|
||||
# Download ESP32 ROM files
|
||||
RUN mkdir -p /opt/esp-rom && \
|
||||
cd /opt/esp-rom && \
|
||||
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32-v3-rom.bin && \
|
||||
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32c3-rom.bin && \
|
||||
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32s2-rom.bin && \
|
||||
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32s3-rom.bin
|
||||
|
||||
# Set ROM path environment variable
|
||||
ENV ESP_ROM_PATH=/opt/esp-rom
|
||||
|
||||
# Create entrypoint script
|
||||
RUN cat > /entrypoint.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Default values
|
||||
FIRMWARE=${FIRMWARE:-/workspace/firmware.bin}
|
||||
MACHINE=${MACHINE:-esp32}
|
||||
FLASH_SIZE=${FLASH_SIZE:-4}
|
||||
TIMEOUT=${TIMEOUT:-30}
|
||||
|
||||
# Select ROM based on machine type
|
||||
case "$MACHINE" in
|
||||
esp32) ROM_FILE=/opt/esp-rom/esp32-v3-rom.bin ;;
|
||||
esp32c3) ROM_FILE=/opt/esp-rom/esp32c3-rom.bin ;;
|
||||
esp32s2) ROM_FILE=/opt/esp-rom/esp32s2-rom.bin ;;
|
||||
esp32s3) ROM_FILE=/opt/esp-rom/esp32s3-rom.bin ;;
|
||||
*) echo "Unknown machine: $MACHINE"; exit 1 ;;
|
||||
esac
|
||||
|
||||
# Check if firmware exists
|
||||
if [ ! -f "$FIRMWARE" ]; then
|
||||
echo "ERROR: Firmware not found: $FIRMWARE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create flash image
|
||||
FLASH_IMG=/tmp/flash.bin
|
||||
dd if=/dev/zero of=$FLASH_IMG bs=1M count=$FLASH_SIZE 2>/dev/null
|
||||
dd if=$FIRMWARE of=$FLASH_IMG bs=1 seek=$((0x10000)) conv=notrunc 2>/dev/null
|
||||
|
||||
# Run QEMU with timeout
|
||||
timeout $TIMEOUT qemu-system-xtensa \
|
||||
-nographic \
|
||||
-machine $MACHINE \
|
||||
-bios $ROM_FILE \
|
||||
-drive file=$FLASH_IMG,if=mtd,format=raw \
|
||||
-global driver=timer.esp32.timg,property=wdt_disable,value=true \
|
||||
"$@"
|
||||
EOF
|
||||
|
||||
RUN chmod +x /entrypoint.sh
|
||||
|
||||
# Set entrypoint
|
||||
ENTRYPOINT ["/entrypoint.sh"]
|
||||
|
||||
# Labels
|
||||
LABEL maintainer="FastLED Team"
|
||||
LABEL description="Lightweight ESP32 QEMU Docker image"
|
||||
LABEL version="1.0.0"
|
||||
@@ -0,0 +1,205 @@
|
||||
# Docker-based QEMU ESP32 Testing
|
||||
|
||||
This directory contains Docker-based QEMU ESP32 emulation tools for FastLED testing.
|
||||
|
||||
## Overview
|
||||
|
||||
The Docker QEMU solution provides:
|
||||
- Consistent testing environment across different platforms
|
||||
- No need for local QEMU installation
|
||||
- Isolated execution environment
|
||||
- Support for multiple ESP32 variants (ESP32, ESP32-C3, ESP32-S2, ESP32-S3)
|
||||
|
||||
## Components
|
||||
|
||||
### Core Files
|
||||
|
||||
- `DockerManager.py` - Docker container management utilities
|
||||
- `qemu_esp32_docker.py` - Main Docker QEMU runner script
|
||||
- `qemu_test_integration.py` - Integration with existing test framework
|
||||
- `Dockerfile.qemu-esp32` - Full ESP-IDF + QEMU environment
|
||||
- `Dockerfile.qemu-esp32-lite` - Lightweight QEMU-only environment
|
||||
- `docker-compose.yml` - Docker Compose configuration
|
||||
|
||||
## Usage
|
||||
|
||||
### Quick Start
|
||||
|
||||
1. **Run a firmware test with Docker:**
|
||||
```bash
|
||||
uv run ci/docker/qemu_esp32_docker.py path/to/firmware.bin
|
||||
```
|
||||
|
||||
2. **Run with specific timeout and interrupt pattern:**
|
||||
```bash
|
||||
uv run ci/docker/qemu_esp32_docker.py firmware.bin \
|
||||
--timeout 60 \
|
||||
--interrupt-regex "Test passed"
|
||||
```
|
||||
|
||||
3. **Use docker-compose:**
|
||||
```bash
|
||||
cd ci/docker
|
||||
FIRMWARE_DIR=../../.pio/build/esp32dev docker-compose up qemu-esp32
|
||||
```
|
||||
|
||||
### Integration with Test Framework
|
||||
|
||||
The Docker QEMU runner integrates with the existing test framework:
|
||||
|
||||
```bash
|
||||
# Check available runners
|
||||
uv run ci/docker/qemu_test_integration.py check
|
||||
|
||||
# Run test with automatic runner selection
|
||||
uv run ci/docker/qemu_test_integration.py test --firmware path/to/firmware.bin
|
||||
|
||||
# Force Docker runner
|
||||
uv run ci/docker/qemu_test_integration.py test --firmware firmware.bin --docker
|
||||
```
|
||||
|
||||
### Building Docker Images
|
||||
|
||||
```bash
|
||||
# Build lightweight image
|
||||
docker build -f Dockerfile.qemu-esp32-lite -t fastled/qemu-esp32:lite .
|
||||
|
||||
# Build full ESP-IDF image
|
||||
docker build -f Dockerfile.qemu-esp32 -t fastled/qemu-esp32:full .
|
||||
|
||||
# Or use docker-compose
|
||||
docker-compose build
|
||||
```
|
||||
|
||||
## Docker Images
|
||||
|
||||
### Pre-built Images
|
||||
|
||||
The runner can use pre-built images:
|
||||
- `espressif/qemu:esp-develop-8.2.0-20240122` (Official Espressif image)
|
||||
- `mluis/qemu-esp32:latest` (Community image)
|
||||
|
||||
### Custom Images
|
||||
|
||||
Two Dockerfiles are provided:
|
||||
|
||||
1. **Dockerfile.qemu-esp32-lite** (Recommended)
|
||||
- Minimal Ubuntu base
|
||||
- QEMU binaries only
|
||||
- ~200MB image size
|
||||
- Fast startup
|
||||
|
||||
2. **Dockerfile.qemu-esp32**
|
||||
- Full ESP-IDF environment
|
||||
- Development tools included
|
||||
- ~2GB image size
|
||||
- Complete toolchain
|
||||
|
||||
## Features
|
||||
|
||||
### Automatic Fallback
|
||||
|
||||
The integration module automatically selects the best available runner:
|
||||
1. Native QEMU (if installed)
|
||||
2. Docker QEMU (if Docker available)
|
||||
3. Error if neither available
|
||||
|
||||
### Platform Support
|
||||
|
||||
Works on:
|
||||
- Linux (native Docker)
|
||||
- macOS (Docker Desktop)
|
||||
- Windows (Docker Desktop, WSL2)
|
||||
|
||||
### Firmware Formats
|
||||
|
||||
Supports:
|
||||
- Direct `firmware.bin` files
|
||||
- PlatformIO build directories
|
||||
- ESP-IDF build directories
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `FIRMWARE_DIR` - Directory containing firmware files
|
||||
- `FIRMWARE_FILE` - Specific firmware file to run
|
||||
- `MACHINE` - ESP32 variant (esp32, esp32c3, esp32s2, esp32s3)
|
||||
- `FLASH_SIZE` - Flash size in MB (default: 4)
|
||||
- `TIMEOUT` - Execution timeout in seconds (default: 30)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Docker Not Found
|
||||
|
||||
Install Docker:
|
||||
- **Windows/Mac**: Install Docker Desktop
|
||||
- **Linux**: `sudo apt-get install docker.io` or equivalent
|
||||
|
||||
### Permission Denied
|
||||
|
||||
Add user to docker group:
|
||||
```bash
|
||||
sudo usermod -aG docker $USER
|
||||
# Log out and back in
|
||||
```
|
||||
|
||||
### Image Pull Failed
|
||||
|
||||
Check internet connection and Docker Hub access. The runner will automatically try alternative images if the primary fails.
|
||||
|
||||
### QEMU Timeout
|
||||
|
||||
Increase timeout value:
|
||||
```bash
|
||||
uv run ci/docker/qemu_esp32_docker.py firmware.bin --timeout 120
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions
|
||||
|
||||
```yaml
|
||||
- name: Run QEMU Test in Docker
|
||||
run: |
|
||||
uv run ci/docker/qemu_esp32_docker.py \
|
||||
.pio/build/esp32dev/firmware.bin \
|
||||
--timeout 60 \
|
||||
--interrupt-regex "Setup complete"
|
||||
```
|
||||
|
||||
### Local CI
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
uv pip install -r requirements.txt
|
||||
|
||||
# Run tests
|
||||
uv run ci/docker/qemu_test_integration.py test \
|
||||
--firmware .pio/build/esp32dev/firmware.bin
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Adding New ESP32 Variants
|
||||
|
||||
1. Update Dockerfiles with new ROM files
|
||||
2. Add machine type to runner scripts
|
||||
3. Test with sample firmware
|
||||
|
||||
### Debugging
|
||||
|
||||
Run interactively:
|
||||
```bash
|
||||
uv run ci/docker/qemu_esp32_docker.py firmware.bin --interactive
|
||||
```
|
||||
|
||||
Enable verbose output:
|
||||
```bash
|
||||
docker-compose up qemu-esp32-interactive
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
- Docker adds ~2-5 seconds overhead vs native QEMU
|
||||
- First run pulls image (one-time ~100MB download)
|
||||
- Subsequent runs use cached image
|
||||
- Consider using lightweight image for CI/CD
|
||||
Whitespace-only changes.
@@ -0,0 +1,61 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
qemu-esp32:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.qemu-esp32-lite
|
||||
image: fastled/qemu-esp32:latest
|
||||
container_name: fastled-qemu-esp32
|
||||
volumes:
|
||||
# Mount firmware directory
|
||||
- ${FIRMWARE_DIR:-./firmware}:/workspace:ro
|
||||
# Mount output directory for logs
|
||||
- ${OUTPUT_DIR:-./output}:/output
|
||||
environment:
|
||||
- FIRMWARE=${FIRMWARE_FILE:-/workspace/firmware.bin}
|
||||
- MACHINE=${MACHINE:-esp32}
|
||||
- FLASH_SIZE=${FLASH_SIZE:-4}
|
||||
- TIMEOUT=${TIMEOUT:-30}
|
||||
command: []
|
||||
stdin_open: false
|
||||
tty: false
|
||||
networks:
|
||||
- qemu-network
|
||||
|
||||
qemu-esp32-interactive:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.qemu-esp32-lite
|
||||
image: fastled/qemu-esp32:latest
|
||||
container_name: fastled-qemu-esp32-interactive
|
||||
volumes:
|
||||
- ${FIRMWARE_DIR:-./firmware}:/workspace:ro
|
||||
environment:
|
||||
- FIRMWARE=${FIRMWARE_FILE:-/workspace/firmware.bin}
|
||||
- MACHINE=${MACHINE:-esp32}
|
||||
- FLASH_SIZE=${FLASH_SIZE:-4}
|
||||
- TIMEOUT=${TIMEOUT:-300}
|
||||
stdin_open: true
|
||||
tty: true
|
||||
networks:
|
||||
- qemu-network
|
||||
|
||||
# Service for building the Docker image
|
||||
builder:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.qemu-esp32-lite
|
||||
image: fastled/qemu-esp32:latest
|
||||
command: echo "Image built successfully"
|
||||
|
||||
networks:
|
||||
qemu-network:
|
||||
driver: bridge
|
||||
|
||||
# Volume definitions (optional, for persistent storage)
|
||||
volumes:
|
||||
firmware:
|
||||
driver: local
|
||||
output:
|
||||
driver: local
|
||||
@@ -0,0 +1,502 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Docker-based QEMU ESP32 Runner
|
||||
|
||||
Runs ESP32 firmware in QEMU using Docker containers for better isolation
|
||||
and consistency across different environments.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
from ci.dockerfiles.DockerManager import DockerManager
|
||||
from ci.util.running_process import RunningProcess
|
||||
|
||||
|
||||
def get_docker_env() -> Dict[str, str]:
|
||||
"""Get environment for Docker commands, handling Git Bash/MSYS2 path conversion."""
|
||||
env = os.environ.copy()
|
||||
# Set UTF-8 encoding environment variables for Windows
|
||||
env["PYTHONIOENCODING"] = "utf-8"
|
||||
env["PYTHONUTF8"] = "1"
|
||||
# Only set MSYS_NO_PATHCONV if we're in a Git Bash/MSYS2 environment
|
||||
if (
|
||||
"MSYSTEM" in os.environ
|
||||
or os.environ.get("TERM") == "xterm"
|
||||
or "bash.exe" in os.environ.get("SHELL", "")
|
||||
):
|
||||
env["MSYS_NO_PATHCONV"] = "1"
|
||||
return env
|
||||
|
||||
|
||||
def run_subprocess_safe(
|
||||
cmd: List[str], **kwargs: Any
|
||||
) -> subprocess.CompletedProcess[Any]:
|
||||
"""Run subprocess with safe UTF-8 handling and error replacement."""
|
||||
kwargs.setdefault("encoding", "utf-8")
|
||||
kwargs.setdefault("errors", "replace")
|
||||
kwargs.setdefault("env", get_docker_env())
|
||||
return subprocess.run(cmd, **kwargs) # type: ignore[misc]
|
||||
|
||||
|
||||
def run_docker_command_streaming(cmd: List[str]) -> int:
|
||||
"""Run docker command with streaming output using RunningProcess."""
|
||||
print(f"Executing: {' '.join(cmd)}")
|
||||
proc = RunningProcess(cmd, env=get_docker_env(), auto_run=True)
|
||||
|
||||
# Stream all output to stdout
|
||||
with proc.line_iter(timeout=None) as it:
|
||||
for line in it:
|
||||
print(line)
|
||||
|
||||
returncode = proc.wait()
|
||||
if returncode != 0:
|
||||
raise subprocess.CalledProcessError(returncode, cmd)
|
||||
return returncode
|
||||
|
||||
|
||||
def run_docker_command_no_fail(cmd: List[str]) -> int:
|
||||
"""Run docker command with streaming output, but don't raise exceptions on failure."""
|
||||
print(f"Executing: {' '.join(cmd)}")
|
||||
proc = RunningProcess(cmd, env=get_docker_env(), auto_run=True)
|
||||
|
||||
# Stream all output to stdout
|
||||
with proc.line_iter(timeout=None) as it:
|
||||
for line in it:
|
||||
print(line)
|
||||
|
||||
return proc.wait()
|
||||
|
||||
|
||||
class DockerQEMURunner:
|
||||
"""Runner for ESP32 QEMU emulation using Docker containers."""
|
||||
|
||||
DEFAULT_IMAGE = "mluis/qemu-esp32:latest"
|
||||
ALTERNATIVE_IMAGE = "espressif/idf:latest"
|
||||
FALLBACK_IMAGE = "espressif/idf:release-v5.2"
|
||||
|
||||
def __init__(self, docker_image: Optional[str] = None):
|
||||
"""Initialize Docker QEMU runner.
|
||||
|
||||
Args:
|
||||
docker_image: Docker image to use, defaults to espressif/qemu
|
||||
"""
|
||||
self.docker_manager = DockerManager()
|
||||
self.docker_image = docker_image or self.DEFAULT_IMAGE
|
||||
self.container_name = None
|
||||
# Use Linux-style paths for all containers since we're using Ubuntu/Alpine
|
||||
self.firmware_mount_path = "/workspace/firmware"
|
||||
|
||||
def check_docker_available(self) -> bool:
|
||||
"""Check if Docker is available and running."""
|
||||
try:
|
||||
proc = RunningProcess(
|
||||
["docker", "version"], env=get_docker_env(), auto_run=True
|
||||
)
|
||||
returncode = proc.wait(timeout=5)
|
||||
return returncode == 0
|
||||
except (Exception, FileNotFoundError):
|
||||
return False
|
||||
|
||||
def pull_image(self):
|
||||
"""Pull the Docker image if not already available."""
|
||||
print(f"Ensuring Docker image {self.docker_image} is available...")
|
||||
try:
|
||||
# Check if image exists locally
|
||||
proc = RunningProcess(
|
||||
["docker", "images", "-q", self.docker_image],
|
||||
env=get_docker_env(),
|
||||
auto_run=True,
|
||||
)
|
||||
stdout_lines: List[str] = []
|
||||
with proc.line_iter(timeout=None) as it:
|
||||
for line in it:
|
||||
stdout_lines.append(line)
|
||||
result_stdout = "\n".join(stdout_lines)
|
||||
if not result_stdout.strip():
|
||||
# Image doesn't exist, pull it directly using docker command
|
||||
print(f"Pulling Docker image: {self.docker_image}")
|
||||
run_docker_command_streaming(["docker", "pull", self.docker_image])
|
||||
print(f"Successfully pulled {self.docker_image}")
|
||||
else:
|
||||
print(f"Image {self.docker_image} already available locally")
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"Warning: Failed to pull image {self.docker_image}: {e}")
|
||||
if self.docker_image == self.DEFAULT_IMAGE:
|
||||
print(f"Trying alternative image: {self.ALTERNATIVE_IMAGE}")
|
||||
self.docker_image = self.ALTERNATIVE_IMAGE
|
||||
try:
|
||||
print(f"Pulling alternative Docker image: {self.docker_image}")
|
||||
run_docker_command_streaming(["docker", "pull", self.docker_image])
|
||||
print(f"Successfully pulled {self.docker_image}")
|
||||
except subprocess.CalledProcessError as e2:
|
||||
print(f"Failed to pull alternative image: {e2}")
|
||||
print(f"Trying fallback image: {self.FALLBACK_IMAGE}")
|
||||
self.docker_image = self.FALLBACK_IMAGE
|
||||
try:
|
||||
print(f"Pulling fallback Docker image: {self.docker_image}")
|
||||
run_docker_command_streaming(
|
||||
["docker", "pull", self.docker_image]
|
||||
)
|
||||
print(f"Successfully pulled {self.docker_image}")
|
||||
except subprocess.CalledProcessError as e3:
|
||||
print(f"Failed to pull fallback image: {e3}")
|
||||
raise e3
|
||||
else:
|
||||
raise e
|
||||
|
||||
def prepare_firmware(self, firmware_path: Path) -> Path:
|
||||
"""Prepare firmware files for mounting into Docker container.
|
||||
|
||||
Args:
|
||||
firmware_path: Path to firmware.bin or build directory
|
||||
|
||||
Returns:
|
||||
Path to the prepared firmware directory
|
||||
"""
|
||||
# Create temporary directory for firmware files
|
||||
temp_dir = Path(tempfile.mkdtemp(prefix="qemu_firmware_"))
|
||||
|
||||
try:
|
||||
if firmware_path.is_file() and firmware_path.suffix == ".bin":
|
||||
# Copy single firmware file
|
||||
shutil.copy2(firmware_path, temp_dir / "firmware.bin")
|
||||
# Create proper 4MB flash image for QEMU
|
||||
self._create_flash_image(firmware_path, temp_dir / "flash.bin")
|
||||
else:
|
||||
# Copy entire build directory
|
||||
if firmware_path.is_dir():
|
||||
# Copy relevant files from build directory
|
||||
patterns = ["*.bin", "*.elf", "partitions.csv", "flash_args"]
|
||||
for pattern in patterns:
|
||||
for file in firmware_path.glob(pattern):
|
||||
shutil.copy2(file, temp_dir)
|
||||
|
||||
# Also check for PlatformIO build structure
|
||||
pio_build = firmware_path / ".pio" / "build" / "esp32dev"
|
||||
if pio_build.exists():
|
||||
for pattern in patterns:
|
||||
for file in pio_build.glob(pattern):
|
||||
shutil.copy2(file, temp_dir)
|
||||
|
||||
# Create proper flash.bin file from firmware.bin
|
||||
firmware_bin = temp_dir / "firmware.bin"
|
||||
if firmware_bin.exists():
|
||||
# Create proper 4MB flash image for QEMU
|
||||
self._create_flash_image(firmware_bin, temp_dir / "flash.bin")
|
||||
else:
|
||||
# Create a minimal 4MB flash.bin for testing
|
||||
flash_bin = temp_dir / "flash.bin"
|
||||
flash_bin.write_bytes(
|
||||
b"\xff" * (4 * 1024 * 1024)
|
||||
) # 4MB of 0xFF
|
||||
else:
|
||||
raise ValueError(f"Invalid firmware path: {firmware_path}")
|
||||
|
||||
return temp_dir
|
||||
except Exception as e:
|
||||
# Clean up on error
|
||||
shutil.rmtree(temp_dir, ignore_errors=True)
|
||||
raise e
|
||||
|
||||
def _create_flash_image(
|
||||
self, firmware_path: Path, output_path: Path, flash_size_mb: int = 4
|
||||
):
|
||||
"""Create a proper flash image for QEMU ESP32.
|
||||
|
||||
Args:
|
||||
firmware_path: Path to the firmware.bin file
|
||||
output_path: Path where to write the flash.bin
|
||||
flash_size_mb: Flash size in MB (must be 2, 4, 8, or 16)
|
||||
"""
|
||||
if flash_size_mb not in [2, 4, 8, 16]:
|
||||
raise ValueError(
|
||||
f"Flash size must be 2, 4, 8, or 16 MB, got {flash_size_mb}"
|
||||
)
|
||||
|
||||
flash_size = flash_size_mb * 1024 * 1024
|
||||
|
||||
# Read firmware content
|
||||
firmware_data = firmware_path.read_bytes()
|
||||
|
||||
# Create flash image: firmware at beginning, rest filled with 0xFF
|
||||
flash_data = firmware_data + b"\xff" * (flash_size - len(firmware_data))
|
||||
|
||||
# Ensure we have exactly the right size
|
||||
if len(flash_data) > flash_size:
|
||||
raise ValueError(
|
||||
f"Firmware size ({len(firmware_data)} bytes) exceeds flash size ({flash_size} bytes)"
|
||||
)
|
||||
|
||||
flash_data = flash_data[:flash_size] # Truncate to exact size
|
||||
|
||||
output_path.write_bytes(flash_data)
|
||||
|
||||
def build_qemu_command(
|
||||
self,
|
||||
firmware_name: str = "firmware.bin",
|
||||
flash_size: int = 4,
|
||||
machine: str = "esp32",
|
||||
) -> List[str]:
|
||||
"""Build QEMU command to run inside Docker container.
|
||||
|
||||
Args:
|
||||
firmware_name: Name of firmware file in mounted directory
|
||||
flash_size: Flash size in MB
|
||||
machine: QEMU machine type (esp32, esp32c3, esp32s3, etc.)
|
||||
|
||||
Returns:
|
||||
List of command arguments for QEMU
|
||||
"""
|
||||
# For Linux containers, use a workaround to run QEMU through Docker Desktop's Windows integration
|
||||
firmware_path = f"{self.firmware_mount_path}/flash.bin"
|
||||
|
||||
# Determine QEMU system and machine based on target
|
||||
if machine == "esp32c3":
|
||||
qemu_system = "/usr/local/bin/qemu-system-riscv32"
|
||||
qemu_machine = "esp32c3"
|
||||
echo_target = "ESP32C3"
|
||||
elif machine == "esp32s3":
|
||||
qemu_system = "/usr/local/bin/qemu-system-xtensa"
|
||||
qemu_machine = "esp32s3"
|
||||
echo_target = "ESP32S3"
|
||||
else:
|
||||
# Default to ESP32 (Xtensa)
|
||||
qemu_system = "/usr/local/bin/qemu-system-xtensa"
|
||||
qemu_machine = "esp32"
|
||||
echo_target = "ESP32"
|
||||
|
||||
# Use container's QEMU with proper configuration
|
||||
wrapper_script = f'''#!/bin/bash
|
||||
set -e
|
||||
echo "Starting {echo_target} QEMU emulation..."
|
||||
echo "Firmware: {firmware_path}"
|
||||
echo "Machine: {qemu_machine}"
|
||||
echo "QEMU system: {qemu_system}"
|
||||
echo "Container: $(cat /etc/os-release | head -1)"
|
||||
|
||||
# Check if firmware file exists
|
||||
if [ ! -f "{firmware_path}" ]; then
|
||||
echo "ERROR: Firmware file not found: {firmware_path}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check firmware size
|
||||
FIRMWARE_SIZE=$(stat -c%s "{firmware_path}")
|
||||
echo "Firmware size: $FIRMWARE_SIZE bytes"
|
||||
|
||||
# Copy firmware to writable location since QEMU needs write access
|
||||
cp "{firmware_path}" /tmp/flash.bin
|
||||
echo "Copied firmware to writable location: /tmp/flash.bin"
|
||||
|
||||
# Try different QEMU configurations depending on machine type
|
||||
if [ "{qemu_machine}" = "esp32c3" ]; then
|
||||
# ESP32C3 uses RISC-V architecture
|
||||
echo "Running {qemu_system} for {qemu_machine}"
|
||||
{qemu_system} \\
|
||||
-nographic \\
|
||||
-machine {qemu_machine} \\
|
||||
-drive file="/tmp/flash.bin",if=mtd,format=raw \\
|
||||
-monitor none \\
|
||||
-serial mon:stdio
|
||||
else
|
||||
# ESP32 uses Xtensa architecture
|
||||
echo "Running {qemu_system} for {qemu_machine}"
|
||||
{qemu_system} \\
|
||||
-nographic \\
|
||||
-machine {qemu_machine} \\
|
||||
-drive file="/tmp/flash.bin",if=mtd,format=raw \\
|
||||
-global driver=timer.esp32.timg,property=wdt_disable,value=true \\
|
||||
-monitor none \\
|
||||
-serial mon:stdio
|
||||
fi
|
||||
|
||||
echo "QEMU execution completed"
|
||||
exit 0
|
||||
'''
|
||||
|
||||
cmd = ["bash", "-c", wrapper_script]
|
||||
|
||||
return cmd
|
||||
|
||||
def run(
|
||||
self,
|
||||
firmware_path: Path,
|
||||
timeout: int = 30,
|
||||
flash_size: int = 4,
|
||||
interrupt_regex: Optional[str] = None,
|
||||
interactive: bool = False,
|
||||
output_file: Optional[str] = None,
|
||||
machine: str = "esp32",
|
||||
) -> int:
|
||||
"""Run ESP32/ESP32C3/ESP32S3 firmware in QEMU using Docker.
|
||||
|
||||
Args:
|
||||
firmware_path: Path to firmware.bin or build directory
|
||||
timeout: Timeout in seconds (timeout is treated as success)
|
||||
flash_size: Flash size in MB
|
||||
interrupt_regex: Regex pattern to detect in output (informational only)
|
||||
interactive: Run in interactive mode (attach to container)
|
||||
output_file: Optional file path to write QEMU output to
|
||||
machine: QEMU machine type (esp32, esp32c3, esp32s3, etc.)
|
||||
|
||||
Returns:
|
||||
Exit code: actual QEMU/container exit code, except timeout returns 0
|
||||
"""
|
||||
if not self.check_docker_available():
|
||||
print("ERROR: Docker is not available or not running", file=sys.stderr)
|
||||
print("Please install Docker and ensure it's running", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
# Pull image if needed
|
||||
self.pull_image()
|
||||
|
||||
# Prepare firmware files
|
||||
print(f"Preparing firmware from: {firmware_path}")
|
||||
temp_firmware_dir = None
|
||||
|
||||
try:
|
||||
temp_firmware_dir = self.prepare_firmware(firmware_path)
|
||||
|
||||
# Generate unique container name
|
||||
self.container_name = f"qemu_esp32_{int(time.time())}"
|
||||
|
||||
# Convert Windows paths to Docker-compatible paths
|
||||
def windows_to_docker_path(path_str: Union[str, Path]) -> str:
|
||||
"""Convert Windows path to Docker volume mount format."""
|
||||
import os
|
||||
|
||||
# Check if we're in Git Bash/MSYS2 environment
|
||||
is_git_bash = (
|
||||
"MSYSTEM" in os.environ
|
||||
or os.environ.get("TERM") == "xterm"
|
||||
or "bash.exe" in os.environ.get("SHELL", "")
|
||||
)
|
||||
|
||||
if os.name == "nt" and is_git_bash:
|
||||
# Convert C:\path\to\dir to /c/path/to/dir for Git Bash
|
||||
path_str = str(path_str).replace("\\", "/")
|
||||
if len(path_str) > 2 and path_str[1:3] == ":/": # Drive letter
|
||||
path_str = "/" + path_str[0].lower() + path_str[2:]
|
||||
else:
|
||||
# For cmd.exe on Windows, keep native Windows paths
|
||||
path_str = str(path_str)
|
||||
|
||||
return path_str
|
||||
|
||||
docker_firmware_path = windows_to_docker_path(temp_firmware_dir)
|
||||
|
||||
# Only mount the firmware directory - QEMU is built into the container
|
||||
volumes = {
|
||||
docker_firmware_path: {
|
||||
"bind": self.firmware_mount_path,
|
||||
"mode": "ro", # Read-only mount
|
||||
},
|
||||
}
|
||||
|
||||
# Build QEMU command
|
||||
qemu_cmd = self.build_qemu_command(
|
||||
firmware_name="firmware.bin", flash_size=flash_size, machine=machine
|
||||
)
|
||||
|
||||
print(f"Running QEMU in Docker container: {self.container_name}")
|
||||
print(f"QEMU command: {' '.join(qemu_cmd)}")
|
||||
|
||||
if interactive:
|
||||
# Run interactively with streaming
|
||||
return self.docker_manager.run_container_streaming(
|
||||
image_name=self.docker_image,
|
||||
command=qemu_cmd,
|
||||
volumes=volumes,
|
||||
name=self.container_name,
|
||||
timeout=timeout,
|
||||
interrupt_pattern=interrupt_regex,
|
||||
output_file=output_file,
|
||||
)
|
||||
else:
|
||||
# Run with streaming output and return actual exit code
|
||||
return self.docker_manager.run_container_streaming(
|
||||
image_name=self.docker_image,
|
||||
command=qemu_cmd,
|
||||
volumes=volumes,
|
||||
name=self.container_name,
|
||||
timeout=timeout,
|
||||
interrupt_pattern=interrupt_regex,
|
||||
output_file=output_file,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
print(f"ERROR: {e}", file=sys.stderr)
|
||||
return 1
|
||||
finally:
|
||||
# Cleanup
|
||||
if temp_firmware_dir and temp_firmware_dir.exists():
|
||||
shutil.rmtree(temp_firmware_dir, ignore_errors=True)
|
||||
|
||||
# Ensure container is stopped and removed
|
||||
if self.container_name:
|
||||
try:
|
||||
# Stop and remove container
|
||||
run_docker_command_no_fail(["docker", "stop", self.container_name])
|
||||
run_docker_command_no_fail(["docker", "rm", self.container_name])
|
||||
except:
|
||||
pass
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run ESP32 firmware in QEMU using Docker"
|
||||
)
|
||||
parser.add_argument(
|
||||
"firmware_path", type=Path, help="Path to firmware.bin file or build directory"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--docker-image",
|
||||
type=str,
|
||||
help=f"Docker image to use (default: {DockerQEMURunner.DEFAULT_IMAGE})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--flash-size", type=int, default=4, help="Flash size in MB (default: 4)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--timeout", type=int, default=30, help="Timeout in seconds (default: 30)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--interrupt-regex", type=str, help="Regex pattern to interrupt execution"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--interactive", action="store_true", help="Run in interactive mode"
|
||||
)
|
||||
parser.add_argument("--output-file", type=str, help="File to write QEMU output to")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.firmware_path.exists():
|
||||
print(
|
||||
f"ERROR: Firmware path does not exist: {args.firmware_path}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
runner = DockerQEMURunner(docker_image=args.docker_image)
|
||||
return runner.run(
|
||||
firmware_path=args.firmware_path,
|
||||
timeout=args.timeout,
|
||||
flash_size=args.flash_size,
|
||||
interrupt_regex=args.interrupt_regex,
|
||||
interactive=args.interactive,
|
||||
output_file=args.output_file,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,283 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Integration module for Docker-based QEMU ESP32 testing.
|
||||
|
||||
This module provides a bridge between the existing test infrastructure
|
||||
and the Docker-based QEMU runner.
|
||||
"""
|
||||
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional, Union
|
||||
|
||||
from ci.dockerfiles.qemu_esp32_docker import DockerQEMURunner
|
||||
|
||||
|
||||
class QEMUTestIntegration:
|
||||
"""Integration class for QEMU testing with Docker fallback."""
|
||||
|
||||
def __init__(self, prefer_docker: bool = False):
|
||||
"""Initialize QEMU test integration.
|
||||
|
||||
Args:
|
||||
prefer_docker: If True, prefer Docker even if native QEMU is available
|
||||
"""
|
||||
self.prefer_docker = prefer_docker
|
||||
self.docker_available = self._check_docker()
|
||||
self.native_qemu_available = self._check_native_qemu()
|
||||
|
||||
def _check_docker(self) -> bool:
|
||||
"""Check if Docker is available."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["docker", "version"], capture_output=True, timeout=5
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (subprocess.SubprocessError, FileNotFoundError):
|
||||
return False
|
||||
|
||||
def _check_native_qemu(self) -> bool:
|
||||
"""Check if native QEMU ESP32 is available."""
|
||||
try:
|
||||
# Import the native QEMU module to check
|
||||
from ci.qemu_esp32 import find_qemu_binary # type: ignore[import-untyped]
|
||||
|
||||
return find_qemu_binary() is not None
|
||||
except ImportError:
|
||||
return False
|
||||
|
||||
def select_runner(self) -> str:
|
||||
"""Select the best available runner.
|
||||
|
||||
Returns:
|
||||
'docker' or 'native' based on availability and preference
|
||||
"""
|
||||
if self.prefer_docker and self.docker_available:
|
||||
return "docker"
|
||||
elif self.native_qemu_available:
|
||||
return "native"
|
||||
elif self.docker_available:
|
||||
return "docker"
|
||||
else:
|
||||
return "none"
|
||||
|
||||
def run_qemu_test(
|
||||
self,
|
||||
firmware_path: Union[str, Path],
|
||||
timeout: int = 30,
|
||||
interrupt_regex: Optional[str] = None,
|
||||
flash_size: int = 4,
|
||||
force_runner: Optional[str] = None,
|
||||
) -> int:
|
||||
"""Run QEMU test with automatic runner selection.
|
||||
|
||||
Args:
|
||||
firmware_path: Path to firmware or build directory
|
||||
timeout: Test timeout in seconds
|
||||
interrupt_regex: Pattern to interrupt on success
|
||||
flash_size: Flash size in MB
|
||||
force_runner: Force specific runner ('docker' or 'native')
|
||||
|
||||
Returns:
|
||||
Exit code (0 for success)
|
||||
"""
|
||||
firmware_path = Path(firmware_path)
|
||||
|
||||
# Determine which runner to use
|
||||
if force_runner:
|
||||
runner_type = force_runner
|
||||
else:
|
||||
runner_type = self.select_runner()
|
||||
|
||||
print(f"Selected QEMU runner: {runner_type}")
|
||||
|
||||
if runner_type == "docker":
|
||||
return self._run_docker_qemu(
|
||||
firmware_path, timeout, interrupt_regex, flash_size
|
||||
)
|
||||
elif runner_type == "native":
|
||||
return self._run_native_qemu(
|
||||
firmware_path, timeout, interrupt_regex, flash_size
|
||||
)
|
||||
else:
|
||||
print("ERROR: No QEMU runner available!", file=sys.stderr)
|
||||
print("Install Docker or native QEMU ESP32", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
def _run_docker_qemu(
|
||||
self,
|
||||
firmware_path: Path,
|
||||
timeout: int,
|
||||
interrupt_regex: Optional[str],
|
||||
flash_size: int,
|
||||
) -> int:
|
||||
"""Run QEMU test using Docker."""
|
||||
print("Running QEMU test in Docker container...")
|
||||
|
||||
runner = DockerQEMURunner()
|
||||
return runner.run(
|
||||
firmware_path=firmware_path,
|
||||
timeout=timeout,
|
||||
interrupt_regex=interrupt_regex,
|
||||
flash_size=flash_size,
|
||||
)
|
||||
|
||||
def _run_native_qemu(
|
||||
self,
|
||||
firmware_path: Path,
|
||||
timeout: int,
|
||||
interrupt_regex: Optional[str],
|
||||
flash_size: int,
|
||||
) -> int:
|
||||
"""Run QEMU test using native installation."""
|
||||
print("Running QEMU test with native installation...")
|
||||
|
||||
# Import and use the native runner
|
||||
from ci.qemu_esp32 import QEMURunner # type: ignore[import-untyped]
|
||||
|
||||
runner = QEMURunner() # type: ignore[no-untyped-call]
|
||||
return runner.run( # type: ignore[no-untyped-call]
|
||||
firmware_path=firmware_path,
|
||||
timeout=timeout,
|
||||
interrupt_regex=interrupt_regex,
|
||||
flash_size=flash_size,
|
||||
)
|
||||
|
||||
def install_qemu(self, use_docker: bool = False) -> bool:
|
||||
"""Install QEMU (native or pull Docker image).
|
||||
|
||||
Args:
|
||||
use_docker: If True, pull Docker image instead of native install
|
||||
|
||||
Returns:
|
||||
True if installation successful
|
||||
"""
|
||||
if use_docker:
|
||||
print("Pulling Docker QEMU image...")
|
||||
try:
|
||||
runner = DockerQEMURunner()
|
||||
runner.pull_image()
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Failed to pull Docker image: {e}", file=sys.stderr)
|
||||
return False
|
||||
else:
|
||||
print("Installing native QEMU...")
|
||||
try:
|
||||
# Run the native install script
|
||||
result = subprocess.run(
|
||||
[sys.executable, "ci/install-qemu.py"],
|
||||
cwd=Path(__file__).parent.parent.parent,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except Exception as e:
|
||||
print(f"Failed to install native QEMU: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def integrate_with_test_framework():
|
||||
"""Integrate Docker QEMU with the existing test framework.
|
||||
|
||||
This function modifies the test.py to add Docker support.
|
||||
"""
|
||||
test_file = Path(__file__).parent.parent.parent / "test.py"
|
||||
|
||||
if not test_file.exists():
|
||||
print(f"ERROR: test.py not found at {test_file}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
print(f"Integrating Docker QEMU support into {test_file}")
|
||||
|
||||
# Read the test.py file
|
||||
content = test_file.read_text()
|
||||
|
||||
# Check if already integrated
|
||||
if "docker.qemu_test_integration" in content:
|
||||
print("Docker QEMU support already integrated")
|
||||
return True
|
||||
|
||||
# Find the QEMU test section and add Docker support
|
||||
integration_code = """
|
||||
# Docker QEMU integration
|
||||
try:
|
||||
from ci.dockerfiles.qemu_test_integration import QEMUTestIntegration
|
||||
DOCKER_QEMU_AVAILABLE = True
|
||||
except ImportError:
|
||||
DOCKER_QEMU_AVAILABLE = False
|
||||
"""
|
||||
|
||||
# Add the import at the top of the file after other imports
|
||||
import_marker = "from ci.qemu_esp32 import"
|
||||
if import_marker in content:
|
||||
# Add after the existing QEMU import
|
||||
content = content.replace(
|
||||
import_marker, integration_code + "\n" + import_marker
|
||||
)
|
||||
|
||||
# Save the modified file
|
||||
test_file.write_text(content)
|
||||
print("Successfully integrated Docker QEMU support")
|
||||
return True
|
||||
else:
|
||||
print("WARNING: Could not find appropriate location to add integration")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function for testing the integration."""
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="QEMU Test Integration with Docker support"
|
||||
)
|
||||
parser.add_argument(
|
||||
"command",
|
||||
choices=["test", "install", "integrate", "check"],
|
||||
help="Command to execute",
|
||||
)
|
||||
parser.add_argument("--firmware", type=Path, help="Firmware path for test command")
|
||||
parser.add_argument(
|
||||
"--docker", action="store_true", help="Prefer Docker over native QEMU"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--timeout", type=int, default=30, help="Test timeout in seconds"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
integration = QEMUTestIntegration(prefer_docker=args.docker)
|
||||
|
||||
if args.command == "check":
|
||||
print(f"Docker available: {integration.docker_available}")
|
||||
print(f"Native QEMU available: {integration.native_qemu_available}")
|
||||
print(f"Selected runner: {integration.select_runner()}")
|
||||
return 0
|
||||
|
||||
elif args.command == "install":
|
||||
success = integration.install_qemu(use_docker=args.docker)
|
||||
return 0 if success else 1
|
||||
|
||||
elif args.command == "integrate":
|
||||
success = integrate_with_test_framework()
|
||||
return 0 if success else 1
|
||||
|
||||
elif args.command == "test":
|
||||
if not args.firmware:
|
||||
print("ERROR: --firmware required for test command", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
return integration.run_qemu_test(
|
||||
firmware_path=args.firmware,
|
||||
timeout=args.timeout,
|
||||
force_runner="docker" if args.docker else None,
|
||||
)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,186 @@
|
||||
"""
|
||||
Work in progress to generate doxygen via a script instead of a GitHub action.
|
||||
"""
|
||||
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from download import download # type: ignore
|
||||
|
||||
|
||||
# Configs
|
||||
DOXYGEN_VERSION = (
|
||||
"1.13.2" # DOXYGEN_AWESOME styler is has certain restrictions with doxygen version
|
||||
)
|
||||
DOXYGEN_AWESOME_VERSION = "2.3.4" # deprecating
|
||||
DOXYFILE_PATH = Path("docs/Doxyfile")
|
||||
HTML_OUTPUT_DIR = Path("docs/html")
|
||||
DOXYGEN_CSS_REPO = "https://github.com/jothepro/doxygen-awesome-css" # deprecating
|
||||
|
||||
|
||||
HERE = Path(__file__).parent.resolve()
|
||||
PROJECT_ROOT = HERE.parent
|
||||
|
||||
DOCS_ROOT = PROJECT_ROOT / "docs"
|
||||
DOCS_TOOL_PATH = PROJECT_ROOT / ".tools_cache"
|
||||
DOCS_OUTPUT_PATH = DOCS_ROOT / "html"
|
||||
|
||||
|
||||
def run(
|
||||
cmd: str,
|
||||
cwd: Optional[str] = None,
|
||||
shell: bool = True,
|
||||
check: bool = True,
|
||||
capture: bool = True,
|
||||
) -> str:
|
||||
print(f"Running: {cmd}")
|
||||
result = subprocess.run(
|
||||
cmd, shell=shell, cwd=cwd, check=False, capture_output=capture, text=False
|
||||
)
|
||||
if capture:
|
||||
stdout = result.stdout.decode("utf-8") if result.stdout else ""
|
||||
stderr = result.stderr.decode("utf-8") if result.stderr else ""
|
||||
else:
|
||||
stdout = ""
|
||||
stderr = ""
|
||||
if result.returncode != 0:
|
||||
msg = f"Command failed with exit code {result.returncode}:\nstdout:\n{stdout}\n\nstderr:\n{stderr}"
|
||||
warnings.warn(msg)
|
||||
if check:
|
||||
raise subprocess.CalledProcessError(
|
||||
result.returncode, cmd, output=result.stdout
|
||||
)
|
||||
return stdout.strip()
|
||||
|
||||
|
||||
def get_git_info() -> Tuple[str, str]:
|
||||
release_tag = os.environ.get("RELEASE_TAG", "")
|
||||
|
||||
try:
|
||||
latest_tag = run("git tag | grep -E '^[0-9]' | sort -V | tail -1")
|
||||
latest_tag = latest_tag if latest_tag else ""
|
||||
except subprocess.CalledProcessError:
|
||||
latest_tag = ""
|
||||
|
||||
git_sha_short = run("git rev-parse --short HEAD")
|
||||
full_sha = run("git rev-parse HEAD")
|
||||
project_number = release_tag or latest_tag or git_sha_short
|
||||
commit_message = (
|
||||
f"{project_number} ({full_sha})"
|
||||
if project_number != git_sha_short
|
||||
else project_number
|
||||
)
|
||||
|
||||
print(f"Project number: {project_number}")
|
||||
print(f"Commit message: {commit_message}")
|
||||
return project_number, commit_message
|
||||
|
||||
|
||||
def install_doxygen_windows() -> Path:
|
||||
print("Installing Doxygen...")
|
||||
doxygen_url = (
|
||||
f"https://www.doxygen.nl/files/doxygen-{DOXYGEN_VERSION}.windows.x64.bin.zip"
|
||||
)
|
||||
zip_path = DOCS_TOOL_PATH / f"doxygen-{DOXYGEN_VERSION}.zip"
|
||||
extract_dir = DOCS_TOOL_PATH / f"doxygen-{DOXYGEN_VERSION}"
|
||||
|
||||
# Create tool path if it doesn't exist
|
||||
DOCS_TOOL_PATH.mkdir(exist_ok=True, parents=True)
|
||||
|
||||
download(doxygen_url, zip_path)
|
||||
shutil.unpack_archive(str(zip_path), extract_dir)
|
||||
bin_path = next(extract_dir.glob("**/doxygen.exe"), None)
|
||||
if not bin_path:
|
||||
raise FileNotFoundError("Doxygen executable not found after extraction.")
|
||||
print(f"Doxygen installed at: {bin_path}")
|
||||
return bin_path
|
||||
|
||||
|
||||
def install_doxygen_unix() -> Path:
|
||||
print("Installing Doxygen...")
|
||||
archive = f"doxygen-{DOXYGEN_VERSION}.linux.bin.tar.gz"
|
||||
url = f"https://www.doxygen.nl/files/{archive}"
|
||||
|
||||
# Create tool path if it doesn't exist
|
||||
DOCS_TOOL_PATH.mkdir(exist_ok=True, parents=True)
|
||||
|
||||
# Change to tool directory for download and extraction
|
||||
original_dir = os.getcwd()
|
||||
os.chdir(str(DOCS_TOOL_PATH))
|
||||
|
||||
try:
|
||||
run(f"wget -q {url}")
|
||||
run(f"tar -xf {archive}")
|
||||
bin_dir = DOCS_TOOL_PATH / f"doxygen-{DOXYGEN_VERSION}"
|
||||
return bin_dir / "bin" / "doxygen"
|
||||
finally:
|
||||
os.chdir(original_dir)
|
||||
|
||||
|
||||
def install_theme() -> Path:
|
||||
print("Installing Doxygen Awesome Theme...")
|
||||
theme_path = DOCS_ROOT / "doxygen-awesome-css"
|
||||
if theme_path.exists():
|
||||
return theme_path
|
||||
run(
|
||||
f"git clone --depth 1 -b v{DOXYGEN_AWESOME_VERSION} {DOXYGEN_CSS_REPO}",
|
||||
cwd=str(DOCS_ROOT),
|
||||
)
|
||||
return theme_path
|
||||
|
||||
|
||||
def generate_docs(doxygen_bin: Path) -> None:
|
||||
print("Generating documentation...")
|
||||
cmd_str = f'"{doxygen_bin}" {DOXYFILE_PATH.name}'
|
||||
run(cmd_str, cwd=str(DOCS_ROOT), capture=False)
|
||||
|
||||
|
||||
# def install_graphviz() -> None:
|
||||
# url: str = get_latest_release_for_platform()
|
||||
# print(url)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
is_windows = platform.system() == "Windows"
|
||||
# is_macos = platform.system() == "Darwin"
|
||||
_, commit_msg = get_git_info()
|
||||
|
||||
if is_windows:
|
||||
doxygen_bin = install_doxygen_windows()
|
||||
# add to path C:\Program Files\Graphviz\bin\
|
||||
os.environ["PATH"] += os.pathsep + r"C:\Program Files\Graphviz\bin"
|
||||
else:
|
||||
doxygen_bin = install_doxygen_unix()
|
||||
|
||||
# install_theme()
|
||||
|
||||
# install_graphviz() # Work in progress
|
||||
|
||||
# Verify Graphviz installation
|
||||
try:
|
||||
dot_version = run("dot -V", check=False)
|
||||
print(f"Graphviz detected: {dot_version}")
|
||||
except Exception:
|
||||
warnings.warn(
|
||||
"Graphviz (dot) not found in PATH. Diagrams may not be generated."
|
||||
)
|
||||
|
||||
# Check it graphviz is installed
|
||||
# if linux
|
||||
if not is_windows:
|
||||
run("dot -Tsvg -Kneato -Grankdir=LR", check=True)
|
||||
|
||||
generate_docs(doxygen_bin=doxygen_bin)
|
||||
|
||||
print(f"\n✅ Docs generated in: {HTML_OUTPUT_DIR}")
|
||||
print(f"📄 Commit message: {commit_msg}")
|
||||
print("✨ You can now manually deploy to GitHub Pages or automate this step.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ESP32 Symbol Analysis Runner for GitHub Actions
|
||||
Wrapper script to run ESP32 symbol analysis with proper error handling
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from ci.util.esp32_symbol_analysis import main as esp32_symbol_analysis
|
||||
|
||||
|
||||
def is_esp32_board(board_name: str) -> bool:
|
||||
"""Check if the board is ESP32-based"""
|
||||
esp32_boards = [
|
||||
"esp32dev",
|
||||
"esp32",
|
||||
"esp32s2",
|
||||
"esp32s3",
|
||||
"esp32c3",
|
||||
"esp32c6",
|
||||
"esp32h2",
|
||||
"esp32p4",
|
||||
"esp32c2",
|
||||
]
|
||||
return any(esp32_board in board_name.lower() for esp32_board in esp32_boards)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run ESP32 symbol analysis for GitHub Actions"
|
||||
)
|
||||
parser.add_argument("--board", required=True, help="Board name being analyzed")
|
||||
parser.add_argument("--example", default="Blink", help="Example that was compiled")
|
||||
parser.add_argument(
|
||||
"--skip-on-failure",
|
||||
action="store_true",
|
||||
help="Don't fail the build if symbol analysis fails",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
print(
|
||||
f"ESP32 Symbol Analysis Runner - Board: {args.board}, Example: {args.example}"
|
||||
)
|
||||
|
||||
# Only run for ESP32 boards
|
||||
if not is_esp32_board(args.board):
|
||||
print(f"Skipping symbol analysis - {args.board} is not an ESP32-based board")
|
||||
return 0
|
||||
|
||||
# Check if build_info.json exists
|
||||
build_info_path = Path(".build") / args.board / "build_info.json"
|
||||
if not build_info_path.exists():
|
||||
message = (
|
||||
f"Build info not found at {build_info_path}. Skipping symbol analysis."
|
||||
)
|
||||
if args.skip_on_failure:
|
||||
print(f"Warning: {message}")
|
||||
return 0
|
||||
else:
|
||||
print(f"Error: {message}")
|
||||
return 1
|
||||
|
||||
try:
|
||||
# TODO - Remove this.
|
||||
# Change to the ci/util directory since the script expects to be run from there
|
||||
original_cwd = os.getcwd()
|
||||
ci_ci_dir = Path(__file__).parent / "ci"
|
||||
os.chdir(ci_ci_dir)
|
||||
|
||||
# Import and run the ESP32 symbol analysis
|
||||
try:
|
||||
print("Running ESP32 symbol analysis...")
|
||||
esp32_symbol_analysis()
|
||||
print("ESP32 symbol analysis completed successfully!")
|
||||
return 0
|
||||
except ImportError as e:
|
||||
print(f"Failed to import esp32_symbol_analysis module: {e}")
|
||||
if args.skip_on_failure:
|
||||
return 0
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(f"ESP32 symbol analysis failed: {e}")
|
||||
if args.skip_on_failure:
|
||||
return 0
|
||||
return 1
|
||||
finally:
|
||||
os.chdir(original_cwd)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Unexpected error in ESP32 symbol analysis runner: {e}")
|
||||
if args.skip_on_failure:
|
||||
return 0
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,232 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Demonstration of the Fingerprint Cache Feature
|
||||
|
||||
This script shows how to use the fingerprint cache for efficient file change detection
|
||||
in build systems and development workflows.
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ci.ci.fingerprint_cache import FingerprintCache, has_changed
|
||||
|
||||
|
||||
def demo_basic_usage() -> None:
|
||||
"""Demonstrate basic fingerprint cache usage."""
|
||||
print("=== Basic Fingerprint Cache Demo ===")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
cache_file = temp_path / "demo_cache.json"
|
||||
cache = FingerprintCache(cache_file)
|
||||
|
||||
# Create a source file
|
||||
source_file = temp_path / "example.cpp"
|
||||
with open(source_file, "w") as f:
|
||||
f.write("#include <iostream>\nint main() { return 0; }")
|
||||
|
||||
baseline_time = time.time() - 3600 # 1 hour ago
|
||||
current_modtime = os.path.getmtime(source_file)
|
||||
|
||||
print(f"Source file: {source_file}")
|
||||
print(f"Baseline time: {baseline_time}")
|
||||
print(f"Current modtime: {current_modtime}")
|
||||
|
||||
# First check - file is newer than baseline
|
||||
changed = cache.has_changed(source_file, baseline_time)
|
||||
print(f"First check (vs baseline): {changed} (expected: True)")
|
||||
|
||||
# Second check - same modtime
|
||||
changed = cache.has_changed(source_file, current_modtime)
|
||||
print(f"Second check (same modtime): {changed} (expected: False)")
|
||||
|
||||
# Touch file (change modtime but not content)
|
||||
time.sleep(0.01)
|
||||
source_file.touch()
|
||||
|
||||
# Third check - modtime changed but content same
|
||||
changed = cache.has_changed(source_file, current_modtime)
|
||||
print(
|
||||
f"Third check (touched file): {changed} (expected: False - content unchanged)"
|
||||
)
|
||||
|
||||
# Actually modify content
|
||||
time.sleep(0.01)
|
||||
with open(source_file, "w") as f:
|
||||
f.write(
|
||||
'#include <iostream>\nint main() { std::cout << "Hello"; return 0; }'
|
||||
)
|
||||
|
||||
# Fourth check - content actually changed
|
||||
changed = cache.has_changed(source_file, current_modtime)
|
||||
print(f"Fourth check (content changed): {changed} (expected: True)")
|
||||
|
||||
|
||||
def demo_build_system_workflow() -> None:
|
||||
"""Demonstrate build system integration workflow."""
|
||||
print("\n=== Build System Workflow Demo ===")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
cache_file = temp_path / "build_cache.json"
|
||||
cache = FingerprintCache(cache_file)
|
||||
|
||||
# Simulate multiple source files
|
||||
source_files = [
|
||||
temp_path / "main.cpp",
|
||||
temp_path / "utils.cpp",
|
||||
temp_path / "config.h",
|
||||
]
|
||||
|
||||
# Create source files
|
||||
contents = [
|
||||
'#include "config.h"\n#include "utils.h"\nint main() { return 0; }',
|
||||
'#include "utils.h"\nvoid utility_function() {}',
|
||||
'#pragma once\n#define VERSION "1.0"',
|
||||
]
|
||||
|
||||
for src_file, content in zip(source_files, contents):
|
||||
with open(src_file, "w") as f:
|
||||
f.write(content)
|
||||
|
||||
print(f"Created {len(source_files)} source files")
|
||||
|
||||
# Simulate first build
|
||||
print("\n--- First Build (all files new) ---")
|
||||
last_build_time = time.time() - 3600 # 1 hour ago
|
||||
changed_files: List[str] = []
|
||||
|
||||
for src_file in source_files:
|
||||
if cache.has_changed(src_file, last_build_time):
|
||||
changed_files.append(src_file.name)
|
||||
|
||||
print(
|
||||
f"Files to rebuild: {changed_files} ({len(changed_files)}/{len(source_files)})"
|
||||
)
|
||||
|
||||
# Update last build time
|
||||
current_modtimes = [os.path.getmtime(f) for f in source_files]
|
||||
|
||||
# Simulate second build (no changes)
|
||||
print("\n--- Second Build (no changes) ---")
|
||||
changed_files: List[str] = []
|
||||
|
||||
for src_file, modtime in zip(source_files, current_modtimes):
|
||||
if cache.has_changed(src_file, modtime):
|
||||
changed_files.append(src_file.name)
|
||||
|
||||
print(
|
||||
f"Files to rebuild: {changed_files} ({len(changed_files)}/{len(source_files)})"
|
||||
)
|
||||
|
||||
# Modify one file
|
||||
print("\n--- Third Build (one file modified) ---")
|
||||
time.sleep(0.01)
|
||||
with open(source_files[0], "a") as f:
|
||||
f.write("\n// Added comment")
|
||||
|
||||
changed_files: List[str] = []
|
||||
for src_file, modtime in zip(source_files, current_modtimes):
|
||||
if cache.has_changed(src_file, modtime):
|
||||
changed_files.append(src_file.name)
|
||||
|
||||
print(
|
||||
f"Files to rebuild: {changed_files} ({len(changed_files)}/{len(source_files)})"
|
||||
)
|
||||
|
||||
|
||||
def demo_performance() -> None:
|
||||
"""Demonstrate cache performance characteristics."""
|
||||
print("\n=== Performance Demo ===")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
cache_file = temp_path / "perf_cache.json"
|
||||
cache = FingerprintCache(cache_file)
|
||||
|
||||
# Create multiple test files
|
||||
num_files = 20
|
||||
test_files: List[Path] = []
|
||||
|
||||
for i in range(num_files):
|
||||
test_file = temp_path / f"file_{i:03d}.cpp"
|
||||
with open(test_file, "w") as f:
|
||||
f.write(
|
||||
f"// File {i}\n#include <iostream>\nint func_{i}() {{ return {i}; }}"
|
||||
)
|
||||
test_files.append(test_file)
|
||||
|
||||
print(f"Created {num_files} test files")
|
||||
|
||||
# Measure cache miss performance (first check)
|
||||
start_time = time.time()
|
||||
baseline_time = time.time() - 3600
|
||||
|
||||
for test_file in test_files:
|
||||
cache.has_changed(test_file, baseline_time)
|
||||
|
||||
miss_time = time.time() - start_time
|
||||
print(
|
||||
f"Cache miss time: {miss_time * 1000:.1f}ms ({miss_time * 1000 / num_files:.2f}ms per file)"
|
||||
)
|
||||
|
||||
# Measure cache hit performance (second check with same modtime)
|
||||
modtimes = [os.path.getmtime(f) for f in test_files]
|
||||
|
||||
start_time = time.time()
|
||||
for test_file, modtime in zip(test_files, modtimes):
|
||||
cache.has_changed(test_file, modtime)
|
||||
|
||||
hit_time = time.time() - start_time
|
||||
print(
|
||||
f"Cache hit time: {hit_time * 1000:.1f}ms ({hit_time * 1000 / num_files:.2f}ms per file)"
|
||||
)
|
||||
print(f"Speedup: {miss_time / hit_time:.1f}x faster")
|
||||
|
||||
# Show cache statistics
|
||||
stats = cache.get_cache_stats()
|
||||
print(f"Cache entries: {stats['total_entries']}")
|
||||
print(f"Cache file size: {stats['cache_file_size_bytes']} bytes")
|
||||
|
||||
|
||||
def demo_convenience_function() -> None:
|
||||
"""Demonstrate the convenience has_changed function."""
|
||||
print("\n=== Convenience Function Demo ===")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
cache_file = temp_path / "convenience_cache.json"
|
||||
|
||||
# Create test file
|
||||
test_file = temp_path / "test.h"
|
||||
with open(test_file, "w") as f:
|
||||
f.write("#pragma once\n#define API_VERSION 1")
|
||||
|
||||
baseline_time = time.time() - 1800 # 30 minutes ago
|
||||
|
||||
# Use convenience function
|
||||
changed = has_changed(test_file, baseline_time, cache_file)
|
||||
print(f"Convenience function result: {changed}")
|
||||
print(f"Cache file created: {cache_file.exists()}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("Fingerprint Cache Feature Demonstration")
|
||||
print("=" * 50)
|
||||
|
||||
demo_basic_usage()
|
||||
demo_build_system_workflow()
|
||||
demo_performance()
|
||||
demo_convenience_function()
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print("Demo completed successfully!")
|
||||
print("\nKey Benefits:")
|
||||
print("- Fast modification time checks avoid expensive MD5 computation")
|
||||
print("- Content-based verification prevents false positives")
|
||||
print("- Persistent cache survives across tool invocations")
|
||||
print("- Significant speedup for incremental builds and testing")
|
||||
@@ -0,0 +1,205 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
from typing import Any, Dict
|
||||
|
||||
from ci.util.bin_2_elf import bin_to_elf
|
||||
from ci.util.elf import dump_symbol_sizes
|
||||
from ci.util.map_dump import map_dump
|
||||
|
||||
|
||||
def cpp_filt(cpp_filt_path: Path, input_text: str) -> str:
|
||||
"""
|
||||
Demangle C++ symbols using c++filt.
|
||||
|
||||
Args:
|
||||
cpp_filt_path (Path): Path to c++filt executable.
|
||||
input_text (str): Text to demangle.
|
||||
|
||||
Returns:
|
||||
str: Demangled text.
|
||||
"""
|
||||
if not cpp_filt_path.exists():
|
||||
raise FileNotFoundError(f"cppfilt not found at '{cpp_filt_path}'")
|
||||
command = [str(cpp_filt_path), "-t", "-n"]
|
||||
process = subprocess.Popen(
|
||||
command,
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
)
|
||||
stdout, stderr = process.communicate(input=input_text)
|
||||
if process.returncode != 0:
|
||||
raise RuntimeError(f"Error running c++filt: {stderr}")
|
||||
return stdout
|
||||
|
||||
|
||||
def demangle_gnu_linkonce_symbols(cpp_filt_path: Path, map_text: str) -> str:
|
||||
"""
|
||||
Demangle .gnu.linkonce.t symbols in the map file.
|
||||
|
||||
Args:
|
||||
cpp_filt_path (Path): Path to c++filt executable.
|
||||
map_text (str): Content of the map file.
|
||||
|
||||
Returns:
|
||||
str: Map file content with demangled symbols.
|
||||
"""
|
||||
# Extract all .gnu.linkonce.t symbols
|
||||
pattern = r"\.gnu\\.linkonce\\.t\\.(.+?)\\s"
|
||||
matches = re.findall(pattern, map_text)
|
||||
|
||||
if not matches:
|
||||
return map_text
|
||||
|
||||
# Create a block of text with the extracted symbols
|
||||
symbols_block = "\n".join(matches)
|
||||
|
||||
# Demangle the symbols
|
||||
demangled_block = cpp_filt(cpp_filt_path, symbols_block)
|
||||
|
||||
# Create a dictionary of mangled to demangled symbols
|
||||
demangled_dict = dict(zip(matches, demangled_block.strip().split("\n")))
|
||||
|
||||
# Replace the mangled symbols with demangled ones in the original text
|
||||
for mangled, demangled in demangled_dict.items():
|
||||
map_text = map_text.replace(
|
||||
f".gnu.linkonce.t.{mangled}", f".gnu.linkonce.t.{demangled}"
|
||||
)
|
||||
|
||||
return map_text
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Convert a binary file to ELF using map file."
|
||||
)
|
||||
parser.add_argument("--first", action="store_true", help="Inspect the first board")
|
||||
parser.add_argument("--cwd", type=Path, help="Custom working directory")
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def load_build_info(build_info_path: Path) -> Dict[str, Any]:
|
||||
"""
|
||||
Load build information from a JSON file.
|
||||
|
||||
Args:
|
||||
build_info_path (Path): Path to the build_info.json file.
|
||||
|
||||
Returns:
|
||||
dict: Parsed JSON data.
|
||||
"""
|
||||
if not build_info_path.exists():
|
||||
raise FileNotFoundError(f"Build info JSON not found at '{build_info_path}'")
|
||||
return json.loads(build_info_path.read_text())
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
if args.cwd:
|
||||
root_build_dir = args.cwd / ".build"
|
||||
else:
|
||||
root_build_dir = Path(".build")
|
||||
|
||||
# Support nested PlatformIO structure: .build/pio/<board>
|
||||
nested_pio_dir = root_build_dir / "pio"
|
||||
if nested_pio_dir.is_dir():
|
||||
root_build_dir = nested_pio_dir
|
||||
|
||||
board_dirs = [d for d in root_build_dir.iterdir() if d.is_dir()]
|
||||
if not board_dirs:
|
||||
print(f"No board directories found in {root_build_dir.absolute()}")
|
||||
return 1
|
||||
|
||||
print("Available boards:")
|
||||
for i, board_dir in enumerate(board_dirs):
|
||||
print(f"[{i}]: {board_dir.name}")
|
||||
|
||||
which = (
|
||||
0
|
||||
if args.first
|
||||
else int(input("Enter the number of the board you want to inspect: "))
|
||||
)
|
||||
board_dir = board_dirs[which]
|
||||
build_info_json = board_dir / "build_info.json"
|
||||
|
||||
build_info = load_build_info(build_info_json)
|
||||
board = board_dir.name
|
||||
board_info = build_info.get(board) or build_info[next(iter(build_info))]
|
||||
|
||||
# Validate paths from build_info.json
|
||||
elf_path = Path(board_info.get("prog_path", ""))
|
||||
if not elf_path.exists():
|
||||
print(
|
||||
f"Error: ELF path '{elf_path}' does not exist. Check the 'prog_path' in build_info.json."
|
||||
)
|
||||
return 1
|
||||
|
||||
bin_file = elf_path.with_suffix(".bin")
|
||||
if not bin_file.exists():
|
||||
# use .hex or .uf2 if .bin doesn't exist
|
||||
bin_file = elf_path.with_suffix(".hex")
|
||||
if not bin_file.exists():
|
||||
bin_file = elf_path.with_suffix(".uf2")
|
||||
if not bin_file.exists():
|
||||
print(f"Error: Binary file not found for '{elf_path}'")
|
||||
return 1
|
||||
cpp_filt_path = Path(board_info["aliases"]["c++filt"])
|
||||
ld_path = Path(board_info["aliases"]["ld"])
|
||||
as_path = Path(board_info["aliases"]["as"])
|
||||
nm_path = Path(board_info["aliases"]["nm"])
|
||||
objcopy_path = Path(board_info["aliases"]["objcopy"])
|
||||
nm_path = Path(board_info["aliases"]["nm"])
|
||||
map_file = board_dir / "firmware.map"
|
||||
if not map_file.exists():
|
||||
# Search for the map file
|
||||
map_file = bin_file.with_suffix(".map")
|
||||
if not map_file.exists():
|
||||
possible_map_files = list(board_dir.glob("**/firmware.map"))
|
||||
if possible_map_files:
|
||||
map_file = possible_map_files[0]
|
||||
else:
|
||||
print("Error: firmware.map file not found")
|
||||
return 1
|
||||
|
||||
try:
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
temp_dir_path = Path(temp_dir)
|
||||
output_elf = bin_to_elf(
|
||||
bin_file,
|
||||
map_file,
|
||||
as_path,
|
||||
ld_path,
|
||||
objcopy_path,
|
||||
temp_dir_path / "output.elf",
|
||||
)
|
||||
out = dump_symbol_sizes(nm_path, cpp_filt_path, output_elf)
|
||||
print(out)
|
||||
except Exception as e:
|
||||
print(
|
||||
f"Error while converting binary to ELF, binary analysis will not work on this build: {e}"
|
||||
)
|
||||
|
||||
map_dump(map_file)
|
||||
|
||||
# Demangle .gnu.linkonce.t symbols and print map file
|
||||
print("\n##################################################")
|
||||
print("# Map file dump:")
|
||||
print("##################################################\n")
|
||||
map_text = map_file.read_text()
|
||||
demangled_map_text = demangle_gnu_linkonce_symbols(cpp_filt_path, map_text)
|
||||
print(demangled_map_text)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,78 @@
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from ci.util.elf import dump_symbol_sizes
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
PROJECT_ROOT = HERE.parent
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Inspect a compiled binary")
|
||||
parser.add_argument("--first", action="store_true", help="Inspect the first board")
|
||||
parser.add_argument("--cwd", type=str, help="Custom working directory")
|
||||
parser.add_argument("--elf", type=str, help="Path to the ELF file to inspect")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
if args.elf:
|
||||
firmware_path = Path(args.elf)
|
||||
if not firmware_path.exists():
|
||||
print(f"ELF file not found: {firmware_path}")
|
||||
return 1
|
||||
|
||||
if args.cwd:
|
||||
# os.chdir(args.cwd)
|
||||
root_build_dir = Path(args.cwd) / ".build"
|
||||
else:
|
||||
root_build_dir = Path(".build")
|
||||
|
||||
# Support nested PlatformIO structure: .build/pio/<board>
|
||||
nested_pio_dir = root_build_dir / "pio"
|
||||
if nested_pio_dir.is_dir():
|
||||
root_build_dir = nested_pio_dir
|
||||
|
||||
# Find the first board directory
|
||||
board_dirs = [d for d in root_build_dir.iterdir() if d.is_dir()]
|
||||
if not board_dirs:
|
||||
# print("No board directories found in .build")
|
||||
print(f"No board directories found in {root_build_dir.absolute()}")
|
||||
return 1
|
||||
|
||||
# display all the boards to the user and ask them to select which one they want by number
|
||||
print("Available boards:")
|
||||
for i, board_dir in enumerate(board_dirs):
|
||||
print(f"[{i}]: {board_dir.name}")
|
||||
|
||||
if args.first:
|
||||
which = 0
|
||||
else:
|
||||
which = int(input("Enter the number of the board you want to inspect: "))
|
||||
|
||||
board_dir = board_dirs[which]
|
||||
board = board_dir.name
|
||||
|
||||
build_info_json = board_dir / "build_info.json"
|
||||
build_info = json.loads(build_info_json.read_text())
|
||||
board_info = build_info.get(board) or build_info[next(iter(build_info))]
|
||||
|
||||
firmware_path = Path(board_info["prog_path"])
|
||||
cpp_filt_path = Path(board_info["aliases"]["c++filt"])
|
||||
|
||||
print(f"Dumping symbol sizes for {board} firmware: {firmware_path}")
|
||||
try:
|
||||
nm_path = Path(board_info["aliases"]["nm"])
|
||||
symbol_sizes = dump_symbol_sizes(nm_path, cpp_filt_path, firmware_path)
|
||||
print(symbol_sizes)
|
||||
except Exception as e:
|
||||
print(f"Error while dumping symbol sizes: {e}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ci.util.paths import BUILD
|
||||
from ci.util.tools import load_tools
|
||||
|
||||
|
||||
def _list_builds() -> List[Path]:
|
||||
str_paths = os.listdir(BUILD)
|
||||
paths = [BUILD / p for p in str_paths]
|
||||
dirs = [p for p in paths if p.is_dir()]
|
||||
return dirs
|
||||
|
||||
|
||||
def _check_build(build: Path) -> bool:
|
||||
# 1. should contain a build_info.json file
|
||||
# 2. should contain a .pio/build directory
|
||||
has_build_info = (build / "build_info.json").exists()
|
||||
has_pio_build = (build / ".pio" / "build").exists()
|
||||
return has_build_info and has_pio_build
|
||||
|
||||
|
||||
def _prompt_build() -> Path:
|
||||
builds = _list_builds()
|
||||
if not builds:
|
||||
print("Error: No builds found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print("Select a build:")
|
||||
for i, build in enumerate(builds):
|
||||
print(f" [{i}]: {build}")
|
||||
while True:
|
||||
try:
|
||||
which = int(input("Enter the number of the build to use: "))
|
||||
if 0 <= which < len(builds):
|
||||
valid = _check_build(BUILD / builds[which])
|
||||
if valid:
|
||||
return BUILD / builds[which]
|
||||
print("Error: Invalid build", file=sys.stderr)
|
||||
else:
|
||||
print("Error: Invalid selection", file=sys.stderr)
|
||||
continue
|
||||
except ValueError:
|
||||
print("Error: Invalid input", file=sys.stderr)
|
||||
continue
|
||||
|
||||
|
||||
def _prompt_object_file(build: Path) -> Path:
|
||||
# Look for object files in .pio/build directory
|
||||
build_dir = build / ".pio" / "build"
|
||||
object_files: List[Path] = list(build_dir.rglob("*.o"))
|
||||
|
||||
if not object_files:
|
||||
print("Error: No object files found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
print("\nSelect an object file:")
|
||||
for i, obj_file in enumerate(object_files):
|
||||
print(f" [{i}]: {obj_file.relative_to(build_dir)}")
|
||||
|
||||
while True:
|
||||
try:
|
||||
which = int(input("Enter the number of the object file to use: "))
|
||||
if 0 <= which < len(object_files):
|
||||
return object_files[which]
|
||||
print("Error: Invalid selection", file=sys.stderr)
|
||||
except ValueError:
|
||||
print("Error: Invalid input", file=sys.stderr)
|
||||
continue
|
||||
|
||||
|
||||
def cli() -> None:
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Dump object file information using build tools"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"build_path",
|
||||
type=Path,
|
||||
nargs="?",
|
||||
help="Path to build directory containing build info JSON file",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
build_path = args.build_path
|
||||
|
||||
# Check if object file was provided and exists
|
||||
if build_path is None:
|
||||
build_path = _prompt_build()
|
||||
else:
|
||||
if not _check_build(build_path):
|
||||
print("Error: Invalid build directory", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
assert build_path is not None
|
||||
assert build_path
|
||||
|
||||
build_info_path = build_path / "build_info.json"
|
||||
assert build_info_path.exists(), f"File not found: {build_info_path}"
|
||||
|
||||
tools = load_tools(build_info_path)
|
||||
|
||||
object_file = _prompt_object_file(build_path)
|
||||
|
||||
cmd = [str(tools.objdump_path), "--syms", str(object_file)]
|
||||
if sys.platform == "win32":
|
||||
cmd = ["cmd", "/c"] + cmd
|
||||
cmd_str = subprocess.list2cmdline(cmd)
|
||||
subprocess.run(cmd, check=True)
|
||||
print("\nDone. Command used:", cmd_str)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
cli()
|
||||
except KeyboardInterrupt:
|
||||
print("Exiting...")
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Standalone QEMU Installation Script
|
||||
|
||||
This is a convenience wrapper around install_qemu_esp32.py
|
||||
that can be called directly to install QEMU for ESP32 emulation.
|
||||
|
||||
Usage:
|
||||
uv run ci/install-qemu.py
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from install_qemu_esp32 import main as install_qemu_main
|
||||
|
||||
|
||||
def main():
|
||||
"""Run the QEMU installation script."""
|
||||
print("=== FastLED QEMU Installation ===")
|
||||
print("Starting QEMU installation process...")
|
||||
print(f"Python executable: {sys.executable}")
|
||||
print(f"Working directory: {os.getcwd()}")
|
||||
print()
|
||||
|
||||
try:
|
||||
# Call the main function directly
|
||||
print("Calling install_qemu_esp32.main()...")
|
||||
install_qemu_main()
|
||||
print("install_qemu_esp32.main() completed successfully")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\nInstallation cancelled by user")
|
||||
sys.exit(130)
|
||||
except SystemExit as e:
|
||||
print(f"SystemExit caught with code: {e.code}")
|
||||
# Re-raise SystemExit to preserve exit codes
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"ERROR: Installation failed: {e}")
|
||||
import traceback
|
||||
|
||||
print("Full traceback:")
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,732 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ESP32 QEMU Installation Script
|
||||
|
||||
Installs ESP32-compatible QEMU emulator using ESP-IDF tools.
|
||||
Cross-platform support for Linux, macOS, and Windows.
|
||||
|
||||
WARNING: Never use sys.stdout.flush() in this file!
|
||||
It causes blocking issues on Windows that hang QEMU processes.
|
||||
Python's default buffering behavior works correctly across platforms.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional, Union
|
||||
|
||||
from download import download # type: ignore
|
||||
|
||||
from ci.util.running_process import RunningProcess
|
||||
|
||||
|
||||
def get_platform_info():
|
||||
"""Get platform-specific information for QEMU installation."""
|
||||
print("IMMEDIATE DEBUG: get_platform_info() called")
|
||||
print("Detecting platform information...")
|
||||
system = platform.system().lower()
|
||||
print(f"IMMEDIATE DEBUG: platform.system() = {system}")
|
||||
machine = platform.machine().lower()
|
||||
print(f"IMMEDIATE DEBUG: platform.machine() = {machine}")
|
||||
print(f"System: {system}, Machine: {machine}")
|
||||
|
||||
# Normalize machine architecture
|
||||
if machine in ["x86_64", "amd64"]:
|
||||
arch = "x86_64"
|
||||
elif machine in ["aarch64", "arm64"]:
|
||||
arch = "arm64"
|
||||
elif machine.startswith("arm"):
|
||||
arch = "arm"
|
||||
else:
|
||||
arch = machine
|
||||
|
||||
return system, arch
|
||||
|
||||
|
||||
def install_system_dependencies():
|
||||
"""Install system dependencies required for QEMU."""
|
||||
system, _ = get_platform_info()
|
||||
|
||||
print("Installing system dependencies...")
|
||||
|
||||
if system == "linux":
|
||||
# Ubuntu/Debian dependencies
|
||||
deps = [
|
||||
"libgcrypt20",
|
||||
"libglib2.0-0",
|
||||
"libpixman-1-0",
|
||||
"libsdl2-2.0-0",
|
||||
"libslirp0",
|
||||
]
|
||||
|
||||
try:
|
||||
# Check if we can install packages (requires sudo)
|
||||
print("Updating package list...")
|
||||
process = RunningProcess(["sudo", "apt-get", "update"])
|
||||
result = process.wait(echo=True)
|
||||
if result == 0:
|
||||
print("Installing system dependencies:", " ".join(deps))
|
||||
process = RunningProcess(["sudo", "apt-get", "install", "-y"] + deps)
|
||||
result = process.wait(echo=True)
|
||||
if result != 0:
|
||||
raise subprocess.CalledProcessError(
|
||||
result if result is not None else 1,
|
||||
"apt-get install",
|
||||
)
|
||||
print("System dependencies installed")
|
||||
else:
|
||||
print("WARNING: Could not install system dependencies (sudo required)")
|
||||
print("Please install manually:", " ".join(deps))
|
||||
except subprocess.CalledProcessError:
|
||||
print("WARNING: Failed to install system dependencies")
|
||||
|
||||
elif system == "darwin":
|
||||
# macOS - dependencies usually available via system
|
||||
print("macOS system dependencies should be available")
|
||||
|
||||
elif system == "windows":
|
||||
# Windows - QEMU binaries are self-contained
|
||||
print("Windows - no additional dependencies required")
|
||||
|
||||
|
||||
def download_esp_idf_tools():
|
||||
"""Download and setup ESP-IDF tools for QEMU installation."""
|
||||
tools_dir = Path.home() / ".espressif"
|
||||
tools_dir.mkdir(exist_ok=True)
|
||||
|
||||
# ESP-IDF tools script URL
|
||||
tools_url = (
|
||||
"https://raw.githubusercontent.com/espressif/esp-idf/master/tools/idf_tools.py"
|
||||
)
|
||||
tools_script = tools_dir / "idf_tools.py"
|
||||
|
||||
if not tools_script.exists():
|
||||
print("Downloading ESP-IDF tools...")
|
||||
print(f"Source: {tools_url}")
|
||||
print(f"Destination: {tools_script}")
|
||||
download(tools_url, tools_script)
|
||||
print(f"Downloaded ESP-IDF tools to {tools_script}")
|
||||
else:
|
||||
print(f"ESP-IDF tools already exist at {tools_script}")
|
||||
|
||||
return tools_script
|
||||
|
||||
|
||||
def check_command_available(command: str) -> bool:
|
||||
"""Check if a command is available in PATH."""
|
||||
print(f"Checking if {command} is available...")
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[command, "--version"], capture_output=True, text=True, timeout=10
|
||||
)
|
||||
available = result.returncode == 0
|
||||
print(f"{command} {'found' if available else 'not found'}")
|
||||
return available
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError, subprocess.SubprocessError):
|
||||
print(f"{command} not found or timed out")
|
||||
return False
|
||||
|
||||
|
||||
def verify_file_hash(file_path: Path, expected_hash: str) -> bool:
|
||||
"""Verify SHA256 hash of a downloaded file."""
|
||||
print(f"Verifying SHA256 hash of {file_path}...")
|
||||
try:
|
||||
with open(file_path, "rb") as f:
|
||||
file_hash = hashlib.sha256(f.read()).hexdigest()
|
||||
|
||||
if file_hash == expected_hash:
|
||||
print(f"SHA256 verification passed: {file_hash}")
|
||||
return True
|
||||
else:
|
||||
print(f"SHA256 verification failed!")
|
||||
print(f"Expected: {expected_hash}")
|
||||
print(f"Got: {file_hash}")
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f"Error verifying hash: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def try_espressif_binary_install():
|
||||
"""Download and install QEMU using Espressif official precompiled binaries."""
|
||||
try:
|
||||
print("\n--- Espressif Official Binary Installation ---")
|
||||
print("Downloading QEMU from Espressif official releases...")
|
||||
|
||||
# Create local QEMU directory (project-local)
|
||||
qemu_dir = Path(".cache") / "qemu"
|
||||
qemu_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Check if already installed
|
||||
qemu_xtensa_binary = qemu_dir / "qemu-system-xtensa.exe"
|
||||
qemu_riscv32_binary = qemu_dir / "qemu-system-riscv32.exe"
|
||||
|
||||
print(f"Checking for existing QEMU at {qemu_xtensa_binary}...")
|
||||
if qemu_xtensa_binary.exists() and qemu_riscv32_binary.exists():
|
||||
print(
|
||||
f"QEMU already installed at {qemu_xtensa_binary} and {qemu_riscv32_binary}"
|
||||
)
|
||||
return True
|
||||
else:
|
||||
print("QEMU not found, proceeding with download...")
|
||||
|
||||
# Espressif official QEMU binaries for Windows x64
|
||||
binaries = [
|
||||
{
|
||||
"name": "QEMU-Xtensa (ESP32/ESP32-S2/ESP32-S3)",
|
||||
"url": "https://github.com/espressif/qemu/releases/download/esp-develop-9.2.2-20250817/qemu-xtensa-softmmu-esp_develop_9.2.2_20250817-x86_64-w64-mingw32.tar.xz",
|
||||
"sha256": "ef550b912726997f3c1ff4a4fb13c1569e2b692efdc5c9f9c3c926a8f7c540fa",
|
||||
"binary_name": "qemu-system-xtensa.exe",
|
||||
},
|
||||
{
|
||||
"name": "QEMU-RISCV32 (ESP32-C3/ESP32-H2/ESP32-C2)",
|
||||
"url": "https://github.com/espressif/qemu/releases/download/esp-develop-9.2.2-20250817/qemu-riscv32-softmmu-esp_develop_9.2.2_20250817-x86_64-w64-mingw32.tar.xz",
|
||||
"sha256": "9474015f24d27acb7516955ec932e5307226bd9d6652cdc870793ed36010ab73",
|
||||
"binary_name": "qemu-system-riscv32.exe",
|
||||
},
|
||||
]
|
||||
|
||||
for binary_info in binaries:
|
||||
print(f"\nDownloading {binary_info['name']}...")
|
||||
print("This may take a few minutes depending on your internet connection.")
|
||||
|
||||
archive_path = qemu_dir / f"{binary_info['binary_name']}.tar.xz"
|
||||
|
||||
print(f"Downloading from: {binary_info['url']}")
|
||||
print(f"Saving to: {archive_path}")
|
||||
|
||||
try:
|
||||
# Download the archive
|
||||
download(binary_info["url"], archive_path)
|
||||
print(f"Download completed: {archive_path}")
|
||||
|
||||
# Verify SHA256 hash
|
||||
if not verify_file_hash(archive_path, binary_info["sha256"]):
|
||||
print(f"Hash verification failed for {binary_info['name']}")
|
||||
archive_path.unlink() # Remove corrupted file
|
||||
return False
|
||||
|
||||
# Extract the archive
|
||||
print(f"Extracting {binary_info['name']}...")
|
||||
with tarfile.open(archive_path, "r:xz") as tar:
|
||||
# List contents to understand structure
|
||||
members = tar.getnames()
|
||||
print(f"Archive contains {len(members)} files")
|
||||
|
||||
# Extract to temporary directory first
|
||||
temp_extract_dir = qemu_dir / "temp_extract"
|
||||
temp_extract_dir.mkdir(exist_ok=True)
|
||||
|
||||
tar.extractall(temp_extract_dir)
|
||||
print(f"Extracted to temporary directory: {temp_extract_dir}")
|
||||
|
||||
# Find the binary in the extracted contents
|
||||
found_binary = None
|
||||
for root, dirs, files in os.walk(temp_extract_dir):
|
||||
for file in files:
|
||||
if file == binary_info["binary_name"]:
|
||||
found_binary = Path(root) / file
|
||||
break
|
||||
if found_binary:
|
||||
break
|
||||
|
||||
if found_binary:
|
||||
target_binary = qemu_dir / binary_info["binary_name"]
|
||||
print(
|
||||
f"Copying {binary_info['binary_name']} to {target_binary}"
|
||||
)
|
||||
shutil.copy2(found_binary, target_binary)
|
||||
|
||||
# Make sure it's executable (though Windows doesn't use execute bit)
|
||||
target_binary.chmod(0o755)
|
||||
|
||||
print(f"Successfully installed {binary_info['binary_name']}")
|
||||
else:
|
||||
print(
|
||||
f"ERROR: Could not find {binary_info['binary_name']} in extracted archive"
|
||||
)
|
||||
return False
|
||||
|
||||
# Clean up temporary extraction directory
|
||||
shutil.rmtree(temp_extract_dir)
|
||||
|
||||
# Clean up downloaded archive
|
||||
archive_path.unlink()
|
||||
print(f"Cleaned up {archive_path}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error downloading/extracting {binary_info['name']}: {e}")
|
||||
# Clean up on error
|
||||
if archive_path.exists():
|
||||
archive_path.unlink()
|
||||
return False
|
||||
|
||||
print("Espressif binary installation completed successfully")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"Espressif binary installation error: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def try_direct_download_install():
|
||||
"""Download and install QEMU directly from the official source."""
|
||||
print("\n--- Legacy Installer Method (Requires Admin) ---")
|
||||
print("WARNING: This method requires administrator privileges and may fail")
|
||||
print("Consider using the Espressif binary method instead")
|
||||
|
||||
try:
|
||||
print("Downloading QEMU directly from official source...")
|
||||
|
||||
# Create local QEMU directory (project-local)
|
||||
qemu_dir = Path(".cache") / "qemu"
|
||||
qemu_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Check if already installed
|
||||
qemu_binary = qemu_dir / "qemu-system-xtensa.exe"
|
||||
print(f"Checking for existing QEMU at {qemu_binary}...")
|
||||
if qemu_binary.exists():
|
||||
print(f"QEMU already installed at {qemu_binary}")
|
||||
return True
|
||||
else:
|
||||
print("QEMU not found, proceeding with download...")
|
||||
|
||||
# Download QEMU from official source
|
||||
print("Downloading QEMU for Windows...")
|
||||
print("This may take several minutes depending on your internet connection.")
|
||||
|
||||
# QEMU download URL from the official Windows builds
|
||||
qemu_url = "https://qemu.weilnetz.de/w64/2024/qemu-w64-setup-20241220.exe"
|
||||
installer_path = qemu_dir / "qemu-installer.exe"
|
||||
|
||||
print(f"Downloading from: {qemu_url}")
|
||||
print(f"Saving to: {installer_path}")
|
||||
|
||||
try:
|
||||
# Download the installer
|
||||
download(qemu_url, installer_path)
|
||||
print(f"Download completed: {installer_path}")
|
||||
|
||||
# Run the installer in silent mode
|
||||
print("Running QEMU installer in silent mode...")
|
||||
print("Note: This may require administrator privileges")
|
||||
|
||||
# Try different installation approaches
|
||||
install_attempts = [
|
||||
# Attempt 1: Install to custom directory (may require admin)
|
||||
[
|
||||
str(installer_path),
|
||||
"/VERYSILENT", # Very silent install
|
||||
"/SUPPRESSMSGBOXES", # Suppress message boxes
|
||||
f"/DIR={qemu_dir.absolute()}", # Install directory
|
||||
],
|
||||
# Attempt 2: Install to default location (may require admin)
|
||||
[
|
||||
str(installer_path),
|
||||
"/VERYSILENT", # Very silent install
|
||||
"/SUPPRESSMSGBOXES", # Suppress message boxes
|
||||
],
|
||||
# Attempt 3: Use basic silent install
|
||||
[
|
||||
str(installer_path),
|
||||
"/S", # Silent install
|
||||
],
|
||||
]
|
||||
|
||||
result = None
|
||||
process = None
|
||||
for i, install_cmd in enumerate(install_attempts, 1):
|
||||
print(
|
||||
f"Installation attempt {i}/{len(install_attempts)}: {' '.join(install_cmd)}"
|
||||
)
|
||||
try:
|
||||
process = RunningProcess(
|
||||
install_cmd,
|
||||
timeout=300, # 5 minute timeout
|
||||
)
|
||||
result = process.wait(echo=True)
|
||||
|
||||
if result == 0:
|
||||
print(f"Installation attempt {i} succeeded")
|
||||
break
|
||||
else:
|
||||
print(
|
||||
f"Installation attempt {i} failed with exit code: {result}"
|
||||
)
|
||||
if i < len(install_attempts):
|
||||
print("Trying next installation method...")
|
||||
except Exception as e:
|
||||
print(f"Installation attempt {i} failed with error: {e}")
|
||||
if i < len(install_attempts):
|
||||
print("Trying next installation method...")
|
||||
|
||||
if result == 0:
|
||||
print("QEMU installer completed successfully")
|
||||
|
||||
# Look for the installed qemu-system-xtensa.exe
|
||||
for qemu_path in qemu_dir.rglob("qemu-system-xtensa.exe"):
|
||||
if qemu_path.is_file():
|
||||
print(f"Found QEMU binary at: {qemu_path}")
|
||||
|
||||
# If it's not in the expected location, copy it there
|
||||
if qemu_path != qemu_binary:
|
||||
print(
|
||||
f"Copying QEMU binary to standard location: {qemu_binary}"
|
||||
)
|
||||
shutil.copy2(qemu_path, qemu_binary)
|
||||
|
||||
# Clean up installer
|
||||
if installer_path.exists():
|
||||
installer_path.unlink()
|
||||
|
||||
print("Direct download installation completed successfully")
|
||||
return True
|
||||
|
||||
print("QEMU installer completed but qemu-system-xtensa.exe not found")
|
||||
else:
|
||||
print(f"QEMU installer failed with exit code: {result}")
|
||||
if process:
|
||||
stderr_output = process.stdout
|
||||
if stderr_output:
|
||||
print(f"Error output: {stderr_output[:200]}")
|
||||
|
||||
except Exception as download_error:
|
||||
print(f"Download/install error: {download_error}")
|
||||
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"Direct download installation error: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def try_windows_installation():
|
||||
"""Try installing QEMU using Espressif binaries for Windows."""
|
||||
print("Attempting automatic QEMU installation on Windows...")
|
||||
|
||||
# Try Espressif official binaries first (recommended method)
|
||||
print("Using Espressif official binaries...")
|
||||
if try_espressif_binary_install():
|
||||
print("Espressif binary installation completed, verifying...")
|
||||
# Verify installation worked
|
||||
if find_qemu_binary():
|
||||
print("SUCCESS: QEMU installed via Espressif official binaries")
|
||||
return True
|
||||
else:
|
||||
print(
|
||||
"WARNING: Espressif binary installation reported success but QEMU binary not found"
|
||||
)
|
||||
else:
|
||||
print("Espressif binary installation failed")
|
||||
|
||||
# Fall back to legacy installer method (requires admin privileges)
|
||||
print("Falling back to legacy installer method...")
|
||||
if try_direct_download_install():
|
||||
print("Legacy installer completed, verifying...")
|
||||
# Verify installation worked
|
||||
if find_qemu_binary():
|
||||
print("SUCCESS: QEMU installed via legacy installer")
|
||||
return True
|
||||
else:
|
||||
print(
|
||||
"WARNING: Legacy installer reported success but QEMU binary not found"
|
||||
)
|
||||
else:
|
||||
print("Legacy installer failed")
|
||||
|
||||
print("Windows QEMU installation failed")
|
||||
return False
|
||||
|
||||
|
||||
def install_qemu_esp32():
|
||||
"""Install ESP32 QEMU using ESP-IDF tools."""
|
||||
print("\n=== Starting ESP32 QEMU Installation ===")
|
||||
system, arch = get_platform_info()
|
||||
|
||||
print(f"\nChecking for ESP32 QEMU for {system}-{arch}...")
|
||||
|
||||
# Check if already installed
|
||||
print("Searching for existing QEMU installation...")
|
||||
qemu_binary = find_qemu_binary()
|
||||
print(f"IMMEDIATE DEBUG: find_qemu_binary returned: {qemu_binary}")
|
||||
if qemu_binary:
|
||||
print("ESP32 QEMU already installed")
|
||||
return True
|
||||
else:
|
||||
print("No existing QEMU installation found")
|
||||
|
||||
# Try automated installation
|
||||
print("Attempting automatic installation...")
|
||||
|
||||
if system == "linux":
|
||||
# Check if we're in CI first for sudo operations
|
||||
if os.getenv("CI") == "true" or os.getenv("GITHUB_ACTIONS") == "true":
|
||||
try:
|
||||
# Try to install QEMU from package manager in CI
|
||||
print("Installing QEMU via apt-get...")
|
||||
print("Updating package list...")
|
||||
process = RunningProcess(["sudo", "apt-get", "update"])
|
||||
result = process.wait(echo=True)
|
||||
|
||||
if result == 0:
|
||||
print("Package list updated successfully")
|
||||
print("Installing QEMU packages...")
|
||||
print("Installing: qemu-system-misc qemu-system-xtensa")
|
||||
process = RunningProcess(
|
||||
[
|
||||
"sudo",
|
||||
"apt-get",
|
||||
"install",
|
||||
"-y",
|
||||
"qemu-system-misc",
|
||||
"qemu-system-xtensa",
|
||||
]
|
||||
)
|
||||
result = process.wait(echo=True)
|
||||
|
||||
if result == 0:
|
||||
print("QEMU packages installed successfully via apt-get")
|
||||
return True
|
||||
else:
|
||||
print("Failed to install QEMU packages via apt-get")
|
||||
print("Return code:", result)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Automatic installation failed: {e}")
|
||||
else:
|
||||
print(
|
||||
"Local Linux environment - automatic installation requires sudo privileges"
|
||||
)
|
||||
|
||||
elif system == "windows":
|
||||
print("Windows platform detected, using direct download...")
|
||||
# Try direct download installation (works in both CI and local environments)
|
||||
if try_windows_installation():
|
||||
print("Windows QEMU installation successful")
|
||||
return True
|
||||
else:
|
||||
print("Windows installation failed")
|
||||
|
||||
elif system == "darwin":
|
||||
# Try macOS installation methods
|
||||
print("macOS platform detected")
|
||||
print("Attempting macOS installation...")
|
||||
print("Note: Homebrew installation could be added here in the future")
|
||||
# Could add homebrew installation here in the future
|
||||
|
||||
# If not found, provide installation instructions
|
||||
print("ESP32 QEMU not found. Please install using one of these methods:")
|
||||
print()
|
||||
|
||||
if system == "windows":
|
||||
print("Windows Installation Options:")
|
||||
print()
|
||||
print("Option 1: Espressif Official Binaries (Recommended)")
|
||||
print(
|
||||
" This is what the automatic installer uses - no administrator privileges required"
|
||||
)
|
||||
print(
|
||||
" Espressif QEMU Xtensa: https://github.com/espressif/qemu/releases/download/esp-develop-9.2.2-20250817/qemu-xtensa-softmmu-esp_develop_9.2.2_20250817-x86_64-w64-mingw32.tar.xz"
|
||||
)
|
||||
print(
|
||||
" Espressif QEMU RISC-V32: https://github.com/espressif/qemu/releases/download/esp-develop-9.2.2-20250817/qemu-riscv32-softmmu-esp_develop_9.2.2_20250817-x86_64-w64-mingw32.tar.xz"
|
||||
)
|
||||
print(" Extract to .cache/qemu/ directory")
|
||||
print()
|
||||
print("Option 2: Official QEMU Installer (Requires Administrator)")
|
||||
print(
|
||||
" Download QEMU from: https://qemu.weilnetz.de/w64/2024/qemu-w64-setup-20241220.exe"
|
||||
)
|
||||
print(" Run the installer and ensure qemu-system-xtensa.exe is in your PATH")
|
||||
print()
|
||||
print("Option 3: ESP-IDF (includes QEMU)")
|
||||
print(
|
||||
" Download ESP-IDF from: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/windows-setup.html"
|
||||
)
|
||||
print()
|
||||
print("Option 4: Manual Installation")
|
||||
print(" Download QEMU from: https://www.qemu.org/download/")
|
||||
print(" Make sure qemu-system-xtensa.exe is in your PATH")
|
||||
|
||||
elif system == "linux":
|
||||
print("Option 1: Install via ESP-IDF")
|
||||
print(" git clone --recursive https://github.com/espressif/esp-idf.git")
|
||||
print(" cd esp-idf && ./install.sh esp32s3")
|
||||
print(" source ./export.sh")
|
||||
print()
|
||||
print("Option 2: Install QEMU from package manager")
|
||||
print(" sudo apt-get install qemu-system-misc qemu-system-xtensa")
|
||||
|
||||
elif system == "darwin":
|
||||
print("Option 1: Install via ESP-IDF")
|
||||
print(" git clone --recursive https://github.com/espressif/esp-idf.git")
|
||||
print(" cd esp-idf && ./install.sh esp32s3")
|
||||
print(" source ./export.sh")
|
||||
print()
|
||||
print("Option 2: Install via Homebrew")
|
||||
print(" brew install qemu")
|
||||
|
||||
print()
|
||||
print("For CI/GitHub Actions, QEMU will be cached automatically once installed.")
|
||||
return False
|
||||
|
||||
|
||||
def find_qemu_binary() -> Optional[Path]:
|
||||
"""Find the installed QEMU binary."""
|
||||
print("IMMEDIATE DEBUG: find_qemu_binary() called")
|
||||
system, _ = get_platform_info()
|
||||
print(f"IMMEDIATE DEBUG: got system: {system}")
|
||||
|
||||
# Look for both Xtensa and RISC-V32 binaries
|
||||
binary_names = []
|
||||
if system == "windows":
|
||||
binary_names = ["qemu-system-xtensa.exe", "qemu-system-riscv32.exe"]
|
||||
else:
|
||||
binary_names = ["qemu-system-xtensa", "qemu-system-riscv32"]
|
||||
|
||||
print(f"IMMEDIATE DEBUG: looking for binaries: {binary_names}")
|
||||
|
||||
# ESP-IDF installation paths and portable installation
|
||||
esp_paths = [
|
||||
Path(".cache") / "qemu", # Project-local portable installation directory
|
||||
Path.home()
|
||||
/ ".cache"
|
||||
/ "qemu", # Legacy user cache location (for backward compatibility)
|
||||
Path.home() / ".espressif" / "tools" / "qemu-xtensa",
|
||||
Path.home() / ".espressif" / "python_env",
|
||||
Path("/opt/espressif/tools/qemu-xtensa"),
|
||||
]
|
||||
|
||||
# Search in ESP-IDF paths first
|
||||
for base_path in esp_paths:
|
||||
print(f"Searching in: {base_path}")
|
||||
try:
|
||||
if base_path.exists():
|
||||
print(f" Directory exists, searching for QEMU binaries...")
|
||||
for binary_name in binary_names:
|
||||
print(
|
||||
f" Starting recursive search for {binary_name} in {base_path}..."
|
||||
)
|
||||
for qemu_path in base_path.rglob(binary_name):
|
||||
print(f" Checking: {qemu_path}")
|
||||
if qemu_path.is_file():
|
||||
print(f"Found QEMU binary at: {qemu_path}")
|
||||
return qemu_path
|
||||
print(f" No QEMU binaries found in {base_path}")
|
||||
else:
|
||||
print(f" Directory does not exist")
|
||||
except Exception as e:
|
||||
print(f" Error searching {base_path}: {e}")
|
||||
|
||||
# Check system PATH for each binary
|
||||
print("Checking system PATH...")
|
||||
for binary_name in binary_names:
|
||||
try:
|
||||
cmd = ["where" if system == "windows" else "which", binary_name]
|
||||
print(f"Running: {' '.join(cmd)}")
|
||||
print("About to call subprocess.run...")
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
|
||||
print("subprocess.run completed")
|
||||
if result.returncode == 0:
|
||||
# Take first line in case of multiple results (Windows)
|
||||
first_line = result.stdout.strip().split("\n")[0]
|
||||
qemu_path = Path(first_line)
|
||||
print(f"Found QEMU binary in PATH: {qemu_path}")
|
||||
return qemu_path
|
||||
else:
|
||||
print(f"Command returned code {result.returncode}")
|
||||
except subprocess.TimeoutExpired:
|
||||
print(f"PATH check for {binary_name} timed out after 10 seconds")
|
||||
except (subprocess.SubprocessError, FileNotFoundError) as e:
|
||||
print(f"Error checking PATH for {binary_name}: {e}")
|
||||
|
||||
print("ERROR: Could not find any QEMU binary")
|
||||
return None
|
||||
|
||||
|
||||
def create_qemu_wrapper():
|
||||
"""Create a wrapper script for easy QEMU access."""
|
||||
# DISABLED: This function was overwriting the actual qemu-esp32.py runner script
|
||||
# The qemu-esp32.py script already exists and handles all the necessary
|
||||
# functionality for running ESP32 firmware in QEMU with proper argument parsing
|
||||
print("Skipping wrapper creation - using existing qemu-esp32.py runner script")
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
"""Main installation routine."""
|
||||
print("IMMEDIATE DEBUG: main() function called")
|
||||
print("=== ESP32 QEMU Installation ===")
|
||||
print("Step 1/5: Starting installation process...")
|
||||
|
||||
# Show platform info
|
||||
print("Step 2/5: Detecting platform...")
|
||||
system, arch = get_platform_info()
|
||||
print(f"IMMEDIATE DEBUG: get_platform_info completed, got: {system}, {arch}")
|
||||
print(f"Platform: {system}-{arch}")
|
||||
|
||||
# Install system dependencies
|
||||
print("Step 3/5: Installing system dependencies...")
|
||||
install_system_dependencies()
|
||||
print("IMMEDIATE DEBUG: install_system_dependencies completed")
|
||||
print("System dependencies step completed")
|
||||
|
||||
# Install ESP32 QEMU
|
||||
print("Step 4/5: Installing ESP32 QEMU...")
|
||||
qemu_installed = install_qemu_esp32()
|
||||
print(
|
||||
f"IMMEDIATE DEBUG: install_qemu_esp32 completed with result: {qemu_installed}"
|
||||
)
|
||||
print(f"ESP32 QEMU installation step completed (success: {qemu_installed})")
|
||||
|
||||
if qemu_installed:
|
||||
# Create wrapper script
|
||||
print("Step 5/5: Creating wrapper scripts...")
|
||||
if not create_qemu_wrapper():
|
||||
print("WARNING: Could not create wrapper script, but QEMU is available")
|
||||
else:
|
||||
print("QEMU installation required for ESP32 emulation")
|
||||
sys.exit(1)
|
||||
|
||||
print("=== Installation Complete ===")
|
||||
print("ESP32 QEMU is ready to use!")
|
||||
|
||||
# Test installation
|
||||
print("\n=== Verifying Installation ===")
|
||||
print("Searching for QEMU binary...")
|
||||
qemu_binary = find_qemu_binary()
|
||||
if qemu_binary:
|
||||
print(f"Testing QEMU binary: {qemu_binary}")
|
||||
try:
|
||||
print("Running version check...")
|
||||
result = subprocess.run(
|
||||
[str(qemu_binary), "--version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
print(f"QEMU version: {result.stdout.strip()}")
|
||||
print("Installation verification successful!")
|
||||
sys.exit(0) # Explicit success exit
|
||||
else:
|
||||
print("WARNING: QEMU installed but version check failed")
|
||||
print(f"Version check return code: {result.returncode}")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"WARNING: Could not verify QEMU installation: {e}")
|
||||
sys.exit(1)
|
||||
else:
|
||||
print("ERROR: QEMU installation verification failed")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,57 @@
|
||||
# FastLED Include What You Use Mapping File
|
||||
# This file tells IWYU about FastLED's specific include patterns and structure
|
||||
|
||||
[
|
||||
# FastLED main headers - these are the public API
|
||||
{ include: ['"FastLED.h"', 'public', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"fastled.h"', 'public', '"FastLED.h"', 'public'] },
|
||||
|
||||
# Core FastLED headers that should be included directly
|
||||
{ include: ['"lib8tion.h"', 'public', '"lib8tion.h"', 'public'] },
|
||||
{ include: ['"pixeltypes.h"', 'public', '"pixeltypes.h"', 'public'] },
|
||||
{ include: ['"hsv2rgb.h"', 'public', '"hsv2rgb.h"', 'public'] },
|
||||
{ include: ['"colorutils.h"', 'public', '"colorutils.h"', 'public'] },
|
||||
{ include: ['"colorpalettes.h"', 'public', '"colorpalettes.h"', 'public'] },
|
||||
|
||||
# Platform-specific headers should not be suggested directly
|
||||
{ include: ['"platforms/*/fastled_*.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"platforms/*/clockless_*.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"platforms/*/*"', 'private', '"FastLED.h"', 'public'] },
|
||||
|
||||
# Internal headers should redirect to public API
|
||||
{ include: ['"controller.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"led_sysdefs.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"fastled_config.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"fastled_delay.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"fastled_progmem.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
|
||||
# fl/ namespace headers - the modern FastLED API
|
||||
{ include: ['"fl/namespace.h"', 'public', '"fl/namespace.h"', 'public'] },
|
||||
{ include: ['"fl/ptr.h"', 'public', '"fl/ptr.h"', 'public'] },
|
||||
{ include: ['"fl/vector.h"', 'public', '"fl/vector.h"', 'public'] },
|
||||
{ include: ['"fl/allocator.h"', 'public', '"fl/allocator.h"', 'public'] },
|
||||
{ include: ['"fl/math.h"', 'public', '"fl/math.h"', 'public'] },
|
||||
{ include: ['"fl/force_inline.h"', 'public', '"fl/force_inline.h"', 'public'] },
|
||||
|
||||
# FX headers - effects system
|
||||
{ include: ['"fx/fx.h"', 'public', '"fx/fx.h"', 'public'] },
|
||||
{ include: ['"fx/fx_engine.h"', 'public', '"fx/fx_engine.h"', 'public'] },
|
||||
{ include: ['"fx/frame.h"', 'public', '"fx/frame.h"', 'public'] },
|
||||
{ include: ['"fx/detail/*"', 'private', '"fx/fx.h"', 'public'] },
|
||||
|
||||
# Third-party headers should not be suggested for user code
|
||||
{ include: ['"third_party/*/*"', 'private', '"FastLED.h"', 'public'] },
|
||||
|
||||
# Arduino-specific headers
|
||||
{ include: ['<Arduino.h>', 'public', '<Arduino.h>', 'public'] },
|
||||
{ include: ['<avr/pgmspace.h>', 'public', '<avr/pgmspace.h>', 'public'] },
|
||||
|
||||
# Chipset-specific headers redirect to main header
|
||||
{ include: ['"chipsets.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"ws2812_controller.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"apa102_controller.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
|
||||
# Testing headers - keep as-is for test files
|
||||
{ include: ['"test_*.h"', 'public', '"test_*.h"', 'public'] },
|
||||
{ include: ['"doctest.h"', 'public', '"doctest.h"', 'public'] },
|
||||
]
|
||||
@@ -0,0 +1,67 @@
|
||||
# Standard Library Include What You Use Mapping File
|
||||
# This file provides standard library header mappings for better IWYU suggestions
|
||||
|
||||
[
|
||||
# C++ standard library headers
|
||||
{ include: ['<cstdint>', 'public', '<cstdint>', 'public'] },
|
||||
{ include: ['<cstddef>', 'public', '<cstddef>', 'public'] },
|
||||
{ include: ['<cstring>', 'public', '<cstring>', 'public'] },
|
||||
{ include: ['<cstdlib>', 'public', '<cstdlib>', 'public'] },
|
||||
{ include: ['<cmath>', 'public', '<cmath>', 'public'] },
|
||||
{ include: ['<cassert>', 'public', '<cassert>', 'public'] },
|
||||
{ include: ['<climits>', 'public', '<climits>', 'public'] },
|
||||
{ include: ['<cfloat>', 'public', '<cfloat>', 'public'] },
|
||||
|
||||
# C++ containers and algorithms
|
||||
{ include: ['<vector>', 'public', '<vector>', 'public'] },
|
||||
{ include: ['<array>', 'public', '<array>', 'public'] },
|
||||
{ include: ['<string>', 'public', '<string>', 'public'] },
|
||||
{ include: ['<algorithm>', 'public', '<algorithm>', 'public'] },
|
||||
{ include: ['<memory>', 'public', '<memory>', 'public'] },
|
||||
{ include: ['<functional>', 'public', '<functional>', 'public'] },
|
||||
{ include: ['<utility>', 'public', '<utility>', 'public'] },
|
||||
{ include: ['<type_traits>', 'public', '<type_traits>', 'public'] },
|
||||
|
||||
# I/O headers
|
||||
{ include: ['<iostream>', 'public', '<iostream>', 'public'] },
|
||||
{ include: ['<iomanip>', 'public', '<iomanip>', 'public'] },
|
||||
{ include: ['<sstream>', 'public', '<sstream>', 'public'] },
|
||||
{ include: ['<fstream>', 'public', '<fstream>', 'public'] },
|
||||
|
||||
# Common type mappings
|
||||
{ symbol: ['std::uint8_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::uint16_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::uint32_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::uint64_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::int8_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::int16_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::int32_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::int64_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::size_t', 'public', '<cstddef>', 'public'] },
|
||||
{ symbol: ['std::ptrdiff_t', 'public', '<cstddef>', 'public'] },
|
||||
|
||||
# Common functions
|
||||
{ symbol: ['std::memcpy', 'public', '<cstring>', 'public'] },
|
||||
{ symbol: ['std::memset', 'public', '<cstring>', 'public'] },
|
||||
{ symbol: ['std::strlen', 'public', '<cstring>', 'public'] },
|
||||
{ symbol: ['std::malloc', 'public', '<cstdlib>', 'public'] },
|
||||
{ symbol: ['std::free', 'public', '<cstdlib>', 'public'] },
|
||||
{ symbol: ['std::abs', 'public', '<cmath>', 'public'] },
|
||||
{ symbol: ['std::sin', 'public', '<cmath>', 'public'] },
|
||||
{ symbol: ['std::cos', 'public', '<cmath>', 'public'] },
|
||||
{ symbol: ['std::sqrt', 'public', '<cmath>', 'public'] },
|
||||
|
||||
# C standard library compatibility (prefer C++ versions)
|
||||
{ include: ['<stdint.h>', 'public', '<cstdint>', 'public'] },
|
||||
{ include: ['<stddef.h>', 'public', '<cstddef>', 'public'] },
|
||||
{ include: ['<string.h>', 'public', '<cstring>', 'public'] },
|
||||
{ include: ['<stdlib.h>', 'public', '<cstdlib>', 'public'] },
|
||||
{ include: ['<math.h>', 'public', '<cmath>', 'public'] },
|
||||
{ include: ['<assert.h>', 'public', '<cassert>', 'public'] },
|
||||
{ include: ['<limits.h>', 'public', '<climits>', 'public'] },
|
||||
{ include: ['<float.h>', 'public', '<cfloat>', 'public'] },
|
||||
|
||||
# Platform-specific but common headers
|
||||
{ include: ['<sys/types.h>', 'public', '<sys/types.h>', 'public'] },
|
||||
{ include: ['<unistd.h>', 'public', '<unistd.h>', 'public'] },
|
||||
]
|
||||
@@ -0,0 +1,56 @@
|
||||
# FastLED JavaScript Linting Tools
|
||||
|
||||
This directory contains JavaScript linting tools for the FastLED project, using Node.js + ESLint for **fast** JavaScript validation.
|
||||
|
||||
|
||||
### Via Main Lint Script (Recommended)
|
||||
|
||||
```bash
|
||||
# JavaScript linting is included by default (fast only!)
|
||||
bash lint
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
The fast JavaScript linter is automatically installed during project setup:
|
||||
|
||||
```bash
|
||||
# Installs everything including fast JS linting
|
||||
./install
|
||||
```
|
||||
|
||||
Or manually install just the JavaScript linter:
|
||||
|
||||
```bash
|
||||
uv run ci/setup-js-linting-fast.py
|
||||
```
|
||||
|
||||
This will:
|
||||
- Download Node.js binary to `.js-tools/node/`
|
||||
- Install ESLint to `.js-tools/node_modules/`
|
||||
- Create ESLint configuration in `.js-tools/.eslintrc.js`
|
||||
- Create fast linting script `.cache/js-tools/lint-js-fast`
|
||||
|
||||
## Philosophy: Fast Only
|
||||
|
||||
**JavaScript linting follows a "fast only" policy:**
|
||||
- ✅ **Fast linting available**: Uses Node.js + ESLint (~0.9 seconds)
|
||||
- ⚠️ **Fast linting unavailable**: Skips JavaScript linting entirely
|
||||
- ❌ **No slow fallback**: Fast linting or no linting at all
|
||||
|
||||
## Configuration
|
||||
|
||||
The fast linter only checks for **critical runtime issues**:
|
||||
- `no-debugger` - Prevents debugger statements in production
|
||||
- `no-eval` - Prevents eval() security vulnerabilities
|
||||
|
||||
**No style enforcement** - focuses on critical issues only for maximum speed.
|
||||
|
||||
## Integration
|
||||
|
||||
JavaScript linting is included by default in `bash lint` when fast linting is available.
|
||||
|
||||
## Performance
|
||||
|
||||
- **Fast linting**: ~0.9 seconds (53x faster than previous solution)
|
||||
- **Total lint time**: ~1.6 seconds (Python + C++ + JavaScript)
|
||||
@@ -0,0 +1,16 @@
|
||||
#include "FastLED.h"
|
||||
|
||||
// Let's include a bunch of stuff and see if it breaks the build.
|
||||
#include <WiFi.h>
|
||||
#include <ESPmDNS.h>
|
||||
#include <NetworkUdp.h>
|
||||
#include <ArduinoOTA.h>
|
||||
#include <ArduinoJson.h>
|
||||
|
||||
void setup() {
|
||||
|
||||
}
|
||||
|
||||
void loop() {
|
||||
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
[platformio]
|
||||
src_dir = symlink://../../../src
|
||||
|
||||
[env:dev]
|
||||
platform = https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip
|
||||
board = esp32-s3-devkitc-1
|
||||
framework = arduino
|
||||
lib_deps =
|
||||
https://github.com/dvarrel/AsyncTCP
|
||||
https://github.com/mathieucarbou/ESPAsyncWebServer
|
||||
ArduinoJson
|
||||
FS
|
||||
ArduinoOTA
|
||||
ESPmDNS
|
||||
|
||||
lib_ldf_mode=deep
|
||||
build_flags=-DFASTLED_STUB_MAIN_INCLUDE_INO="../ci/kitchensink/kitchensink.ino.cpp"
|
||||
@@ -0,0 +1,98 @@
|
||||
#!/bin/bash
|
||||
# FastLED JavaScript Linting Script (Node.js + ESLint + JSDoc Type Checking - FAST!)
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo -e "${BLUE}FAST FastLED JavaScript Linting (Node.js + ESLint + JSDoc Type Checking - FAST!)${NC}"
|
||||
|
||||
# Check if ESLint is installed
|
||||
if [ ! -f ".cache/js-tools/node_modules/.bin/eslint.cmd" ]; then
|
||||
echo -e "${RED}ERROR: ESLint not found. Run: uv run ci/setup-js-linting-fast.py${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Find JavaScript files in WASM platform
|
||||
JS_FILES=$(find src/platforms/wasm -name "*.js" -type f 2>/dev/null)
|
||||
|
||||
if [ -z "$JS_FILES" ]; then
|
||||
echo -e "${YELLOW}WARNING: No JavaScript files found in src/platforms/wasm/${NC}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo -e "${BLUE}Found JavaScript files:${NC}"
|
||||
echo "$JS_FILES" | sed 's/^/ /'
|
||||
|
||||
# Run ESLint using config from ci/ directory
|
||||
echo -e "${BLUE}Running ESLint...${NC}"
|
||||
cd .cache/js-tools
|
||||
if "./node_modules/.bin/eslint.cmd" --no-eslintrc --no-inline-config -c ../../ci/.eslintrc.js ../../src/platforms/wasm/compiler/*.js ../../src/platforms/wasm/compiler/modules/*.js; then
|
||||
echo -e "${GREEN}✅ ESLint completed successfully${NC}"
|
||||
else
|
||||
echo -e "${RED}ERROR: ESLint failed${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Run JSDoc Type Checking using TypeScript
|
||||
echo -e "${BLUE}Running JSDoc Type Checking...${NC}"
|
||||
cd ../../src/platforms/wasm/compiler
|
||||
|
||||
# Check if TypeScript is available (local installation first, then global)
|
||||
if [ -f "../../../../.cache/js-tools/node_modules/.bin/tsc.cmd" ]; then
|
||||
echo -e "${BLUE}Using local TypeScript for JSDoc type checking...${NC}"
|
||||
# Run TypeScript and filter out Node.js events library errors
|
||||
TSC_OUTPUT=$(../../../../.cache/js-tools/node_modules/.bin/tsc.cmd --noEmit --project jsconfig.json 2>&1)
|
||||
TSC_EXIT_CODE=$?
|
||||
# Filter out the specific Node.js events error
|
||||
FILTERED_OUTPUT=$(echo "$TSC_OUTPUT" | grep -v "node_modules/events/events.js.*Property 'context' does not exist on type 'Error'" || true)
|
||||
|
||||
if [ -n "$FILTERED_OUTPUT" ]; then
|
||||
echo "$FILTERED_OUTPUT"
|
||||
if [ $TSC_EXIT_CODE -ne 0 ]; then
|
||||
echo -e "${RED}❌ JSDoc type checking failed with errors${NC}"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
echo -e "${GREEN}✅ JSDoc type checking completed successfully${NC}"
|
||||
elif [ -f "../../../../.cache/js-tools/node_modules/.bin/tsc" ]; then
|
||||
echo -e "${BLUE}Using local TypeScript for JSDoc type checking...${NC}"
|
||||
# Run TypeScript and filter out Node.js events library errors
|
||||
TSC_OUTPUT=$(../../../../.cache/js-tools/node_modules/.bin/tsc --noEmit --project jsconfig.json 2>&1)
|
||||
TSC_EXIT_CODE=$?
|
||||
# Filter out the specific Node.js events error
|
||||
FILTERED_OUTPUT=$(echo "$TSC_OUTPUT" | grep -v "node_modules/events/events.js.*Property 'context' does not exist on type 'Error'" || true)
|
||||
|
||||
if [ -n "$FILTERED_OUTPUT" ]; then
|
||||
echo "$FILTERED_OUTPUT"
|
||||
if [ $TSC_EXIT_CODE -ne 0 ]; then
|
||||
echo -e "${RED}❌ JSDoc type checking failed with errors${NC}"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
echo -e "${GREEN}✅ JSDoc type checking completed successfully${NC}"
|
||||
elif command -v npx >/dev/null 2>&1 && npx tsc --version >/dev/null 2>&1; then
|
||||
echo -e "${BLUE}Using global TypeScript for JSDoc type checking...${NC}"
|
||||
# Run TypeScript and filter out Node.js events library errors
|
||||
TSC_OUTPUT=$(npx tsc --noEmit --project jsconfig.json 2>&1)
|
||||
TSC_EXIT_CODE=$?
|
||||
# Filter out the specific Node.js events error
|
||||
FILTERED_OUTPUT=$(echo "$TSC_OUTPUT" | grep -v "node_modules/events/events.js.*Property 'context' does not exist on type 'Error'" || true)
|
||||
|
||||
if [ -n "$FILTERED_OUTPUT" ]; then
|
||||
echo "$FILTERED_OUTPUT"
|
||||
if [ $TSC_EXIT_CODE -ne 0 ]; then
|
||||
echo -e "${RED}❌ JSDoc type checking failed with errors${NC}"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
echo -e "${GREEN}✅ JSDoc type checking completed successfully${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}⚠️ TypeScript not found, skipping JSDoc type checking${NC}"
|
||||
echo -e "${BLUE}💡 To enable JSDoc checking: Run the ./install script or npm install -g typescript${NC}"
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}SUCCESS: JavaScript linting and type checking completed successfully${NC}"
|
||||
@@ -0,0 +1,38 @@
|
||||
#!/bin/bash
|
||||
# FastLED JavaScript Linting Script (Node.js + ESLint - FAST!)
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo -e "${BLUE}FAST FastLED JavaScript Linting (Node.js + ESLint - FAST!)${NC}"
|
||||
|
||||
# Check if ESLint is installed
|
||||
if [ ! -f ".cache/js-tools/node_modules/.bin/eslint.cmd" ]; then
|
||||
echo -e "${RED}ERROR: ESLint not found. Run: uv run ci/setup-js-linting-fast.py${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Find JavaScript files in WASM platform
|
||||
JS_FILES=$(find src/platforms/wasm -name "*.js" -type f 2>/dev/null)
|
||||
|
||||
if [ -z "$JS_FILES" ]; then
|
||||
echo -e "${YELLOW}WARNING: No JavaScript files found in src/platforms/wasm/${NC}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo -e "${BLUE}Found JavaScript files:${NC}"
|
||||
echo "$JS_FILES" | sed 's/^/ /'
|
||||
|
||||
# Run ESLint using config from ci/ directory
|
||||
echo -e "${BLUE}Running ESLint...${NC}"
|
||||
cd .cache/js-tools
|
||||
if "./node_modules/.bin/eslint.cmd" --no-eslintrc --no-inline-config -c ../../ci/.eslintrc.js ../../src/platforms/wasm/compiler/*.js ../../src/platforms/wasm/compiler/modules/*.js; then
|
||||
echo -e "${GREEN}SUCCESS: JavaScript linting completed successfully${NC}"
|
||||
else
|
||||
echo -e "${RED}ERROR: JavaScript linting failed${NC}"
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,252 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Build System Migration Example
|
||||
|
||||
This shows how to migrate from the old scattered build logic
|
||||
to the new unified BuildAPI.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ci.build_api import create_example_builder, create_unit_test_builder
|
||||
|
||||
|
||||
def old_way_unit_test_building():
|
||||
"""
|
||||
OLD WAY: Scattered logic with manual PCH and library building
|
||||
|
||||
This is how unit test building worked before the refactor:
|
||||
- Manual PCH building in multiple places
|
||||
- Manual library building with duplicated logic
|
||||
- Mixed build flags between unit tests and examples
|
||||
- Manual directory management
|
||||
"""
|
||||
|
||||
# OLD CODE (before refactor) - DO NOT USE
|
||||
"""
|
||||
# Manual PCH building (scattered in multiple files)
|
||||
from ci.compiler.test_example_compilation import create_fastled_compiler
|
||||
from ci.compiler.cpp_test_compile import create_unit_test_fastled_library
|
||||
|
||||
compiler = create_fastled_compiler(use_pch=True, use_sccache=False, parallel=True)
|
||||
|
||||
# Manual library building (duplicated logic)
|
||||
lib_file = create_unit_test_fastled_library(clean=False, use_pch=True)
|
||||
|
||||
# Manual target compilation loop
|
||||
for test_file in test_files:
|
||||
success = compiler.compile_cpp_file(test_file, output_obj)
|
||||
if success:
|
||||
link_executable(output_obj, lib_file, output_exe)
|
||||
"""
|
||||
|
||||
|
||||
def new_way_unit_test_building():
|
||||
"""
|
||||
NEW WAY: Unified BuildAPI with automatic PCH and library building
|
||||
|
||||
This is how unit test building works after the refactor:
|
||||
- Automatic PCH building (once, reused by all targets)
|
||||
- Automatic library building (once, reused by all targets)
|
||||
- Clean separation of unit test build flags
|
||||
- Automatic directory organization
|
||||
"""
|
||||
|
||||
# NEW CODE (after refactor) - USE THIS
|
||||
unit_builder = create_unit_test_builder(
|
||||
build_dir=".build/unit", use_pch=True, parallel=True
|
||||
)
|
||||
|
||||
test_files = [
|
||||
Path("tests/test_json.cpp"),
|
||||
Path("tests/test_color.cpp"),
|
||||
Path("tests/test_math.cpp"),
|
||||
]
|
||||
|
||||
# Single call handles everything:
|
||||
# - PCH building (automatically, once)
|
||||
# - Library building (automatically, once)
|
||||
# - Individual target compilation and linking
|
||||
results = unit_builder.build_targets(test_files)
|
||||
|
||||
# Clean result handling
|
||||
for result in results:
|
||||
if result.success:
|
||||
print(f"✓ {result.target.name} built successfully")
|
||||
else:
|
||||
print(f"✗ {result.target.name} failed: {result.message}")
|
||||
|
||||
|
||||
def old_way_example_building():
|
||||
"""
|
||||
OLD WAY: Mixed build configuration and manual management
|
||||
|
||||
Examples used the same build_flags.toml as unit tests, causing:
|
||||
- Wrong defines (FASTLED_FORCE_NAMESPACE=1 for examples)
|
||||
- Mixed configuration between unit tests and examples
|
||||
- Manual PCH and library building
|
||||
"""
|
||||
|
||||
# OLD CODE (before refactor) - DO NOT USE
|
||||
"""
|
||||
# Used same build_flags.toml as unit tests (wrong!)
|
||||
compiler = create_fastled_compiler(use_pch=True, use_sccache=False, parallel=True)
|
||||
|
||||
# Manual example compilation with wrong defines
|
||||
for example_file in example_files:
|
||||
success = compiler.compile_ino_file(example_file, output_obj)
|
||||
# ... manual linking logic ...
|
||||
"""
|
||||
|
||||
|
||||
def new_way_example_building():
|
||||
"""
|
||||
NEW WAY: Clean separation with dedicated build configuration
|
||||
|
||||
Examples now have their own build_example.toml with:
|
||||
- Correct defines (no FASTLED_FORCE_NAMESPACE=1)
|
||||
- Arduino compatibility defines
|
||||
- Automatic PCH and library building
|
||||
"""
|
||||
|
||||
# NEW CODE (after refactor) - USE THIS
|
||||
example_builder = create_example_builder(
|
||||
build_dir=".build/examples", use_pch=True, parallel=True
|
||||
)
|
||||
|
||||
example_files = [
|
||||
Path("examples/Blink/Blink.ino"),
|
||||
Path("examples/DemoReel100/DemoReel100.ino"),
|
||||
]
|
||||
|
||||
# Single call handles everything with correct configuration
|
||||
results = example_builder.build_targets(example_files)
|
||||
|
||||
for result in results:
|
||||
if result.success:
|
||||
print(f"✓ {result.target.name} built successfully")
|
||||
else:
|
||||
print(f"✗ {result.target.name} failed: {result.message}")
|
||||
|
||||
|
||||
def migration_benefits():
|
||||
"""
|
||||
Benefits of the new unified BuildAPI:
|
||||
"""
|
||||
|
||||
print("MIGRATION BENEFITS:")
|
||||
print("=" * 50)
|
||||
|
||||
print("1. UNIFIED API:")
|
||||
print(
|
||||
" Before: create_fastled_compiler() + create_unit_test_fastled_library() + manual loops"
|
||||
)
|
||||
print(" After: create_unit_test_builder().build_targets(files)")
|
||||
|
||||
print("\n2. CLEAN SEPARATION:")
|
||||
print(" Before: build_flags.toml used by both unit tests and examples")
|
||||
print(" After: build_unit.toml vs build_example.toml")
|
||||
|
||||
print("\n3. AUTOMATIC ARTIFACT BUILDING:")
|
||||
print(" Before: Manual PCH and library building in callers")
|
||||
print(" After: Automatic PCH and library building in BuildAPI")
|
||||
|
||||
print("\n4. ORGANIZED DIRECTORIES:")
|
||||
print(" Before: Mixed artifacts in .build/")
|
||||
print(" After: Organized .build/unit/ and .build/examples/ with subdirectories")
|
||||
|
||||
print("\n5. MULTIPLE TARGET EFFICIENCY:")
|
||||
print(" Before: PCH and library built for each target")
|
||||
print(" After: PCH and library built once, reused by all targets")
|
||||
|
||||
print("\n6. SIMPLE ERROR HANDLING:")
|
||||
print(" Before: Manual error handling in compilation loops")
|
||||
print(" After: Structured SketchResult objects with clear success/failure")
|
||||
|
||||
|
||||
def migration_checklist():
|
||||
"""
|
||||
Checklist for migrating existing code to the new BuildAPI:
|
||||
"""
|
||||
|
||||
print("\nMIGRATION CHECKLIST:")
|
||||
print("=" * 50)
|
||||
|
||||
print("✓ COMPLETED:")
|
||||
print(" - Split ci/build_flags.toml into build_unit.toml and build_example.toml")
|
||||
print(" - Created unified BuildAPI class")
|
||||
print(" - Implemented automatic PCH and library building")
|
||||
print(" - Implemented organized build directory structure")
|
||||
print(
|
||||
" - Created convenience functions create_unit_test_builder() and create_example_builder()"
|
||||
)
|
||||
|
||||
print("\n⚠ TODO:")
|
||||
print(
|
||||
" - Update existing unit test compilation code to use create_unit_test_builder()"
|
||||
)
|
||||
print(
|
||||
" - Update existing example compilation code to use create_example_builder()"
|
||||
)
|
||||
print(" - Remove old scattered PCH and library building logic")
|
||||
print(" - Update test scripts to use new API")
|
||||
print(" - Deprecate old build_flags.toml (marked as legacy)")
|
||||
|
||||
|
||||
def show_api_simplification():
|
||||
"""
|
||||
Show how the API is dramatically simplified:
|
||||
"""
|
||||
|
||||
print("\nAPI SIMPLIFICATION:")
|
||||
print("=" * 50)
|
||||
|
||||
print("BEFORE (scattered across multiple files):")
|
||||
print("""
|
||||
# In ci/compiler/test_example_compilation.py
|
||||
compiler = create_fastled_compiler(use_pch=True, use_sccache=False, parallel=True)
|
||||
|
||||
# In ci/compiler/cpp_test_compile.py
|
||||
lib_file = create_unit_test_fastled_library(clean=False, use_pch=True)
|
||||
|
||||
# In test runner
|
||||
for test_file in test_files:
|
||||
obj_file = build_dir / f"{test_file.stem}.o"
|
||||
success = compiler.compile_cpp_file(test_file, obj_file)
|
||||
if success:
|
||||
exe_file = build_dir / f"{test_file.stem}.exe"
|
||||
link_success = link_executable(obj_file, lib_file, exe_file)
|
||||
""")
|
||||
|
||||
print("AFTER (unified API):")
|
||||
print("""
|
||||
# Single unified API
|
||||
builder = create_unit_test_builder()
|
||||
results = builder.build_targets(test_files)
|
||||
|
||||
# That's it! Everything else is automatic:
|
||||
# - PCH building
|
||||
# - Library building
|
||||
# - Individual compilation
|
||||
# - Linking
|
||||
# - Directory organization
|
||||
# - Error handling
|
||||
""")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("FastLED Build System Migration Example")
|
||||
print("=" * 60)
|
||||
|
||||
migration_benefits()
|
||||
migration_checklist()
|
||||
show_api_simplification()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("NEXT STEPS:")
|
||||
print("1. Update existing unit test compilation to use create_unit_test_builder()")
|
||||
print("2. Update existing example compilation to use create_example_builder()")
|
||||
print("3. Remove old scattered build logic")
|
||||
print("4. Test the new unified system")
|
||||
print("=" * 60)
|
||||
Loaded 100 of 1839 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user