Signed-off-by: myperworld <myperworld@126.com>
This commit is contained in:
myperworld committed
1 parent
0986832fd1
commit
aacd052c06
3403 files changed
+597376
-12
No files matched your search
@@ -0,0 +1,66 @@
|
||||
# Advanced Development with FastLED
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## GDB On Unit Tests
|
||||
|
||||
Yes, we have step through debugging with FastLED.
|
||||
|
||||
* VSCode
|
||||
* Install Plugin: GDB Debugger - Beyond
|
||||
* Navigate to one of the tests in `tests/` and open in
|
||||
* Hit `F5`
|
||||
|
||||
If the Python Debugger pops up, then manually switch the VSCode debugger using `Launch(gdb)`
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Enabling 3-second compile times using our `web-compiler`
|
||||
|
||||
* You must have `docker` installed for fastest compile times. It's free.
|
||||
* `cd <FASTLED FOLDER>`
|
||||
* `pip install fastled`
|
||||
* `fastled examples/Blink/Blink.ino`
|
||||
* Cpp changes to the fastled source can be compiled by the live fastled compiler.
|
||||
|
||||
## Testing your changes
|
||||
|
||||
Most of this is in the basic CONTRIBUTING.md guide. But as a reminder
|
||||
* Unit Testing: `./test`
|
||||
* Linting: `./lint`
|
||||
* Compiling on platforms `./compile uno,teensy41,esp32s3 --examples Blink,Apa102HD`
|
||||
|
||||
## Enabling AI coding
|
||||
|
||||
`aider.chat` is available for advanced and high velocity coding with FastLED. To use it, have your open-ai or Anthropic api key ready. It's recommended to use Anthropic as it's performance is much better than Open-AI for coding.
|
||||
|
||||
At the root of the project type:
|
||||
|
||||
`./ai` and follow the prompts. Once the key is installed you will get a prompt that looks like this:
|
||||
|
||||
```bash
|
||||
architect>
|
||||
```
|
||||
|
||||
There are two modes to use this AI, a watch mode which watches your files for changes and launches automatically, and a slow way which you add target files then instruct it to make changes:
|
||||
|
||||
* watch mode (best for implimenting a function or two)
|
||||
* Edit any file in the repo. Add a comment with `AI!` at the end. The ai will see this and start implementing what you just typed.
|
||||
* Example: Edit `src/fl/vector.h` and put in a comment `// Add more comments AI!`, then say yes to the changes in the prompt.
|
||||
* Slow mode (much better for bigger changes across the file or several)
|
||||
* While you are in the `architect> ` prompt you will add a file to the chat
|
||||
* `/add src/fl/vector.h`
|
||||
* Now tell the AI what you want it to do, and it will do it.
|
||||
* Making the AI fix it's own problems it introduced.
|
||||
* At the AI command prompt, have it run the following
|
||||
* Linux/Mac: `/run ./test`
|
||||
* On Windows: `/run uv run test.py`
|
||||
* After the test concludes, the AI will ask you if you want to add the output back into the chat, agree to it then let it try to correct it's mistakes.
|
||||
|
||||
Every single time you do a change, make sure and thoroughly check it. I recommend VSCodes built in git diff tool.
|
||||
|
||||
Although the AI is pretty amazing, it will inject entropy into your code and this is the source of a lot of problems. So watch all changes it makes very thoroughly. Under almost all circumstances you will have to revert unnecessary changes (like comments) line by line.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Special Notes on APA102 and the 'High Definition' Mode in FastLED
|
||||
|
||||
The APA102 LED driver includes a 5-bit per-LED brightness component. Previously, this feature was not fully utilized, except through a workaround that defined a global brightness affecting all LEDs uniformly rather than individually.
|
||||
|
||||
In FastLED the APA102 chipset will have extra resolution in comparison to the WS2812 RGB8 mode.
|
||||
|
||||
There are two modes:
|
||||
* APA102 "Regular Mode"
|
||||
* Has enhanced color resolution when using the "global brightness" factor
|
||||
* APA102HD Mode
|
||||
* Applies automatic gamma correction at the driver level using "pseudo 13-bit" color mixing.
|
||||
|
||||
**APA102HD Mode**
|
||||
|
||||
[example: examples/APA102HD](examples/Apa102HD/)
|
||||
|
||||
By introducing a 5-bit gamma bit-shift algorithm, we now effectively leverage this per-LED brightness control. Faced with the decision to either rewrite the entire `CRGB` library to expose the 5-bit brightness—including adaptations for formats like RGBW—or to retain the existing RGB8 format used by FastLED and implement the enhancement at the driver level, the latter option was chosen. This approach avoids widespread changes and maintains compatibility; if RGB8 suffices for game development, it is adequate for LED development as well.
|
||||
|
||||
The term "Pseudo-13-bit" arises because the additional resolution becomes significant only when all color components are at low values. For example, colors like `CRGB(255, 255, 254)` or `CRGB(255, 1, 1)` do not benefit from increased resolution due to the dominance of the brighter components. However, in low-light conditions with colors such as `CRGB(8, 8, 8)`, where the maximum component value is low, the pseudo-13-bit algorithm significantly enhances resolution—precisely where increased resolution is most desired.
|
||||
|
||||
Gamma correction is applied to preserve the RGB8 format and because future LEDs are expected to support gamma correction inherently. In game development, the 0-255 color values are based on the gamma scale rather than the linear power scale. LEDs like the WS2812 operate on a linear power scale, which results in washed-out, undersaturated colors when displaying captured video directly. Implementing software gamma correction for RGB8 severely reduces color resolution.
|
||||
|
||||
To address this, an internal gamma scale mapping is applied:
|
||||
|
||||
```
|
||||
RGB8 → RGB16 + 5-bit gamma → RGB8 + 5-bit gamma
|
||||
```
|
||||
|
||||
During the conversion back to RGB8, the brightness from the 5-bit gamma is optimally distributed using a closed-form mathematical algorithm. Rather than the iterative bit-shifting approach used in earlier versions, the current implementation uses a sophisticated quantization and scaling method that provides superior accuracy.
|
||||
|
||||
**Closed-Form Algorithm (Current Implementation)**
|
||||
|
||||
The algorithm works by:
|
||||
|
||||
1. **Finding the maximum component**: Determines which RGB component requires the highest brightness level
|
||||
2. **Quantizing to 5-bit scale**: Computes the optimal 5-bit brightness value that accommodates the maximum component
|
||||
3. **Applying pre-computed scaling**: Uses a lookup table of precise scaling factors to convert the 16-bit gamma-corrected values back to 8-bit RGB components
|
||||
|
||||
The key insight is the quantization formula:
|
||||
```
|
||||
scale = (max_component + (2047 - (max_component >> 5))) >> 11
|
||||
```
|
||||
|
||||
This ensures the 5-bit brightness value is always sufficient to represent the maximum component while minimizing quantization error.
|
||||
|
||||
The scaling factors in the lookup table are computed as:
|
||||
```
|
||||
bright_scale[i] = (i/31) * (255/65536) * 256
|
||||
```
|
||||
|
||||
These pre-computed values eliminate floating-point arithmetic while maintaining precision. The final RGB values are calculated with proper rounding:
|
||||
```
|
||||
rgb_out = (rgb16 * scale_factor + 0x808000) >> 24
|
||||
```
|
||||
|
||||
**Algorithm Performance**
|
||||
|
||||
This closed-form approach delivers exceptional accuracy:
|
||||
- **Worst-case error**: 0.2% over the full 16-bit input range
|
||||
- **Average error**: 0.05% across the range
|
||||
- **Perfect match**: to floating-point reference implementation
|
||||
|
||||
By eliminating iterative bit-shifting, this method also provides consistent performance regardless of input values and significantly reduces code size on embedded platforms.
|
||||
|
||||
**Version History**
|
||||
|
||||
In version 3.9.0, the algorithm was completely rewritten to function natively on 8-bit controllers like the `__AVR__` chipsets without significant performance loss. The current closed-form implementation replaced the previous iterative approach, providing both better accuracy and smaller code footprint.
|
||||
|
||||
**Further Enhancements in Version 3.9.0**
|
||||
|
||||
Additionally, version 3.9.0 separated the color temperature from the global brightness scale. Before this update, global brightness was pre-mixed with the component scales—a method suitable for the WS2812's RGB8 format but not for the APA102's RGB8 plus 5-bit brightness. The update saw the global brightness and color scales separated for non-AVR chipsets. While the WS2812 continues to use pre-mixed values for performance reasons on AVR chipsets, the APA102 now performs component mixing within the "pseudo-13-bit space."
|
||||
|
||||
Although APA102HD mode offers the highest dynamic range, the standard APA102 mode also benefits from increased resolution when adjusting global brightness. In this mode, instead of pre-mixing scales and multiplying them against each `CRGB` value, the global brightness is applied to the 5-bit brightness component, and only the color scales are multiplied against the `CRGB` values. This approach is superior because each component of the color scale typically exceeds 127, providing ample high-order bits to preserve color information.
|
||||
@@ -0,0 +1,168 @@
|
||||
# 🚨🚨🚨 **CRITICAL REQUIREMENTS FOR BACKGROUND AGENTS** 🚨🚨🚨
|
||||
|
||||
# ⚠️ **STOP! READ THIS FIRST!** ⚠️
|
||||
|
||||
## **MANDATORY TESTING REQUIREMENT - NO EXCEPTIONS!**
|
||||
|
||||
### 🔥 **RULE #1: ALWAYS RUN `bash test` BEFORE COMPLETION** 🔥
|
||||
|
||||
**THIS IS THE MOST IMPORTANT RULE FOR BACKGROUND AGENTS:**
|
||||
|
||||
- **🚨 EVERY BACKGROUND AGENT MUST RUN `bash test` BEFORE SAYING THEY ARE DONE**
|
||||
- **🚨 NO EXCEPTIONS - THIS IS MANDATORY AND NON-NEGOTIABLE**
|
||||
- **🚨 IF YOU DON'T RUN `bash test`, YOUR WORK IS INCOMPLETE**
|
||||
- **🚨 SAYING "DONE" WITHOUT RUNNING `bash test` IS BREAKING THE RULES**
|
||||
|
||||
### **WHY THIS IS CRITICAL:**
|
||||
- The `bash test` command runs the full test suite including unit tests and compilation checks
|
||||
- It catches breaking changes, compilation errors, and regressions
|
||||
- It ensures the codebase remains stable and functional
|
||||
- **WITHOUT THIS, YOU WILL BREAK THE PROJECT**
|
||||
|
||||
### **WHAT `bash test` DOES:**
|
||||
- Runs all C++ unit tests
|
||||
- Runs Python linting and formatting checks
|
||||
- Validates code compilation across platforms
|
||||
- Checks for syntax errors and warnings
|
||||
- Ensures all examples still compile
|
||||
|
||||
### 🚨 **MANDATORY STEPS FOR COMPLETION - FOLLOW EXACTLY:** 🚨
|
||||
|
||||
1. **🔥 RUN `bash test` COMMAND**
|
||||
- Execute: `bash test`
|
||||
- Wait for ALL tests to complete
|
||||
- Check that ALL tests PASS
|
||||
|
||||
2. **🔥 VERIFY TEST RESULTS**
|
||||
- If ANY test fails, you MUST fix the issues
|
||||
- Do NOT ignore warnings or errors
|
||||
- Do NOT proceed until ALL tests pass
|
||||
|
||||
3. **🔥 USE MCP SERVER VALIDATION (RECOMMENDED)**
|
||||
- Use the `validate_completion` tool from MCP server: `uv run mcp_server.py`
|
||||
- This tool runs `bash test` and validates results
|
||||
- Only indicate completion if this tool returns SUCCESS
|
||||
|
||||
4. **🔥 ZERO TOLERANCE FOR FAILURES**
|
||||
- If ANY test fails, the background agent MUST fix the issues
|
||||
- Do NOT indicate completion with failing tests
|
||||
- Do NOT ignore test errors or warnings
|
||||
- Fix the code and re-run `bash test` until ALL tests pass
|
||||
|
||||
### **COMPLETION CHECKLIST - MUST COMPLETE ALL ITEMS:**
|
||||
- [ ] ✅ All code changes have been made
|
||||
- [ ] ✅ **`bash test` has been run and ALL tests PASS**
|
||||
- [ ] ✅ No compilation errors or warnings remain
|
||||
- [ ] ✅ MCP server `validate_completion` tool shows SUCCESS (if used)
|
||||
- [ ] ✅ All test output has been reviewed for any issues
|
||||
- [ ] ✅ **ONLY THEN** indicate task completion
|
||||
|
||||
### 🔥 **CONSEQUENCES OF NOT FOLLOWING THIS RULE:** 🔥
|
||||
|
||||
- **YOUR CHANGES WILL BREAK THE PROJECT**
|
||||
- **OTHER DEVELOPERS WILL HAVE TO FIX YOUR MISTAKES**
|
||||
- **THE CONTINUOUS INTEGRATION SYSTEM WILL FAIL**
|
||||
- **YOU WILL CAUSE REGRESSIONS AND BUGS**
|
||||
|
||||
### 🚨 **FINAL WARNING:** 🚨
|
||||
|
||||
**IF YOU ARE A BACKGROUND AGENT AND YOU INDICATE COMPLETION WITHOUT RUNNING `bash test`, YOU HAVE FAILED TO FOLLOW THE MOST BASIC REQUIREMENT OF THIS PROJECT.**
|
||||
|
||||
**NO EXCEPTIONS. NO SHORTCUTS. ALWAYS RUN `bash test` FIRST.**
|
||||
|
||||
---
|
||||
|
||||
## **Examples of CORRECT Background Agent Behavior:**
|
||||
|
||||
### ✅ **GOOD Example:**
|
||||
```
|
||||
1. Made code changes
|
||||
2. Ran `bash test`
|
||||
3. All tests passed
|
||||
4. Indicated completion: "Task completed successfully. All tests pass."
|
||||
```
|
||||
|
||||
### ❌ **BAD Example:**
|
||||
```
|
||||
1. Made code changes
|
||||
2. Skipped running tests
|
||||
3. Indicated completion: "Task completed successfully."
|
||||
```
|
||||
**^ THIS IS WRONG AND BREAKS THE PROJECT**
|
||||
|
||||
### ✅ **GOOD Example with Test Failures:**
|
||||
```
|
||||
1. Made code changes
|
||||
2. Ran `bash test`
|
||||
3. Some tests failed
|
||||
4. Fixed the failing code
|
||||
5. Ran `bash test` again
|
||||
6. All tests passed
|
||||
7. Indicated completion: "Task completed successfully. All tests pass."
|
||||
```
|
||||
|
||||
### ❌ **BAD Example with Test Failures:**
|
||||
```
|
||||
1. Made code changes
|
||||
2. Ran `bash test`
|
||||
3. Some tests failed
|
||||
4. Ignored failures and indicated completion anyway
|
||||
```
|
||||
**^ THIS IS WRONG AND BREAKS THE PROJECT**
|
||||
|
||||
---
|
||||
|
||||
## **HOW TO RUN `bash test`:**
|
||||
|
||||
### **Command:**
|
||||
```bash
|
||||
bash test
|
||||
```
|
||||
|
||||
### **Expected Output (Success):**
|
||||
```
|
||||
Running C++ tests...
|
||||
All C++ tests passed ✅
|
||||
Running Python linting...
|
||||
All Python checks passed ✅
|
||||
Running compilation checks...
|
||||
All compilation checks passed ✅
|
||||
```
|
||||
|
||||
### **What to do if tests fail:**
|
||||
1. **READ THE ERROR MESSAGES CAREFULLY**
|
||||
2. **FIX THE CODE THAT'S CAUSING THE FAILURES**
|
||||
3. **RUN `bash test` AGAIN**
|
||||
4. **REPEAT UNTIL ALL TESTS PASS**
|
||||
5. **ONLY THEN INDICATE COMPLETION**
|
||||
|
||||
---
|
||||
|
||||
## **MCP SERVER VALIDATION TOOL:**
|
||||
|
||||
### **How to use:**
|
||||
```bash
|
||||
# Start MCP server
|
||||
uv run mcp_server.py
|
||||
|
||||
# Use the validate_completion tool
|
||||
# This automatically runs `bash test` and validates results
|
||||
```
|
||||
|
||||
### **Benefits:**
|
||||
- Automatically runs `bash test`
|
||||
- Provides clear pass/fail results
|
||||
- Ensures you don't forget to run tests
|
||||
- **RECOMMENDED FOR ALL BACKGROUND AGENTS**
|
||||
|
||||
---
|
||||
|
||||
## 🚨 **REMEMBER: THIS IS NOT OPTIONAL** 🚨
|
||||
|
||||
**EVERY BACKGROUND AGENT WORKING ON THIS PROJECT MUST:**
|
||||
1. **RUN `bash test` BEFORE COMPLETION**
|
||||
2. **ENSURE ALL TESTS PASS**
|
||||
3. **FIX ANY FAILURES**
|
||||
4. **ONLY THEN INDICATE COMPLETION**
|
||||
|
||||
**NO SHORTCUTS. NO EXCEPTIONS. ALWAYS TEST FIRST.**
|
||||
@@ -0,0 +1,39 @@
|
||||
# FastLED AI Agent Guidelines
|
||||
|
||||
## Quick Reference
|
||||
|
||||
This project uses directory-specific agent guidelines. See:
|
||||
|
||||
- **CI/Build Tasks**: `ci/AGENTS.md` - Python build system, compilation, MCP server tools
|
||||
- **Testing**: `tests/AGENTS.md` - Unit tests, test execution, validation requirements
|
||||
- **Examples**: `examples/AGENTS.md` - Arduino sketch compilation, .ino file rules
|
||||
|
||||
## Key Commands
|
||||
|
||||
- `uv run test.py` - Run all tests
|
||||
- `uv run test.py --cpp` - Run C++ tests only
|
||||
- `uv run test.py TestName` - Run specific C++ test (e.g., `uv run test.py xypath`)
|
||||
- `bash lint` - Run code formatting/linting
|
||||
- `uv run ci/ci-compile.py uno --examples Blink` - Compile examples for specific platform
|
||||
- `uv run ci/wasm_compile.py examples/Blink --just-compile` - Compile Arduino sketches to WASM
|
||||
- `uv run mcp_server.py` - Start MCP server for advanced tools
|
||||
|
||||
## Core Rules
|
||||
|
||||
### Command Execution (ALL AGENTS)
|
||||
- **Python**: Always use `uv run python script.py` (never just `python`)
|
||||
- **Stay in project root** - never `cd` to subdirectories
|
||||
- **Git-bash compatibility**: Prefix commands with space: `bash test`
|
||||
|
||||
### C++ Code Standards
|
||||
- **Use `fl::` namespace** instead of `std::`
|
||||
- **If you want to use a stdlib header like <type_traits>, look check for equivalent in `fl/type_traits.h`
|
||||
- **Use proper warning macros** from `fl/compiler_control.h`
|
||||
- **Follow existing code patterns** and naming conventions
|
||||
|
||||
### Memory Refresh Rule
|
||||
**🚨 ALL AGENTS: Read the relevant AGENTS.md file before concluding work to refresh memory about current project rules and requirements.**
|
||||
|
||||
---
|
||||
|
||||
*This file intentionally kept minimal. Detailed guidelines are in directory-specific AGENTS.md files.*
|
||||
@@ -0,0 +1,25 @@
|
||||
# FastLED
|
||||
# https://github.com/FastLED/FastLED
|
||||
# MIT License
|
||||
|
||||
cmake_minimum_required(VERSION 3.5)
|
||||
|
||||
# Collect all source files
|
||||
file(GLOB FastLED_SRCS "src/*.cpp")
|
||||
file(GLOB FastLED_FL_SRCS "src/fl/*.cpp")
|
||||
file(GLOB FastLED_SENSORS_SRCS "src/sensors/*.cpp")
|
||||
file(GLOB FastLED_FX_SRCS "src/fx/*.cpp" "src/fx/**/*.cpp")
|
||||
|
||||
file(GLOB ESP32_SRCS "src/platforms/esp/32/*.cpp" "src/platforms/esp/32/rmt_5/*.cpp")
|
||||
file(GLOB ESP32_THIRD_PARTY_SRCS "src/third_party/**/src/*.c" "src/third_party/**/src/*.cpp")
|
||||
file(GLOB ESP32_LED_STRIP_SRCS "src/third_party/espressif/led_strip/src/*.c")
|
||||
|
||||
# Combine all source files into a single list
|
||||
list(APPEND FastLED_SRCS ${FastLED_FL_SRCS} ${FastLED_SENSORS_SRCS} ${FastLED_FX_SRCS} ${ESP32_SRCS} ${ESP32_THIRD_PARTY_SRCS} ${ESP32_LED_STRIP_SRCS})
|
||||
|
||||
# Register the component with ESP-IDF
|
||||
idf_component_register(SRCS ${FastLED_SRCS}
|
||||
INCLUDE_DIRS "src" "src/third_party/espressif/led_strip/src"
|
||||
REQUIRES arduino-esp32 esp_driver_rmt esp_lcd driver)
|
||||
|
||||
project(FastLED)
|
||||
@@ -0,0 +1,193 @@
|
||||
## Contributing
|
||||
|
||||
The most important part about contributing to FastLED is knowing how to test your changes.
|
||||
|
||||
The FastLED library includes a powerful cli that can compile to any device. It will run if you have either [python](https://www.python.org/downloads/) or [uv](https://github.com/astral-sh/uv) installed on the system.
|
||||
|
||||
## FastLED compiler cli
|
||||
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_clone_and_compile.yml)
|
||||
|
||||
The FastLED compiler cli can be invoked at the project root.
|
||||
|
||||
```bash (MacOS/Linux, windows us git-bsh or compile.bat)
|
||||
git clone https://github.com/fastled/fastled
|
||||
cd fastled
|
||||
./compile uno --examples Blink # linux/macos/git-bash
|
||||
# compile.bat # Windows.
|
||||
```
|
||||
|
||||
## Linting and Unit Testing
|
||||
|
||||
```bash
|
||||
./lint
|
||||
./test # runs unit tests
|
||||
# Note that you do NOT need to install the C++ compiler toolchain
|
||||
# for compiling + running unit tests via ./test. If `gcc` is not
|
||||
# found in your system `PATH` then the `ziglang` clang compiler
|
||||
# will be swapped in automatically.
|
||||
````
|
||||
|
||||
|
||||
### Testing a bunch of platforms at once.
|
||||
|
||||
```
|
||||
./compile teensy41,teensy40 --examples Blink
|
||||
./compile esp32dev,esp32s3,esp32c3,esp32c6,esp32s2 --examples Blink,Apa102HD
|
||||
./compiles uno,digix,attiny85 --examples Blink,Apa102HD
|
||||
```
|
||||
|
||||
## Unit Tests
|
||||
|
||||
Shared code is unit-tested on the host machine. They can be found at `tests/` at the root of the repo. Unit testing only requires either `python` or `uv` to be installed. The C++ compiler toolchain will be installed automatically.
|
||||
|
||||
The easiest way to run the tests is just use `./test`
|
||||
|
||||
Alternatively, tests can be built and run for your development machine with CMake:
|
||||
|
||||
```bash
|
||||
cmake -S tests -B tests/.build
|
||||
ctest --test-dir tests/.build --output-on-failure
|
||||
# Not that this will fail if you do not have gcc installed. When in doubt
|
||||
# use ./test to compile the unit tests, as a compiler is guaranteed to be
|
||||
# available via this tool.
|
||||
```
|
||||
|
||||
## VSCode
|
||||
|
||||
We also support VSCode and IntelliSense auto-completion when the free [platformio](https://marketplace.visualstudio.com/items?itemName=platformio.platformio-ide) extension is installed. The development sketch to test library changes can be found at [dev/dev.ino](dev/dev.ino).
|
||||
|
||||
* Make sure you have [platformio](https://marketplace.visualstudio.com/items?itemName=platformio.platformio-ide) installed.
|
||||
* Click the compile button.
|
||||
|
||||
|
||||
*Changes in non platform specific code can be tested quickly in our webcompiler by invoking the script `./wasm` at the project root*
|
||||
|
||||
## VSCode Debugging Guide
|
||||
|
||||
FastLED includes comprehensive VSCode debugging support with GDB and Clang-generated debug symbols, providing professional-grade step-through debugging capabilities for the test suite.
|
||||
|
||||
### Quick Start
|
||||
|
||||
1. **Prerequisites**: Ensure you have VSCode with the C/C++ extension installed
|
||||
2. **Open a test file**: e.g., `tests/test_allocator.cpp`
|
||||
3. **Set breakpoints**: Click the left margin next to line numbers
|
||||
4. **Press F5**: Automatically debugs the corresponding test executable
|
||||
5. **Debug**: Use F10 (step over), F11 (step into), F5 (continue)
|
||||
|
||||
### Available Debug Configurations
|
||||
|
||||
#### 🎯 **"Debug FastLED Test (Current File)"** ⭐ *Most Common*
|
||||
- **When to use**: When you have any test file open (e.g., `test_allocator.cpp`)
|
||||
- **What it does**: Automatically detects and debugs the corresponding test executable
|
||||
- **How to use**: Open any `test_*.cpp` file and press `F5`
|
||||
|
||||
#### 🔧 **Specific Test Configurations**
|
||||
- **"Debug test_allocator"** - Memory allocation and deallocation debugging
|
||||
- **"Debug test_math"** - Mathematical functions and color calculations
|
||||
- **"Debug test_fastled"** - Core FastLED functionality and API
|
||||
- **"Debug test_hsv16"** - Color space conversions and accuracy
|
||||
- **"Debug test_corkscrew"** - LED layout and geometric calculations
|
||||
|
||||
#### 🎛️ **Advanced Configurations**
|
||||
- **"Debug with Specific Test Filter"** - Run only specific test cases
|
||||
- Example: Filter "allocator_inlined" to debug only those tests
|
||||
- **"Debug with Custom Args"** - Pass custom command-line arguments
|
||||
- Example: `--verbose`, `--list-test-cases`, etc.
|
||||
|
||||
### Debugging Features
|
||||
|
||||
#### Step Commands
|
||||
- **F5**: Continue execution
|
||||
- **F10**: Step over (execute current line)
|
||||
- **F11**: Step into (enter function calls)
|
||||
- **Shift+F11**: Step out (exit current function)
|
||||
- **Ctrl+Shift+F5**: Restart debugging
|
||||
- **Shift+F5**: Stop debugging
|
||||
|
||||
#### Variable Inspection
|
||||
- **Variables panel**: See all local variables and their values
|
||||
- **Watch panel**: Add expressions to monitor continuously
|
||||
- **Hover inspection**: Hover over variables to see values
|
||||
- **Debug console**: Type expressions to evaluate
|
||||
|
||||
### Build Tasks for Debugging
|
||||
|
||||
Access via `Ctrl+Shift+P` → "Tasks: Run Task":
|
||||
|
||||
- **"Build FastLED Tests"** - Quick incremental build (default: `Ctrl+Shift+B`)
|
||||
- **"Build FastLED Tests (Full)"** - Complete build including Python tests
|
||||
- **"Build Single Test"** - Build and run specific test only
|
||||
- **"Clean Build"** - Remove all build artifacts and rebuild
|
||||
|
||||
### Debugging Tips for FastLED
|
||||
|
||||
#### Memory Issues
|
||||
```cpp
|
||||
// Use test_allocator for memory debugging
|
||||
// Set breakpoints on allocate/deallocate functions
|
||||
// Watch pointer values in Variables panel
|
||||
// Monitor memory patterns for corruption
|
||||
```
|
||||
|
||||
#### Color Conversion Issues
|
||||
```cpp
|
||||
// Use test_hsv16 for color space debugging
|
||||
// Watch RGB/HSV values during conversion
|
||||
// Step through algorithms with F11
|
||||
// Compare expected vs actual in Watch panel
|
||||
```
|
||||
|
||||
#### Template Debugging
|
||||
```cpp
|
||||
// Clang generates excellent template debug info
|
||||
// Step into template functions with F11
|
||||
// Watch template parameters in Variables panel
|
||||
// Use Call Stack to understand instantiation chain
|
||||
```
|
||||
|
||||
### Technical Setup
|
||||
|
||||
#### Clang + GDB Benefits
|
||||
This setup provides the **best of both worlds**:
|
||||
- **Clang's superior symbol generation**: Better template debugging, modern C++ support
|
||||
- **GDB's mature debugging features**: Robust breakpoint handling, memory inspection
|
||||
- **Cross-platform compatibility**: Works on Linux, macOS, Windows
|
||||
- **FastLED optimization**: Unified compilation testing with `FASTLED_ALL_SRC=1`
|
||||
|
||||
#### Debug Build Configuration
|
||||
The FastLED test system automatically uses optimal debug settings:
|
||||
- **Compiler**: Clang (when available) or GCC fallback
|
||||
- **Debug info**: `-g3` (full debug information including macros)
|
||||
- **Optimization**: `-O0` (no optimization for accurate debugging)
|
||||
- **Frame pointers**: `-fno-omit-frame-pointer` (for accurate stack traces)
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
#### "Program not found" Error
|
||||
1. **Build first**: Run "Build FastLED Tests" task
|
||||
2. **Check executable exists**: `ls tests/.build/bin/test_*`
|
||||
3. **Verify path**: Ensure executable path in launch.json is correct
|
||||
|
||||
#### Breakpoints Not Hit
|
||||
1. **Check file paths**: Ensure source file matches executable
|
||||
2. **Verify compilation**: Code might be optimized out
|
||||
3. **Try function breakpoints**: Sometimes more reliable than line breakpoints
|
||||
|
||||
#### Variables Show "Optimized Out"
|
||||
1. **Use debug build**: Already configured with `-O0` (no optimization)
|
||||
2. **Check variable scope**: Variable might be out of scope
|
||||
3. **Try different breakpoint**: Move breakpoint to where variable is active
|
||||
|
||||
For complete debugging documentation, see [DEBUGGING.md](DEBUGGING.md).
|
||||
|
||||
|
||||
## Once you are done
|
||||
* run `./test`
|
||||
* run `./lint`
|
||||
* Then submit your code via a git pull request.
|
||||
|
||||
|
||||
## Going deeper
|
||||
|
||||
[ADVANCED_DEVELOPMENT.md](https://github.com/FastLED/FastLED/blob/master/ADVANCED_DEVELOPMENT.md)
|
||||
@@ -0,0 +1,203 @@
|
||||
# Corkscrew Pipeline: computeTile and multiSample Integration
|
||||
|
||||
## Overview
|
||||
|
||||
The FastLED corkscrew allows the user to write to a regular rectangular buffer and have it displayd on a dense corkscrew of LEDs.
|
||||
|
||||
Dense 144LED @ 3.28 are cheap and readily avialable. They are cheap, have high density.
|
||||
|
||||
## Pipeline Components
|
||||
|
||||
### 1. User Paints to XY Grid
|
||||
|
||||
Users create patterns on a 2D rectangular grid (`fl::Grid<CRGB>`) using standard XY coordinates.
|
||||
|
||||
```cpp
|
||||
// Grid dimensions calculated from corkscrew parameters
|
||||
uint16_t width = input.calculateWidth(); // LEDs per turn
|
||||
uint16_t height = input.calculateHeight(); // Total vertical segments
|
||||
fl::Grid<CRGB> sourceGrid(width, height);
|
||||
```
|
||||
|
||||
### 2. LED Projection: Corkscrew → XY Grid
|
||||
|
||||
Each LED in the corkscrew has a corresponding floating-point position on the XY grid:
|
||||
|
||||
```cpp
|
||||
// From corkscrew.cpp - calculateLedPositionExtended
|
||||
vec2f calculateLedPosition(uint16_t ledIndex, uint16_t numLeds, uint16_t width) {
|
||||
const float ledProgress = static_cast<float>(ledIndex) / static_cast<float>(numLeds - 1);
|
||||
const uint16_t row = ledIndex / width; // Which turn (vertical position)
|
||||
const uint16_t remainder = ledIndex % width; // Position within turn
|
||||
const float alpha = static_cast<float>(remainder) / static_cast<float>(width);
|
||||
|
||||
const float width_pos = ledProgress * numLeds;
|
||||
const float height_pos = static_cast<float>(row) + alpha;
|
||||
|
||||
return vec2f(width_pos, height_pos);
|
||||
}
|
||||
```
|
||||
|
||||
### 3. computeTile: Splat Pixel Rendering
|
||||
|
||||
The `splat` function implements the "computeTile" concept by converting floating-point positions to `Tile2x2_u8` structures representing neighbor intensities:
|
||||
|
||||
```cpp
|
||||
// From splat.cpp
|
||||
Tile2x2_u8 splat(vec2f xy) {
|
||||
// 1) Get integer cell indices
|
||||
int16_t cx = static_cast<int16_t>(floorf(xy.x));
|
||||
int16_t cy = static_cast<int16_t>(floorf(xy.y));
|
||||
|
||||
// 2) Calculate fractional offsets [0..1)
|
||||
float fx = xy.x - cx;
|
||||
float fy = xy.y - cy;
|
||||
|
||||
// 3) Compute bilinear weights for 4 neighbors
|
||||
float w_ll = (1 - fx) * (1 - fy); // lower-left
|
||||
float w_lr = fx * (1 - fy); // lower-right
|
||||
float w_ul = (1 - fx) * fy; // upper-left
|
||||
float w_ur = fx * fy; // upper-right
|
||||
|
||||
// 4) Build Tile2x2_u8 with weights as intensities [0..255]
|
||||
Tile2x2_u8 out(vec2<int16_t>(cx, cy));
|
||||
out.lower_left() = to_uint8(w_ll);
|
||||
out.lower_right() = to_uint8(w_lr);
|
||||
out.upper_left() = to_uint8(w_ul);
|
||||
out.upper_right() = to_uint8(w_ur);
|
||||
|
||||
return out;
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Tile2x2_u8: Neighbor Intensity Representation
|
||||
|
||||
The `Tile2x2_u8` structure represents the sampling strength from the four nearest neighbors:
|
||||
|
||||
```cpp
|
||||
class Tile2x2_u8 {
|
||||
uint8_t mTile[2][2]; // 4 neighbor intensities [0..255]
|
||||
vec2<int16_t> mOrigin; // Base grid coordinate (cx, cy)
|
||||
|
||||
// Access methods for the 4 neighbors:
|
||||
uint8_t& lower_left(); // (0,0) - weight for pixel at (cx, cy)
|
||||
uint8_t& lower_right(); // (1,0) - weight for pixel at (cx+1, cy)
|
||||
uint8_t& upper_left(); // (0,1) - weight for pixel at (cx, cy+1)
|
||||
uint8_t& upper_right(); // (1,1) - weight for pixel at (cx+1, cy+1)
|
||||
};
|
||||
```
|
||||
|
||||
### 5. Cylindrical Wrapping with Tile2x2_u8_wrap
|
||||
|
||||
For corkscrew mapping, the tile needs cylindrical wrapping:
|
||||
|
||||
```cpp
|
||||
// From corkscrew.cpp - at_wrap()
|
||||
Tile2x2_u8_wrap Corkscrew::at_wrap(float i) const {
|
||||
Tile2x2_u8 tile = at_splat_extrapolate(i); // Get base tile
|
||||
Tile2x2_u8_wrap::Entry data[2][2];
|
||||
vec2i16 origin = tile.origin();
|
||||
|
||||
for (uint8_t x = 0; x < 2; x++) {
|
||||
for (uint8_t y = 0; y < 2; y++) {
|
||||
vec2i16 pos = origin + vec2i16(x, y);
|
||||
// Apply cylindrical wrapping to x-coordinate
|
||||
pos.x = fmodf(pos.x, static_cast<float>(mState.width));
|
||||
data[x][y] = {pos, tile.at(x, y)}; // {position, intensity}
|
||||
}
|
||||
}
|
||||
return Tile2x2_u8_wrap(data);
|
||||
}
|
||||
```
|
||||
|
||||
### 6. multiSample: Weighted Color Sampling
|
||||
|
||||
The `readFromMulti` method implements the "multiSample" concept by using tile intensities to determine sampling strength:
|
||||
|
||||
```cpp
|
||||
// From corkscrew.cpp - readFromMulti()
|
||||
void Corkscrew::readFromMulti(const fl::Grid<CRGB>& source_grid) const {
|
||||
for (size_t led_idx = 0; led_idx < mInput.numLeds; ++led_idx) {
|
||||
// Get wrapped tile for this LED position
|
||||
Tile2x2_u8_wrap tile = at_wrap(static_cast<float>(led_idx));
|
||||
|
||||
uint32_t r_accum = 0, g_accum = 0, b_accum = 0;
|
||||
uint32_t total_weight = 0;
|
||||
|
||||
// Sample from each of the 4 neighbors
|
||||
for (uint8_t x = 0; x < 2; x++) {
|
||||
for (uint8_t y = 0; y < 2; y++) {
|
||||
const auto& entry = tile.at(x, y);
|
||||
vec2i16 pos = entry.first; // Grid position
|
||||
uint8_t weight = entry.second; // Sampling intensity [0..255]
|
||||
|
||||
if (inBounds(source_grid, pos)) {
|
||||
CRGB sample_color = source_grid.at(pos.x, pos.y);
|
||||
|
||||
// Weighted accumulation
|
||||
r_accum += static_cast<uint32_t>(sample_color.r) * weight;
|
||||
g_accum += static_cast<uint32_t>(sample_color.g) * weight;
|
||||
b_accum += static_cast<uint32_t>(sample_color.b) * weight;
|
||||
total_weight += weight;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Final color = weighted average
|
||||
CRGB final_color = CRGB::Black;
|
||||
if (total_weight > 0) {
|
||||
final_color.r = static_cast<uint8_t>(r_accum / total_weight);
|
||||
final_color.g = static_cast<uint8_t>(g_accum / total_weight);
|
||||
final_color.b = static_cast<uint8_t>(b_accum / total_weight);
|
||||
}
|
||||
|
||||
mCorkscrewLeds[led_idx] = final_color;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Complete Pipeline Flow
|
||||
|
||||
```
|
||||
1. User draws → XY Grid (CRGB values at integer coordinates)
|
||||
↓
|
||||
2. LED projection → vec2f position on grid (floating-point)
|
||||
↓
|
||||
3. computeTile → Tile2x2_u8 (4 neighbor intensities)
|
||||
(splat) ↓
|
||||
4. Wrap for → Tile2x2_u8_wrap (cylindrical coordinates + intensities)
|
||||
cylinder ↓
|
||||
5. multiSample → Weighted sampling from 4 neighbors
|
||||
(readFromMulti) ↓
|
||||
6. Final LED color → CRGB value for corkscrew LED
|
||||
```
|
||||
|
||||
## Key Insights
|
||||
|
||||
### Sub-Pixel Accuracy
|
||||
The system achieves sub-pixel accuracy by:
|
||||
- Using floating-point LED positions on the grid
|
||||
- Converting to bilinear weights for 4 nearest neighbors
|
||||
- Performing weighted color sampling instead of nearest-neighbor
|
||||
|
||||
### Cylindrical Mapping
|
||||
- X-coordinates wrap around the cylinder circumference
|
||||
- Y-coordinates represent vertical position along the helix
|
||||
- Width = LEDs per turn, Height = total vertical segments
|
||||
|
||||
### Anti-Aliasing
|
||||
The weighted sampling naturally provides anti-aliasing:
|
||||
- Sharp grid patterns become smoothly interpolated on the corkscrew
|
||||
- Reduces visual artifacts from the discrete→continuous→discrete mapping
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
- **Memory**: O(W×H) for grid (O(N) for corkscrew LEDs where O(N) <= O(WxH) == O(WxW))
|
||||
- **Computation**: O(N) with 4 samples per LED (constant factor)
|
||||
- **Quality**: Sub-pixel accurate with built-in anti-aliasing
|
||||
|
||||
|
||||
## Future Work
|
||||
|
||||
Often led strips are soldered together. These leaves a gap between the other
|
||||
leds on the strip. This gab should be accounted for to maximize spatial accuracy with rendering straight lines (e.g. text).
|
||||
@@ -0,0 +1,90 @@
|
||||
# CRGB Namespace Migration Plan
|
||||
|
||||
## Overview
|
||||
Migrate the CRGB class and related color types from global namespace to `fl` namespace, following the established pattern from EOrder and RGBW migrations.
|
||||
|
||||
## Current State Analysis
|
||||
|
||||
### Files to Review
|
||||
- `src/crgb.h` - Main CRGB class definition
|
||||
|
||||
### Dependencies to Map
|
||||
- CRGB class methods and operators
|
||||
- CHSV class and HSV<->RGB conversions
|
||||
- Color constants (CRGB::Red, CRGB::Blue, etc.)
|
||||
- Pixel type aliases (e.g., CRGB*, pixel arrays)
|
||||
- Color utility functions that operate on CRGB/CHSV
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Phase 1: Create Canonical fl Namespace Version
|
||||
1. **Create `src/fl/crgb.h`**
|
||||
- use `cp` to copy the contents of `src/crgb.h` into `src/fl/crgb.h`
|
||||
- Move core CRGB class definition to fl namespace
|
||||
- Include all operators (+, -, *, /, etc.)
|
||||
- Include all color constants (Red, Green, Blue, etc.)
|
||||
- Include blend functions and color manipulation methods
|
||||
|
||||
|
||||
### Phase 2: Update Global Headers for Backward Compatibility
|
||||
1. **Update `src/crgb.h`**
|
||||
- Include `fl/crgb.h`
|
||||
- Add type alias: `using CRGB = fl::CRGB;`
|
||||
- Add global constant aliases for color constants
|
||||
- Add global function wrappers where needed
|
||||
|
||||
### Phase 3: Handle Special Cases
|
||||
1. **Color Constants**
|
||||
- Ensure `CRGB::Red`, `CRGB::Green`, etc. continue to work
|
||||
- May require `using fl::CRGB::Red;` style imports
|
||||
- Alternative: Global constants like `const CRGB Red = fl::CRGB::Red;`
|
||||
|
||||
2. **Template Functions**
|
||||
- Review template functions that accept CRGB parameters
|
||||
- Ensure they work with both global and namespaced types
|
||||
- May need template specializations or SFINAE
|
||||
|
||||
3. **Array and Pointer Types**
|
||||
- Ensure `CRGB*`, `CRGB[]` continue to work seamlessly
|
||||
- Verify LED strip array declarations remain compatible
|
||||
|
||||
### Step 5: Comprehensive Testing
|
||||
- Run full test suite
|
||||
- Compile all examples
|
||||
- Performance verification
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
### Potential Issues
|
||||
1. **Template Compatibility** - Templates expecting global CRGB may need updates
|
||||
2. **Static Member Access** - `CRGB::Red` style access must continue working
|
||||
3. **Operator Overloads** - Binary operators between global and namespaced types
|
||||
4. **Performance** - Aliasing should not introduce overhead
|
||||
|
||||
### Mitigation Strategies
|
||||
1. **Gradual Migration** - Implement and test one component at a time
|
||||
2. **Comprehensive Testing** - Test both old and new usage patterns
|
||||
3. **Template Specialization** - Add specializations if needed for compatibility
|
||||
4. **Performance Monitoring** - Benchmark before and after migration
|
||||
|
||||
## Success Criteria
|
||||
- [ ] All 91 unit tests pass
|
||||
- [ ] All Arduino examples compile successfully
|
||||
- [ ] No performance regression
|
||||
- [ ] Full backward compatibility maintained
|
||||
- [ ] New fl:: namespace usage works correctly
|
||||
- [ ] Zero breaking changes for existing user code
|
||||
|
||||
## Dependencies
|
||||
This migration depends on the successful completion of:
|
||||
- ✅ EOrder namespace migration
|
||||
- ✅ RGBW namespace migration
|
||||
- ✅ Build system stability
|
||||
|
||||
## Timeline Estimate
|
||||
- Research and planning: 1-2 hours
|
||||
- Core implementation: 3-4 hours
|
||||
- Testing and validation: 2-3 hours
|
||||
- Edge case resolution: 1-2 hours
|
||||
|
||||
**Total estimated time: 7-11 hours**
|
||||
@@ -0,0 +1,494 @@
|
||||
# FastLED Native Platform Configuration
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the comprehensive `platform.json` configuration for FastLED's native build platform. The design leverages the Python `ziglang` package to provide a unified, cross-platform compilation environment that integrates seamlessly with the existing `build_unit.toml` structure.
|
||||
|
||||
## Key Design Principles
|
||||
|
||||
### 1. **Unified Toolchain Integration**
|
||||
- **Ziglang Foundation**: Uses `python -m ziglang` commands matching the current `build_unit.toml` configuration
|
||||
- **Cross-Platform Consistency**: Single toolchain across Linux, macOS, Windows with automatic target detection
|
||||
- **Direct TOML Mapping**: Toolchain commands directly map between `platform.json` and `build_unit.toml`
|
||||
|
||||
### 2. **Comprehensive Architecture Support**
|
||||
- **x86_64 Targets**: Linux, macOS, Windows with appropriate system-specific flags
|
||||
- **ARM64 Targets**: Linux (generic), macOS (Apple Silicon), Windows with optimized CPU settings
|
||||
- **RISC-V 64**: Linux support for emerging architecture
|
||||
- **Auto-Detection**: Intelligent target selection for simplified development workflow
|
||||
|
||||
### 3. **Arduino Package Format Compliance**
|
||||
- **PlatformIO Integration**: Full compatibility with PlatformIO package management
|
||||
- **Framework Support**: Both FastLED-stub and Arduino-stub frameworks
|
||||
- **Package Dependencies**: Proper toolchain, framework, and tool dependency management
|
||||
- **Versioning Strategy**: Semantic versioning with appropriate version constraints
|
||||
|
||||
### 4. **Build System Flexibility**
|
||||
- **Flag Inheritance**: Base, debug, release, and test flag configurations
|
||||
- **Platform-Specific Linking**: Automatic application of OS-specific link flags
|
||||
- **Compiler Caching**: Optional ccache/sccache integration for performance
|
||||
- **Build System Support**: CMake, Ninja, and direct compilation compatibility
|
||||
|
||||
## Complete Platform.json Structure
|
||||
|
||||
The full `platform.json` includes:
|
||||
|
||||
1. **Board Definitions** (8 comprehensive targets):
|
||||
- `native-x64-linux`, `native-x64-macos`, `native-x64-windows`
|
||||
- `native-arm64-linux`, `native-arm64-macos`, `native-arm64-windows`
|
||||
- `native-riscv64-linux`, `native-auto` (intelligent detection)
|
||||
|
||||
2. **Toolchain Configuration**:
|
||||
- Complete ziglang tool mapping (CC, CXX, AR, LD, etc.)
|
||||
- Build flag hierarchies (base, debug, release, test)
|
||||
- Platform-specific link flags for each OS
|
||||
|
||||
3. **Package Management**:
|
||||
- Toolchain dependencies with proper versioning
|
||||
- Optional tool packages (cmake, ninja, caching)
|
||||
- Framework packages for Arduino compatibility
|
||||
|
||||
4. **Development Features**:
|
||||
- Integrated debugger support (GDB for Linux/Windows, LLDB for macOS)
|
||||
- Environment variable customization support
|
||||
- Target compatibility specifications
|
||||
|
||||
## Platform.json Configuration
|
||||
|
||||
The complete platform.json follows Arduino package format with these key sections:
|
||||
|
||||
### Core Metadata
|
||||
```json
|
||||
{
|
||||
"name": "FastLED Native Platform",
|
||||
"title": "Native Host Platform with Zig Toolchain",
|
||||
"description": "Cross-platform native compilation using Zig's bundled Clang",
|
||||
"version": "1.0.0",
|
||||
"packageType": "platform",
|
||||
"spec": {
|
||||
"owner": "fastled",
|
||||
"id": "native-zig",
|
||||
"name": "native-zig"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Framework Definitions
|
||||
```json
|
||||
{
|
||||
"frameworks": {
|
||||
"fastled-stub": {
|
||||
"package": "framework-fastled-stub",
|
||||
"script": "builder/frameworks/fastled-stub.py"
|
||||
},
|
||||
"arduino-stub": {
|
||||
"package": "framework-arduino-stub",
|
||||
"script": "builder/frameworks/arduino-stub.py"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Board Configurations
|
||||
Each board includes complete configuration:
|
||||
- **Hardware specs**: MCU, RAM, Flash, CPU frequency
|
||||
- **Target triple**: Zig compilation target
|
||||
- **Build flags**: Platform-specific preprocessor defines
|
||||
- **Debug tools**: Appropriate debugger (GDB/LLDB) configuration
|
||||
- **Framework support**: Compatible frameworks list
|
||||
|
||||
Example board definition:
|
||||
```json
|
||||
{
|
||||
"native-x64-linux": {
|
||||
"name": "Native x86_64 Linux",
|
||||
"mcu": "x86_64",
|
||||
"fcpu": "3000000000L",
|
||||
"ram": 17179869184,
|
||||
"flash": 1099511627776,
|
||||
"vendor": "Generic",
|
||||
"product": "Native x86_64",
|
||||
"frameworks": ["fastled-stub", "arduino-stub"],
|
||||
"build": {
|
||||
"target": "x86_64-linux-gnu",
|
||||
"toolchain": "ziglang",
|
||||
"cpu": "x86_64",
|
||||
"mcu": "native",
|
||||
"f_cpu": "3000000000L",
|
||||
"extra_flags": [
|
||||
"-DNATIVE_PLATFORM=1",
|
||||
"-DSTUB_PLATFORM=1",
|
||||
"-DFASTLED_NATIVE_X64=1",
|
||||
"-DFASTLED_NATIVE_LINUX=1"
|
||||
]
|
||||
},
|
||||
"upload": { "protocol": "native" },
|
||||
"debug": {
|
||||
"tools": {
|
||||
"gdb": {
|
||||
"server": {
|
||||
"executable": "gdb",
|
||||
"arguments": ["$PROG_PATH"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Toolchain Mapping
|
||||
Direct integration with ziglang commands:
|
||||
```json
|
||||
{
|
||||
"toolchain-ziglang": {
|
||||
"CC": "python -m ziglang cc",
|
||||
"CXX": "python -m ziglang c++",
|
||||
"AR": "python -m ziglang ar",
|
||||
"LD": "python -m ziglang c++",
|
||||
"OBJCOPY": "python -m ziglang objcopy",
|
||||
"NM": "python -m ziglang nm",
|
||||
"STRIP": "python -m ziglang strip",
|
||||
"RANLIB": "python -m ziglang ranlib"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Build Flag Hierarchies
|
||||
```json
|
||||
{
|
||||
"build_flags": {
|
||||
"base": [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-Wno-deprecated-register",
|
||||
"-Wno-backslash-newline-escape",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
"-DSTUB_PLATFORM=1",
|
||||
"-DNATIVE_PLATFORM=1"
|
||||
],
|
||||
"debug": [
|
||||
"-g3",
|
||||
"-Og",
|
||||
"-DDEBUG=1",
|
||||
"-DFASTLED_DEBUG=1"
|
||||
],
|
||||
"release": [
|
||||
"-O3",
|
||||
"-DNDEBUG=1",
|
||||
"-flto"
|
||||
],
|
||||
"test": [
|
||||
"-DFASTLED_UNIT_TEST=1",
|
||||
"-DFASTLED_FORCE_NAMESPACE=1",
|
||||
"-DFASTLED_TESTING=1",
|
||||
"-DFASTLED_USE_PROGMEM=0",
|
||||
"-DARDUINO=10808",
|
||||
"-DFASTLED_USE_STUB_ARDUINO=1",
|
||||
"-DSKETCH_HAS_LOTS_OF_MEMORY=1",
|
||||
"-DFASTLED_STUB_IMPL=1",
|
||||
"-DFASTLED_USE_JSON_UI=1",
|
||||
"-DFASTLED_NO_AUTO_NAMESPACE=1",
|
||||
"-DFASTLED_NO_PINMAP=1",
|
||||
"-DHAS_HARDWARE_PIN_SUPPORT=1",
|
||||
"-DFASTLED_DEBUG_LEVEL=1",
|
||||
"-DFASTLED_NO_ATEXIT=1",
|
||||
"-DDOCTEST_CONFIG_NO_EXCEPTIONS_BUT_WITH_ALL_ASSERTS=1"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Platform-Specific Linking
|
||||
```json
|
||||
{
|
||||
"link_flags": {
|
||||
"base": [],
|
||||
"linux": [
|
||||
"-pthread",
|
||||
"-lm",
|
||||
"-ldl"
|
||||
],
|
||||
"macos": [
|
||||
"-pthread",
|
||||
"-lm"
|
||||
],
|
||||
"windows": [
|
||||
"-mconsole",
|
||||
"-nodefaultlibs",
|
||||
"-unwindlib=libunwind",
|
||||
"-nostdlib++",
|
||||
"-lc++",
|
||||
"-lkernel32",
|
||||
"-luser32",
|
||||
"-lgdi32",
|
||||
"-ladvapi32"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with build_unit.toml
|
||||
|
||||
The `platform.json` is designed to seamlessly integrate with the existing `build_unit.toml` configuration:
|
||||
|
||||
### Toolchain Mapping
|
||||
|
||||
The `[tools]` section in `build_unit.toml` directly maps to the ziglang toolchain defined in `platform.json`:
|
||||
|
||||
- `cpp_compiler = ["python", "-m", "ziglang", "c++"]` → `CXX: "python -m ziglang c++"`
|
||||
- `linker = ["python", "-m", "ziglang", "c++"]` → `LD: "python -m ziglang c++"`
|
||||
- `archiver = ["python", "-m", "ziglang", "ar"]` → `AR: "python -m ziglang ar"`
|
||||
|
||||
### Build Flag Integration
|
||||
|
||||
The platform flags complement the TOML configuration:
|
||||
|
||||
- **Base flags**: Defined in both `platform.json` and `build_unit.toml` for consistency
|
||||
- **Platform-specific flags**: Windows-specific flags from TOML are automatically applied
|
||||
- **Test flags**: Unit test defines from TOML are included in the platform test configuration
|
||||
|
||||
### Target Selection
|
||||
|
||||
The platform supports automatic target detection or explicit target selection:
|
||||
|
||||
```bash
|
||||
# Auto-detect target
|
||||
platformio run -e native-auto
|
||||
|
||||
# Explicit target selection
|
||||
platformio run -e native-x64-linux
|
||||
platformio run -e native-arm64-macos
|
||||
platformio run -e native-x64-windows
|
||||
```
|
||||
|
||||
## Integration Benefits
|
||||
|
||||
### Seamless TOML Integration
|
||||
The design ensures perfect compatibility with existing `build_unit.toml`:
|
||||
- Toolchain commands match exactly: `["python", "-m", "ziglang", "c++"]`
|
||||
- Build flags complement rather than duplicate TOML configuration
|
||||
- Platform-specific flags automatically applied based on detection
|
||||
|
||||
### Enhanced Development Workflow
|
||||
- **Single Command Compilation**: `platformio run -e native-auto`
|
||||
- **Cross-Compilation Support**: Build for different targets from any host
|
||||
- **Testing Integration**: Full unit test support with proper namespace isolation
|
||||
- **Debug Builds**: Integrated debugging with platform-appropriate tools
|
||||
|
||||
### Future-Proof Architecture
|
||||
- **Extensible Design**: Easy addition of new architectures and targets
|
||||
- **WebAssembly Ready**: Structure supports future WASM integration
|
||||
- **Container Builds**: Foundation for Docker/Podman integration
|
||||
- **Static Analysis**: Framework for clang-tidy and analyzer integration
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Compilation
|
||||
|
||||
```bash
|
||||
# Compile for current platform (auto-detect)
|
||||
platformio run -e native-auto
|
||||
|
||||
# Compile for specific platform
|
||||
platformio run -e native-x64-linux
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Run unit tests
|
||||
platformio test -e native-auto
|
||||
|
||||
# Run tests with specific build flags
|
||||
platformio test -e native-x64-linux --test-port=native
|
||||
```
|
||||
|
||||
### Cross-Compilation
|
||||
|
||||
```bash
|
||||
# Compile for ARM64 Linux from x86_64
|
||||
platformio run -e native-arm64-linux
|
||||
|
||||
# Compile for Windows from Linux
|
||||
platformio run -e native-x64-windows
|
||||
```
|
||||
|
||||
### Debug Builds
|
||||
|
||||
```bash
|
||||
# Debug build with symbols
|
||||
platformio run -e native-auto --build-type=debug
|
||||
|
||||
# Release build with optimization
|
||||
platformio run -e native-auto --build-type=release
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Custom Board Definition
|
||||
|
||||
```json
|
||||
{
|
||||
"native-custom": {
|
||||
"name": "Custom Native Target",
|
||||
"mcu": "custom",
|
||||
"build": {
|
||||
"target": "x86_64-linux-musl",
|
||||
"extra_flags": [
|
||||
"-DCUSTOM_DEFINE=1",
|
||||
"-static"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
The platform respects environment variables for customization:
|
||||
|
||||
- `ZIG_TARGET`: Override target triple (e.g., `x86_64-linux-musl`)
|
||||
- `ZIG_CPU`: Override CPU type (e.g., `x86_64_v3`)
|
||||
- `FASTLED_NATIVE_FLAGS`: Additional compiler flags
|
||||
|
||||
### Build System Integration
|
||||
|
||||
The platform integrates with multiple build systems:
|
||||
|
||||
- **PlatformIO**: Primary integration with comprehensive board support
|
||||
- **CMake**: Compatible with existing CMake configurations
|
||||
- **Direct Compilation**: Can be used directly with ziglang commands
|
||||
|
||||
## Complete Board Definitions
|
||||
|
||||
The platform.json includes 8 comprehensive board definitions:
|
||||
|
||||
1. **native-x64-linux**: x86_64 Linux with GNU target
|
||||
2. **native-x64-macos**: x86_64 macOS with Apple-specific configuration
|
||||
3. **native-x64-windows**: x86_64 Windows with MinGW compatibility
|
||||
4. **native-arm64-linux**: ARM64 Linux with Cortex-A76 optimization
|
||||
5. **native-arm64-macos**: ARM64 macOS with Apple M1 optimization
|
||||
6. **native-arm64-windows**: ARM64 Windows with appropriate runtime
|
||||
7. **native-riscv64-linux**: RISC-V 64-bit Linux support
|
||||
8. **native-auto**: Automatic target detection and selection
|
||||
|
||||
## Tool Dependencies
|
||||
|
||||
The platform includes comprehensive tool support:
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": {
|
||||
"python-ziglang": {
|
||||
"type": "compiler",
|
||||
"version": "^0.13.0",
|
||||
"description": "Zig toolchain with bundled Clang/LLVM",
|
||||
"install": { "pip": "ziglang" }
|
||||
},
|
||||
"cmake": {
|
||||
"type": "build-system",
|
||||
"version": "^3.21.0",
|
||||
"optional": true
|
||||
},
|
||||
"ninja": {
|
||||
"type": "build-system",
|
||||
"version": "^1.11.0",
|
||||
"optional": true
|
||||
},
|
||||
"ccache": {
|
||||
"type": "compiler-cache",
|
||||
"version": "^4.0.0",
|
||||
"optional": true
|
||||
},
|
||||
"sccache": {
|
||||
"type": "compiler-cache",
|
||||
"version": "^0.7.0",
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Target Compatibility
|
||||
|
||||
Platform compatibility specifications ensure proper deployment:
|
||||
|
||||
```json
|
||||
{
|
||||
"target_compatibility": {
|
||||
"x86_64-linux-gnu": {
|
||||
"glibc": ">=2.31",
|
||||
"kernel": ">=5.4.0"
|
||||
},
|
||||
"x86_64-macos-none": {
|
||||
"macos": ">=11.0"
|
||||
},
|
||||
"x86_64-windows-gnu": {
|
||||
"windows": ">=10"
|
||||
},
|
||||
"aarch64-linux-gnu": {
|
||||
"glibc": ">=2.31",
|
||||
"kernel": ">=5.4.0"
|
||||
},
|
||||
"aarch64-macos-none": {
|
||||
"macos": ">=11.0"
|
||||
},
|
||||
"aarch64-windows-gnu": {
|
||||
"windows": ">=10"
|
||||
},
|
||||
"riscv64-linux-gnu": {
|
||||
"glibc": ">=2.31",
|
||||
"kernel": ">=5.10.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Ziglang not found**: Ensure `ziglang` Python package is installed
|
||||
2. **Target not supported**: Check `zig targets` for available targets
|
||||
3. **Linking errors**: Verify platform-specific link flags in configuration
|
||||
|
||||
### Debug Information
|
||||
|
||||
```bash
|
||||
# Check available targets
|
||||
zig targets
|
||||
|
||||
# Verify toolchain installation
|
||||
python -m ziglang version
|
||||
|
||||
# Test compilation
|
||||
python -m ziglang c++ --version
|
||||
```
|
||||
|
||||
## Implementation Status
|
||||
|
||||
This design is **ready for implementation** and provides:
|
||||
- ✅ **Complete board definitions** for all major native platforms
|
||||
- ✅ **Full toolchain integration** with existing ziglang setup
|
||||
- ✅ **Comprehensive build configurations** for all use cases
|
||||
- ✅ **Arduino package format compliance** for PlatformIO compatibility
|
||||
- ✅ **Debug and development tool integration** for professional development
|
||||
- ✅ **Future-proof extensible architecture** for additional platforms
|
||||
|
||||
The configuration can be deployed alongside the existing build system for gradual validation and testing before full migration.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- **WebAssembly Support**: Integration with FastLED WASM compilation
|
||||
- **Static Analysis**: Integration with clang-tidy and static analyzers
|
||||
- **Performance Profiling**: Built-in profiling support for optimization
|
||||
- **Container Builds**: Docker/Podman integration for reproducible builds
|
||||
- **Package Registry**: Publication to Arduino package index for distribution
|
||||
|
||||
## Conclusion
|
||||
|
||||
This platform.json design provides a comprehensive, production-ready configuration for FastLED's native build platform. It seamlessly integrates with the existing `build_unit.toml` while providing the standardized Arduino package format required for broader ecosystem compatibility. The design is thoroughly tested conceptually and ready for implementation with proper validation and testing procedures.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,20 @@
|
||||
The MIT License (MIT)
|
||||
|
||||
Copyright (c) 2013 FastLED
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
this software and associated documentation files (the "Software"), to deal in
|
||||
the Software without restriction, including without limitation the rights to
|
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
|
||||
the Software, and to permit persons to whom the Software is furnished to do so,
|
||||
subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
||||
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
||||
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
@@ -0,0 +1,56 @@
|
||||
Platform Porting Guide
|
||||
==========================
|
||||
|
||||
# Fast porting for a new board on existing hardware
|
||||
|
||||
Sometimes "porting" FastLED simply consists of supplying new pin definitions for the given platform. For example, platforms/avr/fastpin_avr.h contains various pin definitions for all the AVR variant chipsets/boards that FastLED supports. Defining a set of pins involves setting up a set of definitions - for example here's one full set from the avr fastpin file:
|
||||
|
||||
```
|
||||
#elif defined(__AVR_ATmega1284P__) || defined(__AVR_ATmega644P__)
|
||||
|
||||
_FL_IO(A); _FL_IO(B); _FL_IO(C); _FL_IO(D);
|
||||
|
||||
#define MAX_PIN 31
|
||||
_FL_DEFPIN(0, 0, B); _FL_DEFPIN(1, 1, B); _FL_DEFPIN(2, 2, B); _FL_DEFPIN(3, 3, B);
|
||||
_FL_DEFPIN(4, 4, B); _FL_DEFPIN(5, 5, B); _FL_DEFPIN(6, 6, B); _FL_DEFPIN(7, 7, B);
|
||||
_FL_DEFPIN(8, 0, D); _FL_DEFPIN(9, 1, D); _FL_DEFPIN(10, 2, D); _FL_DEFPIN(11, 3, D);
|
||||
_FL_DEFPIN(12, 4, D); _FL_DEFPIN(13, 5, D); _FL_DEFPIN(14, 6, D); _FL_DEFPIN(15, 7, D);
|
||||
_FL_DEFPIN(16, 0, C); _FL_DEFPIN(17, 1, C); _FL_DEFPIN(18, 2, C); _FL_DEFPIN(19, 3, C);
|
||||
_FL_DEFPIN(20, 4, C); _FL_DEFPIN(21, 5, C); _FL_DEFPIN(22, 6, C); _FL_DEFPIN(23, 7, C);
|
||||
_FL_DEFPIN(24, 0, A); _FL_DEFPIN(25, 1, A); _FL_DEFPIN(26, 2, A); _FL_DEFPIN(27, 3, A);
|
||||
_FL_DEFPIN(28, 4, A); _FL_DEFPIN(29, 5, A); _FL_DEFPIN(30, 6, A); _FL_DEFPIN(31, 7, A);
|
||||
|
||||
#define HAS_HARDWARE_PIN_SUPPORT 1
|
||||
```
|
||||
|
||||
The ```_FL_IO``` macro is used to define the port registers for the platform while the ```_FL_DEFPIN``` macro is used to define pins. The parameters to the macro are the pin number, the bit on the port that represents that pin, and the port identifier itself. On some platforms, like the AVR, ports are identified by letter. On other platforms, like arm, ports are identified by number.
|
||||
|
||||
The ```HAS_HARDWARE_PIN_SUPPORT``` define tells the rest of the FastLED library that there is hardware pin support available. There may be other platform specific defines for things like hardware SPI ports and such.
|
||||
|
||||
## Setting up the basic files/folders
|
||||
|
||||
* Create platform directory (e.g. platforms/arm/kl26)
|
||||
* Create configuration header led_sysdefs_arm_kl26.h:
|
||||
* Define platform flags (like FASTLED_ARM/FASTLED_TEENSY)
|
||||
* Define configuration parameters re: interrupts, or clock doubling
|
||||
* Include extar system header files if needed
|
||||
* Create main platform include, fastled_arm_kl26.h
|
||||
* Include the various other header files as needed
|
||||
* Modify led_sysdefs.h to conditionally include platform sysdefs header file
|
||||
* Modify platforms.h to conditionally include platform fastled header
|
||||
|
||||
## Porting fastpin.h
|
||||
|
||||
The heart of the FastLED library is the fast pin access. This is a templated class that provides 1-2 cycle pin access, bypassing digital write and other such things. As such, this will usually be the first bit of the library that you will want to port when moving to a new platform. Once you have FastPIN up and running then you can do some basic work like testing toggles or running bit-bang'd SPI output.
|
||||
|
||||
There's two low level FastPin classes. There's the base FastPIN template class, and then there is FastPinBB which is for bit-banded access on those MCUs that support bitbanding. Note that the bitband class is optional and primarily useful in the implementation of other functionality internal to the platform. This file is also where you would do the pin to port/bit mapping defines.
|
||||
|
||||
Explaining how the macros work and should be used is currently beyond the scope of this document.
|
||||
|
||||
## Porting fastspi.h
|
||||
|
||||
This is where you define the low level interface to the hardware SPI system (including a writePixels method that does a bunch of housekeeping for writing led data). Use the fastspi_nop.h file as a reference for the methods that need to be implemented. There are ofteh other useful methods that can help with the internals of the SPI code, I recommend taking a look at how the various platforms implement their SPI classes.
|
||||
|
||||
## Porting clockless.h
|
||||
|
||||
This is where you define the code for the clockless controllers. Across ARM platforms this will usually be fairly similar - though different arm platforms will have different clock sources that you can/should use.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,382 @@
|
||||
# FastLED - The Universal LED Library
|
||||
|
||||
**Transform any microcontroller into an LED powerhouse.**
|
||||
|
||||
Drive **30,000+ LEDs** on high-end devices • **Sub-$1 compatibility** on tiny chips • **Background rendering** for responsive apps • **Nearly every LED chipset supported** • [**#2 most popular Arduino library**](https://docs.arduino.cc/libraries/)
|
||||
|
||||
[](https://www.ardu-badge.com/FastLED) [](https://github.com/FastLED/FastLED/actions/workflows/build.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_unit_test.yml) [](http://fastled.io/docs) [](https://www.reddit.com/r/FastLED/)
|
||||
|
||||
<details>
|
||||
<summary>📊 <strong>Detailed Build Status</strong> (Click to expand all platform badges)</summary>
|
||||
|
||||
**The build system is undergoing optimization and some builds may appear red during migration. Despite any red status, everything still works.**
|
||||
|
||||
### Arduino Family
|
||||
**Core Boards:** [](https://github.com/FastLED/FastLED/actions/workflows/build_uno.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_nano_every.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_uno_r4_wifif.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_yun.yml)
|
||||
|
||||
**ARM Boards:** [](https://github.com/FastLED/FastLED/actions/workflows/build_due.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_digix.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_giga_r1.yml)*
|
||||
|
||||
**ATtiny Series:** [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny85.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny88.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny1604.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny1616.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_attiny4313.yml)**
|
||||
|
||||
*Notes: * Giga-R1 support added in 3.9.14 • ** ATtiny4313 has limited memory (WS2812 Blink + APA102 examples only)
|
||||
|
||||
### Teensy Series
|
||||
**Standard Models:** [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy30.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy31.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensyLC.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy40.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy41.yml)
|
||||
|
||||
**Special Features:** [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy_octo.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_teensy41_ofled.yml)
|
||||
|
||||
### NRF52 (Nordic)
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_adafruit_feather_nrf52840_sense.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_nrf52840_dk.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_adafruit_xiaoblesense.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_nrf52_xiaoblesense.yml)
|
||||
|
||||
*Note: NRF52 XiaoBLE board has mbed engine but doesn't compile against Arduino.h for unknown reasons.
|
||||
|
||||
### Apollo3 (Ambiq)
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_apollo3_red.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_apollo3_thing_explorable.yml)
|
||||
|
||||
*Beta support added in 3.10.2
|
||||
|
||||
### STM32 (STMicroelectronics)
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_bluepill.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_maple_map.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_stm103tb.yml)
|
||||
|
||||
*Note: STM103TB has limited PlatformIO support
|
||||
|
||||
### Raspberry Pi Pico
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_rp2040.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_rp2350.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_rp2350B.yml)
|
||||
|
||||
### ESP32 (Espressif)
|
||||
**ESP8266:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp8622.yml)
|
||||
|
||||
**ESP32 Classic:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32dev.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32wroom.yml)
|
||||
|
||||
**ESP32 S-Series:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32s2.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32s3.yml)
|
||||
|
||||
**ESP32 C-Series:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32c2.yml)* [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32c3.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32c5.yml)[](https://github.com/FastLED/FastLED/actions/workflows/build_esp32c6.yml)
|
||||
|
||||
**ESP32 Advanced:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32h2.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32p4.yml)
|
||||
|
||||
**Special Features:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32_i2s_ws2812.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp_extra_libs.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32dev_namespace.yml)
|
||||
|
||||
**Legacy:** [](https://github.com/FastLED/FastLED/actions/workflows/build_esp32dev_idf3.3.yml)
|
||||
|
||||
*Notes: * ESP32-C2 support added in 3.9.10 • [Espressif compatibility evaluation](https://github.com/espressif/arduino-esp32/blob/gh-pages/LIBRARIES_TEST.md)
|
||||
|
||||
### Specialty Platforms
|
||||
**x86:** [](https://github.com/FastLED/FastLED/actions/workflows/build_linux.yml)
|
||||
|
||||
**WebAssembly:** [](https://github.com/FastLED/FastLED/actions/workflows/build_wasm.yml) [](https://github.com/FastLED/FastLED/actions/workflows/build_wasm_compilers.yml)
|
||||
|
||||
### Library Size Validation
|
||||
**Core Platforms:** [](https://github.com/FastLED/FastLED/actions/workflows/check_attiny85.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_uno_size.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_esp32_size.yml)
|
||||
|
||||
**Teensy Platforms:** [](https://github.com/FastLED/FastLED/actions/workflows/check_teensylc_size.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_teensy30_size.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_teensy31_size.yml) [](https://github.com/FastLED/FastLED/actions/workflows/check_teensy41_size.yml)
|
||||
|
||||
</details>
|
||||
|
||||
## ⚡ Get Blinking in 30 Seconds
|
||||
|
||||
```cpp
|
||||
#include <FastLED.h>
|
||||
#define NUM_LEDS 60
|
||||
CRGB leds[NUM_LEDS];
|
||||
|
||||
void setup() {
|
||||
FastLED.addLeds<WS2812, 6>(leds, NUM_LEDS);
|
||||
}
|
||||
|
||||
void loop() {
|
||||
leds[0] = CRGB::Red; FastLED.show(); delay(500);
|
||||
leds[0] = CRGB::Blue; FastLED.show(); delay(500);
|
||||
}
|
||||
```
|
||||
|
||||
**✅ Works on Arduino, ESP32, Teensy, Raspberry Pi, and 50+ other platforms**
|
||||
|
||||
## 🚀 Why FastLED?
|
||||
|
||||
| **Massive Scale** | **Tiny Footprint** | **Background Rendering** | **Universal** |
|
||||
|-------------------|---------------------|--------------------------|---------------|
|
||||
| Drive 30,000 LEDs on Teensy 4.1 | Runs on $0.50 ATtiny chips | ESP32/Teensy render while you code | Works on 50+ platforms |
|
||||
| 50 parallel strips on Teensy | <2KB on Arduino Uno | Never miss user input | Nearly every LED chipset |
|
||||
|
||||
**🎯 Performance**: Zero-cost global brightness • High-performance 8-bit math, memory efficient on platforms that need it.
|
||||
**🔧 Developer Experience**: Quick platform switching • Extensive examples • Active community support
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [🆕 Latest Feature](#-latest-feature)
|
||||
- [⭐ Community Growth](#-community-growth)
|
||||
- [🆕 Latest Features](#-latest-features)
|
||||
- [🌍 Platform Support](#-platform-support)
|
||||
- [📦 Installation](#-installation)
|
||||
- [📚 Documentation & Support](#-documentation--support)
|
||||
- [🎮 Advanced Features](#-advanced-features)
|
||||
- [🤝 Contributing](#-contributing)
|
||||
|
||||
## ⭐ Community Growth
|
||||
|
||||
<a href="https://star-history.com/#fastled/fastled&Date">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=fastled/fastled&type=Date&theme=dark" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=fastled/fastled&type=Date" />
|
||||
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=fastled/fastled&type=Date" />
|
||||
</picture>
|
||||
</a>
|
||||
|
||||
## 🆕 Latest Features
|
||||
|
||||
### **FastLED 3.10.0: Animartrix Out of Beta**
|
||||
Advanced animation framework for complex LED visualizations and effects.
|
||||
|
||||
[](https://fastled.github.io/assets/fastled_3_10_animartrix.mp4)
|
||||
|
||||
### **FastLED 3.9.16: WaveFx / Multi Layer Compositing**
|
||||
Multi-layer compositing & time-based animation control for tech-artists.
|
||||
|
||||
|
||||
https://github.com/user-attachments/assets/ff8e0432-3e0d-47cc-a444-82ce27f562af
|
||||
|
||||
| **3.9.13** | **3.9.10** | **3.9.8** | **3.9.2** |
|
||||
|------------|------------|-----------|-----------|
|
||||
| [**HD107 Turbo**](#new-in-3913-hd107-turbo-40mhz-led-support)<br/>40MHz LED support | [**ESP32 SPI**](#new-in-3910-super-stable-ws2812-spi-driver-for-esp32)<br/>Super stable WS2812 driver | [**Massive Teensy**](#new-in-398---massive-teensy-41--40-ws2812-led-output)<br/>50 parallel pins on 4.1 | [**WS2812 Overclock**](#new-in-392---overclocking-of-ws2812)<br/>Up to 70% speed boost |
|
||||
|
||||
| **3.7.7** | **More Features** |
|
||||
|-----------|-------------------|
|
||||
| [**RGBW Support**](#new-in-377---rgbw-led-strip-support)<br/>White channel LED strips | [📋 Full Changelog](https://github.com/FastLED/FastLED/releases)<br/>[📺 Demo Videos](https://zackees.github.io/fastled-wasm/) |
|
||||
|
||||
[**📺 Live Demos**](https://zackees.github.io/fastled-wasm/) • [**📋 Full Changelog**](https://github.com/FastLED/FastLED/releases)
|
||||
|
||||
<details>
|
||||
<summary>📖 <strong>Detailed Feature Information</strong> (Click to expand)</summary>
|
||||
|
||||
## New in 3.10.0: Animartrix out of beta
|
||||
|
||||
[](https://fastled.github.io/assets/fastled_3_10_animartrix.mp4)
|
||||
|
||||
|
||||
## New in 3.9.16: WaveFx / Multi Layer Compositing / Time-based animation control
|
||||
|
||||
Video:
|
||||
|
||||
|
||||
|
||||
https://github.com/user-attachments/assets/ff8e0432-3e0d-47cc-a444-82ce27f562af
|
||||
|
||||
|
||||
|
||||
#### Major release for tech-artists!
|
||||
|
||||
Lots of improvements in this release, read the full [change list here](https://github.com/FastLED/FastLED/releases/tag/3.9.16)
|
||||
|
||||
#### Links
|
||||
|
||||
* This demo -> [FxWave2d](https://github.com/FastLED/FastLED/blob/master/examples/FxWave2d/FxWave2d.ino)
|
||||
* [Wave Simulation Library](https://github.com/FastLED/FastLED/blob/master/src/fl/wave_simulation.h)
|
||||
* [FireCylinder](https://github.com/FastLED/FastLED/blob/master/examples/FireCylinder/FireCylinder.ino)
|
||||
* Wraps around so that (0,y) ~= (width-1,y)
|
||||
* [TimeAlpha](https://github.com/FastLED/FastLED/blob/master/src/fl/time_alpha.h)
|
||||
* Precision control of animations with time-based alpha transition.
|
||||
|
||||
|
||||
|
||||
## New in 3.9.13: HD107 "Turbo" 40Mhz LED Support
|
||||
|
||||
|
||||
|
||||
## New in 3.9.12: WS2816 "HD" LED support
|
||||
|
||||
|
||||
|
||||
## New in 3.9.10: Super Stable WS2812 SPI driver for ESP32
|
||||
|
||||
|
||||
|
||||
## New in 3.9.9: 16-way Yves I2S parallel driver for the ESP32-S3
|
||||
|
||||
|
||||
|
||||
*Note some users find that newer versions of the ESP32 Arduino core (3.10) don't work very well, but older versions do, see [issue 1903](https://github.com/FastLED/FastLED/issues/1903)
|
||||
|
||||
## New in 3.9.8 - Massive Teensy 4.1 & 4.0 WS2812 LED output
|
||||
|
||||
* *Teensy 4.1: 50 parallel pins*
|
||||
* *Teensy 4.0: 42 parallel pins*
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## New in 3.9.2 - Overclocking of WS2812
|
||||
|
||||
Update: max overclock has been reported at +70%: https://www.reddit.com/r/FastLED/comments/1gkcb6m/fastled_FASTLED_OVERCLOCK_17/
|
||||
|
||||
### New in 3.7.7 - RGBW LED Strip Support
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
## 🌍 Platform Support
|
||||
|
||||
### Platform Categories
|
||||
| **Arduino Family** | **ESP32 Series** | **Teensy** | **ARM** | **Specialty** |
|
||||
|--------------------|------------------|------------|---------|---------------|
|
||||
| Uno, Nano, Mega<br/>Due, Giga R1, R4 | ESP32, S2, S3, C3<br/>C6, H2, P4 | 3.0, 3.1, 4.0, 4.1<br/>LC + OctoWS2811 | STM32, NRF52<br/>Apollo3 | Raspberry Pi<br/>WASM, x86 |
|
||||
|
||||
**FastLED supports 50+ platforms!** From sub-$1 ATtiny chips to high-end Teensy 4.1 with 50 parallel outputs.
|
||||
|
||||
## 📦 Installation
|
||||
|
||||
### Quick Install Options
|
||||
|
||||
| **Arduino IDE** | **PlatformIO** | **Package Managers** |
|
||||
|-----------------|----------------|----------------------|
|
||||
| Library Manager → Search "FastLED" → Install | `lib_deps = fastled/FastLED` | `pio pkg install --library "fastled/FastLED"` |
|
||||
| Or install [latest release .zip](https://github.com/FastLED/FastLED/releases/latest/) | Add to `platformio.ini` | Command line installation |
|
||||
|
||||
### Template Projects
|
||||
|
||||
- **🎯 [Arduino + PlatformIO Starter](https://github.com/FastLED/PlatformIO-Starter)** - Best of both worlds, works with both IDEs
|
||||
- **🚀 [FastLED Examples](examples)** - 100+ ready-to-run demos
|
||||
- **🌐 [Web Compiler](https://zackees.github.io/fastled-wasm/)** - Test in your browser
|
||||
|
||||
<details>
|
||||
<summary><strong>Arduino IDE Setup Instructions</strong> (Click to expand)</summary>
|
||||
|
||||
After installing the Arduino IDE, add FastLED through the Library Manager:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
|
||||
## 📚 Documentation & Support
|
||||
|
||||
| **📖 Documentation** | **💬 Community** | **🐛 Issues** | **📺 Examples** |
|
||||
|---------------------|------------------|---------------|-----------------|
|
||||
| [API Reference](http://fastled.io/docs) | [Reddit r/FastLED](https://www.reddit.com/r/FastLED/) | [GitHub Issues](http://fastled.io/issues) | [Live Demos](https://zackees.github.io/fastled-wasm/) |
|
||||
| [Doxygen Docs](https://fastled.io/docs/files.html) | 1000s of users & solutions | Bug reports & feature requests | [GitHub Examples](examples) |
|
||||
|
||||
**Need Help?** Visit [r/FastLED](https://reddit.com/r/FastLED) - thousands of knowledgeable users and extensive solution history!
|
||||
|
||||
## 🎮 Advanced Features
|
||||
|
||||
### Performance Leaders: Parallel Output Records
|
||||
|
||||
| **Platform** | **Max Parallel Outputs** | **Performance Notes** |
|
||||
|--------------|---------------------------|----------------------|
|
||||
| **Teensy 4.1** | 50 parallel strips | Current record holder - [Example](examples/TeensyMassiveParallel/TeensyMassiveParallel.ino) |
|
||||
| **Teensy 4.0** | 42 parallel strips | High-performance ARM Cortex-M7 |
|
||||
| **ESP32DEV** | 24 via I2S + 8 via RMT | [I2S Example](examples/EspI2SDemo/EspI2SDemo.ino) |
|
||||
| **ESP32-S3** | 16 via I2S + 4 via RMT | [S3 Example](examples/Esp32S3I2SDemo/Esp32S3I2SDemo.ino) |
|
||||
|
||||
*Note: Some ESP32 Arduino core versions (3.10+) have compatibility issues. Older versions work better - see [issue #1903](https://github.com/FastLED/FastLED/issues/1903)*
|
||||
|
||||
### Exotic Setups (120+ Outputs!)
|
||||
**Custom shift register boards** can achieve extreme parallel output:
|
||||
- **ESP32DEV**: [120-output virtual driver](https://github.com/hpwit/I2SClocklessVirtualLedDriver)
|
||||
- **ESP32-S3**: [120-output S3 driver](https://github.com/hpwit/I2SClockLessLedVirtualDriveresp32s3)
|
||||
|
||||
### Supported LED Chipsets
|
||||
|
||||
FastLED supports virtually every LED chipset available:
|
||||
|
||||
| **Clockless (3-wire)** | **SPI-based (4-wire)** | **Specialty** |
|
||||
|------------------------|------------------------|---------------|
|
||||
| **WS281x Family**: WS2811, WS2812 (NeoPixel), WS2812-V5B, WS2815 | **APA102 / DotStars**: Including HD107s (40MHz turbo) | **SmartMatrix Panels** |
|
||||
| **TM180x Series**: TM1809/4, TM1803 | **High-Speed SPI**: LPD8806, WS2801, SM16716 | **DMX Output** |
|
||||
| **Other 3-wire**: UCS1903, GW6205, SM16824E | **APA102HD**: Driver-level gamma correction | **P9813 Total Control** |
|
||||
|
||||
**RGBW Support**: WS2816 and other white-channel LED strips • **Overclocking**: WS2812 up to 70% speed boost
|
||||
|
||||
More details: [Chipset Reference Wiki](https://github.com/FastLED/FastLED/wiki/Chipset-reference)
|
||||
|
||||
### APA102 High Definition Mode
|
||||
```cpp
|
||||
#define LED_TYPE APA102HD // Enables hardware gamma correction
|
||||
void setup() {
|
||||
FastLED.addLeds<LED_TYPE, DATA_PIN, CLOCK_PIN, RGB>(leds, NUM_LEDS);
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
|
||||
Read more: [APA102 HD Documentation](APA102.md) • [Rust Implementation](https://docs.rs/apa102-spi/latest/apa102_spi/)
|
||||
|
||||
## 🛠️ Development & Contributing
|
||||
|
||||
[](https://github.com/FastLED/FastLED/actions/workflows/build_clone_and_compile.yml)
|
||||
|
||||
**Ready to contribute?** FastLED welcomes code contributions, platform testing, documentation improvements, and community support.
|
||||
|
||||
<details>
|
||||
<summary>🔧 <strong>Development Setup & Contributing Guide</strong> (Click to expand)</summary>
|
||||
|
||||
### Quick Development Setup
|
||||
Zero pain setup - can be done from command line in seconds with `uv` or `python`:
|
||||
|
||||
|
||||
|
||||
**Steps:**
|
||||
1. Fork the repository on GitHub
|
||||
2. Clone your fork locally: `git clone https://github.com/yourusername/FastLED.git`
|
||||
3. Test compilation: `bash compile` (select your target platform)
|
||||
4. Make your changes and test
|
||||
5. Push to your fork and create a pull request
|
||||
|
||||
**See our detailed [Contributing Guide](CONTRIBUTING.md) for more information.**
|
||||
|
||||
### Testing Other Devices
|
||||
Use the `compile` script to test on 20+ platforms:
|
||||
- Teensy series, ESP32 variants, Arduino family, ARM boards, and more
|
||||
- Automated testing ensures compatibility across all supported platforms
|
||||
|
||||
### How to Help
|
||||
|
||||
| **📋 How to Help** | **🔗 Resources** |
|
||||
|-------------------|------------------|
|
||||
| **Code contributions**: Bug fixes, new features, optimizations | [Contributing Guide](CONTRIBUTING.md) |
|
||||
| **Platform support**: Help test on new/existing platforms | [Platform Testing](compile) |
|
||||
| **Documentation**: Improve examples, fix typos, add tutorials | [Documentation](http://fastled.io/docs) |
|
||||
| **Community**: Answer questions on Reddit, report issues | [r/FastLED](https://reddit.com/r/FastLED) |
|
||||
|
||||
**Platform Porting**: Information on porting FastLED to new platforms: [PORTING.md](PORTING.md)
|
||||
|
||||
### About FastLED
|
||||
|
||||
**What's in the name?** Originally "FastSPI_LED" focused on high-speed SPI, but evolved to support all LED types and became "FastLED" - everything fast, for LEDs.
|
||||
|
||||
**Official Site**: [fastled.io](http://fastled.io) - documentation, issues, and news
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Daniel Garcia, Founder of FastLED
|
||||
|
||||
In Memory of Daniel Garcia
|
||||
Daniel Garcia, the brilliant founder of FastLED, tragically passed away in September 2019 in the Conception dive boat fire alongside his partner, Yulia. This heartbreaking loss was felt deeply by the maker and developer community, where Daniel's contributions had left an indelible mark.
|
||||
|
||||
Daniel was more than just a talented programmer; he was a passionate innovator who transformed the way creators interacted with LED technology. His work on FastLED brought high-performance LED control to countless projects, empowering developers to craft breathtaking installations.
|
||||
|
||||
In his personal life, Daniel was known for his kindness and creativity. His pride in FastLED and the vibrant community it fostered was a testament to his dedication to open-source development and his commitment to helping others bring light into the world.
|
||||
|
||||
While Daniel is no longer with us, his legacy continues through the FastLED library and the countless makers who use it. The community he built serves as a living tribute to his ingenuity, generosity, and the joy he found in sharing his work with the world.
|
||||
|
||||
## About the Current Contributor
|
||||
|
||||
Zach Vorhies, the current main contributor to FastLED, briefly worked with Dan in 2014 in San Francisco and was an avid user of the FastLED library for over 13 years. After Daniel Garcia’s untimely passing, Zach stepped up to ensure FastLED’s continued growth and development.
|
||||
|
||||
Zach has this to say about FastLED:
|
||||
|
||||
*"The true power of FastLED lies in its ability to transform programmers into LED artists. Free space becomes their canvas; bending light is their medium. FastLED is a collective effort by programmers who want to manifest the world that science fiction writers promised us. -- To contribute code to FastLED is to leave behind a piece of something immortal."*
|
||||
|
||||
|
||||
---
|
||||
|
||||
**💫 In memory of Daniel Garcia and the vision that transforms programmers into LED artists**
|
||||
|
||||
*To stay updated on the latest feature releases, please click the `Watch` button in the upper right*
|
||||
""
|
||||
@@ -0,0 +1,39 @@
|
||||
# FastLED Release howto
|
||||
|
||||
*Pushing a fastled release, the short version, last updated May 2024*
|
||||
|
||||
## Example
|
||||
|
||||
https://github.com/FastLED/FastLED/commit/4444758ffaf853ba4f8deb973532548c9c1ee231
|
||||
|
||||
## How to
|
||||
|
||||
Edit these files to update the version number
|
||||
* library.json
|
||||
* library.properties
|
||||
* src/FastLED.h
|
||||
* docs/Doxyfile
|
||||
* RELEASE.md
|
||||
* This file: update instructions with the current release.
|
||||
|
||||
|
||||
Edit this file with release notes and version number.
|
||||
* release_notes.md
|
||||
|
||||
Release notes should list highlight changes (not necessarily all minor bug fixes) and thank people for their help.
|
||||
|
||||
Git commands to commit and tag release'
|
||||
```bash
|
||||
$ git commit -am "Rev 3.10.2"
|
||||
$ git tag 3.10.2 master
|
||||
$ git push
|
||||
$ git push origin 3.10.2
|
||||
```
|
||||
|
||||
Then use the GitHub UI to make a new “Release”:
|
||||
|
||||
https://github.com/FastLED/FastLED/releases/new
|
||||
|
||||
Announce new version on subreddit, highlighting major changes and thanking people for helping.
|
||||
|
||||
That’s it.
|
||||
@@ -0,0 +1,261 @@
|
||||
# FastLED TEST_CASE Execution Guide
|
||||
|
||||
## Overview
|
||||
|
||||
FastLED uses the **doctest** framework for C++ unit testing. Tests are organized in files named `test_*.cpp` in the `tests/` directory. Each file can contain multiple `TEST_CASE` macros that define individual test functions.
|
||||
|
||||
## Test Structure
|
||||
|
||||
- **Test Files**: Located in `tests/test_*.cpp` (e.g., `test_algorithm.cpp`, `test_easing.cpp`)
|
||||
- **TEST_CASEs**: Individual test functions defined with `TEST_CASE("name")` macro
|
||||
- **SUBCASEs**: Nested test sections within TEST_CASEs using `SUBCASE("name")`
|
||||
- **Assertions**: `CHECK()`, `CHECK_EQ()`, `CHECK_LT()`, etc. for validation
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Using the MCP Server (Recommended)
|
||||
|
||||
1. **Start the MCP server:**
|
||||
```bash
|
||||
uv run mcp_server.py
|
||||
```
|
||||
|
||||
2. **Available MCP tools:**
|
||||
- `run_tests`: Run tests with various options
|
||||
- `list_test_cases`: List available TEST_CASEs
|
||||
- `test_instructions`: Show detailed instructions
|
||||
|
||||
### Command Line Usage
|
||||
|
||||
1. **Run all tests:**
|
||||
```bash
|
||||
uv run test.py
|
||||
# or using the user rule:
|
||||
bash test
|
||||
```
|
||||
|
||||
2. **Run only C++ tests:**
|
||||
```bash
|
||||
uv run test.py --cpp
|
||||
```
|
||||
|
||||
3. **Run specific test file:**
|
||||
```bash
|
||||
uv run test.py --cpp algorithm
|
||||
# This runs test_algorithm.cpp
|
||||
```
|
||||
|
||||
## Running Specific TEST_CASEs
|
||||
|
||||
### Method 1: Direct Executable (Most Precise)
|
||||
|
||||
First, compile the tests, then run the specific executable with doctest filtering:
|
||||
|
||||
```bash
|
||||
# 1. Compile the test (if not already compiled)
|
||||
uv run test.py --cpp algorithm
|
||||
|
||||
# 2. Run specific TEST_CASE
|
||||
./tests/.build/bin/test_algorithm --test-case="reverse an int list"
|
||||
|
||||
# 3. Run with verbose output
|
||||
./tests/.build/bin/test_algorithm --test-case="reverse an int list" --verbose
|
||||
```
|
||||
|
||||
### Method 2: Using MCP Server
|
||||
|
||||
Use the `run_tests` tool with specific parameters:
|
||||
- `test_type`: "specific"
|
||||
- `specific_test`: "algorithm" (without test_ prefix)
|
||||
- `test_case`: "reverse an int list"
|
||||
- `verbose`: true (optional)
|
||||
|
||||
### Method 3: List Available TEST_CASEs First
|
||||
|
||||
```bash
|
||||
# Using MCP list_test_cases tool, or manually:
|
||||
grep -r "TEST_CASE" tests/test_algorithm.cpp
|
||||
```
|
||||
|
||||
## Common Test Files and Their Purpose
|
||||
|
||||
| File | Purpose | Example TEST_CASEs |
|
||||
|------|---------|-------------------|
|
||||
| `test_algorithm.cpp` | Algorithm utilities | "reverse an int list" |
|
||||
| `test_easing.cpp` | Easing functions | "8-bit easing functions", "easeInOutQuad16" |
|
||||
| `test_hsv16.cpp` | HSV color space | "HSV to RGB conversion", "HSV hue wraparound" |
|
||||
| `test_math.cpp` | Mathematical functions | "sin32 accuracy", "random number generation" |
|
||||
| `test_vector.cpp` | Vector container | "vector push_back", "vector iteration" |
|
||||
| `test_fx.cpp` | Effects framework | "fx engine initialization" |
|
||||
| `test_apa102_hd.cpp` | APA102 HD driver | "APA102 color mapping" |
|
||||
| `test_bitset.cpp` | Bitset operations | "bitset operations" |
|
||||
|
||||
## Example Workflows
|
||||
|
||||
### Debug a Failing Test
|
||||
|
||||
1. **List TEST_CASEs in a specific file:**
|
||||
```bash
|
||||
# Using MCP server list_test_cases tool with test_file: "easing"
|
||||
```
|
||||
|
||||
2. **Run with verbose output:**
|
||||
```bash
|
||||
uv run test.py --cpp easing --verbose
|
||||
```
|
||||
|
||||
3. **Run specific failing TEST_CASE:**
|
||||
```bash
|
||||
./tests/.build/bin/test_easing --test-case="8-bit easing functions" --verbose
|
||||
```
|
||||
|
||||
### Test Development Workflow
|
||||
|
||||
1. **Clean build and run:**
|
||||
```bash
|
||||
uv run test.py --cpp mytest --clean --verbose
|
||||
```
|
||||
|
||||
2. **Run only failed tests:**
|
||||
```bash
|
||||
cd tests/.build && ctest --rerun-failed
|
||||
```
|
||||
|
||||
3. **Test with different compilers:**
|
||||
```bash
|
||||
uv run test.py --cpp mytest --clang
|
||||
```
|
||||
|
||||
### Finding Specific Tests
|
||||
|
||||
1. **Search for TEST_CASEs by pattern:**
|
||||
```bash
|
||||
# Using MCP list_test_cases tool with search_pattern: "easing"
|
||||
```
|
||||
|
||||
2. **Manual search:**
|
||||
```bash
|
||||
grep -r "TEST_CASE.*easing" tests/
|
||||
```
|
||||
|
||||
## Doctest Command Line Options
|
||||
|
||||
When running test executables directly, you can use these doctest options:
|
||||
|
||||
- `--test-case=<name>`: Run specific TEST_CASE
|
||||
- `--test-case-exclude=<name>`: Exclude specific TEST_CASE
|
||||
- `--subcase=<name>`: Run specific SUBCASE
|
||||
- `--subcase-exclude=<name>`: Exclude specific SUBCASE
|
||||
- `--list-test-cases`: List all TEST_CASEs in the executable
|
||||
- `--list-test-suites`: List all test suites
|
||||
- `--verbose`: Show detailed output including successful assertions
|
||||
- `--success`: Show successful assertions too
|
||||
- `--no-colors`: Disable colored output
|
||||
- `--force-colors`: Force colored output
|
||||
- `--no-breaks`: Disable breaking into debugger
|
||||
- `--no-skip`: Don't skip any tests
|
||||
|
||||
### Examples:
|
||||
|
||||
```bash
|
||||
# List all TEST_CASEs in a test file
|
||||
./tests/.build/bin/test_easing --list-test-cases
|
||||
|
||||
# Run multiple TEST_CASEs with pattern
|
||||
./tests/.build/bin/test_easing --test-case="*quad*"
|
||||
|
||||
# Exclude specific TEST_CASEs
|
||||
./tests/.build/bin/test_easing --test-case-exclude="*16-bit*"
|
||||
|
||||
# Run with full verbose output
|
||||
./tests/.build/bin/test_easing --success --verbose
|
||||
```
|
||||
|
||||
## Understanding Test Output
|
||||
|
||||
### Successful Test:
|
||||
```
|
||||
[doctest] run with "--help" for options
|
||||
===============================================================================
|
||||
TEST_CASE: reverse an int list
|
||||
===============================================================================
|
||||
tests/test_algorithm.cpp:12
|
||||
all tests passed!
|
||||
===============================================================================
|
||||
[doctest] test cases: 1 | 1 passed | 0 failed | 0 skipped
|
||||
[doctest] assertions: 5 | 5 passed | 0 failed |
|
||||
[doctest] Status: SUCCESS!
|
||||
```
|
||||
|
||||
### Failed Test:
|
||||
```
|
||||
===============================================================================
|
||||
TEST_CASE: reverse an int list
|
||||
===============================================================================
|
||||
tests/test_algorithm.cpp:12
|
||||
tests/test_algorithm.cpp:20: ERROR: CHECK_EQ(vec[0], 6) is NOT correct!
|
||||
values: CHECK_EQ(5, 6)
|
||||
|
||||
===============================================================================
|
||||
[doctest] test cases: 1 | 0 passed | 1 failed | 0 skipped
|
||||
[doctest] assertions: 5 | 4 passed | 1 failed |
|
||||
[doctest] Status: FAILURE!
|
||||
```
|
||||
|
||||
## Tips and Best Practices
|
||||
|
||||
1. **Use `--verbose`** to see detailed test output and assertions
|
||||
2. **Use `--clean`** when testing after significant code changes
|
||||
3. **List TEST_CASEs first** to see what's available before running specific tests
|
||||
4. **Individual TEST_CASE execution** is useful for debugging specific functionality
|
||||
5. **Check test output carefully** - doctest provides detailed failure information with line numbers
|
||||
6. **Use pattern matching** with `--test-case="*pattern*"` to run related tests
|
||||
7. **Combine with debugger** - tests are compiled with debug symbols (`-g3`)
|
||||
|
||||
## Environment Setup
|
||||
|
||||
- Tests are compiled with Debug flags (`-g3`, `-O0`) for better debugging
|
||||
- GDB integration available for crash analysis
|
||||
- Static analysis warnings enabled as errors
|
||||
- Cross-platform support (Linux, macOS, Windows)
|
||||
- Uses zig/clang as the default compiler for consistency
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues:
|
||||
|
||||
1. **Test executable not found:**
|
||||
```bash
|
||||
# Make sure tests are compiled first
|
||||
uv run test.py --cpp <test_name>
|
||||
```
|
||||
|
||||
2. **TEST_CASE name not found:**
|
||||
```bash
|
||||
# List available TEST_CASEs
|
||||
./tests/.build/bin/test_<name> --list-test-cases
|
||||
```
|
||||
|
||||
3. **Compilation errors:**
|
||||
```bash
|
||||
# Clean build
|
||||
uv run test.py --cpp <test_name> --clean
|
||||
```
|
||||
|
||||
4. **Permission denied on executable:**
|
||||
```bash
|
||||
chmod +x tests/.build/bin/test_<name>
|
||||
```
|
||||
|
||||
## Integration with Development
|
||||
|
||||
### Git Hooks
|
||||
Consider adding a pre-commit hook to run relevant tests:
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Run tests for changed files
|
||||
uv run test.py --cpp
|
||||
```
|
||||
|
||||
### IDE Integration
|
||||
Most IDEs can be configured to run individual TEST_CASEs directly from the editor by configuring the test executable path and doctest arguments.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Testing
|
||||
|
||||
* REPO SYNC MODE: When there is a compiler error there isn't a hard failure. The compiler error needs to get propagate back to the top.
|
||||
|
||||
|
||||
* Esp32 testing
|
||||
* https://github.com/marketplace/actions/esp32-qemu-runner will run a sketch for X seconds and see's if it crashes
|
||||
* There's specific tests we'd like to run with this including the WS2812 and APA102 tests to test the clockless and clocked drivers
|
||||
|
||||
# Feature Enhancements
|
||||
|
||||
[ ] Adafruit converter driver
|
||||
[ ] NeoPixel converter driver
|
||||
|
||||
* I2S driver for ESP32 WS2812
|
||||
* https://github.com/hpwit/I2SClocklessLedDriver
|
||||
* Our copy is here: https://github.com/FastLED/FastLED/blob/master/src/platforms/esp/32/clockless_i2s_esp32.h
|
||||
* S3:
|
||||
* https://github.com/hpwit/I2SClockLessLedDriveresp32s3
|
||||
* Apparently, this driver allows MASSIVE parallelization for WS2812
|
||||
* Timing guide for reducing RMT frequency https://github.com/Makuna/NeoPixelBus/pull/795
|
||||
* ESp32 LED guide
|
||||
* web: https://components.espressif.com/components/espressif/led_strip
|
||||
* repo: https://github.com/espressif/idf-extra-components/tree/60c14263f3b69ac6e98ecae79beecbe5c18d5596/led_strip
|
||||
* adafruit conversation on RMT progress: https://github.com/adafruit/Adafruit_NeoPixel/issues/375
|
||||
|
||||
|
||||
* MIT Licensed SdFat library
|
||||
* https://github.com/greiman/SdFat
|
||||
* YVes LittleFS implementation for ESP
|
||||
* https://github.com/hpwit/ledOS/blob/main/src/fileSystem.h
|
||||
|
||||
* NimBLE for Arduino
|
||||
* https://github.com/h2zero/NimBLE-Arduino?tab=readme-ov-file
|
||||
|
||||
* Arduino test compile
|
||||
* https://github.com/hpwit/arduino-test-compile/blob/master/arduino-test-compile.sh
|
||||
|
||||
|
||||
# Misc:
|
||||
|
||||
* sutaburosu's guide to playing around with FastLED 4
|
||||
* https://github.com/sutaburosu/FastLED4-ESP32-playpen
|
||||
|
||||
|
||||
# Animartrix: try using sinf, cosf and other found trip instead of trying to go
|
||||
full double precision.
|
||||
@@ -0,0 +1,57 @@
|
||||
I was dissatisfied with a gamma corrected ramp using the APA102HD mode (it seemed to visibly step at one or more points), and some analysis of the 8 bit rgb + 5 bit brightness function showed why:
|
||||
https://www.argyllcms.com/APA102_loglog.svg.
|
||||
|
||||
So here is a suggested replacement for five_bit_bitshift() in fl/five_bit_hd_gamma.cpp, that fixed the problem for me:
|
||||
|
||||
// Improved FastLED five_bit_bitshift() function
|
||||
// This appears to exactly match the floating point reference,
|
||||
// with a worst case resulting error of 0.2% over the full 16 bit input range,
|
||||
// and an average error of 0.05% over that range. Errors scale with maximum
|
||||
// magnitude.
|
||||
|
||||
// ix/31 * 255/65536 * 256 scaling factors, valid for indexes 1..31
|
||||
static uint32_t bright_scale[32] = {
|
||||
0, 2023680, 1011840, 674560, 505920, 404736, 337280, 289097,
|
||||
252960, 224853, 202368, 183971, 168640, 155668, 144549, 134912,
|
||||
126480, 119040, 112427, 106509, 101184, 96366, 91985, 87986,
|
||||
84320, 80947, 77834, 74951, 72274, 69782, 67456, 65280
|
||||
};
|
||||
|
||||
// Since the return value wasn't used, it has been omitted.
|
||||
// It's not clear what scale brightness is, or how it is to be applied,
|
||||
// so we assume 8 bits applied over the given rgb values.
|
||||
void five_bit_bitshift(uint16_t r16, uint16_t g16, uint16_t b16, uint8_t brightness,
|
||||
CRGB *out, uint8_t *out_power_5bit) {
|
||||
uint8_t r8 = 0, g8 = 0, b8 = 0;
|
||||
|
||||
// Apply any brightness setting (we assume brightness is 0..255)
|
||||
if (brightness != 0xff) {
|
||||
r16 = scale16by8(r16, brightness);
|
||||
g16 = scale16by8(g16, brightness);
|
||||
b16 = scale16by8(b16, brightness);
|
||||
}
|
||||
|
||||
// Locate the largest value to set the brightness/scale factor
|
||||
uint16_t scale = max3(r16, g16, b16);
|
||||
|
||||
if (scale == 0) {
|
||||
*out = CRGB(0, 0, 0);
|
||||
*out_power_5bit = 0;
|
||||
return;
|
||||
} else {
|
||||
uint32_t scalef;
|
||||
|
||||
// Compute 5 bit quantized scale that is at or above the maximum value.
|
||||
scale = (scale + (2047 - (scale >> 5))) >> 11;
|
||||
|
||||
// Adjust the 16 bit values to account for the scale, then round to 8 bits
|
||||
scalef = bright_scale[scale];
|
||||
r8 = (r16 * scalef + 0x808000) >> 24;
|
||||
g8 = (g16 * scalef + 0x808000) >> 24;
|
||||
b8 = (b16 * scalef + 0x808000) >> 24;
|
||||
|
||||
*out = CRGB(r8, g8, b8);
|
||||
*out_power_5bit = scale;
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,246 @@
|
||||
# FastLED CI/Build System Agent Guidelines
|
||||
|
||||
## 🔧 Python Build System
|
||||
|
||||
### Core Architecture
|
||||
FastLED uses a high-performance Python-based build system:
|
||||
|
||||
**Key Components:**
|
||||
- `ci/compiler/clang_compiler.py` - Core compiler with PCH and fingerprint cache
|
||||
- `ci/ci/fingerprint_cache.py` - 99% faster PCH rebuilds through intelligent caching
|
||||
- `ci/util/build_flags.py` - TOML-based build configuration management
|
||||
- `ci/compiler/test_example_compilation.py` - Example compilation system
|
||||
|
||||
**Performance Features:**
|
||||
- **PCH (Precompiled Headers)** - Dramatically faster compilation
|
||||
- **Fingerprint Cache** - 99% time reduction for unchanged dependencies
|
||||
- **Parallel Compilation** - Automatic CPU-optimized parallel builds
|
||||
- **Smart Rebuilds** - Only rebuild when source files actually change
|
||||
|
||||
### 🚨 Command Execution Rules
|
||||
|
||||
#### FOREGROUND AGENTS (Interactive Development)
|
||||
**Python Code Execution:**
|
||||
- ✅ **Create/modify tmp.py** in project root (NOT subdirectories)
|
||||
- ✅ **Always run: `uv run python tmp.py`** (never just `python`)
|
||||
- ✅ **Correct import paths:** `from ci.ci.module` for ci.ci modules, `from ci.module` for top-level ci modules
|
||||
- ❌ **NEVER cd to subdirectories** - stay in project root
|
||||
|
||||
**Example:**
|
||||
```python
|
||||
# tmp.py (created in project root)
|
||||
from ci.ci.clang_compiler import Compiler # CORRECT import path
|
||||
import subprocess
|
||||
result = subprocess.run(['git', 'status'], capture_output=True, text=True)
|
||||
print(result.stdout)
|
||||
```
|
||||
Then run: `uv run python tmp.py`
|
||||
|
||||
#### BACKGROUND AGENTS (Automated/CI Environments)
|
||||
**Restrictions:**
|
||||
- ❌ **NO tmp.py files** (forbidden for background agents)
|
||||
- ✅ **Use existing scripts:** `uv run python ci/ci-compile.py uno --examples Blink`
|
||||
- ✅ **Use MCP server tools** for programmatic operations
|
||||
- ✅ **Always use `uv run python`** with existing scripts
|
||||
|
||||
## 🤖 MCP Server Configuration
|
||||
|
||||
**Start server:** `uv run mcp_server.py`
|
||||
|
||||
**Available Tools:**
|
||||
- Running tests with various options
|
||||
- Compiling examples for different platforms
|
||||
- Code fingerprinting and change detection
|
||||
- Linting and formatting
|
||||
- **Build info analysis** for platform-specific defines, compiler flags, toolchain information
|
||||
- **Symbol analysis** for binary optimization (all platforms)
|
||||
- Stack trace setup for enhanced debugging
|
||||
- **🌐 FastLED Web Compiler** with Playwright (FOREGROUND AGENTS ONLY)
|
||||
- **🚨 `validate_completion` tool** (MANDATORY for background agents)
|
||||
|
||||
### FastLED Web Compiler (FOREGROUND AGENTS ONLY)
|
||||
|
||||
**Prerequisites:**
|
||||
- `pip install fastled` - FastLED web compiler
|
||||
- `pip install playwright` - Browser automation
|
||||
- Docker (optional, for faster compilation)
|
||||
|
||||
**Usage via MCP Server:**
|
||||
```python
|
||||
# Basic usage
|
||||
use run_fastled_web_compiler tool with:
|
||||
- example_path: "examples/Audio"
|
||||
- capture_duration: 30
|
||||
- headless: false
|
||||
- save_screenshot: true
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- Compiles Arduino sketches to WASM for browser execution
|
||||
- Captures console.log output with playwright automation
|
||||
- Takes screenshots of running visualizations
|
||||
- Monitors FastLED_onFrame calls to verify proper initialization
|
||||
|
||||
**🚫 BACKGROUND AGENT RESTRICTION:** This tool is COMPLETELY DISABLED for background agents and CI environments.
|
||||
|
||||
## 🤖 Linting Guidelines
|
||||
|
||||
### FOREGROUND AGENTS
|
||||
**🚨 ALWAYS USE `bash lint` - DO NOT RUN INDIVIDUAL LINTING SCRIPTS**
|
||||
|
||||
- ✅ **CORRECT:** `bash lint`
|
||||
- ❌ **FORBIDDEN:** `./lint-js`, `./check-js`, `python3 scripts/enhance-js-typing.py`
|
||||
- ❌ **FORBIDDEN:** `uv run ruff check`, `uv run black`, individual tools
|
||||
|
||||
**WHY:** `bash lint` provides comprehensive coverage of Python, C++, JavaScript, and enhancement analysis.
|
||||
|
||||
### BACKGROUND AGENTS
|
||||
**CAN USE FINE-GRAINED LINTING FOR SPECIFIC NEEDS**
|
||||
|
||||
Background agents may use individual linting scripts when needed:
|
||||
- `./lint-js` - JavaScript-only linting
|
||||
- `./check-js` - JavaScript type checking
|
||||
- `python3 scripts/enhance-js-typing.py` - JavaScript enhancement analysis
|
||||
- `uv run ruff check` - Python linting only
|
||||
- MCP server `lint_code` tool - Programmatic access
|
||||
|
||||
**BUT STILL PREFER `bash lint` FOR COMPREHENSIVE CHECKING**
|
||||
|
||||
## Build System Commands
|
||||
|
||||
**Unit Test Compilation:**
|
||||
```bash
|
||||
# Fast Python API (default)
|
||||
bash test --unit --verbose # Uses PCH optimization
|
||||
|
||||
# Disable PCH if needed
|
||||
bash test --unit --no-pch --verbose
|
||||
|
||||
# Legacy CMake system (8x slower)
|
||||
bash test --unit --legacy --verbose
|
||||
```
|
||||
|
||||
**Available Build Options:**
|
||||
- `--no-pch` - Disable precompiled headers
|
||||
- `--clang` - Use Clang compiler (recommended for speed)
|
||||
- `--clean` - Force full rebuild
|
||||
- `--verbose` - Show detailed compilation output
|
||||
- `--legacy` - Use old CMake system (discouraged)
|
||||
|
||||
**Platform Compilation:**
|
||||
```bash
|
||||
# Compile examples for specific platforms
|
||||
uv run ci/ci-compile.py uno --examples Blink
|
||||
uv run ci/ci-compile.py esp32dev --examples Blink
|
||||
uv run ci/ci-compile.py teensy31 --examples Blink
|
||||
```
|
||||
|
||||
**WASM Compilation:**
|
||||
```bash
|
||||
# Compile any sketch directory to WASM
|
||||
uv run ci/wasm_compile.py path/to/your/sketch
|
||||
|
||||
# Quick compile test (compile only, no browser)
|
||||
uv run ci/wasm_compile.py examples/Blink --just-compile
|
||||
|
||||
# Compile and open browser
|
||||
uv run ci/wasm_compile.py examples/Blink -b --open
|
||||
```
|
||||
|
||||
## Build Troubleshooting
|
||||
|
||||
**For Linker Issues:**
|
||||
1. Check `tests/cmake/LinkerCompatibility.cmake` first
|
||||
2. Look for lld-link detection and compatibility functions
|
||||
3. Check GNU→MSVC flag translation logic
|
||||
4. Verify warning suppression settings
|
||||
|
||||
**For Compiler Issues:**
|
||||
1. Check `tests/cmake/CompilerDetection.cmake` for detection logic
|
||||
2. Review `tests/cmake/CompilerFlags.cmake` for flag conflicts
|
||||
3. Verify optimization settings in `OptimizationSettings.cmake`
|
||||
|
||||
**For Build Performance:**
|
||||
1. Check `tests/cmake/ParallelBuild.cmake` for parallel settings
|
||||
2. Review LTO configuration in `OptimizationSettings.cmake`
|
||||
3. Verify linker selection (mold, lld, default)
|
||||
|
||||
## 🚨 Critical User Feedback Corrections
|
||||
|
||||
**Always Use `uv run python` - NEVER just `python` or `uv run`:**
|
||||
- ❌ **WRONG**: `python -c "..."`
|
||||
- ❌ **WRONG**: `uv run -c "..."`
|
||||
- ✅ **CORRECT**: `uv run python -c "..."`
|
||||
|
||||
**Correct Import Paths:**
|
||||
- ✅ **CORRECT**: `from ci.ci.clang_compiler import ...` (for ci.ci modules)
|
||||
- ✅ **CORRECT**: `from ci.clang_compiler import ...` (for top-level ci modules, if they exist)
|
||||
|
||||
**Never Change Directory:**
|
||||
- ❌ **WRONG**: `cd ci && python ...`
|
||||
- ❌ **WRONG**: `cd ci/ci && python ...`
|
||||
- ✅ **CORRECT**: Stay in project root, use full paths: `uv run python ci/script.py`
|
||||
|
||||
**Git-Bash Terminal Compatibility:**
|
||||
Use leading space for git-bash compatibility:
|
||||
- ✅ **CORRECT**: ` bash test` (note the leading space)
|
||||
- ❌ **INCORRECT**: `bash test` (may get truncated to `ash test`)
|
||||
|
||||
## Platform Build Information Analysis
|
||||
|
||||
**Generate Build Info:**
|
||||
```bash
|
||||
# Compile a platform to generate build_info.json
|
||||
uv run ci/ci-compile.py uno --examples Blink
|
||||
uv run ci/ci-compile.py esp32dev --examples Blink
|
||||
```
|
||||
|
||||
**Analyze Platform Defines:**
|
||||
```bash
|
||||
# Command line tool
|
||||
python3 ci/ci/build_info_analyzer.py --board uno --show-defines
|
||||
|
||||
# Via MCP server
|
||||
uv run mcp_server.py
|
||||
# Use build_info_analysis tool with: board="uno", show_defines=true
|
||||
```
|
||||
|
||||
**Compare Platforms:**
|
||||
```bash
|
||||
# Command line
|
||||
python3 ci/ci/build_info_analyzer.py --compare uno esp32dev
|
||||
|
||||
# Via MCP
|
||||
# Use build_info_analysis tool with: board="uno", compare_with="esp32dev"
|
||||
```
|
||||
|
||||
## Symbol Analysis for Binary Optimization
|
||||
|
||||
**Analyze specific platform:**
|
||||
```bash
|
||||
uv run ci/ci/symbol_analysis.py --board uno
|
||||
uv run ci/ci/symbol_analysis.py --board esp32dev
|
||||
uv run ci/ci/symbol_analysis.py --board teensy31
|
||||
```
|
||||
|
||||
**Analyze all available platforms:**
|
||||
```bash
|
||||
uv run ci/demo_symbol_analysis.py
|
||||
```
|
||||
|
||||
**Using MCP Server:**
|
||||
```bash
|
||||
# Use MCP server tools
|
||||
uv run mcp_server.py
|
||||
# Then use symbol_analysis tool with board: "uno" or "auto"
|
||||
# Or use symbol_analysis tool with run_all_platforms: true
|
||||
```
|
||||
|
||||
**JSON Output:**
|
||||
```bash
|
||||
uv run symbol_analysis.py --board esp32dev --output-json
|
||||
# Results saved to: .build/esp32dev_symbol_analysis.json
|
||||
```
|
||||
|
||||
## Memory Refresh Rule
|
||||
**🚨 ALL AGENTS: Read ci/AGENTS.md before concluding CI/build work to refresh memory about current Python build system rules and MCP server requirements.**
|
||||
@@ -0,0 +1,228 @@
|
||||
# FastLED Python Build System - Implementation Summary
|
||||
|
||||
## Overview
|
||||
|
||||
Successfully implemented a new Python-based build system to replace the complex CMake approach for FastLED example compilation testing. The new system delivers on all design goals: **Simple, Fast, and Transparent**.
|
||||
|
||||
## Key Achievements
|
||||
|
||||
### ⚡ Performance Results
|
||||
- **4.3 examples/second** compilation rate
|
||||
- **90%+ faster** than CMake-based approach
|
||||
- **Sub-second builds** for incremental changes
|
||||
- **Intelligent caching** with 30%+ hit rates on repeated builds
|
||||
|
||||
### 🏗️ Architecture Implemented
|
||||
|
||||
```
|
||||
FastLED Python Build System
|
||||
├── ci/build_system/
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ ├── build_config.py # Configuration management
|
||||
│ ├── example_scanner.py # .ino file discovery & analysis
|
||||
│ ├── change_detector.py # Fast file change detection
|
||||
│ ├── compiler.py # Direct Clang compilation
|
||||
│ ├── cache_manager.py # Build cache management
|
||||
│ └── fastled_build.py # Main build engine
|
||||
├── ci/fastled_build # CLI entry point
|
||||
├── ci/test_example_compilation_python.py # New test script
|
||||
└── ci/test_example_compilation.py # Updated original (with --use-python-build flag)
|
||||
```
|
||||
|
||||
### 🎯 Core Features Delivered
|
||||
|
||||
#### 1. **Fast Example Discovery**
|
||||
- Scans 80+ examples in milliseconds
|
||||
- Categorizes by FastLED usage and PCH compatibility
|
||||
- Intelligent filtering of problematic examples
|
||||
|
||||
#### 2. **Content-Based Change Detection**
|
||||
- SHA-256 fingerprinting of source files
|
||||
- Dependency tracking for headers
|
||||
- Smart cache invalidation
|
||||
|
||||
#### 3. **Precompiled Header Optimization**
|
||||
- Automatic PCH generation for FastLED headers
|
||||
- PCH reuse across compatible examples
|
||||
- 50%+ compilation speed improvement
|
||||
|
||||
#### 4. **Intelligent Caching**
|
||||
- Content-addressable build cache
|
||||
- sccache integration for compiler-level caching
|
||||
- Automatic cleanup of stale entries
|
||||
|
||||
#### 5. **Parallel Compilation**
|
||||
- Multi-threaded compilation (16 cores utilized)
|
||||
- ThreadPoolExecutor-based job management
|
||||
- Optimal resource utilization
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### CLI Tool
|
||||
```bash
|
||||
# Quick status check
|
||||
./fastled_build status
|
||||
|
||||
# Build all examples
|
||||
./fastled_build build
|
||||
|
||||
# Build specific examples
|
||||
./fastled_build build Blink DemoReel100
|
||||
|
||||
# Clean build
|
||||
./fastled_build clean
|
||||
./fastled_build build
|
||||
|
||||
# System diagnostics
|
||||
./fastled_build diagnostics
|
||||
```
|
||||
|
||||
### Test Scripts
|
||||
```bash
|
||||
# Default Python build system (recommended)
|
||||
python test_example_compilation.py
|
||||
|
||||
# Alternative Python build system script
|
||||
python test_example_compilation_python.py
|
||||
|
||||
# Both scripts now use Python build system by default
|
||||
python test_example_compilation.py Blink DemoReel100 # Python system
|
||||
```
|
||||
|
||||
## Technical Implementation Details
|
||||
|
||||
### Example Categorization
|
||||
- **FastLED PCH Compatible**: Examples that can use precompiled headers
|
||||
- **FastLED Full Build**: Examples requiring full compilation
|
||||
- **Basic Examples**: Non-FastLED examples
|
||||
- **Platform Specific**: Hardware-dependent examples
|
||||
- **Problematic**: Known issues (excluded by default)
|
||||
|
||||
### Build Configuration
|
||||
- **Compiler**: Auto-detection (Clang preferred)
|
||||
- **Platform**: STUB implementation for testing
|
||||
- **Flags**: C++17, -O0 for fast compilation
|
||||
- **Arduino Compatibility**: WASM Arduino.h integration
|
||||
|
||||
### Performance Optimizations
|
||||
- **Fast Paths**: "No changes" detection in < 1 second
|
||||
- **PCH Reuse**: Share PCH across compatible examples
|
||||
- **Content Hashing**: Only hash changed files
|
||||
- **Parallel Processing**: All I/O and compilation parallelized
|
||||
|
||||
## Migration Status
|
||||
|
||||
### ✅ Phase 1: Parallel Implementation - COMPLETE
|
||||
- ✅ Implemented Python build system alongside CMake
|
||||
- ✅ Added `--use-python-build` flag to test script
|
||||
- ✅ Validated equivalent functionality
|
||||
- ✅ Performance testing shows 4x+ improvement
|
||||
|
||||
### 🚧 Phase 2: Feature Parity - IN PROGRESS
|
||||
- ✅ All CMake features replicated
|
||||
- ✅ Performance optimization completed
|
||||
- ✅ CLI and API documentation
|
||||
- ⏳ Extended platform testing
|
||||
|
||||
### ✅ Phase 3: Migration - COMPLETE
|
||||
- ✅ Switched default to Python build system
|
||||
- ✅ Removed CMake implementation completely
|
||||
- ✅ Updated `test_example_compilation.py` to use Python build system only
|
||||
- ✅ Maintained backward compatibility with same CLI interface
|
||||
|
||||
### 🧹 Phase 4: Cleanup - ⚠️ **INCOMPLETE**
|
||||
- ❌ Remove misleading "CMake" references in function names and error messages
|
||||
- ❌ Clean up dead code paths that reference non-existent CMake directories
|
||||
- ❌ Fix confusing system names (both systems are Python-based)
|
||||
- ❌ Update all documentation to reflect accurate system architecture
|
||||
- ⚠️ **Status**: Misleading naming caused confusion and bugs; cleanup required
|
||||
|
||||
## Performance Comparison
|
||||
|
||||
| Metric | CMake System | Python System | Improvement |
|
||||
|--------|--------------|---------------|-------------|
|
||||
| Cold start | 15-30s | 1-2s | **15x faster** |
|
||||
| Incremental (5 examples) | 8-15s | 1-3s | **5x faster** |
|
||||
| Full build (80 examples) | 45-60s | 10-15s | **4x faster** |
|
||||
| Cache hit rate | ~60% | ~90% | **50% better** |
|
||||
| Memory usage | 2-4GB | 200-500MB | **80% less** |
|
||||
|
||||
## Error Handling & Diagnostics
|
||||
|
||||
### Comprehensive Error Reporting
|
||||
- Clear, actionable error messages
|
||||
- Compilation error aggregation
|
||||
- Performance timing breakdown
|
||||
- Cache statistics and health
|
||||
|
||||
### Diagnostic Tools
|
||||
```bash
|
||||
./fastled_build diagnostics
|
||||
```
|
||||
Shows:
|
||||
- Compiler information and availability
|
||||
- Cache statistics and hit rates
|
||||
- sccache integration status
|
||||
- File change detection stats
|
||||
|
||||
## Integration with Existing Workflow
|
||||
|
||||
### Backward Compatibility
|
||||
- ✅ Original `test_example_compilation.py` still works with same interface
|
||||
- ✅ Same command-line arguments and exit codes
|
||||
- ✅ Same behavior and output format
|
||||
- ✅ Complete drop-in replacement (CMake removed)
|
||||
- ✅ Performance improvement with no API changes
|
||||
|
||||
### Testing Integration
|
||||
```python
|
||||
# Easy integration in existing test scripts
|
||||
from build_system import BuildConfig, FastLEDBuildEngine
|
||||
|
||||
config = BuildConfig.from_environment()
|
||||
engine = FastLEDBuildEngine(config)
|
||||
result = engine.build(examples=["Blink", "DemoReel100"])
|
||||
assert result.success
|
||||
```
|
||||
|
||||
## Benefits Realized
|
||||
|
||||
### 1. **Simplicity**
|
||||
- ✅ Direct compiler control (no CMake abstraction)
|
||||
- ✅ Python stack traces for easy debugging
|
||||
- ✅ Transparent, understandable build logic
|
||||
|
||||
### 2. **Performance**
|
||||
- ✅ 4x+ faster than CMake approach
|
||||
- ✅ Intelligent caching tailored to FastLED
|
||||
- ✅ Efficient incremental builds
|
||||
|
||||
### 3. **Maintainability**
|
||||
- ✅ Python codebase (team expertise)
|
||||
- ✅ Modular, testable design
|
||||
- ✅ Clear separation of concerns
|
||||
|
||||
### 4. **Flexibility**
|
||||
- ✅ Easy FastLED-specific optimizations
|
||||
- ✅ Simple tool integration
|
||||
- ✅ Consistent cross-platform behavior
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Extended Testing**: Test with more examples and edge cases
|
||||
2. **Documentation**: Update development docs to recommend Python system
|
||||
3. **CI Integration**: Consider switching CI to use Python build system
|
||||
4. **Feature Additions**: Add requested features (linking, size analysis, etc.)
|
||||
|
||||
## Conclusion
|
||||
|
||||
⚠️ **MIGRATION STATUS**: The Python build system has successfully replaced the CMake approach, but **Phase 4 cleanup was never completed**, leaving misleading function names and error messages that reference "CMake" when both systems are actually Python-based.
|
||||
|
||||
### Final Results
|
||||
- **🚀 13.3 examples/second** compilation rate (in cached scenarios)
|
||||
- **⚡ Ultra-fast builds** with intelligent caching
|
||||
- **🔄 100% backward compatibility** with existing workflows
|
||||
- **🧹 Clean codebase** with CMake complexity completely removed
|
||||
- **📊 Superior performance** across all metrics
|
||||
|
||||
The system delivers on all original design goals: **Simple, Fast, and Transparent**. However, **incomplete cleanup has left confusing naming** that makes the system less transparent than intended. The naming issues have been identified and are being addressed.
|
||||
@@ -0,0 +1,11 @@
|
||||
This is the Continuous Integration tool that builds the product in various configurations.
|
||||
|
||||
Running
|
||||
|
||||
Install the python `uv` tool.
|
||||
|
||||
`pip install uv`
|
||||
|
||||
Now run the python in this directory
|
||||
|
||||
`uv run ci-compile.py`
|
||||
@@ -0,0 +1,313 @@
|
||||
# FastLED Symbol Analysis
|
||||
|
||||
This document describes the new generic symbol analysis functionality that works across all supported platforms including UNO, ESP32, and others.
|
||||
|
||||
## Overview
|
||||
|
||||
The symbol analysis tool provides comprehensive analysis of compiled ELF files to help identify optimization opportunities and understand binary size allocation. Unlike the previous ESP32-specific implementation, this tool:
|
||||
|
||||
- ✅ **Works on ALL platforms** (UNO, ESP32, Teensy, STM32, etc.)
|
||||
- ✅ **Pulls ALL symbols** without filtering or classification
|
||||
- ✅ **No FastLED-specific filtering** - analyzes everything
|
||||
- ✅ **Generic and extensible** for any platform with build_info.json
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Analyze a specific platform
|
||||
|
||||
```bash
|
||||
cd ci
|
||||
uv run symbol_analysis.py --board uno
|
||||
uv run symbol_analysis.py --board esp32dev
|
||||
```
|
||||
|
||||
### Run analysis on all available platforms
|
||||
|
||||
```bash
|
||||
uv run demo_symbol_analysis.py
|
||||
```
|
||||
|
||||
### Use in GitHub Actions
|
||||
|
||||
```bash
|
||||
uv run symbol_analysis_runner.py --board uno --example Blink
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
- **`symbol_analysis.py`** - Main generic symbol analysis tool
|
||||
- **`symbol_analysis_runner.py`** - GitHub Actions wrapper
|
||||
- **`demo_symbol_analysis.py`** - Demonstration script for multiple platforms
|
||||
- **`SYMBOL_ANALYSIS.md`** - This documentation file
|
||||
|
||||
## Example Output
|
||||
|
||||
### UNO Platform Results
|
||||
|
||||
```
|
||||
================================================================================
|
||||
UNO SYMBOL ANALYSIS REPORT
|
||||
================================================================================
|
||||
|
||||
SUMMARY:
|
||||
Total symbols: 51
|
||||
Total symbol size: 3767 bytes (3.7 KB)
|
||||
|
||||
LARGEST SYMBOLS (all symbols, sorted by size):
|
||||
1. 1204 bytes - ClocklessController<...>::showPixels(...)
|
||||
2. 572 bytes - CFastLED::show(unsigned char) [clone .constprop.18]
|
||||
3. 460 bytes - main
|
||||
4. 204 bytes - CPixelLEDController<...>::show(...)
|
||||
5. 168 bytes - CPixelLEDController<...>::showColor(...)
|
||||
|
||||
SYMBOL TYPE BREAKDOWN:
|
||||
t: 19 symbols, 2710 bytes (2.6 KB)
|
||||
T: 14 symbols, 904 bytes (0.9 KB)
|
||||
b: 15 symbols, 89 bytes (0.1 KB)
|
||||
d: 3 symbols, 64 bytes (0.1 KB)
|
||||
```
|
||||
|
||||
### ESP32 Platform Results
|
||||
|
||||
```
|
||||
================================================================================
|
||||
ESP32DEV SYMBOL ANALYSIS REPORT
|
||||
================================================================================
|
||||
|
||||
SUMMARY:
|
||||
Total symbols: 2503
|
||||
Total symbol size: 237092 bytes (231.5 KB)
|
||||
|
||||
LARGEST SYMBOLS (all symbols, sorted by size):
|
||||
1. 12009 bytes - _vfprintf_r
|
||||
2. 11813 bytes - _svfprintf_r
|
||||
3. 8010 bytes - _vfiprintf_r
|
||||
4. 7854 bytes - _svfiprintf_r
|
||||
5. 4192 bytes - port_IntStack
|
||||
|
||||
SYMBOL TYPE BREAKDOWN:
|
||||
T: 1240 symbols, 156230 bytes (152.6 KB)
|
||||
t: 371 symbols, 45841 bytes (44.8 KB)
|
||||
d: 475 symbols, 13032 bytes (12.7 KB)
|
||||
D: 60 symbols, 9421 bytes (9.2 KB)
|
||||
W: 126 symbols, 5209 bytes (5.1 KB)
|
||||
```
|
||||
|
||||
### ESP32S3 Platform Results
|
||||
|
||||
```
|
||||
================================================================================
|
||||
ESP32S3 SYMBOL ANALYSIS REPORT
|
||||
================================================================================
|
||||
|
||||
SUMMARY:
|
||||
Total symbols: 2831
|
||||
Total symbol size: 260710 bytes (254.6 KB)
|
||||
|
||||
LARGEST SYMBOLS (all symbols, sorted by size):
|
||||
1. 11309 bytes - _vfprintf_r
|
||||
2. 11118 bytes - _svfprintf_r
|
||||
3. 7337 bytes - _svfiprintf_r
|
||||
4. 7265 bytes - _vfiprintf_r
|
||||
5. 4192 bytes - port_IntStack
|
||||
6. 1164 bytes - printBeforeSetupInfo()
|
||||
7. 828 bytes - fl::RmtController5::loadPixelData(PixelIterator&)
|
||||
|
||||
SYMBOL TYPE BREAKDOWN:
|
||||
T: 1416 symbols, 172591 bytes (168.5 KB)
|
||||
t: 429 symbols, 48854 bytes (47.7 KB)
|
||||
d: 529 symbols, 14189 bytes (13.9 KB)
|
||||
D: 72 symbols, 9831 bytes (9.6 KB)
|
||||
W: 122 symbols, 5457 bytes (5.3 KB)
|
||||
```
|
||||
|
||||
## Output Files
|
||||
|
||||
The tool generates JSON files with complete symbol data:
|
||||
|
||||
- `.build/{board}_symbol_analysis.json` - Complete analysis results
|
||||
|
||||
### JSON Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": {
|
||||
"board": "uno",
|
||||
"total_symbols": 51,
|
||||
"total_size": 3767,
|
||||
"largest_symbols": [...],
|
||||
"type_breakdown": {...},
|
||||
"dependencies": {...}
|
||||
},
|
||||
"all_symbols_sorted_by_size": [
|
||||
{
|
||||
"address": "00001240",
|
||||
"size": 1204,
|
||||
"type": "t",
|
||||
"name": "_ZN19ClocklessController...",
|
||||
"demangled_name": "ClocklessController<...>::showPixels(...)"
|
||||
},
|
||||
...
|
||||
],
|
||||
"dependencies": {
|
||||
"module.cpp.o": ["symbol1", "symbol2", ...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Supported Platforms
|
||||
|
||||
The tool automatically detects and works with any platform that has:
|
||||
- A `build_info.json` file in `.build/{platform}/`
|
||||
- Standard toolchain tools (nm, c++filt) in the build info aliases
|
||||
|
||||
Successfully tested platforms:
|
||||
- ✅ **UNO** (AVR) - 51 symbols, 3.7 KB
|
||||
- ✅ **ESP32DEV** (Xtensa) - 2503 symbols, 231.5 KB
|
||||
- ✅ **ESP32S3** (Xtensa) - 2831 symbols, 254.6 KB
|
||||
- ✅ **TEENSY31** (ARM Cortex-M4) - 238 symbols, 10.9 KB
|
||||
- ✅ **TEENSYLC** (ARM Cortex-M0+) - 241 symbols, 9.1 KB
|
||||
- ✅ **DIGIX** (ARM Cortex-M3) - 298 symbols, 17.9 KB
|
||||
- ✅ **STM32** (ARM Cortex-M3) - 311 symbols, 15.6 KB
|
||||
|
||||
## Key Improvements
|
||||
|
||||
### From ESP32-specific to Generic
|
||||
|
||||
**Before (ESP32 only):**
|
||||
```python
|
||||
# Hard-coded ESP32 board detection
|
||||
esp32_boards = ["esp32dev", "esp32", "esp32s2", ...]
|
||||
|
||||
# FastLED-specific filtering
|
||||
if any(keyword in search_text for keyword in [
|
||||
"fastled", "cfastled", "crgb", "hsv", ...]):
|
||||
fastled_symbols.append(symbol_info)
|
||||
```
|
||||
|
||||
**After (Generic):**
|
||||
```python
|
||||
# Auto-detect any platform with build_info.json
|
||||
for item in build_dir.iterdir():
|
||||
if item.is_dir() and (item / "build_info.json").exists():
|
||||
available_boards.append((build_info_file, item.name))
|
||||
|
||||
# No filtering - analyze ALL symbols
|
||||
symbols.append(symbol_info) # Everything included
|
||||
```
|
||||
|
||||
### Symbol Analysis Improvements
|
||||
|
||||
1. **No Classification** - All symbols are included without filtering
|
||||
2. **Complete Demangling** - All C++ symbols are properly demangled
|
||||
3. **Platform Agnostic** - Works with any toolchain (AVR, ARM, Xtensa, etc.)
|
||||
4. **Size-based Sorting** - All results sorted by symbol size for easy optimization targeting
|
||||
|
||||
## GitHub Actions Integration
|
||||
|
||||
The `symbol_analysis_runner.py` script provides GitHub Actions compatibility:
|
||||
|
||||
```yaml
|
||||
- name: Run Symbol Analysis
|
||||
run: python ci/symbol_analysis_runner.py --board uno --example Blink
|
||||
```
|
||||
|
||||
This runner:
|
||||
- Validates build_info.json exists
|
||||
- Handles errors gracefully with `--skip-on-failure`
|
||||
- Provides proper exit codes for CI/CD
|
||||
|
||||
## Migration from ESP32 Analysis
|
||||
|
||||
If you were using the old ESP32-specific analysis:
|
||||
|
||||
**Old command:**
|
||||
```bash
|
||||
cd ci/ci && python esp32_symbol_analysis.py
|
||||
```
|
||||
|
||||
**New command:**
|
||||
```bash
|
||||
cd ci && uv run symbol_analysis.py --board esp32dev
|
||||
```
|
||||
|
||||
The new tool provides all the same information plus:
|
||||
- Support for more platforms
|
||||
- ALL symbols instead of just FastLED ones
|
||||
- Better error handling
|
||||
- Cleaner output format
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Analyzing Custom Platforms
|
||||
|
||||
To add support for a new platform:
|
||||
|
||||
1. Ensure the platform builds correctly with PlatformIO
|
||||
2. Verify `build_info.json` is generated in `.build/{platform}/`
|
||||
3. Run: `uv run symbol_analysis.py --board {platform}`
|
||||
|
||||
### Batch Analysis
|
||||
|
||||
```bash
|
||||
# Analyze all available platforms
|
||||
uv run demo_symbol_analysis.py
|
||||
|
||||
# Analyze specific platforms in sequence
|
||||
for board in uno esp32dev teensy31; do
|
||||
uv run symbol_analysis.py --board $board
|
||||
done
|
||||
```
|
||||
|
||||
### Integration with Build Systems
|
||||
|
||||
```python
|
||||
# In your Python build script
|
||||
from ci.ci.symbol_analysis import main as analyze_symbols
|
||||
import sys
|
||||
|
||||
# Override arguments
|
||||
sys.argv = ["symbol_analysis.py", "--board", "uno"]
|
||||
analyze_symbols()
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **"build_info.json not found"**
|
||||
- Solution: Compile the platform first: `uv run ci/ci-compile.py {board} --examples Blink`
|
||||
|
||||
2. **"KeyError: 'board_name'"**
|
||||
- Solution: The board name in the directory doesn't match the key in build_info.json
|
||||
- Check: `cat .build/{board}/build_info.json | jq 'keys'`
|
||||
|
||||
3. **"Tool not found"**
|
||||
- Solution: The toolchain isn't installed or path is incorrect in build_info.json
|
||||
|
||||
### Debug Mode
|
||||
|
||||
To debug issues, run with verbose Python output:
|
||||
```bash
|
||||
uv run python -v symbol_analysis.py --board uno
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new features:
|
||||
|
||||
1. Maintain platform independence
|
||||
2. Don't add filtering logic
|
||||
3. Test on multiple platforms
|
||||
4. Update this documentation
|
||||
5. Add tests to verify functionality
|
||||
|
||||
## Performance
|
||||
|
||||
Platform analysis times (approximate):
|
||||
- **UNO**: < 1 second (51 symbols)
|
||||
- **Teensy**: < 2 seconds (238-241 symbols)
|
||||
- **STM32**: < 3 seconds (311 symbols)
|
||||
- **ESP32**: < 10 seconds (2503 symbols)
|
||||
|
||||
The tool scales well with symbol count and provides progress feedback during demangling.
|
||||
@@ -0,0 +1,13 @@
|
||||
"""FastLED CI tools and utilities
|
||||
|
||||
Module-level initialization to ensure consistent runtime behavior across tools.
|
||||
"""
|
||||
|
||||
# Configure UTF-8 console output on Windows globally for CI tools
|
||||
try:
|
||||
from ci.util.console_utf8 import configure_utf8_console
|
||||
|
||||
configure_utf8_console()
|
||||
except Exception:
|
||||
# Never fail import due to console configuration
|
||||
pass
|
||||
@@ -0,0 +1,45 @@
|
||||
|
||||
|
||||
|
||||
#define FASTLED_USE_ADAFRUIT_NEOPIXEL
|
||||
#include "FastLED.h"
|
||||
|
||||
// Test the non-template Adafruit bridge
|
||||
#define DATA_PIN 2
|
||||
#define NUM_LEDS 144
|
||||
|
||||
// Test LED arrays with different RGB orders
|
||||
CRGB leds_grb[NUM_LEDS];
|
||||
CRGB leds_rgb[NUM_LEDS];
|
||||
CRGB leds_bgr[NUM_LEDS];
|
||||
|
||||
void setup() {
|
||||
Serial.begin(115200);
|
||||
|
||||
// Test template controllers
|
||||
FastLED.addLeds<WS2812, DATA_PIN, GRB>(leds_grb, NUM_LEDS);
|
||||
FastLED.addLeds<WS2812, DATA_PIN+1, RGB>(leds_rgb, NUM_LEDS);
|
||||
FastLED.addLeds<WS2812, DATA_PIN+2, BGR>(leds_bgr, NUM_LEDS);
|
||||
|
||||
// Fill with test patterns
|
||||
fill_rainbow(leds_grb, NUM_LEDS, 0, 255/NUM_LEDS);
|
||||
fill_solid(leds_rgb, NUM_LEDS, CRGB::Red);
|
||||
fill_solid(leds_bgr, NUM_LEDS, CRGB::Blue);
|
||||
|
||||
FastLED.show();
|
||||
Serial.println("Adafruit bridge test initialized");
|
||||
}
|
||||
|
||||
void loop() {
|
||||
static uint8_t hue = 0;
|
||||
|
||||
// Update colors
|
||||
fill_rainbow(leds_grb, NUM_LEDS, hue, 255/NUM_LEDS);
|
||||
fill_solid(leds_rgb, NUM_LEDS, CHSV(hue, 255, 255));
|
||||
fill_solid(leds_bgr, NUM_LEDS, CHSV(hue + 128, 255, 255));
|
||||
|
||||
FastLED.show();
|
||||
|
||||
hue += 2;
|
||||
delay(50);
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
[platformio]
|
||||
src_dir = symlink://../../../src
|
||||
|
||||
[env:dev]
|
||||
platform = https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip
|
||||
board = esp32-s3-devkitc-1
|
||||
framework = arduino
|
||||
lib_deps =
|
||||
adafruit/Adafruit NeoPixel
|
||||
|
||||
lib_ldf_mode=deep
|
||||
build_flags=-DFASTLED_STUB_MAIN_INCLUDE_INO="../ci/adafruit_bridge/adafruit_bridge.ino.cpp"
|
||||
-DFASTLED_USE_ADAFRUIT_NEOPIXEL=1
|
||||
@@ -0,0 +1,691 @@
|
||||
# dataclasses
|
||||
|
||||
import json
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
|
||||
# An open source version of the esp-idf 5.1 platform for the ESP32 that
|
||||
# gives esp32 boards the same build environment as the Arduino 2.3.1+.
|
||||
|
||||
# Set to a specific release, we may want to update this in the future.
|
||||
ESP32_IDF_5_1_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip"
|
||||
|
||||
# TODO: Upgrade toolkit to 5.3
|
||||
ESP32_IDF_5_3_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/53.03.10/platform-espressif32.zip"
|
||||
ESP32_IDF_5_4_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/54.03.20/platform-espressif32.zip"
|
||||
ESP32_IDF_5_5_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/55.03.30-2/platform-espressif32.zip"
|
||||
ESP32_IDF_5_1_PIOARDUINO_LATEST = (
|
||||
"https://github.com/pioarduino/platform-espressif32.git#develop"
|
||||
)
|
||||
ESP32_IDF_4_4_LATEST = "platformio/espressif32"
|
||||
APOLLO3_2_2_0 = "https://github.com/nigelb/platform-apollo3blue"
|
||||
# Top of trunk.
|
||||
# ESP32_IDF_5_1_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32"
|
||||
|
||||
# Old fork that we were using
|
||||
# ESP32_IDF_5_1_PIOARDUINO = "https://github.com/zackees/platform-espressif32#Arduino/IDF5"
|
||||
|
||||
# ALL will be auto populated in the Board constructor whenever a
|
||||
# board is defined.
|
||||
ALL: list["Board"] = []
|
||||
|
||||
|
||||
class Auto:
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class Board:
|
||||
board_name: str
|
||||
real_board_name: str | None = None
|
||||
platform: str | None = None
|
||||
platform_needs_install: bool = False
|
||||
use_pio_run: bool = (
|
||||
False # some platforms like esp32-c2-devkitm-1 will only work with pio run
|
||||
)
|
||||
platform_packages: str | None = None
|
||||
framework: str | None = None
|
||||
board_build_mcu: str | None = None
|
||||
board_build_core: str | None = None
|
||||
board_build_filesystem_size: str | None = None
|
||||
build_flags: list[str] | None = None # Reserved for future use.
|
||||
build_unflags: list[str] | None = None # New: unflag options
|
||||
defines: list[str] | None = None
|
||||
customsdk: str | None = None
|
||||
board_partitions: str | None = None # Reserved for future use.
|
||||
no_board_spec: bool = (
|
||||
False # For platforms like 'native' that don't need a board specification
|
||||
)
|
||||
add_board_to_all: bool = True
|
||||
lib_compat_mode: str | None = (
|
||||
None # Library compatibility mode (e.g., 'off' for native platform)
|
||||
)
|
||||
lib_ldf_mode: str | None = (
|
||||
None # Library Dependency Finder mode (e.g., 'chain+' for enhanced dependency finding)
|
||||
)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Check if framework is set, warn and auto-set to arduino if missing (except for native/stub platforms)
|
||||
if self.framework is None and not self._is_native_or_stub_platform():
|
||||
self.framework = "arduino"
|
||||
|
||||
ALL.append(self)
|
||||
|
||||
def _is_native_or_stub_platform(self) -> bool:
|
||||
"""Check if this is a native or stub platform that doesn't need a framework"""
|
||||
# Check for native platform
|
||||
if self.platform and (
|
||||
"native" in self.platform.lower() or "stub" in self.platform.lower()
|
||||
):
|
||||
return True
|
||||
|
||||
# Check for stub build flags
|
||||
if self.build_flags:
|
||||
for flag in self.build_flags:
|
||||
if "FASTLED_STUB" in flag or "PLATFORM_NATIVE" in flag:
|
||||
return True
|
||||
|
||||
# Check no_board_spec flag (typically used for native platforms)
|
||||
if self.no_board_spec:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def clone(self) -> "Board":
|
||||
out = Board(
|
||||
board_name=self.board_name,
|
||||
add_board_to_all=False,
|
||||
)
|
||||
for field_name, field_info in self.__dataclass_fields__.items():
|
||||
field_value: Any = getattr(self, field_name)
|
||||
# Create deep copy for mutable types to avoid shared references
|
||||
if isinstance(field_value, (list, dict)):
|
||||
field_value = deepcopy(field_value) # type: ignore[misc]
|
||||
setattr(out, field_name, field_value)
|
||||
return out
|
||||
|
||||
def get_real_board_name(self) -> str:
|
||||
return self.real_board_name if self.real_board_name else self.board_name
|
||||
|
||||
def to_dictionary(self) -> dict[str, list[str]]:
|
||||
out: dict[str, list[str]] = {}
|
||||
if self.real_board_name:
|
||||
out[self.board_name] = [f"board={self.real_board_name}"]
|
||||
options = out.setdefault(self.board_name, [])
|
||||
|
||||
if self.platform:
|
||||
options.append(f"platform={self.platform}")
|
||||
|
||||
if self.platform_needs_install:
|
||||
options.append("platform_needs_install=true")
|
||||
if self.platform_packages:
|
||||
options.append(f"platform_packages={self.platform_packages}")
|
||||
if self.framework:
|
||||
options.append(f"framework={self.framework}")
|
||||
if self.board_build_core:
|
||||
options.append(f"board_build.core={self.board_build_core}")
|
||||
if self.board_build_mcu:
|
||||
options.append(f"board_build.mcu={self.board_build_mcu}")
|
||||
if self.board_build_filesystem_size:
|
||||
options.append(
|
||||
f"board_build.filesystem_size={self.board_build_filesystem_size}"
|
||||
)
|
||||
if self.defines:
|
||||
for define in self.defines:
|
||||
options.append(f"build_flags=-D{define}")
|
||||
|
||||
# Handle build_unflags
|
||||
if self.build_unflags:
|
||||
for uf in self.build_unflags:
|
||||
options.append(f"build_unflags={uf}")
|
||||
|
||||
# Handle explicit build_flags (added for native host compilation and other special cases)
|
||||
if self.build_flags:
|
||||
for bf in self.build_flags:
|
||||
options.append(f"build_flags={bf}")
|
||||
|
||||
if self.customsdk:
|
||||
options.append(f"custom_sdkconfig={self.customsdk}")
|
||||
|
||||
# Add board-specific build cache directory pointing via symlink directive
|
||||
# here = Path(__file__).parent
|
||||
# project_root = here.parent.parent # Move from ci/util/ to project root
|
||||
# cache_dir = project_root / ".pio_cache" / self.board_name
|
||||
# absolute_cache_dir = cache_dir.resolve()
|
||||
# options.append(f"build_cache_dir=symlink://{absolute_cache_dir}")
|
||||
|
||||
return out
|
||||
|
||||
def __repr__(self) -> str:
|
||||
json_str = json.dumps(self.to_dictionary(), indent=4, sort_keys=True)
|
||||
return json_str
|
||||
|
||||
def __hash__(self) -> int:
|
||||
data_str = self.__repr__()
|
||||
return hash(data_str)
|
||||
|
||||
def to_platformio_ini(
|
||||
self,
|
||||
additional_defines: list[str] | None = None,
|
||||
additional_include_dirs: list[str] | None = None,
|
||||
additional_libs: list[str] | None = None,
|
||||
include_platformio_section: bool = False,
|
||||
core_dir: str | None = None,
|
||||
packages_dir: str | None = None,
|
||||
project_root: str | None = None,
|
||||
build_cache_dir: str | None = None,
|
||||
extra_scripts: list[str] | None = None,
|
||||
) -> str:
|
||||
"""Return a `platformio.ini` snippet representing this board.
|
||||
|
||||
The output is suitable for directly appending to a *platformio.ini* file
|
||||
and follows the same semantics used by the PlatformIO CLI. Only
|
||||
parameters understood by PlatformIO are emitted – internal helper
|
||||
fields like ``platform_needs_install`` and ``use_pio_run`` are **not**
|
||||
included because they are consumed exclusively by the build helpers in
|
||||
the *ci/* folder and would be ignored (or flagged as errors) by
|
||||
PlatformIO itself.
|
||||
|
||||
Args:
|
||||
example: Example name for dynamic build flags (currently unused)
|
||||
additional_defines: Additional defines to merge with board defines
|
||||
additional_include_dirs: Additional include directories to merge with build flags
|
||||
include_platformio_section: Whether to include [platformio] section with core_dir/packages_dir
|
||||
core_dir: PlatformIO core directory path
|
||||
packages_dir: PlatformIO packages directory path
|
||||
project_root: FastLED project root for lib_deps symlink
|
||||
"""
|
||||
lines: list[str] = []
|
||||
|
||||
# Optional [platformio] section
|
||||
if include_platformio_section:
|
||||
lines.append("[platformio]")
|
||||
if build_cache_dir:
|
||||
lines.append(f"build_cache_dir = {build_cache_dir}")
|
||||
if core_dir:
|
||||
lines.append(f"core_dir = {core_dir}")
|
||||
if packages_dir:
|
||||
lines.append(f"packages_dir = {packages_dir}")
|
||||
lines.append("")
|
||||
|
||||
# Section header
|
||||
lines.append(f"[env:{self.board_name}]")
|
||||
if extra_scripts:
|
||||
lines.append(f"extra_scripts = {' '.join(extra_scripts)}")
|
||||
|
||||
# Board identifier (skip for platforms that don't need board specification)
|
||||
if not self.no_board_spec:
|
||||
lines.append(f"board = {self.get_real_board_name()}")
|
||||
|
||||
# Optional parameters -------------------------------------------------
|
||||
if self.platform:
|
||||
lines.append(f"platform = {self.platform}")
|
||||
|
||||
if self.platform_packages:
|
||||
lines.append(f"platform_packages = {self.platform_packages}")
|
||||
|
||||
if self.framework:
|
||||
lines.append(f"framework = {self.framework}")
|
||||
|
||||
if self.board_build_core:
|
||||
lines.append(f"board_build.core = {self.board_build_core}")
|
||||
|
||||
if self.board_build_mcu:
|
||||
lines.append(f"board_build.mcu = {self.board_build_mcu}")
|
||||
|
||||
if self.board_build_filesystem_size:
|
||||
lines.append(
|
||||
f"board_build.filesystem_size = {self.board_build_filesystem_size}"
|
||||
)
|
||||
|
||||
if self.board_partitions:
|
||||
lines.append(f"board_partitions = {self.board_partitions}")
|
||||
|
||||
# Build-time flags and unflags ---------------------------------------
|
||||
build_flags_elements: list[str] = []
|
||||
if self.defines:
|
||||
build_flags_elements.extend(f"-D{define}" for define in self.defines)
|
||||
|
||||
# Add additional defines
|
||||
if additional_defines:
|
||||
build_flags_elements.extend(f"-D{define}" for define in additional_defines)
|
||||
|
||||
# Use static build flags
|
||||
if self.build_flags:
|
||||
build_flags_elements.extend(self.build_flags)
|
||||
|
||||
# Add additional include directories
|
||||
if additional_include_dirs:
|
||||
build_flags_elements.extend(
|
||||
f"-I{include_dir}" for include_dir in additional_include_dirs
|
||||
)
|
||||
|
||||
if build_flags_elements:
|
||||
# Put each build flag on a separate line for better readability and debugging
|
||||
lines.append("build_flags =")
|
||||
for flag in build_flags_elements:
|
||||
lines.append(f" {flag}")
|
||||
|
||||
if self.build_unflags:
|
||||
# PlatformIO accepts multiple *build_unflags* separated by spaces.
|
||||
# Emit a single line for readability.
|
||||
lines.append(f"build_unflags = {' '.join(self.build_unflags)}")
|
||||
|
||||
# Custom ESP-IDF sdkconfig override (ESP32-family boards)
|
||||
if self.customsdk:
|
||||
lines.append(f"custom_sdkconfig = {self.customsdk}")
|
||||
|
||||
# Library compatibility mode (for platforms like native that need special handling)
|
||||
if self.lib_compat_mode:
|
||||
lines.append(f"lib_compat_mode = {self.lib_compat_mode}")
|
||||
|
||||
# Library Dependency Finder mode (for enhanced dependency finding)
|
||||
if self.lib_ldf_mode:
|
||||
lines.append(f"lib_ldf_mode = {self.lib_ldf_mode}")
|
||||
|
||||
# Add FastLED-specific configurations if project_root is provided
|
||||
if project_root:
|
||||
# Only add default lib_ldf_mode if board doesn't specify its own
|
||||
current_config = "\n".join(lines)
|
||||
if "lib_ldf_mode" not in current_config:
|
||||
lines.append("lib_ldf_mode = chain")
|
||||
lines.append("lib_archive = true")
|
||||
|
||||
# Build lib_deps with additional libs only (FastLED is copied to lib/FastLED)
|
||||
# PlatformIO supports multiple lib_deps entries on separate lines
|
||||
if additional_libs:
|
||||
# Format as multi-line lib_deps for proper PlatformIO parsing
|
||||
if len(additional_libs) == 1:
|
||||
lines.append(f"lib_deps = {additional_libs[0]}")
|
||||
else:
|
||||
lines.append("lib_deps =")
|
||||
for entry in additional_libs:
|
||||
lines.append(f" {entry}")
|
||||
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# [env:sparkfun_xrp_controller]
|
||||
# platform = https://github.com/maxgerhardt/platform-raspberrypi
|
||||
# board = sparkfun_xrp_controller
|
||||
# framework = arduino
|
||||
# lib_deps = fastled/FastLED @ ^3.9.16
|
||||
|
||||
|
||||
WEBTARGET = Board(
|
||||
board_name="web",
|
||||
)
|
||||
|
||||
# Native host compilation target using PlatformIO's "native" platform.
|
||||
# This allows compiling FastLED for the host machine (Linux/macOS/Windows)
|
||||
# which is useful for CI compile-tests and static analysis. We replicate
|
||||
# the build flags present in ci/native/platformio.ini so that the same
|
||||
# stub implementation and main-file inclusion are used.
|
||||
|
||||
NATIVE = Board(
|
||||
board_name="native",
|
||||
platform="platformio/native",
|
||||
no_board_spec=True, # Native platform doesn't need a board specification
|
||||
lib_compat_mode="off", # Disable library compatibility checking for native platform
|
||||
build_flags=[
|
||||
"-DFASTLED_STUB_IMPL", # Enable stub platform implementations
|
||||
"-DFASTLED_USE_STUB_ARDUINO", # Enable Arduino stub implementations
|
||||
"-DPLATFORM_NATIVE", # Enable native platform stub compilation
|
||||
"-std=c++17",
|
||||
"-I../../../src/platforms/stub", # Include path for Arduino.h and other stub headers (relative to project dir)
|
||||
"-I../../../src", # Include path for FastLED.h and other source headers (relative to project dir)
|
||||
],
|
||||
)
|
||||
|
||||
DUE = Board(
|
||||
board_name="due",
|
||||
platform="atmelsam",
|
||||
)
|
||||
|
||||
|
||||
SPARKFUN_XRP_CONTROLLER_2350B = Board(
|
||||
board_name="sparkfun_xrp_controller",
|
||||
platform="https://github.com/maxgerhardt/platform-raspberrypi",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
APOLLO3_RED_BOARD = Board(
|
||||
board_name="apollo3_red",
|
||||
real_board_name="SparkFun_RedBoard_Artemis_ATP",
|
||||
platform=APOLLO3_2_2_0,
|
||||
platform_packages="framework-arduinoapollo3@https://github.com/sparkfun/Arduino_Apollo3#v2.2.0",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
APOLLO3_SPARKFUN_THING_PLUS_EXPLORABLE = Board(
|
||||
board_name="apollo3_thing_explorable",
|
||||
real_board_name="SparkFun_Thing_Plus_expLoRaBLE",
|
||||
platform=APOLLO3_2_2_0,
|
||||
platform_packages="framework-arduinoapollo3@https://github.com/sparkfun/Arduino_Apollo3#v2.2.0",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
ESP32DEV = Board(
|
||||
board_name="esp32dev",
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32DEV_IDF3_3 = Board(
|
||||
board_name="esp32dev_idf33",
|
||||
real_board_name="esp32dev",
|
||||
platform="espressif32@1.11.2",
|
||||
)
|
||||
|
||||
ESP32DEV_IDF4_4 = Board(
|
||||
board_name="esp32dev_idf44",
|
||||
real_board_name="esp32dev",
|
||||
platform=ESP32_IDF_4_4_LATEST,
|
||||
)
|
||||
|
||||
GIGA_R1 = Board(
|
||||
board_name="giga_r1",
|
||||
platform="ststm32",
|
||||
framework="arduino",
|
||||
real_board_name="giga_r1_m7",
|
||||
)
|
||||
|
||||
# ESP01 = Board(
|
||||
# board_name="esp01",
|
||||
# platform=ESP32_IDF_5_1_PIOARDUINO,
|
||||
# )
|
||||
|
||||
ESP32_C2_DEVKITM_1 = Board(
|
||||
board_name="esp32c2",
|
||||
real_board_name="esp32-c2-devkitm-1",
|
||||
platform="https://github.com/pioarduino/platform-espressif32/releases/download/stable/platform-espressif32.zip",
|
||||
framework="arduino, espidf",
|
||||
)
|
||||
|
||||
ESP32_C3_DEVKITM_1 = Board(
|
||||
board_name="esp32c3",
|
||||
real_board_name="esp32-c3-devkitm-1",
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_C5_DEVKITC_1 = Board(
|
||||
board_name="esp32c5",
|
||||
real_board_name="esp32-c5-devkitc-1",
|
||||
platform=ESP32_IDF_5_5_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_C6_DEVKITC_1 = Board(
|
||||
board_name="esp32c6",
|
||||
real_board_name="esp32-c6-devkitc-1",
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_S3_DEVKITC_1 = Board(
|
||||
board_name="esp32s3",
|
||||
real_board_name="esp32-s3-devkitc-1",
|
||||
platform=ESP32_IDF_5_4_PIOARDUINO,
|
||||
framework="arduino",
|
||||
board_partitions="huge_app.csv",
|
||||
build_unflags=["-DFASTLED_RMT5=0", "-DFASTLED_RMT5"],
|
||||
)
|
||||
|
||||
ESP32_S2_DEVKITM_1 = Board(
|
||||
board_name="esp32s2",
|
||||
real_board_name="lolin_s2_mini ",
|
||||
board_build_mcu="esp32s2",
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_UPESY_WROOM = Board(
|
||||
board_name="upesy_wroom",
|
||||
real_board_name="upesy_wroom",
|
||||
platform="espressif32",
|
||||
)
|
||||
|
||||
ESP32_H2_DEVKITM_1 = Board(
|
||||
board_name="esp32-h2-devkitm-1",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
)
|
||||
|
||||
ESP32_P4 = Board(
|
||||
board_name="esp32p4",
|
||||
real_board_name="esp32-p4-evboard",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
platform="https://github.com/pioarduino/platform-espressif32/releases/download/stable/platform-espressif32.zip",
|
||||
)
|
||||
|
||||
ADA_FEATHER_NRF52840_SENSE = Board(
|
||||
board_name="adafruit_feather_nrf52840_sense",
|
||||
platform="nordicnrf52",
|
||||
)
|
||||
|
||||
XIAOBLESENSE_ADAFRUIT_NRF52 = Board(
|
||||
board_name="xiaoblesense_adafruit",
|
||||
platform="https://github.com/maxgerhardt/platform-nordicnrf52",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
)
|
||||
|
||||
# Alias: handle common misspelling without the trailing 't'
|
||||
XIAOBLESENSE_ADAFRUI_ALIAS = Board(
|
||||
board_name="xiaoblesense_adafrui", # missing 't'
|
||||
real_board_name="xiaoblesense_adafruit", # map to the correct board name
|
||||
platform="https://github.com/maxgerhardt/platform-nordicnrf52",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
XIAOBLESENSE_NRF52 = Board(
|
||||
board_name="xiaoblesense",
|
||||
real_board_name="xiaoble_adafruit",
|
||||
platform="https://github.com/maxgerhardt/platform-nordicnrf52",
|
||||
platform_needs_install=True,
|
||||
)
|
||||
|
||||
# Correct nRF52840 DK board definition
|
||||
# The Nordic nRF52840 DK is directly supported by the default PlatformIO
|
||||
# `nordicnrf52` platform under the board name `nrf52840_dk`, so we don't
|
||||
# need a custom platform package or extra installation steps. Point the
|
||||
# Board definition at the stock platform and use the canonical board name.
|
||||
# This fixes compilation failures introduced during the build-system
|
||||
# migration where the board was temporarily mapped to the XIAO variant.
|
||||
NRF52840 = Board(
|
||||
board_name="nrf52840_dk",
|
||||
real_board_name="nrf52840_dk_adafruit", # Use Adafruit BSP variant which includes full Nordic SDK headers
|
||||
platform="nordicnrf52",
|
||||
framework="arduino",
|
||||
platform_needs_install=False,
|
||||
platform_packages="framework-arduinoadafruitnrf52@^1.10601.0",
|
||||
defines=[
|
||||
"FASTLED_USE_COMPILE_TESTS=0",
|
||||
],
|
||||
board_build_core="nRF5", # Ensure correct core directory
|
||||
)
|
||||
|
||||
RPI_PICO = Board(
|
||||
board_name="rpipico",
|
||||
platform="https://github.com/maxgerhardt/platform-raspberrypi.git",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
platform_packages="framework-arduinopico@https://github.com/earlephilhower/arduino-pico.git",
|
||||
framework="arduino",
|
||||
board_build_core="earlephilhower",
|
||||
board_build_filesystem_size="0.5m",
|
||||
)
|
||||
|
||||
RPI_PICO2 = Board(
|
||||
board_name="rpipico2",
|
||||
real_board_name="rpipico", # Use the existing Pico board definition until PlatformIO adds native Pico 2 support
|
||||
platform="https://github.com/maxgerhardt/platform-raspberrypi.git",
|
||||
platform_needs_install=True, # Install platform package to get the boards
|
||||
platform_packages="framework-arduinopico@https://github.com/earlephilhower/arduino-pico.git",
|
||||
framework="arduino",
|
||||
board_build_core="earlephilhower",
|
||||
board_build_filesystem_size="0.5m",
|
||||
)
|
||||
|
||||
BLUEPILL = Board(
|
||||
board_name="bluepill",
|
||||
real_board_name="bluepill_f103c8",
|
||||
platform="ststm32",
|
||||
)
|
||||
|
||||
# maple_mini_b20
|
||||
MAPLE_MINI = Board(
|
||||
board_name="maple_mini",
|
||||
real_board_name="maple_mini_b20",
|
||||
platform="ststm32",
|
||||
)
|
||||
|
||||
HY_TINYSTM103TB = Board(
|
||||
board_name="hy_tinystm103tb",
|
||||
platform="ststm32",
|
||||
)
|
||||
|
||||
ATTINY88 = Board(
|
||||
board_name="attiny88",
|
||||
platform="atmelavr",
|
||||
)
|
||||
|
||||
# ATtiny1604
|
||||
ATTINY1604 = Board(
|
||||
board_name="ATtiny1604",
|
||||
platform="atmelmegaavr",
|
||||
)
|
||||
|
||||
# attiny4313
|
||||
ATTINY4313 = Board(
|
||||
board_name="attiny4313",
|
||||
platform="atmelavr",
|
||||
)
|
||||
|
||||
ATTINY1616 = Board(
|
||||
board_name="ATtiny1616",
|
||||
platform="atmelmegaavr",
|
||||
)
|
||||
|
||||
UNO_R4_WIFI = Board(
|
||||
board_name="uno_r4_wifi",
|
||||
platform="renesas-ra",
|
||||
)
|
||||
|
||||
NANO_EVERY = Board(
|
||||
board_name="nano_every",
|
||||
platform="atmelmegaavr",
|
||||
)
|
||||
|
||||
ESP32DEV_I2S = Board(
|
||||
board_name="esp32dev_i2s",
|
||||
real_board_name="esp32dev",
|
||||
platform=ESP32_IDF_4_4_LATEST,
|
||||
)
|
||||
|
||||
ESP32S3_RMT51 = Board(
|
||||
board_name="esp32rmt_51",
|
||||
real_board_name="esp32-s3-devkitc-1",
|
||||
platform_needs_install=True,
|
||||
platform=ESP32_IDF_5_3_PIOARDUINO,
|
||||
defines=[
|
||||
"FASTLED_RMT5=1",
|
||||
],
|
||||
)
|
||||
|
||||
# Teensy boards
|
||||
TEENSY_LC = Board(
|
||||
board_name="teensylc",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
TEENSY30 = Board(
|
||||
board_name="teensy30",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
TEENSY31 = Board(
|
||||
board_name="teensy31",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
TEENSY40 = Board(
|
||||
board_name="teensy40",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
TEENSY41 = Board(
|
||||
board_name="teensy41",
|
||||
platform="teensy",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
# Basic Arduino boards
|
||||
UNO = Board(
|
||||
board_name="uno",
|
||||
platform="atmelavr",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
YUN = Board(
|
||||
board_name="yun",
|
||||
platform="atmelavr",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
DIGIX = Board(
|
||||
board_name="digix",
|
||||
real_board_name="due", # Digix is Arduino Due compatible
|
||||
platform="atmelsam",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
# ESP8266 boards
|
||||
ESP01 = Board(
|
||||
board_name="esp01",
|
||||
platform="espressif8266",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
# ATtiny boards
|
||||
ATTINY85 = Board(
|
||||
board_name="attiny85",
|
||||
platform="atmelavr",
|
||||
framework="arduino",
|
||||
)
|
||||
|
||||
# Seeed XIAO ESP32S3 board – same platform, needs FASTLED_RMT5 macro removal
|
||||
XIAO_ESP32S3 = Board(
|
||||
board_name="seeed_xiao_esp32s3",
|
||||
real_board_name="seeed_xiao_esp32s3",
|
||||
platform=ESP32_IDF_5_4_PIOARDUINO,
|
||||
board_partitions="huge_app.csv",
|
||||
defines=None,
|
||||
build_unflags=["-DFASTLED_RMT5=0", "-DFASTLED_RMT5"],
|
||||
)
|
||||
|
||||
|
||||
def _make_board_map(boards: list[Board]) -> dict[str, Board]:
|
||||
# make board map, but assert on duplicate board names
|
||||
board_map: dict[str, Board] = {}
|
||||
for board in boards:
|
||||
assert board.board_name not in board_map, (
|
||||
f"Duplicate board name: {board.board_name}"
|
||||
)
|
||||
board_map[board.board_name] = board
|
||||
return board_map
|
||||
|
||||
|
||||
_BOARD_MAP: dict[str, Board] = _make_board_map(ALL)
|
||||
|
||||
|
||||
def create_board(board_name: str, no_project_options: bool = False) -> Board:
|
||||
board: Board
|
||||
if no_project_options:
|
||||
board = Board(board_name=board_name, add_board_to_all=False)
|
||||
if board_name not in _BOARD_MAP:
|
||||
# empty board without any special overrides, assume platformio will know what to do with it.
|
||||
board = Board(board_name=board_name, add_board_to_all=False)
|
||||
else:
|
||||
board = _BOARD_MAP[board_name]
|
||||
return board.clone()
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"build": {
|
||||
"core": "maple",
|
||||
"cpu": "cortex-m3",
|
||||
"extra_flags": "-DSTM32F103C8 -DSTM32F1 -DARDUINO_GENERIC_STM32F103C8 -DARDUINO_ARCH_STM32",
|
||||
"f_cpu": "72000000L",
|
||||
"hwids": [
|
||||
[
|
||||
"0x1EAF",
|
||||
"0x0003"
|
||||
]
|
||||
],
|
||||
"ldscript": "jtag_c8.ld",
|
||||
"mcu": "stm32f103c8",
|
||||
"product_line": "STM32F103xx",
|
||||
"variant": "STM32F103C8"
|
||||
},
|
||||
"connectivity": [
|
||||
"can"
|
||||
],
|
||||
"debug": {
|
||||
"jlink_device": "STM32F103C8",
|
||||
"openocd_board": "stm32f103c8_blue_pill",
|
||||
"openocd_target": "stm32f1x",
|
||||
"svd_path": "STM32F103.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"stm32cube",
|
||||
"libopencm3"
|
||||
],
|
||||
"name": "Blue Pill STM32F103C8",
|
||||
"upload": {
|
||||
"maximum_ram_size": 20480,
|
||||
"maximum_size": 65536,
|
||||
"protocol": "stlink",
|
||||
"protocols": [
|
||||
"stlink",
|
||||
"dfu",
|
||||
"serial",
|
||||
"blackmagic",
|
||||
"jlink"
|
||||
]
|
||||
},
|
||||
"url": "https://stm32-base.org/boards/STM32F103C8T6-Blue-Pill",
|
||||
"vendor": "Generic"
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"build": {
|
||||
"core": "esp32",
|
||||
"f_cpu": "120000000L",
|
||||
"f_flash": "60000000L",
|
||||
"flash_mode": "qio",
|
||||
"mcu": "esp32c2",
|
||||
"variant": "esp32c2"
|
||||
},
|
||||
"connectivity": [
|
||||
"wifi"
|
||||
],
|
||||
"debug": {
|
||||
"openocd_target": "esp32c2.cfg"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"espidf"
|
||||
],
|
||||
"name": "Espressif ESP32-C2-DevKitM-1",
|
||||
"upload": {
|
||||
"flash_size": "4MB",
|
||||
"maximum_ram_size": 278528,
|
||||
"maximum_size": 4194304,
|
||||
"require_upload_port": true,
|
||||
"speed": 460800
|
||||
},
|
||||
"espidf": {
|
||||
"custom_sdkconfig": [
|
||||
"CONFIG_IDF_TARGET=\"esp32c2\""
|
||||
]
|
||||
},
|
||||
"url": "https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp8684/esp8684-devkitm-1/user_guide.html",
|
||||
"vendor": "Espressif"
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"build": {
|
||||
"core": "esp32",
|
||||
"f_cpu": "160000000L",
|
||||
"f_flash": "80000000L",
|
||||
"flash_mode": "qio",
|
||||
"mcu": "esp32c6",
|
||||
"variant": "esp32c6"
|
||||
},
|
||||
"connectivity": [
|
||||
"bluetooth",
|
||||
"wifi"
|
||||
],
|
||||
"debug": {
|
||||
"openocd_target": "esp32c6.cfg"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"espidf"
|
||||
],
|
||||
"name": "Espressif ESP32-C6-DevKitC-1",
|
||||
"upload": {
|
||||
"flash_size": "8MB",
|
||||
"maximum_ram_size": 327680,
|
||||
"maximum_size": 8388608,
|
||||
"require_upload_port": true,
|
||||
"speed": 460800
|
||||
},
|
||||
"url": "https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp32c6/esp32-c6-devkitc-1/index.html",
|
||||
"vendor": "Espressif"
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"build": {
|
||||
"core": "esp32",
|
||||
"f_cpu": "96000000L",
|
||||
"f_flash": "64000000L",
|
||||
"f_image": "48000000L",
|
||||
"flash_mode": "qio",
|
||||
"mcu": "esp32h2",
|
||||
"variant": "esp32h2"
|
||||
},
|
||||
"connectivity": [
|
||||
"bluetooth"
|
||||
],
|
||||
"debug": {
|
||||
"openocd_target": "esp32h2.cfg"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"espidf"
|
||||
],
|
||||
"name": "Espressif ESP32-H2-DevKit",
|
||||
"upload": {
|
||||
"flash_size": "4MB",
|
||||
"maximum_ram_size": 327680,
|
||||
"maximum_size": 4194304,
|
||||
"require_upload_port": true,
|
||||
"speed": 460800
|
||||
},
|
||||
"url": "https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp32h2/esp32-h2-devkitm-1/index.html",
|
||||
"vendor": "Espressif"
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino": {
|
||||
"ldscript": "linker_script.ld",
|
||||
"flash_layout": "100_0"
|
||||
},
|
||||
"extra_flags": "-DARDUINO_GIGA -DARDUINO_GIGA_PINS -DGIGA_PINS -DGIGA -DCM4_BINARY_START=0x60000000 -DCM4_BINARY_END=0x60040000 -DCM4_RAM_END=0x60080000",
|
||||
"core": "arduino",
|
||||
"cpu": "cortex-m7",
|
||||
"f_cpu": "480000000L",
|
||||
"mcu": "stm32h747xih6",
|
||||
"variant": "GIGA",
|
||||
"product_line": "STM32H747xx",
|
||||
"hwids": [
|
||||
[
|
||||
"0x2341",
|
||||
"0x0266"
|
||||
],
|
||||
[
|
||||
"0x2341",
|
||||
"0x0366"
|
||||
],
|
||||
[
|
||||
"0x2341",
|
||||
"0x0466"
|
||||
]
|
||||
]
|
||||
},
|
||||
"connectivity": [
|
||||
"bluetooth",
|
||||
"wifi"
|
||||
],
|
||||
"debug": {
|
||||
"jlink_device": "STM32H747XI_M7",
|
||||
"openocd_target": "stm32h7x_dual_bank"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino"
|
||||
],
|
||||
"name": "Arduino Giga R1 Wifi",
|
||||
"upload": {
|
||||
"_maximum_ram_size": 294248,
|
||||
"_maximum_size": 1048576,
|
||||
"maximum_ram_size": 523624,
|
||||
"maximum_size": 1966080,
|
||||
"protocol": "dfu",
|
||||
"protocols": [
|
||||
"cmsis-dap",
|
||||
"dfu",
|
||||
"jlink",
|
||||
"stlink",
|
||||
"mbed"
|
||||
],
|
||||
"require_upload_port": true,
|
||||
"use_1200bps_touch": true,
|
||||
"wait_for_upload_port": true,
|
||||
"offset_address": "0x08100000"
|
||||
},
|
||||
"url": "https://docs.arduino.cc/hardware/giga-r1-wifi",
|
||||
"vendor": "Arduino"
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino":{
|
||||
"ldscript": "nrf52_xxaa.ld"
|
||||
},
|
||||
"core": "nRF5",
|
||||
"cpu": "cortex-m4",
|
||||
"extra_flags": "-DARDUINO_NRF52_DK",
|
||||
"f_cpu": "64000000L",
|
||||
"mcu": "nrf52840",
|
||||
"variant": "nRF52DK",
|
||||
"zephyr": {
|
||||
"variant": "nrf52840dk_nrf52840"
|
||||
}
|
||||
},
|
||||
"connectivity": [
|
||||
"bluetooth"
|
||||
],
|
||||
"debug": {
|
||||
"default_tools": [
|
||||
"jlink"
|
||||
],
|
||||
"jlink_device": "nRF52840_xxAA",
|
||||
"onboard_tools": [
|
||||
"cmsis-dap",
|
||||
"jlink"
|
||||
],
|
||||
"svd_path": "nrf52840.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino",
|
||||
"mbed",
|
||||
"zephyr"
|
||||
],
|
||||
"name": "Nordic nRF52840-DK",
|
||||
"upload": {
|
||||
"maximum_ram_size": 262144,
|
||||
"maximum_size": 1048576,
|
||||
"protocol": "jlink",
|
||||
"protocols": [
|
||||
"jlink",
|
||||
"nrfjprog",
|
||||
"stlink",
|
||||
"blackmagic",
|
||||
"cmsis-dap",
|
||||
"mbed"
|
||||
]
|
||||
},
|
||||
"url": "https://os.mbed.com/platforms/Nordic-nRF52840-DK/",
|
||||
"vendor": "Nordic"
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino": {
|
||||
"earlephilhower": {
|
||||
"boot2_source": "boot2_w25q080_2_padded_checksum.S",
|
||||
"usb_vid": "0x2E8A",
|
||||
"usb_pid": "0x000A"
|
||||
}
|
||||
},
|
||||
"core": "earlephilhower",
|
||||
"cpu": "cortex-m0plus",
|
||||
"extra_flags": "-DARDUINO_RASPBERRY_PI_PICO -DARDUINO_ARCH_RP2040 -DUSBD_MAX_POWER_MA=250 ",
|
||||
"f_cpu": "133000000L",
|
||||
"hwids": [
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x00C0"
|
||||
],
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x000A"
|
||||
]
|
||||
],
|
||||
"mcu": "rp2040",
|
||||
"variant": "rpipico"
|
||||
},
|
||||
"debug": {
|
||||
"jlink_device": "RP2040_M0_0",
|
||||
"openocd_target": "rp2040.cfg",
|
||||
"svd_path": "rp2040.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino"
|
||||
],
|
||||
"name": "Pico",
|
||||
"upload": {
|
||||
"maximum_ram_size": 262144,
|
||||
"maximum_size": 2097152,
|
||||
"require_upload_port": true,
|
||||
"native_usb": true,
|
||||
"use_1200bps_touch": true,
|
||||
"wait_for_upload_port": false,
|
||||
"protocol": "picotool",
|
||||
"protocols": [
|
||||
"blackmagic",
|
||||
"cmsis-dap",
|
||||
"jlink",
|
||||
"raspberrypi-swd",
|
||||
"picotool",
|
||||
"picoprobe",
|
||||
"pico-debug"
|
||||
]
|
||||
},
|
||||
"url": "https://www.raspberrypi.org/products/raspberry-pi-pico/",
|
||||
"vendor": "Raspberry Pi"
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino": {
|
||||
"earlephilhower": {
|
||||
"boot2_source": "boot2_generic_03h_2_padded_checksum.S",
|
||||
"usb_vid": "0x2E8A",
|
||||
"usb_pid": "0x000B"
|
||||
}
|
||||
},
|
||||
"core": "earlephilhower",
|
||||
"cpu": "cortex-m33",
|
||||
"extra_flags": "-D ARDUINO_RASPBERRY_PI_PICO_2 -DARDUINO_ARCH_RP2350 -DUSBD_MAX_POWER_MA=250",
|
||||
"f_cpu": "133000000L",
|
||||
"hwids": [
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x00C0"
|
||||
],
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x000B"
|
||||
]
|
||||
],
|
||||
"mcu": "rp2350",
|
||||
"variant": "rpipico2"
|
||||
},
|
||||
"debug": {
|
||||
"jlink_device": "RP2350_0",
|
||||
"openocd_target": "rp2350.cfg",
|
||||
"svd_path": "rp2350.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino"
|
||||
],
|
||||
"name": "Pico 2",
|
||||
"upload": {
|
||||
"maximum_ram_size": 524288,
|
||||
"maximum_size": 2097152,
|
||||
"require_upload_port": true,
|
||||
"native_usb": true,
|
||||
"use_1200bps_touch": true,
|
||||
"wait_for_upload_port": false,
|
||||
"protocol": "picotool",
|
||||
"protocols": [
|
||||
"blackmagic",
|
||||
"cmsis-dap",
|
||||
"jlink",
|
||||
"raspberrypi-swd",
|
||||
"picotool",
|
||||
"picoprobe",
|
||||
"pico-debug"
|
||||
]
|
||||
},
|
||||
"url": "https://www.raspberrypi.org/products/raspberry-pi-pico/",
|
||||
"vendor": "Raspberry Pi"
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"build": {
|
||||
"arduino": {
|
||||
"earlephilhower": {
|
||||
"boot2_source": "boot2_w25q080_2_padded_checksum.S",
|
||||
"usb_vid": "0x2E8A",
|
||||
"usb_pid": "0xF00A"
|
||||
}
|
||||
},
|
||||
"core": "earlephilhower",
|
||||
"cpu": "cortex-m0plus",
|
||||
"extra_flags": "-DARDUINO_RASPBERRY_PI_PICO_W -DARDUINO_ARCH_RP2040 -DUSBD_MAX_POWER_MA=250 ",
|
||||
"f_cpu": "133000000L",
|
||||
"hwids": [
|
||||
[
|
||||
"0x2E8A",
|
||||
"0x00C0"
|
||||
],
|
||||
[
|
||||
"0x2E8A",
|
||||
"0xF00A"
|
||||
]
|
||||
],
|
||||
"mcu": "rp2040",
|
||||
"variant": "rpipicow"
|
||||
},
|
||||
"debug": {
|
||||
"jlink_device": "RP2040_M0_0",
|
||||
"openocd_target": "rp2040.cfg",
|
||||
"svd_path": "rp2040.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino"
|
||||
],
|
||||
"name": "Pico W",
|
||||
"upload": {
|
||||
"maximum_ram_size": 262144,
|
||||
"maximum_size": 2097152,
|
||||
"require_upload_port": true,
|
||||
"native_usb": true,
|
||||
"use_1200bps_touch": true,
|
||||
"wait_for_upload_port": false,
|
||||
"protocol": "picotool",
|
||||
"protocols": [
|
||||
"blackmagic",
|
||||
"cmsis-dap",
|
||||
"jlink",
|
||||
"raspberrypi-swd",
|
||||
"picotool",
|
||||
"picoprobe",
|
||||
"pico-debug"
|
||||
]
|
||||
},
|
||||
"url": "https://www.raspberrypi.org/products/raspberry-pi-pico/",
|
||||
"vendor": "Raspberry Pi"
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"build": {
|
||||
"core": "silabs",
|
||||
"f_cpu": "39000000L",
|
||||
"mcu": "cortex-m33",
|
||||
"variant": "thingplusmatter"
|
||||
},
|
||||
"connectivity": [
|
||||
"bluetooth",
|
||||
"thread",
|
||||
"wifi",
|
||||
"zigbee"
|
||||
],
|
||||
"debug": {
|
||||
"jlink_device": "EFR32MG24B020F1536IM40",
|
||||
"onboard_tools": [
|
||||
"jlink"
|
||||
],
|
||||
"svd_path": "EFR32MG24B020F1536IM40.svd"
|
||||
},
|
||||
"frameworks": [
|
||||
"arduino"
|
||||
],
|
||||
"name": "Sparkfun Thing Plus Matter",
|
||||
"upload": {
|
||||
"flash_size": "1536kB",
|
||||
"maximum_ram_size": 262144,
|
||||
"maximum_size": 1572864,
|
||||
"protocol": "jlink"
|
||||
},
|
||||
"url": "https://www.sparkfun.com/products/20270",
|
||||
"vendor": "Sparkfun"
|
||||
}
|
||||
@@ -0,0 +1,643 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Unified Build API
|
||||
|
||||
This module provides a unified build system for both unit tests and examples,
|
||||
centralizing PCH and library building while providing a simple, elegant API.
|
||||
|
||||
Key Features:
|
||||
- Unified API for both unit tests and examples
|
||||
- Automatic PCH header building and reuse
|
||||
- Automatic libfastled.a building and reuse
|
||||
- Organized build directories under .build/
|
||||
- Efficient multiple target building
|
||||
- Simple, elegant interface
|
||||
|
||||
Usage:
|
||||
# Unit test building
|
||||
unit_builder = BuildAPI(
|
||||
config_file="ci/build_unit.toml",
|
||||
build_dir=".build/unit",
|
||||
build_type=BuildType.UNIT_TEST
|
||||
)
|
||||
unit_builder.build_targets(["test_json.cpp", "test_color.cpp"])
|
||||
|
||||
# Example building
|
||||
example_builder = BuildAPI(
|
||||
config_file="ci/build_example.toml",
|
||||
build_dir=".build/examples",
|
||||
build_type=BuildType.EXAMPLE
|
||||
)
|
||||
example_builder.build_targets(["Blink.ino", "DemoReel100.ino"])
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Sequence, Set, Tuple
|
||||
|
||||
from ci.ci.fingerprint_cache import FingerprintCache
|
||||
from ci.compiler.clang_compiler import (
|
||||
BuildFlags,
|
||||
Compiler,
|
||||
CompilerOptions,
|
||||
LibarchiveOptions,
|
||||
)
|
||||
from ci.compiler.test_example_compilation import get_fastled_core_sources
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
from ci.util.running_process import subprocess_run
|
||||
|
||||
|
||||
class BuildType(Enum):
|
||||
"""Build type enumeration"""
|
||||
|
||||
UNIT_TEST = "unit_test"
|
||||
EXAMPLE = "example"
|
||||
|
||||
|
||||
@dataclass
|
||||
class BuildTarget:
|
||||
"""Represents a single build target (test file or example)"""
|
||||
|
||||
source_file: Path
|
||||
name: str
|
||||
output_dir: Path
|
||||
|
||||
def get_output_executable(self) -> Path:
|
||||
"""Get the expected output executable path"""
|
||||
if self.source_file.suffix == ".cpp":
|
||||
# Unit test: test_name -> test_name.exe
|
||||
base_name = self.source_file.stem
|
||||
return self.output_dir / f"{base_name}.exe"
|
||||
else:
|
||||
# Example: Blink.ino -> Blink.exe
|
||||
base_name = self.source_file.stem
|
||||
return self.output_dir / f"{base_name}.exe"
|
||||
|
||||
|
||||
@dataclass
|
||||
class BuildResult:
|
||||
"""Result of a build operation"""
|
||||
|
||||
target: BuildTarget
|
||||
success: bool
|
||||
message: str
|
||||
build_time: float
|
||||
|
||||
|
||||
class BuildAPI:
|
||||
"""
|
||||
Unified build API for both unit tests and examples.
|
||||
|
||||
This class centralizes all build operations including PCH building,
|
||||
library creation, and target compilation. It provides a simple,
|
||||
elegant interface while handling the complexity internally.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
build_flags_toml: str | Path,
|
||||
build_dir: str | Path,
|
||||
build_type: BuildType,
|
||||
use_pch: bool = True,
|
||||
parallel: bool = True,
|
||||
clean: bool = False,
|
||||
):
|
||||
"""
|
||||
Initialize the build API.
|
||||
|
||||
Args:
|
||||
build_flags_toml: Path to build_flags.toml file (build_unit.toml or build_example.toml)
|
||||
build_dir: Directory for build outputs (e.g., .build/unit or .build/examples)
|
||||
build_type: Type of build (UNIT_TEST or EXAMPLE)
|
||||
use_pch: Whether to use precompiled headers
|
||||
parallel: Whether to enable parallel compilation
|
||||
clean: Whether to force a clean build
|
||||
"""
|
||||
self.build_flags_toml = Path(build_flags_toml)
|
||||
self.build_dir = Path(build_dir)
|
||||
self.build_type = build_type
|
||||
self.use_pch = use_pch
|
||||
self.parallel = parallel
|
||||
self.clean = clean
|
||||
|
||||
# Ensure build directory exists
|
||||
self.build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Set up subdirectories for organization
|
||||
self.artifacts_dir = self.build_dir / "artifacts" # PCH, lib files
|
||||
self.targets_dir = self.build_dir / "targets" # Individual target builds
|
||||
self.cache_dir = self.build_dir / "cache" # Fingerprint caches
|
||||
|
||||
for directory in [self.artifacts_dir, self.targets_dir, self.cache_dir]:
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Initialize internal state
|
||||
self._compiler: Optional[Compiler] = None
|
||||
self._pch_file: Optional[Path] = None
|
||||
self._library_file: Optional[Path] = None
|
||||
self._fingerprint_cache: Optional[FingerprintCache] = None
|
||||
|
||||
# Parse build flags from TOML file
|
||||
self._parse_build_flags()
|
||||
|
||||
# Create compiler from TOML configuration
|
||||
self._create_compiler()
|
||||
|
||||
def _parse_build_flags(self) -> None:
|
||||
"""Parse build flags from TOML file"""
|
||||
if not self.build_flags_toml.exists():
|
||||
raise RuntimeError(
|
||||
f"Build flags TOML file not found: {self.build_flags_toml}"
|
||||
)
|
||||
|
||||
print(f"[BUILD API] Parsing build flags from {self.build_flags_toml}")
|
||||
|
||||
# Parse build flags from TOML
|
||||
self.build_flags = BuildFlags.from_toml_file(
|
||||
self.build_flags_toml,
|
||||
quick_build=True, # Use quick build mode for faster iteration
|
||||
strict_mode=False,
|
||||
)
|
||||
|
||||
# Initialize fingerprint cache
|
||||
cache_file = self.cache_dir / "fingerprint_cache.json"
|
||||
self._fingerprint_cache = FingerprintCache(cache_file)
|
||||
|
||||
def _create_compiler(self) -> None:
|
||||
"""Create compiler from TOML configuration ONLY - no inline settings allowed"""
|
||||
from ci.compiler.clang_compiler import create_compiler_options_from_toml
|
||||
|
||||
# ALL settings must come from the TOML file - NO hardcoded/inline settings allowed
|
||||
print(f"[BUILD API] Creating compiler from TOML configuration")
|
||||
|
||||
# Set up PCH output path
|
||||
pch_output_path: Optional[str] = None
|
||||
if self.use_pch:
|
||||
pch_output_path = str(
|
||||
self.artifacts_dir / f"{self.build_type.value}_pch.hpp.pch"
|
||||
)
|
||||
|
||||
# Use the proper function to create CompilerOptions from TOML - NO inline settings
|
||||
self.compiler_settings = create_compiler_options_from_toml(
|
||||
toml_path=self.build_flags_toml,
|
||||
include_path=str(PROJECT_ROOT / "src"), # Base include path only
|
||||
quick_build=True, # Use quick build mode for faster iteration
|
||||
strict_mode=False,
|
||||
# Pass PCH and parallel settings from constructor
|
||||
use_pch=self.use_pch,
|
||||
pch_output_path=pch_output_path,
|
||||
parallel=self.parallel,
|
||||
)
|
||||
|
||||
# Create compiler instance with TOML-loaded settings
|
||||
self._compiler = Compiler(self.compiler_settings, self.build_flags)
|
||||
|
||||
def _ensure_pch_built(self) -> bool:
|
||||
"""
|
||||
Ensure PCH header is built and ready.
|
||||
|
||||
Returns:
|
||||
bool: True if PCH is ready, False otherwise
|
||||
"""
|
||||
if not self.use_pch or not self._compiler:
|
||||
return True
|
||||
|
||||
print(f"[BUILD API] Building PCH header for {self.build_type.value}")
|
||||
|
||||
# Check if we need to rebuild PCH
|
||||
if not self.clean and self._pch_file and self._pch_file.exists():
|
||||
# TODO: Add proper dependency checking here
|
||||
print(f"[BUILD API] Reusing existing PCH: {self._pch_file}")
|
||||
return True
|
||||
|
||||
# Build PCH
|
||||
success = self._compiler.create_pch_file()
|
||||
if success and self.compiler_settings.pch_output_path:
|
||||
self._pch_file = Path(self.compiler_settings.pch_output_path)
|
||||
print(f"[BUILD API] PCH built successfully: {self._pch_file}")
|
||||
else:
|
||||
print(f"[BUILD API] PCH build failed")
|
||||
|
||||
return success
|
||||
|
||||
def _ensure_library_built(self) -> bool:
|
||||
"""
|
||||
Ensure libfastled.a is built and ready.
|
||||
|
||||
Returns:
|
||||
bool: True if library is ready, False otherwise
|
||||
"""
|
||||
if not self._compiler:
|
||||
return False
|
||||
|
||||
lib_name = f"libfastled_{self.build_type.value}.a"
|
||||
self._library_file = self.artifacts_dir / lib_name
|
||||
|
||||
print(f"[BUILD API] Building FastLED library for {self.build_type.value}")
|
||||
|
||||
# Check if we need to rebuild library
|
||||
if not self.clean and self._library_file.exists():
|
||||
# TODO: Add proper dependency checking here
|
||||
print(f"[BUILD API] Reusing existing library: {self._library_file}")
|
||||
return True
|
||||
|
||||
# Prefer UNITY chunk build for faster library creation
|
||||
try:
|
||||
unity_dir = self.artifacts_dir / "unity"
|
||||
unity_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Collect FastLED core sources (consistent with example compilation)
|
||||
all_sources = [
|
||||
p for p in get_fastled_core_sources() if p.name != "stub_main.cpp"
|
||||
]
|
||||
if not all_sources:
|
||||
print("[BUILD API] No FastLED sources found for unity build")
|
||||
return False
|
||||
|
||||
# Deterministic sort relative to project root
|
||||
project_root = PROJECT_ROOT
|
||||
|
||||
def sort_key(p: Path) -> str:
|
||||
try:
|
||||
rel = p.relative_to(project_root)
|
||||
except Exception:
|
||||
rel = p
|
||||
return rel.as_posix()
|
||||
|
||||
all_sources = sorted(all_sources, key=sort_key)
|
||||
|
||||
# Partition into chunks (1..4 typical)
|
||||
total = len(all_sources)
|
||||
max_chunks = min(4, max(1, (os.cpu_count() or 2) // 2))
|
||||
chunks = max_chunks if total >= max_chunks else max(1, total)
|
||||
|
||||
# Prepare compile options (PCH off for unity)
|
||||
compile_opts = CompilerOptions(
|
||||
include_path=self._compiler.settings.include_path,
|
||||
compiler=self._compiler.settings.compiler,
|
||||
defines=self._compiler.settings.defines,
|
||||
std_version=self._compiler.settings.std_version,
|
||||
compiler_args=self._compiler.settings.compiler_args,
|
||||
use_pch=False,
|
||||
additional_flags=["-c"],
|
||||
parallel=self.parallel,
|
||||
)
|
||||
|
||||
# Compute chunk sizes
|
||||
base = total // chunks
|
||||
rem = total % chunks
|
||||
start = 0
|
||||
unity_objects: list[Path] = []
|
||||
|
||||
for i in range(chunks):
|
||||
size = base + (1 if i < rem else 0)
|
||||
end = start + size
|
||||
group = all_sources[start:end]
|
||||
start = end
|
||||
|
||||
if not group:
|
||||
continue
|
||||
|
||||
unity_cpp = unity_dir / f"unity{i + 1}.cpp"
|
||||
# Use synchronous path to simplify error handling here
|
||||
result = self._compiler._compile_unity_sync(
|
||||
compile_opts,
|
||||
group,
|
||||
unity_output_path=unity_cpp,
|
||||
)
|
||||
if not result.ok:
|
||||
print(f"[BUILD API] Unity chunk {i + 1} failed: {result.stderr}")
|
||||
return False
|
||||
unity_objects.append(unity_cpp.with_suffix(".o"))
|
||||
|
||||
if not unity_objects:
|
||||
print("[BUILD API] No unity objects produced")
|
||||
return False
|
||||
|
||||
# Archive unity objects into the final library
|
||||
archive_result = self._compiler.create_archive(
|
||||
unity_objects, self._library_file, LibarchiveOptions()
|
||||
).result()
|
||||
if not archive_result.ok:
|
||||
print(f"[BUILD API] Archive creation failed: {archive_result.stderr}")
|
||||
return False
|
||||
|
||||
print(
|
||||
f"[BUILD API] Library built successfully (UNITY): {self._library_file}"
|
||||
)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"[BUILD API] UNITY library build failed with exception: {e}")
|
||||
return False
|
||||
|
||||
def _create_static_library(
|
||||
self, object_files: List[Path], output_lib: Path
|
||||
) -> bool:
|
||||
"""
|
||||
Create static library from object files.
|
||||
|
||||
Args:
|
||||
object_files: List of object files to archive
|
||||
output_lib: Output library file path
|
||||
|
||||
Returns:
|
||||
bool: True if successful, False otherwise
|
||||
"""
|
||||
if not object_files:
|
||||
return False
|
||||
|
||||
# Use archiver from build flags
|
||||
archiver_cmd = self.build_flags.tools.archiver
|
||||
archive_flags = self.build_flags.archive.flags
|
||||
|
||||
# Build archiver command
|
||||
cmd = (
|
||||
archiver_cmd
|
||||
+ [archive_flags]
|
||||
+ [str(output_lib)]
|
||||
+ [str(f) for f in object_files]
|
||||
)
|
||||
|
||||
print(f"[BUILD API] Creating static library: {' '.join(cmd)}")
|
||||
|
||||
# Execute archiver command using stdout-pumping wrapper
|
||||
try:
|
||||
_completed = subprocess_run(
|
||||
command=cmd,
|
||||
cwd=None,
|
||||
check=True,
|
||||
timeout=900,
|
||||
enable_stack_trace=False,
|
||||
)
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"[BUILD API] Archiver failed: {e}")
|
||||
print(f"[BUILD API] Archiver stdout: {e.stdout}")
|
||||
print(f"[BUILD API] Archiver stderr: {e.stderr}")
|
||||
return False
|
||||
|
||||
def build_targets(self, target_files: Sequence[str | Path]) -> List[BuildResult]:
|
||||
"""
|
||||
Build multiple targets efficiently.
|
||||
|
||||
Args:
|
||||
target_files: List of source files to build
|
||||
|
||||
Returns:
|
||||
List[BuildResult]: Results for each target
|
||||
"""
|
||||
print(
|
||||
f"[BUILD API] Building {len(target_files)} targets for {self.build_type.value}"
|
||||
)
|
||||
|
||||
# Ensure shared artifacts are built first
|
||||
if not self._ensure_pch_built():
|
||||
return [
|
||||
BuildResult(
|
||||
target=BuildTarget(Path(f), Path(f).stem, self.targets_dir),
|
||||
success=False,
|
||||
message="PCH build failed",
|
||||
build_time=0.0,
|
||||
)
|
||||
for f in target_files
|
||||
]
|
||||
|
||||
if not self._ensure_library_built():
|
||||
return [
|
||||
BuildResult(
|
||||
target=BuildTarget(Path(f), Path(f).stem, self.targets_dir),
|
||||
success=False,
|
||||
message="Library build failed",
|
||||
build_time=0.0,
|
||||
)
|
||||
for f in target_files
|
||||
]
|
||||
|
||||
# Build individual targets
|
||||
results: List[BuildResult] = []
|
||||
for target_file in target_files:
|
||||
result = self._build_single_target(Path(target_file))
|
||||
results.append(result)
|
||||
|
||||
return results
|
||||
|
||||
def _build_single_target(self, target_file: Path) -> BuildResult:
|
||||
"""
|
||||
Build a single target.
|
||||
|
||||
Args:
|
||||
target_file: Source file to build
|
||||
|
||||
Returns:
|
||||
BuildResult: Result of the build
|
||||
"""
|
||||
import time
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Create build target
|
||||
target_dir = self.targets_dir / target_file.stem
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
target = BuildTarget(
|
||||
source_file=target_file, name=target_file.stem, output_dir=target_dir
|
||||
)
|
||||
|
||||
print(f"[BUILD API] Building target: {target.name}")
|
||||
|
||||
try:
|
||||
# Compile and link the target
|
||||
success = self._compile_and_link_target(target)
|
||||
|
||||
build_time = time.time() - start_time
|
||||
message = "Build successful" if success else "Build failed"
|
||||
|
||||
return BuildResult(
|
||||
target=target, success=success, message=message, build_time=build_time
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
build_time = time.time() - start_time
|
||||
return BuildResult(
|
||||
target=target,
|
||||
success=False,
|
||||
message=f"Build error: {e}",
|
||||
build_time=build_time,
|
||||
)
|
||||
|
||||
def _compile_and_link_target(self, target: BuildTarget) -> bool:
|
||||
"""
|
||||
Compile and link a single target.
|
||||
|
||||
Args:
|
||||
target: Target to build
|
||||
|
||||
Returns:
|
||||
bool: True if successful, False otherwise
|
||||
"""
|
||||
if not self._compiler:
|
||||
return False
|
||||
|
||||
# Compile target source file
|
||||
obj_file = target.output_dir / f"{target.name}.o"
|
||||
if target.source_file.suffix == ".cpp":
|
||||
success = self._compiler.compile_cpp_file(target.source_file, obj_file)
|
||||
else:
|
||||
# For .ino files
|
||||
success = self._compiler.compile_ino_file(target.source_file, obj_file)
|
||||
|
||||
if not success:
|
||||
print(f"[BUILD API] Failed to compile {target.source_file}")
|
||||
return False
|
||||
|
||||
# Link with library to create executable
|
||||
exe_file = target.get_output_executable()
|
||||
success = self._link_executable([obj_file], exe_file)
|
||||
|
||||
if not success:
|
||||
print(f"[BUILD API] Failed to link {exe_file}")
|
||||
return False
|
||||
|
||||
print(f"[BUILD API] Successfully built {exe_file}")
|
||||
return True
|
||||
|
||||
def _link_executable(self, object_files: List[Path], output_exe: Path) -> bool:
|
||||
"""
|
||||
Link object files into an executable.
|
||||
|
||||
Args:
|
||||
object_files: Object files to link
|
||||
output_exe: Output executable path
|
||||
|
||||
Returns:
|
||||
bool: True if successful, False otherwise
|
||||
"""
|
||||
if not self._library_file or not self._library_file.exists():
|
||||
print(f"[BUILD API] Library not available for linking")
|
||||
return False
|
||||
|
||||
# Build linker command
|
||||
linker_cmd = self.build_flags.tools.linker
|
||||
link_flags = self.build_flags.link_flags
|
||||
|
||||
cmd = (
|
||||
linker_cmd
|
||||
+ link_flags
|
||||
+ [str(f) for f in object_files]
|
||||
+ [str(self._library_file)]
|
||||
+ ["-o", str(output_exe)]
|
||||
)
|
||||
cmd_str = subprocess.list2cmdline(cmd)
|
||||
|
||||
print(f"[BUILD API] Linking: {cmd_str}")
|
||||
|
||||
# Execute linker using stdout-pumping wrapper
|
||||
try:
|
||||
_completed = subprocess_run(
|
||||
command=cmd,
|
||||
cwd=None,
|
||||
check=True,
|
||||
timeout=900,
|
||||
enable_stack_trace=False,
|
||||
)
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"[BUILD API] Linker failed: {e}")
|
||||
print(f"[BUILD API] Linker stdout: {e.stdout}")
|
||||
print(f"[BUILD API] Linker stderr: {e.stderr}")
|
||||
return False
|
||||
|
||||
def get_build_info(self) -> Dict[str, Any]:
|
||||
"""Get information about the current build configuration"""
|
||||
return {
|
||||
"build_flags_toml": str(self.build_flags_toml),
|
||||
"build_dir": str(self.build_dir),
|
||||
"build_type": self.build_type.value,
|
||||
"use_pch": self.use_pch,
|
||||
"parallel": self.parallel,
|
||||
"pch_file": str(self._pch_file) if self._pch_file else None,
|
||||
"library_file": str(self._library_file) if self._library_file else None,
|
||||
}
|
||||
|
||||
def clean_build_artifacts(self) -> None:
|
||||
"""Clean all build artifacts"""
|
||||
print(f"[BUILD API] Cleaning build artifacts in {self.build_dir}")
|
||||
|
||||
if self.build_dir.exists():
|
||||
shutil.rmtree(self.build_dir)
|
||||
self.build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Recreate subdirectories
|
||||
for directory in [self.artifacts_dir, self.targets_dir, self.cache_dir]:
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Reset internal state
|
||||
self._pch_file = None
|
||||
self._library_file = None
|
||||
if self._fingerprint_cache:
|
||||
# Clear the cache by resetting it
|
||||
cache_file = self.cache_dir / "fingerprint_cache.json"
|
||||
self._fingerprint_cache = FingerprintCache(cache_file)
|
||||
|
||||
|
||||
def create_unit_test_builder(
|
||||
build_dir: str | Path = ".build/unit",
|
||||
use_pch: bool = True,
|
||||
parallel: bool = True,
|
||||
clean: bool = False,
|
||||
) -> BuildAPI:
|
||||
"""
|
||||
Create a BuildAPI instance for unit tests.
|
||||
|
||||
Args:
|
||||
build_dir: Build directory for unit test outputs
|
||||
use_pch: Whether to use precompiled headers
|
||||
parallel: Whether to enable parallel compilation
|
||||
clean: Whether to force a clean build
|
||||
|
||||
Returns:
|
||||
BuildAPI: Configured build API for unit tests
|
||||
"""
|
||||
build_flags_toml = PROJECT_ROOT / "ci" / "build_unit.toml"
|
||||
return BuildAPI(
|
||||
build_flags_toml=build_flags_toml,
|
||||
build_dir=build_dir,
|
||||
build_type=BuildType.UNIT_TEST,
|
||||
use_pch=use_pch,
|
||||
parallel=parallel,
|
||||
clean=clean,
|
||||
)
|
||||
|
||||
|
||||
def create_example_builder(
|
||||
build_dir: str | Path = ".build/examples",
|
||||
use_pch: bool = True,
|
||||
parallel: bool = True,
|
||||
clean: bool = False,
|
||||
) -> BuildAPI:
|
||||
"""
|
||||
Create a BuildAPI instance for examples.
|
||||
|
||||
Args:
|
||||
build_dir: Build directory for example outputs
|
||||
use_pch: Whether to use precompiled headers
|
||||
parallel: Whether to enable parallel compilation
|
||||
clean: Whether to force a clean build
|
||||
|
||||
Returns:
|
||||
BuildAPI: Configured build API for examples
|
||||
"""
|
||||
build_flags_toml = PROJECT_ROOT / "ci" / "build_example.toml"
|
||||
return BuildAPI(
|
||||
build_flags_toml=build_flags_toml,
|
||||
build_dir=build_dir,
|
||||
build_type=BuildType.EXAMPLE,
|
||||
use_pch=use_pch,
|
||||
parallel=parallel,
|
||||
clean=clean,
|
||||
)
|
||||
@@ -0,0 +1,32 @@
|
||||
[all]
|
||||
# Universal defines and compiler flags for compile-commands generation
|
||||
defines = [
|
||||
"STUB_PLATFORM",
|
||||
"__EMSCRIPTEN__"
|
||||
]
|
||||
compiler_flags = [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra"
|
||||
]
|
||||
# Include flags are passed verbatim; use -I and -isystem as needed
|
||||
include_flags = [
|
||||
"-Isrc",
|
||||
"-isystemsrc/platforms/stub"
|
||||
]
|
||||
|
||||
[tools]
|
||||
# Tool configuration required by BuildFlags.parse
|
||||
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
linker = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"]
|
||||
c_compiler = ["uv", "run", "python", "-m", "ziglang", "cc"]
|
||||
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
|
||||
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
|
||||
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
|
||||
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
|
||||
|
||||
[archive]
|
||||
# Required by BuildFlags.parse; not used for compile-commands generation
|
||||
flags = "rcsD"
|
||||
@@ -0,0 +1,181 @@
|
||||
# ================================================================================================
|
||||
# FastLED Example Build Flags Configuration
|
||||
# ================================================================================================
|
||||
#
|
||||
# This file centralizes ALL compilation flags used by example compilation to ensure
|
||||
# they remain synchronized and compatible across the entire build system.
|
||||
#
|
||||
# Examples use the STUB platform for Arduino-compatible functionality without hardware
|
||||
# dependencies, but do NOT need unit test specific flags like FASTLED_FORCE_NAMESPACE.
|
||||
|
||||
[tools]
|
||||
# Build tool configuration - Full compiler command with arguments
|
||||
# CRITICAL: Must use 'uv run python' for subprocess execution (subprocess doesn't inherit uv environment)
|
||||
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
linker = ["uv", "run", "python", "-m", "ziglang", "c++"] # Separate linker config for consistency
|
||||
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"] # Ziglang archiver for unified toolchain
|
||||
c_compiler = ["uv", "run", "python", "-m", "ziglang", "cc"]
|
||||
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
|
||||
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
|
||||
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
|
||||
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
|
||||
|
||||
[all]
|
||||
# Universal compilation flags (compiler command handled separately)
|
||||
compiler_flags = [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-Wno-deprecated-register", # Suppress 'register' keyword warnings for C++17 compatibility
|
||||
"-Wno-backslash-newline-escape", # Suppress backslash newline escape warnings
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
]
|
||||
|
||||
defines = [
|
||||
"STUB_PLATFORM",
|
||||
# NOTE: Examples do NOT get FASTLED_FORCE_NAMESPACE=1 - they use global namespace
|
||||
]
|
||||
|
||||
include_flags = [
|
||||
"-I.",
|
||||
"-Isrc",
|
||||
]
|
||||
|
||||
[windows]
|
||||
# ================================================================================================
|
||||
# WINDOWS-SPECIFIC BUILD FLAGS
|
||||
# ================================================================================================
|
||||
# Comprehensive Windows build configuration using Zig's bundled Clang with GNU target
|
||||
|
||||
# C compiler flags for Windows
|
||||
cc_flags = [
|
||||
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
|
||||
"-fuse-ld=lld", # Use Zig's bundled LLD linker
|
||||
"-std=c17", # C17 standard
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
]
|
||||
|
||||
# C++ compiler flags for Windows
|
||||
cpp_flags = [
|
||||
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
|
||||
"-fuse-ld=lld-link", # Use lld-link (Windows MSVC-compatible linker)
|
||||
"-std=c++17", # C++17 standard
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
|
||||
# Fix Arduino INPUT macro conflict with Windows headers
|
||||
"-DNOMINMAX", # Prevent Windows min/max macros
|
||||
"-DWIN32_LEAN_AND_MEAN", # Reduce Windows header conflicts
|
||||
]
|
||||
|
||||
# Linker flags for Windows
|
||||
link_flags = [
|
||||
"-mconsole", # Console application
|
||||
|
||||
# Prevent automatic library inclusion to avoid CRT collisions
|
||||
"-nodefaultlibs", # Don't automatically include default libraries
|
||||
|
||||
# Fix CRT collision between Zig/Clang and MSYS2/MinGW runtimes
|
||||
# This prevents "atexit was replaced" errors from conflicting CRT implementations
|
||||
# "-rtlib=compiler-rt", # Removed - was causing conflicts
|
||||
"-unwindlib=libunwind", # Use Clang's libunwind (avoid MinGW unwind)
|
||||
"-nostdlib++", # Don't auto-link standard C++ library
|
||||
"-lc++", # Manually link libc++ (Clang's C++ standard library)
|
||||
|
||||
"-lkernel32", # Windows kernel functions
|
||||
"-luser32", # Windows user interface
|
||||
"-lgdi32", # Graphics device interface
|
||||
"-ladvapi32", # Advanced Windows API
|
||||
]
|
||||
|
||||
[linking.base]
|
||||
# Base linking flags (Unix-like systems)
|
||||
flags = [
|
||||
"-pthread", # POSIX threads (not used on Windows)
|
||||
]
|
||||
|
||||
[build_modes.quick]
|
||||
flags = [
|
||||
"-O1",
|
||||
"-g0",
|
||||
"-fno-inline-functions",
|
||||
"-fno-vectorize",
|
||||
"-fno-unroll-loops",
|
||||
"-fno-strict-aliasing",
|
||||
]
|
||||
|
||||
link_flags = []
|
||||
|
||||
[strict_mode]
|
||||
flags = [
|
||||
"-Werror",
|
||||
"-Wextra",
|
||||
"-Wconversion",
|
||||
"-Wsign-conversion",
|
||||
"-Werror=unused-variable", # Force error on unused variables
|
||||
"-Werror=unused-parameter", # Force error on unused parameters
|
||||
"-Werror=unused-function", # Force error on unused functions
|
||||
"-Werror=unused-but-set-variable", # Force error on variables that are set but never used
|
||||
"-Wuninitialized",
|
||||
"-Wdouble-promotion",
|
||||
"-Wformat=2",
|
||||
"-Wcast-align",
|
||||
"-Wcast-qual",
|
||||
"-Werror=return-type",
|
||||
]
|
||||
|
||||
[stub_platform]
|
||||
# ================================================================================================
|
||||
# STUB PLATFORM DEFINES FOR EXAMPLE COMPILATION
|
||||
# ================================================================================================
|
||||
# These defines are used for compiling examples on the STUB platform
|
||||
# which provides Arduino-compatible functionality without hardware dependencies.
|
||||
|
||||
defines = [
|
||||
"STUB_PLATFORM", # Enable STUB platform
|
||||
"ARDUINO=10808", # Arduino version compatibility
|
||||
"FASTLED_USE_STUB_ARDUINO", # Use stub Arduino implementation
|
||||
"FASTLED_STUB_IMPL", # Enable STUB platform Arduino function implementations
|
||||
"SKETCH_HAS_LOTS_OF_MEMORY=1", # Enable memory-intensive features for STUB platform
|
||||
"FASTLED_HAS_ENGINE_EVENTS=1", # Enable EngineEvents for UI components
|
||||
]
|
||||
|
||||
# Include paths required for stub platform compilation
|
||||
include_paths = [
|
||||
"src/platforms/stub", # Arduino.h stub for examples
|
||||
]
|
||||
|
||||
# Additional compiler arguments for stub platform
|
||||
compiler_args = [
|
||||
# No additional args currently needed - all flags come from [all] and [build_modes.quick] sections
|
||||
]
|
||||
|
||||
[archive]
|
||||
# ================================================================================================
|
||||
# ARCHIVE CREATION FLAGS
|
||||
# ================================================================================================
|
||||
# Flags for creating static libraries (.a files) with deterministic builds.
|
||||
# Using Zig's unified toolchain (ziglang ar) which supports deterministic builds on all platforms.
|
||||
|
||||
flags = "rcsD" # r=insert, c=create if needed, s=write symbol table, D=deterministic (zero timestamps)
|
||||
|
||||
[linking.unix]
|
||||
# ================================================================================================
|
||||
# UNIX-SPECIFIC LINKING FLAGS
|
||||
# ================================================================================================
|
||||
# Linking flags specific to Unix-like systems (Linux, macOS).
|
||||
|
||||
flags = [
|
||||
# Core Libraries
|
||||
"-pthread", # POSIX threads support
|
||||
"-lm", # Math library
|
||||
"-ldl", # Dynamic loading library
|
||||
|
||||
# Platform-specific libraries (added as needed)
|
||||
# Note: Additional platform-specific flags can be added here
|
||||
]
|
||||
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Unified Build API - Example Usage
|
||||
|
||||
This script demonstrates how to use the new unified build API to replace
|
||||
the old scattered build logic for both unit tests and examples.
|
||||
|
||||
Before (scattered logic):
|
||||
- PCH building logic in multiple places
|
||||
- libfastled.a building in callers
|
||||
- Build flags mixed between unit tests and examples
|
||||
- Manual directory management
|
||||
|
||||
After (unified API):
|
||||
- Single API for both unit tests and examples
|
||||
- Automatic PCH and library building
|
||||
- Clean separation of build configurations
|
||||
- Organized build directory structure
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from ci.build_api import BuildType, create_example_builder, create_unit_test_builder
|
||||
|
||||
|
||||
def demo_unit_test_building():
|
||||
"""Demonstrate unit test building with the unified API"""
|
||||
print("=" * 60)
|
||||
print("UNIT TEST BUILDING DEMO")
|
||||
print("=" * 60)
|
||||
|
||||
# Create unit test builder - automatically uses ci/build_unit.toml
|
||||
unit_builder = create_unit_test_builder(
|
||||
build_dir=".build/unit", use_pch=True, parallel=True, clean=False
|
||||
)
|
||||
|
||||
# Get build info
|
||||
info = unit_builder.get_build_info()
|
||||
print(f"Build configuration: {info}")
|
||||
|
||||
# Build multiple unit test targets efficiently
|
||||
# The API automatically handles:
|
||||
# - PCH building (once, reused by all targets)
|
||||
# - libfastled.a building (once, reused by all targets)
|
||||
# - Individual target compilation and linking
|
||||
test_files = [
|
||||
Path("tests/test_json.cpp"),
|
||||
Path("tests/test_color.cpp"),
|
||||
Path("tests/test_math.cpp"),
|
||||
]
|
||||
|
||||
print(f"\nBuilding {len(test_files)} unit test targets...")
|
||||
results = unit_builder.build_targets(test_files)
|
||||
|
||||
# Report results
|
||||
for result in results:
|
||||
status = "SUCCESS" if result.success else "FAILED"
|
||||
print(f" {result.target.name}: {status} ({result.build_time:.2f}s)")
|
||||
if not result.success:
|
||||
print(f" Error: {result.message}")
|
||||
|
||||
print(f"\nBuild artifacts organized in: {unit_builder.build_dir}")
|
||||
print(f" - PCH file: {unit_builder.artifacts_dir}")
|
||||
print(f" - Library: {unit_builder.artifacts_dir}")
|
||||
print(f" - Targets: {unit_builder.targets_dir}")
|
||||
print(f" - Cache: {unit_builder.cache_dir}")
|
||||
|
||||
|
||||
def demo_example_building():
|
||||
"""Demonstrate example building with the unified API"""
|
||||
print("\n" + "=" * 60)
|
||||
print("EXAMPLE BUILDING DEMO")
|
||||
print("=" * 60)
|
||||
|
||||
# Create example builder - automatically uses ci/build_example.toml
|
||||
example_builder = create_example_builder(
|
||||
build_dir=".build/examples", use_pch=True, parallel=True, clean=False
|
||||
)
|
||||
|
||||
# Get build info
|
||||
info = example_builder.get_build_info()
|
||||
print(f"Build configuration: {info}")
|
||||
|
||||
# Build multiple example targets efficiently
|
||||
# The API automatically handles:
|
||||
# - PCH building (once, reused by all targets)
|
||||
# - libfastled.a building (once, with example-specific flags)
|
||||
# - Individual target compilation and linking
|
||||
example_files = [
|
||||
Path("examples/Blink/Blink.ino"),
|
||||
Path("examples/DemoReel100/DemoReel100.ino"),
|
||||
Path("examples/ColorPalette/ColorPalette.ino"),
|
||||
]
|
||||
|
||||
print(f"\nBuilding {len(example_files)} example targets...")
|
||||
results = example_builder.build_targets(example_files)
|
||||
|
||||
# Report results
|
||||
for result in results:
|
||||
status = "SUCCESS" if result.success else "FAILED"
|
||||
print(f" {result.target.name}: {status} ({result.build_time:.2f}s)")
|
||||
if not result.success:
|
||||
print(f" Error: {result.message}")
|
||||
|
||||
print(f"\nBuild artifacts organized in: {example_builder.build_dir}")
|
||||
print(f" - PCH file: {example_builder.artifacts_dir}")
|
||||
print(f" - Library: {example_builder.artifacts_dir}")
|
||||
print(f" - Targets: {example_builder.targets_dir}")
|
||||
print(f" - Cache: {example_builder.cache_dir}")
|
||||
|
||||
|
||||
def demo_build_configuration_differences():
|
||||
"""Demonstrate the differences between unit test and example configurations"""
|
||||
print("\n" + "=" * 60)
|
||||
print("BUILD CONFIGURATION DIFFERENCES")
|
||||
print("=" * 60)
|
||||
|
||||
unit_builder = create_unit_test_builder()
|
||||
example_builder = create_example_builder()
|
||||
|
||||
unit_info = unit_builder.get_build_info()
|
||||
example_info = example_builder.get_build_info()
|
||||
|
||||
print("Unit Test Configuration:")
|
||||
print(f" - Build flags TOML: {unit_info['build_flags_toml']}")
|
||||
print(f" - Build directory: {unit_info['build_dir']}")
|
||||
print(f" - Build type: {unit_info['build_type']}")
|
||||
print(f" - Includes FASTLED_FORCE_NAMESPACE=1")
|
||||
print(f" - Includes FASTLED_TESTING=1")
|
||||
|
||||
print("\nExample Configuration:")
|
||||
print(f" - Build flags TOML: {example_info['build_flags_toml']}")
|
||||
print(f" - Build directory: {example_info['build_dir']}")
|
||||
print(f" - Build type: {example_info['build_type']}")
|
||||
print(f" - Uses global namespace (no FASTLED_FORCE_NAMESPACE)")
|
||||
print(f" - Arduino compatibility defines")
|
||||
|
||||
|
||||
def demo_clean_builds():
|
||||
"""Demonstrate clean build functionality"""
|
||||
print("\n" + "=" * 60)
|
||||
print("CLEAN BUILD DEMO")
|
||||
print("=" * 60)
|
||||
|
||||
# Create builders with clean=True to force rebuilding of all artifacts
|
||||
unit_builder = create_unit_test_builder(clean=True)
|
||||
example_builder = create_example_builder(clean=True)
|
||||
|
||||
print("Unit test builder:")
|
||||
print(f" - Will clean and rebuild all artifacts in {unit_builder.build_dir}")
|
||||
|
||||
print("Example builder:")
|
||||
print(f" - Will clean and rebuild all artifacts in {example_builder.build_dir}")
|
||||
|
||||
# Demonstrate manual cleaning
|
||||
print("\nManual cleaning:")
|
||||
unit_builder.clean_build_artifacts()
|
||||
print(f" - Cleaned unit test artifacts")
|
||||
|
||||
example_builder.clean_build_artifacts()
|
||||
print(f" - Cleaned example artifacts")
|
||||
|
||||
|
||||
def demo_advanced_usage():
|
||||
"""Demonstrate advanced usage patterns"""
|
||||
print("\n" + "=" * 60)
|
||||
print("ADVANCED USAGE DEMO")
|
||||
print("=" * 60)
|
||||
|
||||
# Custom build directory structure
|
||||
unit_builder = create_unit_test_builder(
|
||||
build_dir=".build/my_custom_unit_tests",
|
||||
use_pch=False, # Disable PCH for debugging
|
||||
parallel=False, # Disable parallel for single-threaded debugging
|
||||
)
|
||||
|
||||
print("Custom configuration:")
|
||||
print(f" - Custom build directory: {unit_builder.build_dir}")
|
||||
print(f" - PCH disabled for easier debugging")
|
||||
print(f" - Parallel compilation disabled for sequential output")
|
||||
|
||||
# Show the organized directory structure
|
||||
print(f"\nOrganized directory structure:")
|
||||
print(f" {unit_builder.build_dir}/")
|
||||
print(f" ├── artifacts/ # PCH files and libfastled.a")
|
||||
print(f" ├── targets/ # Individual target builds")
|
||||
print(f" │ ├── test_json/")
|
||||
print(f" │ │ ├── test_json.o")
|
||||
print(f" │ │ └── test_json.exe")
|
||||
print(f" │ └── test_color/")
|
||||
print(f" │ ├── test_color.o")
|
||||
print(f" │ └── test_color.exe")
|
||||
print(f" └── cache/ # Fingerprint caches")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("FastLED Unified Build API - Example Usage")
|
||||
print("This demonstrates the new unified build system")
|
||||
|
||||
# Run all demos (commented out to avoid actual compilation during demo)
|
||||
|
||||
# Uncomment these to run actual builds:
|
||||
# demo_unit_test_building()
|
||||
# demo_example_building()
|
||||
|
||||
# These can run without actual building:
|
||||
demo_build_configuration_differences()
|
||||
demo_clean_builds()
|
||||
demo_advanced_usage()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("DEMO COMPLETE")
|
||||
print("=" * 60)
|
||||
print("The unified build API provides:")
|
||||
print("✓ Single API for both unit tests and examples")
|
||||
print("✓ Automatic PCH and library building")
|
||||
print("✓ Clean separation of build configurations")
|
||||
print("✓ Organized build directory structure")
|
||||
print("✓ Efficient multiple target building")
|
||||
print("✓ Simple, elegant interface")
|
||||
@@ -0,0 +1,276 @@
|
||||
# ================================================================================================
|
||||
# FastLED Unit Test Build Flags Configuration
|
||||
# ================================================================================================
|
||||
#
|
||||
# This file centralizes ALL compilation flags used by unit test compilation to ensure
|
||||
# they remain synchronized and compatible across the entire build system.
|
||||
#
|
||||
# Unit tests require special flags like FASTLED_FORCE_NAMESPACE=1 to put all symbols
|
||||
# in the fl:: namespace that the tests expect.
|
||||
|
||||
[tools]
|
||||
# Build tool configuration - Full compiler command with arguments
|
||||
# CRITICAL: Must use 'uv run python' for subprocess execution (subprocess doesn't inherit uv environment)
|
||||
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
linker = ["uv", "run", "python", "-m", "ziglang", "c++"] # Separate linker config for consistency
|
||||
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"] # Ziglang archiver for unified toolchain
|
||||
c_compiler = ["uv", "run", "python", "-m", "ziglang", "cc"]
|
||||
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
|
||||
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
|
||||
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
|
||||
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
|
||||
|
||||
[all]
|
||||
# Universal compilation flags (compiler command handled separately)
|
||||
compiler_flags = [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-Wno-deprecated-register", # Suppress 'register' keyword warnings for C++17 compatibility
|
||||
"-Wno-backslash-newline-escape", # Suppress backslash newline escape warnings
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
]
|
||||
|
||||
defines = [
|
||||
"STUB_PLATFORM",
|
||||
"FASTLED_UNIT_TEST=1",
|
||||
"FASTLED_FORCE_NAMESPACE=1", # CRITICAL: Required for unit tests to access fl:: namespace
|
||||
"FASTLED_TESTING=1", # Enable test-specific functionality like MockTimeProvider
|
||||
]
|
||||
|
||||
include_flags = [
|
||||
"-I.",
|
||||
"-Isrc",
|
||||
"-Itests",
|
||||
]
|
||||
|
||||
[windows]
|
||||
# ================================================================================================
|
||||
# WINDOWS-SPECIFIC BUILD FLAGS
|
||||
# ================================================================================================
|
||||
# Comprehensive Windows build configuration using Zig's bundled Clang with GNU target
|
||||
|
||||
# C compiler flags for Windows
|
||||
cc_flags = [
|
||||
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
|
||||
"-fuse-ld=lld", # Use Zig's bundled LLD linker
|
||||
"-std=c17", # C17 standard
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
]
|
||||
|
||||
# C++ compiler flags for Windows
|
||||
cpp_flags = [
|
||||
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
|
||||
"-fuse-ld=lld-link", # Use lld-link (Windows MSVC-compatible linker)
|
||||
"-std=c++17", # C++17 standard
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
|
||||
# Fix Arduino INPUT macro conflict with Windows headers
|
||||
"-DNOMINMAX", # Prevent Windows min/max macros
|
||||
"-DWIN32_LEAN_AND_MEAN", # Reduce Windows header conflicts
|
||||
]
|
||||
|
||||
# Linker flags for Windows
|
||||
link_flags = [
|
||||
"-mconsole", # Console application
|
||||
|
||||
# Prevent automatic library inclusion to avoid CRT collisions
|
||||
"-nodefaultlibs", # Don't automatically include default libraries
|
||||
|
||||
# Fix CRT collision between Zig/Clang and MSYS2/MinGW runtimes
|
||||
# This prevents "atexit was replaced" errors from conflicting CRT implementations
|
||||
# "-rtlib=compiler-rt", # Removed - was causing conflicts
|
||||
"-unwindlib=libunwind", # Use Clang's libunwind (avoid MinGW unwind)
|
||||
"-nostdlib++", # Don't auto-link standard C++ library
|
||||
"-lc++", # Manually link libc++ (Clang's C++ standard library)
|
||||
|
||||
"-lkernel32", # Windows kernel functions
|
||||
"-luser32", # Windows user interface
|
||||
"-lgdi32", # Graphics device interface
|
||||
"-ladvapi32", # Advanced Windows API
|
||||
]
|
||||
|
||||
[linking.base]
|
||||
# Base linking flags (Unix-like systems)
|
||||
flags = [
|
||||
"-pthread", # POSIX threads (not used on Windows)
|
||||
]
|
||||
|
||||
[linux]
|
||||
# Linux-specific build configuration – libunwind disabled (fallback to execinfo)
|
||||
link_flags = [
|
||||
]
|
||||
|
||||
defines = [
|
||||
]
|
||||
|
||||
[build_modes.quick]
|
||||
flags = [
|
||||
"-O1",
|
||||
"-fno-inline-functions",
|
||||
"-fno-vectorize",
|
||||
"-fno-unroll-loops",
|
||||
"-fno-strict-aliasing",
|
||||
"-fno-omit-frame-pointer", # Preserve stack frames for debugging
|
||||
]
|
||||
|
||||
link_flags = []
|
||||
|
||||
[build_modes.debug_asan]
|
||||
# AddressSanitizer and UndefinedBehaviorSanitizer for debugging memory issues
|
||||
flags = [
|
||||
"-fsanitize=address", # AddressSanitizer for memory error detection
|
||||
"-fsanitize=undefined", # UndefinedBehaviorSanitizer for undefined behavior
|
||||
"-O1", # Light optimization for performance
|
||||
"-fno-omit-frame-pointer", # Preserve stack frames for better debugging
|
||||
"-fno-optimize-sibling-calls", # Better stack traces
|
||||
]
|
||||
|
||||
link_flags = [
|
||||
"-fsanitize=address",
|
||||
"-fsanitize=undefined",
|
||||
]
|
||||
|
||||
[strict_mode]
|
||||
flags = [
|
||||
"-Werror",
|
||||
"-Wextra",
|
||||
"-Wconversion",
|
||||
"-Wsign-conversion",
|
||||
"-Werror=unused-variable", # Force error on unused variables
|
||||
"-Werror=unused-parameter", # Force error on unused parameters
|
||||
"-Werror=unused-function", # Force error on unused functions
|
||||
"-Werror=unused-but-set-variable", # Force error on variables that are set but never used
|
||||
"-Wuninitialized",
|
||||
"-Wdouble-promotion",
|
||||
"-Wformat=2",
|
||||
"-Wcast-align",
|
||||
"-Wcast-qual",
|
||||
"-Werror=return-type",
|
||||
]
|
||||
|
||||
[test]
|
||||
# Test-specific preprocessor defines
|
||||
defines = [
|
||||
# FastLED Test Configuration
|
||||
"-DFASTLED_UNIT_TEST=1", # Identify as unit test build
|
||||
"-DFASTLED_FORCE_NAMESPACE=1", # Force namespace usage to avoid conflicts
|
||||
"-DFASTLED_USE_PROGMEM=0", # Disable PROGMEM usage in tests
|
||||
"-DSTUB_PLATFORM", # Use stub platform for testing
|
||||
"-DARDUINO=10808", # Arduino version compatibility
|
||||
"-DFASTLED_USE_STUB_ARDUINO", # Use stub Arduino implementation
|
||||
"-DSKETCH_HAS_LOTS_OF_MEMORY=1", # Assume lots of memory available
|
||||
"-DFASTLED_STUB_IMPL", # Use stub implementations
|
||||
"-DFASTLED_USE_JSON_UI=1", # Enable JSON UI features
|
||||
"-DFASTLED_TESTING", # Enable testing features
|
||||
"-DFASTLED_NO_AUTO_NAMESPACE", # Disable automatic namespace
|
||||
"-DFASTLED_NO_PINMAP", # Disable pin mapping
|
||||
"-DHAS_HARDWARE_PIN_SUPPORT", # Enable hardware pin support
|
||||
"-DFASTLED_DEBUG_LEVEL=1", # Set debug level
|
||||
"-DFASTLED_NO_ATEXIT=1", # Disable atexit to fix Windows lld-link errors
|
||||
"-DDOCTEST_CONFIG_NO_EXCEPTIONS_BUT_WITH_ALL_ASSERTS", # Doctest configuration
|
||||
"-DENABLE_CRASH_HANDLER", # Enable global crash handler in tests
|
||||
]
|
||||
|
||||
# Test-specific compiler flags
|
||||
compiler_flags = [
|
||||
# Language Standard and Compatibility
|
||||
"-std=c++17", # Base C++17 standard
|
||||
"-std=gnu++17", # Use GNU C++17 standard with extensions
|
||||
"-fpermissive", # Allow some non-conforming code patterns
|
||||
|
||||
# Warning Configuration
|
||||
"-Wall", # Enable all warnings
|
||||
"-Wextra", # Enable extra warnings
|
||||
"-Werror=return-type", # Error on missing return statements
|
||||
|
||||
# Debug and Crash Handling
|
||||
"-fno-omit-frame-pointer", # Preserve stack frames for debugging
|
||||
|
||||
# AddressSanitizer and Memory Debugging
|
||||
"-fsanitize=address", # AddressSanitizer for memory error detection
|
||||
"-fsanitize=undefined", # UndefinedBehaviorSanitizer for undefined behavior
|
||||
|
||||
# Thread Safety and Runtime Behavior
|
||||
"-fno-threadsafe-statics", # Disable thread-safe static initialization
|
||||
"-fno-exceptions", # Disable C++ exceptions
|
||||
"-fno-rtti", # Disable RTTI
|
||||
|
||||
# PCH Support
|
||||
"-fpch-instantiate-templates", # Required for proper PCH template handling
|
||||
]
|
||||
|
||||
# Test-specific include flags
|
||||
include_flags = [
|
||||
"-I.", # Current directory
|
||||
"-Isrc", # FastLED source directory
|
||||
"-Itests", # Test directory
|
||||
]
|
||||
|
||||
# Test-specific linking flags for AddressSanitizer and libunwind
|
||||
link_flags = [
|
||||
"-fsanitize=address", # Link AddressSanitizer runtime
|
||||
"-fsanitize=undefined", # Link UndefinedBehaviorSanitizer runtime
|
||||
"-lunwind", # Link libunwind for stack traces
|
||||
"-lunwind-x86_64", # Platform-specific libunwind functions
|
||||
]
|
||||
|
||||
[archive]
|
||||
# ================================================================================================
|
||||
# ARCHIVE CREATION FLAGS
|
||||
# ================================================================================================
|
||||
# Flags for creating static libraries (.a files) with deterministic builds.
|
||||
# Using Zig's unified toolchain (ziglang ar) which supports deterministic builds on all platforms.
|
||||
|
||||
flags = "rcsD" # r=insert, c=create if needed, s=write symbol table, D=deterministic (zero timestamps)
|
||||
|
||||
[stub_platform]
|
||||
# ================================================================================================
|
||||
# STUB PLATFORM DEFINES FOR UNIT TEST COMPILATION
|
||||
# ================================================================================================
|
||||
# These defines are used when unit tests need to compile example code on the STUB platform
|
||||
# which provides Arduino-compatible functionality without hardware dependencies.
|
||||
|
||||
defines = [
|
||||
"STUB_PLATFORM", # Enable STUB platform
|
||||
"ARDUINO=10808", # Arduino version compatibility
|
||||
"FASTLED_USE_STUB_ARDUINO", # Use stub Arduino implementation
|
||||
"FASTLED_STUB_IMPL", # Enable STUB platform Arduino function implementations
|
||||
"SKETCH_HAS_LOTS_OF_MEMORY=1", # Enable memory-intensive features for STUB platform
|
||||
"FASTLED_HAS_ENGINE_EVENTS=1", # Enable EngineEvents for UI components
|
||||
"FASTLED_UNIT_TEST=1", # Identify as unit test build
|
||||
"FASTLED_FORCE_NAMESPACE=1", # Force namespace usage for unit tests
|
||||
"FASTLED_TESTING=1", # Enable testing features
|
||||
]
|
||||
|
||||
# Include paths required for stub platform compilation in unit tests
|
||||
include_paths = [
|
||||
"src/platforms/stub", # Arduino.h stub for examples
|
||||
]
|
||||
|
||||
# Additional compiler arguments for stub platform
|
||||
compiler_args = [
|
||||
# No additional args currently needed - all flags come from [all] and other sections
|
||||
]
|
||||
|
||||
[linking.unix]
|
||||
# ================================================================================================
|
||||
# UNIX-SPECIFIC LINKING FLAGS
|
||||
# ================================================================================================
|
||||
# Linking flags specific to Unix-like systems (Linux, macOS).
|
||||
|
||||
flags = [
|
||||
# Core Libraries
|
||||
"-pthread", # POSIX threads support
|
||||
"-lm", # Math library
|
||||
"-ldl", # Dynamic loading library
|
||||
|
||||
# Platform-specific libraries (added as needed)
|
||||
# Note: Additional platform-specific flags can be added here
|
||||
]
|
||||
@@ -0,0 +1,123 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, List, Match, Optional
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
PROJECT_ROOT = HERE.parent
|
||||
|
||||
IS_GITHUB = "GITHUB_ACTIONS" in os.environ
|
||||
|
||||
|
||||
def run_command(
|
||||
cmd_list: List[str],
|
||||
capture_output: bool = True,
|
||||
shell: bool = False,
|
||||
check: Optional[bool] = None,
|
||||
) -> Optional[str]:
|
||||
"""Run a command and return its output."""
|
||||
actual_check = check if check is not None else False
|
||||
cmd = cmd_list if not shell else subprocess.list2cmdline(cmd_list)
|
||||
|
||||
result: subprocess.CompletedProcess[str] = subprocess.run(
|
||||
cmd, capture_output=capture_output, text=True, shell=shell, check=actual_check
|
||||
)
|
||||
|
||||
if not capture_output:
|
||||
return None
|
||||
|
||||
stdout: str = result.stdout
|
||||
stdout = stdout.strip()
|
||||
return stdout
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check compiled program size for a board"
|
||||
)
|
||||
parser.add_argument("board", help="Board name")
|
||||
parser.add_argument(
|
||||
"--max-size", type=int, required=False, help="Maximum allowed size"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-build",
|
||||
action="store_true",
|
||||
help="Skip compilation and check existing build",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--example",
|
||||
default="Blink",
|
||||
help="Example to compile (default: Blink)",
|
||||
)
|
||||
|
||||
# Parse known args first
|
||||
args, unknown = parser.parse_known_args()
|
||||
|
||||
# Add remaining arguments as extra_args
|
||||
args.extra_args = unknown
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def main():
|
||||
os.chdir(str(PROJECT_ROOT))
|
||||
args = parse_args()
|
||||
|
||||
if not args.no_build:
|
||||
cmd_list = [
|
||||
"uv",
|
||||
"run",
|
||||
"python",
|
||||
"-m",
|
||||
"ci.ci-compile",
|
||||
args.board,
|
||||
"--examples",
|
||||
args.example,
|
||||
] + args.extra_args
|
||||
try:
|
||||
run_command(cmd_list, shell=True, capture_output=IS_GITHUB, check=True)
|
||||
except subprocess.CalledProcessError:
|
||||
run_command(cmd_list, shell=True, capture_output=False, check=True)
|
||||
|
||||
output = run_command(
|
||||
["uv", "run", "-m", "ci.compiled_size", "--board", args.board],
|
||||
capture_output=True,
|
||||
)
|
||||
size_match = re.search(r": *(\d+)", output) # type: ignore
|
||||
|
||||
if not size_match:
|
||||
print("Error: Unable to extract size from output")
|
||||
print(f"Output: {output}")
|
||||
sys.exit(1)
|
||||
|
||||
size_str = size_match.group(1) # type: ignore
|
||||
if not size_str:
|
||||
print("Error: Size group is empty")
|
||||
sys.exit(1)
|
||||
size = int(size_str) # type: ignore
|
||||
|
||||
if args.max_size is not None and args.max_size > 0:
|
||||
max_size = args.max_size
|
||||
if size > max_size:
|
||||
print(f"{args.board} size {size} is greater than max size {max_size}")
|
||||
print("::error::Compiled size exceeds maximum allowed size")
|
||||
sys.exit(1)
|
||||
else:
|
||||
print(f"{args.board} size {size} is within the limit of {max_size}")
|
||||
else:
|
||||
if not args.max_size:
|
||||
print("Warning: No max size specified")
|
||||
elif args.max_size <= 0:
|
||||
print("Warning: max size was <= 0 so no check was performed")
|
||||
print(f"{args.board} size: {size}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,5 @@
|
||||
#!/bin/bash
|
||||
|
||||
# cd to the directory of the script
|
||||
cd "$(dirname "$0")"
|
||||
uv run python ci-compile.py "$@"
|
||||
@@ -0,0 +1,424 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
BACKUP: Original concurrent build system - REPLACED by pio ci approach
|
||||
|
||||
This file is a backup of the original ci-compile.py that used concurrent builds.
|
||||
It has been replaced by a simpler pio ci approach in the main ci-compile.py.
|
||||
|
||||
The original system used ThreadPoolExecutor to run multiple board compilations
|
||||
in parallel with build directory creation and artifact recycling.
|
||||
|
||||
The new system uses pio ci command directly for each board/example combination,
|
||||
which is simpler and more straightforward.
|
||||
|
||||
This backup is kept for reference and potential future needs.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import io
|
||||
import multiprocessing
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
from typing import List, Set
|
||||
|
||||
from ci.boards import Board, create_board # type: ignore
|
||||
from ci.util.concurrent_run import ConcurrentRunArgs, concurrent_run
|
||||
from ci.util.locked_print import locked_print
|
||||
|
||||
|
||||
HERE = Path(__file__).parent.resolve()
|
||||
|
||||
LIBS = ["src"]
|
||||
EXTRA_LIBS = [
|
||||
"https://github.com/me-no-dev/ESPAsyncWebServer.git",
|
||||
"ArduinoOTA",
|
||||
"SD",
|
||||
"FS",
|
||||
"ESPmDNS",
|
||||
"WiFi",
|
||||
"WebSockets",
|
||||
]
|
||||
BUILD_FLAGS = ["-Wl,-Map,firmware.map", "-fopt-info-all=optimization_report.txt"]
|
||||
|
||||
# Default boards to compile for. You can use boards not defined here but
|
||||
# if the board isn't part of the officially supported platformio boards then
|
||||
# you will need to add the board to the ~/.platformio/platforms directory.
|
||||
# prior to running this script. This happens automatically as of 2024-08-20
|
||||
# with the github workflow scripts.
|
||||
DEFAULT_BOARDS_NAMES = [
|
||||
"apollo3_red",
|
||||
"apollo3_thing_explorable",
|
||||
"web", # work in progress
|
||||
"uno", # Build is faster if this is first, because it's used for global init.
|
||||
"esp32dev",
|
||||
"esp01", # ESP8266
|
||||
"esp32c3",
|
||||
"attiny85",
|
||||
"ATtiny1616",
|
||||
"esp32c6",
|
||||
"esp32s3",
|
||||
"esp32p4",
|
||||
"yun",
|
||||
"digix",
|
||||
"teensylc",
|
||||
"teensy30",
|
||||
"teensy31",
|
||||
"teensy41",
|
||||
"adafruit_feather_nrf52840_sense",
|
||||
"xiaoblesense_adafruit",
|
||||
"rpipico",
|
||||
"rpipico2",
|
||||
"uno_r4_wifi",
|
||||
"esp32rmt_51",
|
||||
"esp32dev_idf44",
|
||||
"bluepill",
|
||||
"esp32rmt_51",
|
||||
"giga_r1",
|
||||
"sparkfun_xrp_controller",
|
||||
]
|
||||
|
||||
OTHER_BOARDS_NAMES = [
|
||||
"nano_every",
|
||||
"esp32-c2-devkitm-1",
|
||||
]
|
||||
|
||||
# Examples to compile.
|
||||
DEFAULT_EXAMPLES = [
|
||||
"Animartrix",
|
||||
"Apa102",
|
||||
"Apa102HD",
|
||||
"Apa102HDOverride",
|
||||
"Audio",
|
||||
"Blink",
|
||||
"Blur",
|
||||
"Chromancer",
|
||||
"ColorPalette",
|
||||
"ColorTemperature",
|
||||
"Corkscrew",
|
||||
"Cylon",
|
||||
"DemoReel100",
|
||||
# "Downscale",
|
||||
"FestivalStick",
|
||||
"FirstLight",
|
||||
"Fire2012",
|
||||
"Multiple/MultipleStripsInOneArray",
|
||||
"Multiple/ArrayOfLedArrays",
|
||||
"Noise",
|
||||
"NoisePlayground",
|
||||
"NoisePlusPalette",
|
||||
# "LuminescentGrand",
|
||||
"Pacifica",
|
||||
"Pride2015",
|
||||
"RGBCalibrate",
|
||||
"RGBSetDemo",
|
||||
"RGBW",
|
||||
"Overclock",
|
||||
"RGBWEmulated",
|
||||
"TwinkleFox",
|
||||
"XYMatrix",
|
||||
"FireMatrix",
|
||||
"FireCylinder",
|
||||
"FxGfx2Video",
|
||||
"FxSdCard",
|
||||
"FxCylon",
|
||||
"FxDemoReel100",
|
||||
"FxTwinkleFox",
|
||||
"FxFire2012",
|
||||
"FxNoisePlusPalette",
|
||||
"FxPacifica",
|
||||
"FxEngine",
|
||||
"WS2816",
|
||||
]
|
||||
|
||||
EXTRA_EXAMPLES: dict[Board, list[str]] = {
|
||||
# ESP32DEV: ["EspI2SDemo"],
|
||||
# ESP32_S3_DEVKITC_1: ["EspS3I2SDemo"],
|
||||
}
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compile FastLED examples for various boards."
|
||||
)
|
||||
# parser.add_argument(
|
||||
# "--boards", type=str, help="Comma-separated list of boards to compile for"
|
||||
# )
|
||||
# needs to be a positional argument instead
|
||||
parser.add_argument(
|
||||
"boards",
|
||||
type=str,
|
||||
help="Comma-separated list of boards to compile for",
|
||||
nargs="?",
|
||||
)
|
||||
parser.add_argument(
|
||||
"positional_examples",
|
||||
type=str,
|
||||
help="Examples to compile (positional arguments after board name)",
|
||||
nargs="*",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--examples", type=str, help="Comma-separated list of examples to compile"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--exclude-examples", type=str, help="Examples that should be excluded"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-init", action="store_true", help="Skip the initialization step"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--defines", type=str, help="Comma-separated list of compiler definitions"
|
||||
)
|
||||
parser.add_argument("--customsdk", type=str, help="custom_sdkconfig project option")
|
||||
parser.add_argument(
|
||||
"--extra-packages",
|
||||
type=str,
|
||||
help="Comma-separated list of extra packages to install",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--add-extra-esp32-libs",
|
||||
action="store_true",
|
||||
help="Add extra libraries to the libraries list to check against compiler errors.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--build-dir", type=str, help="Override the default build directory"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-project-options",
|
||||
action="store_true",
|
||||
help="Don't use custom project options",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--interactive",
|
||||
action="store_true",
|
||||
help="Enable interactive mode to choose a board",
|
||||
)
|
||||
# Passed by the github action to disable interactive mode.
|
||||
parser.add_argument(
|
||||
"--no-interactive", action="store_true", help="Disable interactive mode"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v", "--verbose", action="store_true", help="Enable verbose output"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--supported-boards",
|
||||
action="store_true",
|
||||
help="Print the list of supported boards and exit",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--symbols",
|
||||
action="store_true",
|
||||
help="Run symbol analysis on compiled output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allsrc",
|
||||
action="store_true",
|
||||
help="Enable all-source build (adds FASTLED_ALL_SRC=1 define)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-allsrc",
|
||||
action="store_true",
|
||||
help="Disable all-source build (adds FASTLED_ALL_SRC=0 define)",
|
||||
)
|
||||
try:
|
||||
args = parser.parse_intermixed_args()
|
||||
unknown = []
|
||||
except SystemExit:
|
||||
# If parse_intermixed_args fails, fall back to parse_known_args
|
||||
args, unknown = parser.parse_known_args()
|
||||
|
||||
# Handle unknown arguments intelligently - treat non-flag arguments as examples
|
||||
unknown_examples = []
|
||||
unknown_flags = []
|
||||
for arg in unknown:
|
||||
if arg.startswith("-"):
|
||||
unknown_flags.append(arg)
|
||||
else:
|
||||
unknown_examples.append(arg)
|
||||
|
||||
# Add unknown examples to positional_examples
|
||||
if unknown_examples:
|
||||
if not hasattr(args, "positional_examples") or args.positional_examples is None:
|
||||
args.positional_examples = []
|
||||
args.positional_examples.extend(unknown_examples)
|
||||
|
||||
# Only warn about actual unknown flags, not examples
|
||||
if unknown_flags:
|
||||
warnings.warn(f"Unknown arguments: {unknown_flags}")
|
||||
|
||||
# Check for FASTLED_CI_NO_INTERACTIVE environment variable
|
||||
# This allows test.py and other scripts to force non-interactive mode
|
||||
if os.environ.get("FASTLED_CI_NO_INTERACTIVE") == "true":
|
||||
args.interactive = False
|
||||
args.no_interactive = True
|
||||
|
||||
# if --interactive and --no-interative are both passed, --no-interactive takes precedence.
|
||||
if args.interactive and args.no_interactive:
|
||||
warnings.warn(
|
||||
"Both --interactive and --no-interactive were passed, --no-interactive takes precedence."
|
||||
)
|
||||
args.interactive = False
|
||||
|
||||
# Validate that --allsrc and --no-allsrc are not both specified
|
||||
if args.allsrc and args.no_allsrc:
|
||||
warnings.warn(
|
||||
"Both --allsrc and --no-allsrc were passed, this is contradictory. Please specify only one."
|
||||
)
|
||||
sys.exit(1) # Exit with error
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def remove_duplicates(items: List[str]) -> List[str]:
|
||||
seen: Set[str] = set()
|
||||
out: List[str] = []
|
||||
for item in items:
|
||||
if item not in seen:
|
||||
seen.add(item)
|
||||
out.append(item)
|
||||
return out
|
||||
|
||||
|
||||
def choose_board_interactively(boards: List[str]) -> List[str]:
|
||||
print("Available boards:")
|
||||
boards = remove_duplicates(sorted(boards))
|
||||
for i, board in enumerate(boards):
|
||||
print(f"[{i}]: {board}")
|
||||
print("[all]: All boards")
|
||||
out: List[str] = []
|
||||
while True:
|
||||
try:
|
||||
# choice = int(input("Enter the number of the board(s) you want to compile to: "))
|
||||
input_str = input(
|
||||
"Enter the number of the board(s) you want to compile to, or it's name(s): "
|
||||
)
|
||||
if "all" in input_str:
|
||||
return boards
|
||||
for board in input_str.split(","):
|
||||
if board == "":
|
||||
continue
|
||||
if not board.isdigit():
|
||||
out.append(board) # Assume it's a board name.
|
||||
else:
|
||||
index = int(board) # Find the board from the index.
|
||||
if 0 <= index < len(boards):
|
||||
out.append(boards[index])
|
||||
else:
|
||||
warnings.warn(f"invalid board index: {index}, skipping")
|
||||
if not out:
|
||||
print("Please try again.")
|
||||
continue
|
||||
return out
|
||||
except ValueError:
|
||||
print("Invalid input. Please enter a number.")
|
||||
|
||||
|
||||
def resolve_example_path(example: str) -> Path:
|
||||
example_path = HERE.parent / "examples" / example
|
||||
if not example_path.exists():
|
||||
raise FileNotFoundError(f"Example '{example}' not found at '{example_path}'")
|
||||
return example_path
|
||||
|
||||
|
||||
def create_concurrent_run_args(args: argparse.Namespace) -> ConcurrentRunArgs:
|
||||
skip_init = args.skip_init
|
||||
if args.interactive:
|
||||
boards = choose_board_interactively(DEFAULT_BOARDS_NAMES + OTHER_BOARDS_NAMES)
|
||||
else:
|
||||
boards = args.boards.split(",") if args.boards else DEFAULT_BOARDS_NAMES
|
||||
projects: list[Board] = []
|
||||
for board in boards:
|
||||
projects.append(create_board(board, no_project_options=args.no_project_options))
|
||||
extra_examples: dict[Board, list[Path]] = {}
|
||||
# Handle both positional and named examples
|
||||
if args.positional_examples:
|
||||
# Convert positional examples, handling both "examples/Blink" and "Blink" formats
|
||||
examples = []
|
||||
for example in args.positional_examples:
|
||||
# Remove "examples/" prefix if present
|
||||
if example.startswith("examples/"):
|
||||
example = example[len("examples/") :]
|
||||
examples.append(example)
|
||||
elif args.examples:
|
||||
examples = args.examples.split(",")
|
||||
else:
|
||||
examples = DEFAULT_EXAMPLES
|
||||
# Only add extra examples when using defaults
|
||||
for b, _examples in EXTRA_EXAMPLES.items():
|
||||
resolved_examples = [resolve_example_path(example) for example in _examples]
|
||||
extra_examples[b] = resolved_examples
|
||||
examples_paths = [resolve_example_path(example) for example in examples]
|
||||
# now process example exclusions.
|
||||
if args.exclude_examples:
|
||||
exclude_examples = args.exclude_examples.split(",")
|
||||
examples_paths = [
|
||||
example
|
||||
for example in examples_paths
|
||||
if example.name not in exclude_examples
|
||||
]
|
||||
for exclude in exclude_examples:
|
||||
examples.remove(exclude)
|
||||
customsdk = args.customsdk
|
||||
defines: list[str] = []
|
||||
if args.defines:
|
||||
defines.extend(args.defines.split(","))
|
||||
# Add FASTLED_ALL_SRC define when --allsrc or --no-allsrc flag is specified
|
||||
if args.allsrc:
|
||||
defines.append("FASTLED_ALL_SRC=1")
|
||||
elif args.no_allsrc:
|
||||
defines.append("FASTLED_ALL_SRC=0")
|
||||
extra_packages: list[str] = []
|
||||
if args.extra_packages:
|
||||
extra_packages.extend(args.extra_packages.split(","))
|
||||
build_dir = args.build_dir
|
||||
extra_scripts = "pre:lib/ci/util-flags.py"
|
||||
verbose = args.verbose
|
||||
|
||||
out: ConcurrentRunArgs = ConcurrentRunArgs(
|
||||
projects=projects,
|
||||
examples=examples_paths,
|
||||
skip_init=skip_init,
|
||||
defines=defines,
|
||||
customsdk=customsdk,
|
||||
extra_packages=extra_packages,
|
||||
libs=LIBS,
|
||||
build_dir=build_dir,
|
||||
extra_scripts=extra_scripts,
|
||||
cwd=str(HERE.parent),
|
||||
board_dir=(HERE / "boards").absolute().as_posix(),
|
||||
build_flags=BUILD_FLAGS,
|
||||
verbose=verbose,
|
||||
extra_examples=extra_examples,
|
||||
symbols=args.symbols,
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main function."""
|
||||
args = parse_args()
|
||||
if args.supported_boards:
|
||||
print(",".join(DEFAULT_BOARDS_NAMES))
|
||||
return 0
|
||||
if args.add_extra_esp32_libs:
|
||||
LIBS.extend(EXTRA_LIBS)
|
||||
|
||||
# Set the working directory to the script's parent directory.
|
||||
run_args = create_concurrent_run_args(args)
|
||||
start_time = time.time()
|
||||
rtn = concurrent_run(args=run_args)
|
||||
time_taken = time.strftime("%Mm:%Ss", time.gmtime(time.time() - start_time))
|
||||
locked_print(f"Compilation finished in {time_taken}.")
|
||||
return rtn
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,21 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def main():
|
||||
# Change to the directory of the script
|
||||
os.chdir(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# Change to the 'native' directory and run 'pio run'
|
||||
os.chdir("native")
|
||||
result = subprocess.run(["pio", "run"], check=True)
|
||||
|
||||
# Exit with the same status as the pio run command
|
||||
sys.exit(result.returncode)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,596 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
Runs the compilation process for examples on boards using pio ci command.
|
||||
This replaces the previous concurrent build system with a simpler pio ci approach.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
from typing import List, Set
|
||||
|
||||
from ci.boards import Board, create_board # type: ignore
|
||||
from ci.util.locked_print import locked_print
|
||||
|
||||
|
||||
HERE = Path(__file__).parent.resolve()
|
||||
|
||||
# Default boards to compile for
|
||||
DEFAULT_BOARDS_NAMES = [
|
||||
"apollo3_red",
|
||||
"apollo3_thing_explorable",
|
||||
"web", # work in progress
|
||||
"uno", # Build is faster if this is first, because it's used for global init.
|
||||
"esp32dev",
|
||||
"esp01", # ESP8266
|
||||
"esp32c3",
|
||||
"attiny85",
|
||||
"ATtiny1616",
|
||||
"esp32c6",
|
||||
"esp32s3",
|
||||
"esp32p4",
|
||||
"yun",
|
||||
"digix",
|
||||
"teensylc",
|
||||
"teensy30",
|
||||
"teensy31",
|
||||
"teensy41",
|
||||
"adafruit_feather_nrf52840_sense",
|
||||
"xiaoblesense_adafruit",
|
||||
"rpipico",
|
||||
"rpipico2",
|
||||
"uno_r4_wifi",
|
||||
"esp32rmt_51",
|
||||
"esp32dev_idf44",
|
||||
"bluepill",
|
||||
"esp32rmt_51",
|
||||
"giga_r1",
|
||||
"sparkfun_xrp_controller",
|
||||
]
|
||||
|
||||
OTHER_BOARDS_NAMES = [
|
||||
"nano_every",
|
||||
"esp32-c2-devkitm-1",
|
||||
]
|
||||
|
||||
# Examples to compile.
|
||||
DEFAULT_EXAMPLES = [
|
||||
"Animartrix",
|
||||
"Apa102",
|
||||
"Apa102HD",
|
||||
"Apa102HDOverride",
|
||||
"Audio",
|
||||
"Blink",
|
||||
"Blur",
|
||||
"Chromancer",
|
||||
"ColorPalette",
|
||||
"ColorTemperature",
|
||||
"Corkscrew",
|
||||
"Cylon",
|
||||
"DemoReel100",
|
||||
# "Downscale",
|
||||
"FestivalStick",
|
||||
"FirstLight",
|
||||
"Fire2012",
|
||||
"Multiple/MultipleStripsInOneArray",
|
||||
"Multiple/ArrayOfLedArrays",
|
||||
"Noise",
|
||||
"NoisePlayground",
|
||||
"NoisePlusPalette",
|
||||
# "LuminescentGrand",
|
||||
"Pacifica",
|
||||
"Pride2015",
|
||||
"RGBCalibrate",
|
||||
"RGBSetDemo",
|
||||
"RGBW",
|
||||
"Overclock",
|
||||
"RGBWEmulated",
|
||||
"TwinkleFox",
|
||||
"XYMatrix",
|
||||
"FireMatrix",
|
||||
"FireCylinder",
|
||||
"FxGfx2Video",
|
||||
"FxSdCard",
|
||||
"FxCylon",
|
||||
"FxDemoReel100",
|
||||
"FxTwinkleFox",
|
||||
"FxFire2012",
|
||||
"FxNoisePlusPalette",
|
||||
"FxPacifica",
|
||||
"FxEngine",
|
||||
"WS2816",
|
||||
]
|
||||
|
||||
EXTRA_EXAMPLES: dict[Board, list[str]] = {
|
||||
# ESP32DEV: ["EspI2SDemo"],
|
||||
# ESP32_S3_DEVKITC_1: ["EspS3I2SDemo"],
|
||||
}
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compile FastLED examples for various boards using pio ci.util."
|
||||
)
|
||||
parser.add_argument(
|
||||
"boards",
|
||||
type=str,
|
||||
help="Comma-separated list of boards to compile for",
|
||||
nargs="?",
|
||||
)
|
||||
parser.add_argument(
|
||||
"positional_examples",
|
||||
type=str,
|
||||
help="Examples to compile (positional arguments after board name)",
|
||||
nargs="*",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--examples", type=str, help="Comma-separated list of examples to compile"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--exclude-examples", type=str, help="Examples that should be excluded"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--defines", type=str, help="Comma-separated list of compiler definitions"
|
||||
)
|
||||
parser.add_argument("--customsdk", type=str, help="custom_sdkconfig project option")
|
||||
parser.add_argument(
|
||||
"--extra-packages",
|
||||
type=str,
|
||||
help="Comma-separated list of extra packages to install",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--build-dir", type=str, help="Override the default build directory"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--interactive",
|
||||
action="store_true",
|
||||
help="Enable interactive mode to choose a board",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-interactive", action="store_true", help="Disable interactive mode"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v", "--verbose", action="store_true", help="Enable verbose output"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--supported-boards",
|
||||
action="store_true",
|
||||
help="Print the list of supported boards and exit",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--symbols",
|
||||
action="store_true",
|
||||
help="Run symbol analysis on compiled output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--allsrc",
|
||||
action="store_true",
|
||||
help="Enable all-source build (adds FASTLED_ALL_SRC=1 define)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-allsrc",
|
||||
action="store_true",
|
||||
help="Disable all-source build (adds FASTLED_ALL_SRC=0 define)",
|
||||
)
|
||||
try:
|
||||
args = parser.parse_intermixed_args()
|
||||
unknown = []
|
||||
except SystemExit:
|
||||
# If parse_intermixed_args fails, fall back to parse_known_args
|
||||
args, unknown = parser.parse_known_args()
|
||||
|
||||
# Handle unknown arguments intelligently - treat non-flag arguments as examples
|
||||
unknown_examples = []
|
||||
unknown_flags = []
|
||||
for arg in unknown:
|
||||
if arg.startswith("-"):
|
||||
unknown_flags.append(arg)
|
||||
else:
|
||||
unknown_examples.append(arg)
|
||||
|
||||
# Add unknown examples to positional_examples
|
||||
if unknown_examples:
|
||||
if not hasattr(args, "positional_examples") or args.positional_examples is None:
|
||||
args.positional_examples = []
|
||||
args.positional_examples.extend(unknown_examples)
|
||||
|
||||
# Only warn about actual unknown flags, not examples
|
||||
if unknown_flags:
|
||||
warnings.warn(f"Unknown arguments: {unknown_flags}")
|
||||
|
||||
# Check for FASTLED_CI_NO_INTERACTIVE environment variable
|
||||
if os.environ.get("FASTLED_CI_NO_INTERACTIVE") == "true":
|
||||
args.interactive = False
|
||||
args.no_interactive = True
|
||||
|
||||
# if --interactive and --no-interative are both passed, --no-interactive takes precedence.
|
||||
if args.interactive and args.no_interactive:
|
||||
warnings.warn(
|
||||
"Both --interactive and --no-interactive were passed, --no-interactive takes precedence."
|
||||
)
|
||||
args.interactive = False
|
||||
|
||||
# Validate that --allsrc and --no-allsrc are not both specified
|
||||
if args.allsrc and args.no_allsrc:
|
||||
warnings.warn(
|
||||
"Both --allsrc and --no-allsrc were passed, this is contradictory. Please specify only one."
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def remove_duplicates(items: List[str]) -> List[str]:
|
||||
seen: Set[str] = set()
|
||||
out: List[str] = []
|
||||
for item in items:
|
||||
if item not in seen:
|
||||
seen.add(item)
|
||||
out.append(item)
|
||||
return out
|
||||
|
||||
|
||||
def choose_board_interactively(boards: List[str]) -> List[str]:
|
||||
print("Available boards:")
|
||||
boards = remove_duplicates(sorted(boards))
|
||||
for i, board in enumerate(boards):
|
||||
print(f"[{i}]: {board}")
|
||||
print("[all]: All boards")
|
||||
out: list[str] = []
|
||||
while True:
|
||||
try:
|
||||
input_str = input(
|
||||
"Enter the number of the board(s) you want to compile to, or it's name(s): "
|
||||
)
|
||||
if "all" in input_str:
|
||||
return boards
|
||||
for board in input_str.split(","):
|
||||
if board == "":
|
||||
continue
|
||||
if not board.isdigit():
|
||||
out.append(board) # Assume it's a board name.
|
||||
else:
|
||||
index = int(board) # Find the board from the index.
|
||||
if 0 <= index < len(boards):
|
||||
out.append(boards[index])
|
||||
else:
|
||||
warnings.warn(f"invalid board index: {index}, skipping")
|
||||
if not out:
|
||||
print("Please try again.")
|
||||
continue
|
||||
return out
|
||||
except ValueError:
|
||||
print("Invalid input. Please enter a number.")
|
||||
|
||||
|
||||
def resolve_example_path(example: str) -> Path:
|
||||
example_path = HERE.parent / "examples" / example
|
||||
if not example_path.exists():
|
||||
raise FileNotFoundError(f"Example '{example}' not found at '{example_path}'")
|
||||
return example_path
|
||||
|
||||
|
||||
def compile_with_pio_ci(
|
||||
board: Board,
|
||||
example_paths: list[Path],
|
||||
build_dir: str | None,
|
||||
defines: list[str],
|
||||
verbose: bool,
|
||||
) -> tuple[bool, str]:
|
||||
"""Compile examples for a board using pio ci command."""
|
||||
|
||||
# Skip web boards
|
||||
if board.board_name == "web":
|
||||
locked_print(f"Skipping web target for board {board.board_name}")
|
||||
return True, ""
|
||||
|
||||
board_name = board.board_name
|
||||
real_board_name = board.get_real_board_name()
|
||||
|
||||
# Set up build directory
|
||||
if build_dir:
|
||||
board_build_dir = Path(build_dir) / board_name
|
||||
else:
|
||||
board_build_dir = Path(".build") / board_name
|
||||
|
||||
board_build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
locked_print(f"*** Compiling examples for board {board_name} using pio ci ***")
|
||||
|
||||
errors: list[str] = []
|
||||
|
||||
for example_path in example_paths:
|
||||
locked_print(
|
||||
f"*** Building example {example_path.name} for board {board_name} ***"
|
||||
)
|
||||
|
||||
# Find the .ino file in the example directory
|
||||
ino_files = list(example_path.glob("*.ino"))
|
||||
if not ino_files:
|
||||
error_msg = f"No .ino file found in {example_path}"
|
||||
locked_print(f"ERROR: {error_msg}")
|
||||
errors.append(error_msg)
|
||||
continue
|
||||
|
||||
# Use the first .ino file found
|
||||
ino_file = ino_files[0]
|
||||
|
||||
# Build pio ci command
|
||||
cmd_list = [
|
||||
"pio",
|
||||
"ci",
|
||||
str(ino_file),
|
||||
"--board",
|
||||
real_board_name,
|
||||
"--lib",
|
||||
"src", # FastLED source directory
|
||||
"--keep-build-dir",
|
||||
"--build-dir",
|
||||
str(board_build_dir / example_path.name),
|
||||
]
|
||||
|
||||
# Add platform-specific options
|
||||
if board.platform:
|
||||
cmd_list.extend(["--project-option", f"platform={board.platform}"])
|
||||
|
||||
if board.platform_packages:
|
||||
cmd_list.extend(
|
||||
["--project-option", f"platform_packages={board.platform_packages}"]
|
||||
)
|
||||
|
||||
if board.framework:
|
||||
cmd_list.extend(["--project-option", f"framework={board.framework}"])
|
||||
|
||||
if board.board_build_core:
|
||||
cmd_list.extend(
|
||||
["--project-option", f"board_build.core={board.board_build_core}"]
|
||||
)
|
||||
|
||||
if board.board_build_filesystem_size:
|
||||
cmd_list.extend(
|
||||
[
|
||||
"--project-option",
|
||||
f"board_build.filesystem_size={board.board_build_filesystem_size}",
|
||||
]
|
||||
)
|
||||
|
||||
# Add defines
|
||||
all_defines = defines.copy()
|
||||
if board.defines:
|
||||
all_defines.extend(board.defines)
|
||||
|
||||
if all_defines:
|
||||
build_flags = " ".join(f"-D{define}" for define in all_defines)
|
||||
cmd_list.extend(["--project-option", f"build_flags={build_flags}"])
|
||||
|
||||
# Add custom SDK config if specified
|
||||
if board.customsdk:
|
||||
cmd_list.extend(["--project-option", f"custom_sdkconfig={board.customsdk}"])
|
||||
|
||||
# Add verbose flag if requested
|
||||
if verbose:
|
||||
cmd_list.append("--verbose")
|
||||
|
||||
# Execute the command
|
||||
cmd_str = subprocess.list2cmdline(cmd_list)
|
||||
locked_print(f"Running command: {cmd_str}")
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd_list,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=str(HERE.parent),
|
||||
timeout=300, # 5 minute timeout per example
|
||||
)
|
||||
|
||||
elapsed_time = time.time() - start_time
|
||||
|
||||
if result.returncode == 0:
|
||||
locked_print(
|
||||
f"*** Successfully built {example_path.name} for {board_name} in {elapsed_time:.2f}s ***"
|
||||
)
|
||||
if verbose and result.stdout:
|
||||
locked_print(f"Build output:\n{result.stdout}")
|
||||
else:
|
||||
error_msg = f"Failed to build {example_path.name} for {board_name}"
|
||||
locked_print(f"ERROR: {error_msg}")
|
||||
locked_print(f"Command: {cmd_str}")
|
||||
if result.stdout:
|
||||
locked_print(f"STDOUT:\n{result.stdout}")
|
||||
if result.stderr:
|
||||
locked_print(f"STDERR:\n{result.stderr}")
|
||||
errors.append(f"{error_msg}: {result.stderr}")
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
error_msg = f"Timeout building {example_path.name} for {board_name}"
|
||||
locked_print(f"ERROR: {error_msg}")
|
||||
errors.append(error_msg)
|
||||
except Exception as e:
|
||||
error_msg = f"Exception building {example_path.name} for {board_name}: {e}"
|
||||
locked_print(f"ERROR: {error_msg}")
|
||||
errors.append(error_msg)
|
||||
|
||||
if errors:
|
||||
return False, "\n".join(errors)
|
||||
|
||||
return True, f"Successfully compiled all examples for {board_name}"
|
||||
|
||||
|
||||
def run_symbol_analysis(boards: list[Board]) -> None:
|
||||
"""Run symbol analysis on compiled outputs if requested."""
|
||||
locked_print("\nRunning symbol analysis on compiled outputs...")
|
||||
|
||||
for board in boards:
|
||||
if board.board_name == "web":
|
||||
continue
|
||||
|
||||
try:
|
||||
locked_print(f"Running symbol analysis for board: {board.board_name}")
|
||||
|
||||
cmd = [
|
||||
"uv",
|
||||
"run",
|
||||
"ci/util/symbol_analysis.py",
|
||||
"--board",
|
||||
board.board_name,
|
||||
]
|
||||
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, cwd=str(HERE.parent)
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
locked_print(
|
||||
f"ERROR: Symbol analysis failed for board {board.board_name}: {result.stderr}"
|
||||
)
|
||||
else:
|
||||
locked_print(f"Symbol analysis completed for board: {board.board_name}")
|
||||
if result.stdout:
|
||||
print(result.stdout)
|
||||
|
||||
except Exception as e:
|
||||
locked_print(
|
||||
f"ERROR: Exception during symbol analysis for board {board.board_name}: {e}"
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main function."""
|
||||
args = parse_args()
|
||||
|
||||
if args.supported_boards:
|
||||
print(",".join(DEFAULT_BOARDS_NAMES))
|
||||
return 0
|
||||
|
||||
# Determine which boards to compile for
|
||||
if args.interactive:
|
||||
boards_names = choose_board_interactively(
|
||||
DEFAULT_BOARDS_NAMES + OTHER_BOARDS_NAMES
|
||||
)
|
||||
else:
|
||||
boards_names = args.boards.split(",") if args.boards else DEFAULT_BOARDS_NAMES
|
||||
|
||||
# Get board objects
|
||||
boards: list[Board] = []
|
||||
for board_name in boards_names:
|
||||
try:
|
||||
board = create_board(board_name, no_project_options=False)
|
||||
boards.append(board)
|
||||
except Exception as e:
|
||||
locked_print(f"ERROR: Failed to get board '{board_name}': {e}")
|
||||
return 1
|
||||
|
||||
# Determine which examples to compile
|
||||
if args.positional_examples:
|
||||
# Convert positional examples, handling both "examples/Blink" and "Blink" formats
|
||||
examples = []
|
||||
for example in args.positional_examples:
|
||||
# Remove "examples/" prefix if present
|
||||
if example.startswith("examples/"):
|
||||
example = example[len("examples/") :]
|
||||
examples.append(example)
|
||||
elif args.examples:
|
||||
examples = args.examples.split(",")
|
||||
else:
|
||||
examples = DEFAULT_EXAMPLES
|
||||
|
||||
# Process example exclusions
|
||||
if args.exclude_examples:
|
||||
exclude_examples = args.exclude_examples.split(",")
|
||||
examples = [ex for ex in examples if ex not in exclude_examples]
|
||||
|
||||
# Resolve example paths
|
||||
example_paths: list[Path] = []
|
||||
for example in examples:
|
||||
try:
|
||||
example_path = resolve_example_path(example)
|
||||
example_paths.append(example_path)
|
||||
except FileNotFoundError as e:
|
||||
locked_print(f"ERROR: {e}")
|
||||
return 1
|
||||
|
||||
# Add extra examples for specific boards
|
||||
extra_examples: dict[Board, list[Path]] = {}
|
||||
for board in boards:
|
||||
if board in EXTRA_EXAMPLES:
|
||||
board_examples = []
|
||||
for example in EXTRA_EXAMPLES[board]:
|
||||
try:
|
||||
board_examples.append(resolve_example_path(example))
|
||||
except FileNotFoundError as e:
|
||||
locked_print(f"WARNING: {e}")
|
||||
if board_examples:
|
||||
extra_examples[board] = board_examples
|
||||
|
||||
# Set up defines
|
||||
defines: list[str] = []
|
||||
if args.defines:
|
||||
defines.extend(args.defines.split(","))
|
||||
|
||||
# Add FASTLED_ALL_SRC define when --allsrc or --no-allsrc flag is specified
|
||||
if args.allsrc:
|
||||
defines.append("FASTLED_ALL_SRC=1")
|
||||
elif args.no_allsrc:
|
||||
defines.append("FASTLED_ALL_SRC=0")
|
||||
|
||||
# Start compilation
|
||||
start_time = time.time()
|
||||
locked_print(
|
||||
f"Starting compilation for {len(boards)} boards with {len(example_paths)} examples"
|
||||
)
|
||||
|
||||
compilation_errors: List[str] = []
|
||||
|
||||
# Compile for each board
|
||||
for board in boards:
|
||||
board_examples = example_paths.copy()
|
||||
if board in extra_examples:
|
||||
board_examples.extend(extra_examples[board])
|
||||
|
||||
success, message = compile_with_pio_ci(
|
||||
board=board,
|
||||
example_paths=board_examples,
|
||||
build_dir=args.build_dir,
|
||||
defines=defines,
|
||||
verbose=args.verbose,
|
||||
)
|
||||
|
||||
if not success:
|
||||
compilation_errors.append(f"Board {board.board_name}: {message}")
|
||||
locked_print(f"ERROR: Compilation failed for board {board.board_name}")
|
||||
# Continue with other boards instead of stopping
|
||||
|
||||
# Run symbol analysis if requested
|
||||
if args.symbols:
|
||||
run_symbol_analysis(boards)
|
||||
|
||||
# Report results
|
||||
elapsed_time = time.time() - start_time
|
||||
time_str = time.strftime("%Mm:%Ss", time.gmtime(elapsed_time))
|
||||
|
||||
if compilation_errors:
|
||||
locked_print(
|
||||
f"\nCompilation finished in {time_str} with {len(compilation_errors)} error(s):"
|
||||
)
|
||||
for error in compilation_errors:
|
||||
locked_print(f" - {error}")
|
||||
return 1
|
||||
else:
|
||||
locked_print(f"\nAll compilations completed successfully in {time_str}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
locked_print("\nInterrupted by user")
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,465 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Example Compiler
|
||||
|
||||
Streamlined compiler that uses the PioCompiler to build FastLED examples for various boards.
|
||||
This replaces the previous complex compilation system with a simpler approach using the Pio compiler.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import Future, as_completed
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import List, Optional, cast
|
||||
|
||||
from typeguard import typechecked
|
||||
|
||||
from ci.boards import ALL, Board, create_board
|
||||
from ci.compiler.compiler import CacheType, SketchResult
|
||||
from ci.compiler.pio import PioCompiler
|
||||
|
||||
|
||||
def green_text(text: str) -> str:
|
||||
"""Return text in green color."""
|
||||
return f"\033[32m{text}\033[0m"
|
||||
|
||||
|
||||
def red_text(text: str) -> str:
|
||||
"""Return text in red color."""
|
||||
return f"\033[31m{text}\033[0m"
|
||||
|
||||
|
||||
def get_default_boards() -> List[str]:
|
||||
"""Get all board names from the ALL boards list, with preferred boards first."""
|
||||
# These are the boards we want to compile first (preferred order)
|
||||
# Order matters: UNO first because it's used for global init and builds faster
|
||||
preferred_board_names = [
|
||||
"uno", # Build is faster if this is first, because it's used for global init.
|
||||
"esp32dev",
|
||||
"esp01", # ESP8266
|
||||
"esp32c3",
|
||||
"esp32c6",
|
||||
"esp32s3",
|
||||
"teensylc",
|
||||
"teensy31",
|
||||
"teensy41",
|
||||
"digix",
|
||||
"rpipico",
|
||||
"rpipico2",
|
||||
]
|
||||
|
||||
# Get all available board names from the ALL list
|
||||
available_board_names = {board.board_name for board in ALL}
|
||||
|
||||
# Start with preferred boards that exist, warn about missing ones
|
||||
default_boards: List[str] = []
|
||||
for board_name in preferred_board_names:
|
||||
if board_name in available_board_names:
|
||||
default_boards.append(board_name)
|
||||
else:
|
||||
print(
|
||||
f"WARNING: Preferred board '{board_name}' not found in available boards"
|
||||
)
|
||||
|
||||
# Add all remaining boards (sorted for consistency)
|
||||
remaining_boards = sorted(available_board_names - set(default_boards))
|
||||
default_boards.extend(remaining_boards)
|
||||
|
||||
return default_boards
|
||||
|
||||
|
||||
def get_all_examples() -> List[str]:
|
||||
"""Get all available example names from the examples directory."""
|
||||
project_root = Path(__file__).parent.parent.resolve()
|
||||
examples_dir = project_root / "examples"
|
||||
|
||||
if not examples_dir.exists():
|
||||
return []
|
||||
|
||||
examples: List[str] = []
|
||||
for item in examples_dir.iterdir():
|
||||
if item.is_dir():
|
||||
# Check if it contains a .ino file with the same name
|
||||
ino_file = item / f"{item.name}.ino"
|
||||
if ino_file.exists():
|
||||
examples.append(item.name)
|
||||
|
||||
# Sort for consistent ordering
|
||||
examples.sort()
|
||||
return examples
|
||||
|
||||
|
||||
def parse_args(args: Optional[list[str]] = None) -> argparse.Namespace:
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compile FastLED examples for various boards using PioCompiler"
|
||||
)
|
||||
parser.add_argument(
|
||||
"boards",
|
||||
type=str,
|
||||
help="Comma-separated list of boards to compile for",
|
||||
nargs="?",
|
||||
)
|
||||
parser.add_argument(
|
||||
"positional_examples",
|
||||
type=str,
|
||||
help="Examples to compile (positional arguments after board name)",
|
||||
nargs="*",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--examples", type=str, help="Comma-separated list of examples to compile"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--exclude-examples", type=str, help="Examples that should be excluded"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--defines", type=str, help="Comma-separated list of compiler definitions"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v", "--verbose", action="store_true", help="Enable verbose output"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-cache",
|
||||
action="store_true",
|
||||
help="Disable sccache for faster compilation (default is already disabled)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable-cache",
|
||||
action="store_true",
|
||||
help="Enable sccache for faster compilation",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--cache",
|
||||
action="store_true",
|
||||
help="(Deprecated) Enable sccache for faster compilation - use --enable-cache instead",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--supported-boards",
|
||||
action="store_true",
|
||||
help="Print the list of supported boards and exit",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-interactive",
|
||||
action="store_true",
|
||||
help="Disables interactive mode (deprecated)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--log-failures",
|
||||
type=str,
|
||||
help="Directory to write per-example failure logs (created on first failure)",
|
||||
)
|
||||
|
||||
try:
|
||||
parsed_args = parser.parse_intermixed_args(args)
|
||||
unknown = []
|
||||
except SystemExit:
|
||||
# If parse_intermixed_args fails, fall back to parse_known_args
|
||||
parsed_args, unknown = parser.parse_known_args(args)
|
||||
|
||||
# Handle unknown arguments intelligently - treat non-flag arguments as examples
|
||||
unknown_examples: List[str] = []
|
||||
for arg in unknown:
|
||||
if not arg.startswith("-"):
|
||||
unknown_examples.append(arg)
|
||||
|
||||
# Add unknown examples to positional_examples
|
||||
if unknown_examples:
|
||||
if (
|
||||
not hasattr(parsed_args, "positional_examples")
|
||||
or parsed_args.positional_examples is None
|
||||
):
|
||||
parsed_args.positional_examples = []
|
||||
# Type assertion to help the type checker - argparse returns Any but we know it's List[str]
|
||||
positional_examples: List[str] = cast(
|
||||
List[str], getattr(parsed_args, "positional_examples", [])
|
||||
)
|
||||
positional_examples.extend(unknown_examples)
|
||||
parsed_args.positional_examples = positional_examples
|
||||
|
||||
return parsed_args
|
||||
|
||||
|
||||
def resolve_example_path(example: str) -> str:
|
||||
"""Resolve example name to path, ensuring it exists."""
|
||||
project_root = Path(__file__).parent.parent.resolve()
|
||||
examples_dir = project_root / "examples"
|
||||
|
||||
# Handle both "Blink" and "examples/Blink" formats
|
||||
if example.startswith("examples/"):
|
||||
example = example[len("examples/") :]
|
||||
|
||||
example_path = examples_dir / example
|
||||
if not example_path.exists():
|
||||
raise FileNotFoundError(f"Example not found: {example_path}")
|
||||
|
||||
return example
|
||||
|
||||
|
||||
@typechecked
|
||||
@dataclass
|
||||
class BoardCompilationResult:
|
||||
"""Aggregated result for compiling a set of examples on a single board."""
|
||||
|
||||
ok: bool
|
||||
sketch_results: List[SketchResult]
|
||||
|
||||
|
||||
def compile_board_examples(
|
||||
board: Board,
|
||||
examples: List[str],
|
||||
defines: List[str],
|
||||
verbose: bool,
|
||||
enable_cache: bool,
|
||||
) -> BoardCompilationResult:
|
||||
"""Compile examples for a single board using PioCompiler."""
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"COMPILING BOARD: {board.board_name}")
|
||||
print(f"EXAMPLES: {', '.join(examples)}")
|
||||
print(f"{'=' * 60}")
|
||||
|
||||
try:
|
||||
# Determine cache type based on flag and board frameworks
|
||||
frameworks = [f.strip().lower() for f in (board.framework or "").split(",")]
|
||||
mixed_frameworks = "arduino" in frameworks and "espidf" in frameworks
|
||||
cache_type = (
|
||||
CacheType.SCCACHE
|
||||
if enable_cache and not mixed_frameworks
|
||||
else CacheType.NO_CACHE
|
||||
)
|
||||
|
||||
# Create PioCompiler instance
|
||||
compiler = PioCompiler(
|
||||
board=board,
|
||||
verbose=verbose,
|
||||
additional_defines=defines,
|
||||
cache_type=cache_type,
|
||||
)
|
||||
|
||||
# Build all examples
|
||||
futures: List[Future[SketchResult]] = compiler.build(examples)
|
||||
|
||||
# Wait for completion and collect results
|
||||
results: List[SketchResult] = []
|
||||
|
||||
for future in futures:
|
||||
try:
|
||||
result = future.result()
|
||||
results.append(result)
|
||||
# SUCCESS/FAILED messages are printed by worker threads
|
||||
except KeyboardInterrupt:
|
||||
print("Keyboard interrupt detected, cancelling builds")
|
||||
compiler.cancel_all()
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
# Represent unexpected exception as a failed SketchResult for consistency
|
||||
from pathlib import Path as _Path
|
||||
|
||||
results.append(
|
||||
SketchResult(
|
||||
success=False,
|
||||
output=f"Build exception: {str(e)}",
|
||||
build_dir=_Path("."),
|
||||
example="<exception>",
|
||||
)
|
||||
)
|
||||
print(f"EXCEPTION during build: {e}")
|
||||
# Cleanup
|
||||
compiler.cancel_all()
|
||||
|
||||
# Show compiler statistics after all builds complete
|
||||
try:
|
||||
stats = compiler.get_cache_stats()
|
||||
if stats:
|
||||
print("\n" + "=" * 60)
|
||||
print("COMPILER STATISTICS")
|
||||
print("=" * 60)
|
||||
print(stats)
|
||||
print("=" * 60)
|
||||
except Exception as e:
|
||||
print(f"Warning: Could not retrieve compiler statistics: {e}")
|
||||
|
||||
any_failures = False
|
||||
for r in results:
|
||||
if not r.success:
|
||||
any_failures = True
|
||||
break
|
||||
return BoardCompilationResult(ok=not any_failures, sketch_results=results)
|
||||
except KeyboardInterrupt:
|
||||
print("Keyboard interrupt detected, cancelling builds")
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
# Compiler could not be set up; return a single failed result to carry message
|
||||
from pathlib import Path as _Path
|
||||
|
||||
return BoardCompilationResult(
|
||||
ok=False,
|
||||
sketch_results=[
|
||||
SketchResult(
|
||||
success=False,
|
||||
output=f"Compiler setup failed: {str(e)}",
|
||||
build_dir=_Path("."),
|
||||
example="<setup>",
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main function."""
|
||||
args = parse_args()
|
||||
if args.verbose:
|
||||
os.environ["VERBOSE"] = "1"
|
||||
|
||||
if args.supported_boards:
|
||||
print(",".join(get_default_boards()))
|
||||
return 0
|
||||
|
||||
# Determine which boards to compile for
|
||||
if args.boards:
|
||||
boards_names = args.boards.split(",")
|
||||
else:
|
||||
boards_names = get_default_boards()
|
||||
|
||||
# Get board objects
|
||||
boards: List[Board] = []
|
||||
for board_name in boards_names:
|
||||
try:
|
||||
board = create_board(board_name, no_project_options=False)
|
||||
boards.append(board)
|
||||
except Exception as e:
|
||||
print(f"ERROR: Failed to get board '{board_name}': {e}")
|
||||
return 1
|
||||
|
||||
# Determine which examples to compile
|
||||
if args.positional_examples:
|
||||
# Convert positional examples, handling both "examples/Blink" and "Blink" formats
|
||||
examples: List[str] = []
|
||||
for example in args.positional_examples:
|
||||
# Remove "examples/" prefix if present
|
||||
if example.startswith("examples/"):
|
||||
example = example[len("examples/") :]
|
||||
examples.append(example)
|
||||
elif args.examples:
|
||||
examples = args.examples.split(",")
|
||||
else:
|
||||
# Compile all available examples since builds are fast now!
|
||||
examples = get_all_examples()
|
||||
|
||||
# Process example exclusions
|
||||
if args.exclude_examples:
|
||||
exclude_examples = args.exclude_examples.split(",")
|
||||
examples = [ex for ex in examples if ex not in exclude_examples]
|
||||
|
||||
# Validate examples exist
|
||||
for example in examples:
|
||||
try:
|
||||
resolve_example_path(example)
|
||||
except FileNotFoundError as e:
|
||||
print(f"ERROR: {e}")
|
||||
return 1
|
||||
|
||||
# Set up defines
|
||||
defines: List[str] = []
|
||||
if args.defines:
|
||||
defines.extend(args.defines.split(","))
|
||||
|
||||
# Start compilation
|
||||
start_time = time.time()
|
||||
print(
|
||||
f"Starting compilation for {len(boards)} boards with {len(examples)} examples"
|
||||
)
|
||||
|
||||
compilation_errors: List[str] = []
|
||||
failed_example_names: List[str] = []
|
||||
failure_logs_dir: Optional[Path] = (
|
||||
Path(args.log_failures) if getattr(args, "log_failures", None) else None
|
||||
)
|
||||
|
||||
# Compile for each board
|
||||
for board in boards:
|
||||
result = compile_board_examples(
|
||||
board=board,
|
||||
examples=examples,
|
||||
defines=defines,
|
||||
verbose=args.verbose,
|
||||
enable_cache=(args.enable_cache or args.cache) and not args.no_cache,
|
||||
)
|
||||
|
||||
if not result.ok:
|
||||
# Record board-level error
|
||||
compilation_errors.append(f"Board {board.board_name} failed")
|
||||
print(red_text(f"ERROR: Compilation failed for board {board.board_name}"))
|
||||
# Print each failing sketch's stdout and collect names for summary
|
||||
for sketch in result.sketch_results:
|
||||
if not sketch.success:
|
||||
if sketch.example and sketch.example not in failed_example_names:
|
||||
failed_example_names.append(sketch.example)
|
||||
# Write per-example failure logs when requested
|
||||
if failure_logs_dir is not None:
|
||||
try:
|
||||
failure_logs_dir.mkdir(parents=True, exist_ok=True)
|
||||
safe_name = f"{sketch.example}.log"
|
||||
log_path = failure_logs_dir / safe_name
|
||||
with log_path.open(
|
||||
"a", encoding="utf-8", errors="ignore"
|
||||
) as f:
|
||||
f.write(sketch.output)
|
||||
f.write("\n")
|
||||
except KeyboardInterrupt:
|
||||
print("Keyboard interrupt detected, cancelling builds")
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
print(
|
||||
f"Warning: Could not write failure log for {sketch.example}: {e}"
|
||||
)
|
||||
print(f"\n{'-' * 60}")
|
||||
print(f"Sketch: {sketch.example}")
|
||||
print(f"{'-' * 60}")
|
||||
# Print the collected output for this sketch
|
||||
print(sketch.output)
|
||||
# Continue with other boards instead of stopping
|
||||
|
||||
# Report results
|
||||
elapsed_time = time.time() - start_time
|
||||
time_str = time.strftime("%Mm:%Ss", time.gmtime(elapsed_time))
|
||||
|
||||
if compilation_errors:
|
||||
print(
|
||||
f"\nCompilation finished in {time_str} with {len(compilation_errors)} error(s):"
|
||||
)
|
||||
for error in compilation_errors:
|
||||
print(f" - {error}")
|
||||
if failed_example_names:
|
||||
print("")
|
||||
print(red_text(f"ERROR! There were {len(failed_example_names)} failures:"))
|
||||
# Sort for stable output
|
||||
for name in sorted(failed_example_names):
|
||||
# print(f" {name}")
|
||||
# same but in red
|
||||
print(red_text(f" {name}"))
|
||||
print("")
|
||||
return 1
|
||||
else:
|
||||
print(f"\nAll compilations completed successfully in {time_str}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
print("\nInterrupted by user")
|
||||
sys.exit(1)
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,134 @@
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
MINIMUM_REPORT_SEVERTIY = "medium"
|
||||
MINIMUM_FAIL_SEVERTIY = "high"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Run cppcheck on the project")
|
||||
parser.add_argument("board", nargs="?", help="Board to check, optional")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def find_platformio_project_dir(build_root: Path, board_name: str) -> Path | None:
|
||||
"""Find a directory containing platformio.ini file for the specified board.
|
||||
|
||||
Handles both old and new build directory structures:
|
||||
Old: .build/uno/Blink/platformio.ini
|
||||
New: .build/fled/examples/uno/Blink/platformio.ini
|
||||
|
||||
Returns the directory containing platformio.ini, or None if not found.
|
||||
"""
|
||||
if not build_root.exists():
|
||||
return None
|
||||
|
||||
# Search patterns for different build structures
|
||||
search_patterns = [
|
||||
# New structure: .build/fled/examples/{board}/**/*.ini
|
||||
build_root / "fled" / "examples" / board_name,
|
||||
# Old structure: .build/{board}/**/*.ini
|
||||
build_root / board_name,
|
||||
# Fallback: search all subdirectories for board name
|
||||
]
|
||||
|
||||
# Try each search pattern
|
||||
for base_dir in search_patterns:
|
||||
if not base_dir.exists():
|
||||
continue
|
||||
|
||||
# Look for platformio.ini in this directory and its subdirectories
|
||||
project_dir = find_platformio_ini_in_tree(base_dir)
|
||||
if project_dir:
|
||||
return project_dir
|
||||
|
||||
# Last resort: search the entire build directory for board-related platformio.ini
|
||||
for platformio_ini in build_root.rglob("platformio.ini"):
|
||||
# Check if this platformio.ini is for our board by looking at the path
|
||||
if board_name in str(platformio_ini):
|
||||
print(f"Found platformio.ini in {platformio_ini.parent}")
|
||||
return platformio_ini.parent
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def find_platformio_ini_in_tree(base_dir: Path) -> Path | None:
|
||||
"""Recursively search for platformio.ini in a directory tree."""
|
||||
if not base_dir.exists():
|
||||
return None
|
||||
|
||||
# Check direct subdirectories first (most common case)
|
||||
for subdir in base_dir.iterdir():
|
||||
if subdir.is_dir():
|
||||
platformio_ini = subdir / "platformio.ini"
|
||||
if platformio_ini.exists():
|
||||
print(f"Found platformio.ini in {subdir}")
|
||||
return subdir
|
||||
|
||||
# Check if there's a platformio.ini directly in the base directory
|
||||
if (base_dir / "platformio.ini").exists():
|
||||
print(f"Found platformio.ini directly in {base_dir}")
|
||||
return base_dir
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
here = Path(__file__).parent
|
||||
project_root = here.parent
|
||||
build = project_root / ".build"
|
||||
|
||||
if not build.exists():
|
||||
print(f"Build directory {build} not found")
|
||||
return 1
|
||||
|
||||
if args.board:
|
||||
# Search for the specified board
|
||||
project_dir = find_platformio_project_dir(build, args.board)
|
||||
if not project_dir:
|
||||
print(f"No platformio.ini found for board '{args.board}'")
|
||||
print("This usually means the board hasn't been compiled yet.")
|
||||
print(f"Try running: uv run ci/ci-compile.py {args.board} --examples Blink")
|
||||
return 1
|
||||
else:
|
||||
# Auto-detect available board by searching for any platformio.ini
|
||||
project_dir = None
|
||||
|
||||
# Try to find any platformio.ini file in the build directory
|
||||
for platformio_ini in build.rglob("platformio.ini"):
|
||||
project_dir = platformio_ini.parent
|
||||
print(f"Auto-detected project directory: {project_dir}")
|
||||
break
|
||||
|
||||
if not project_dir:
|
||||
print(f"No platformio.ini files found in {build}")
|
||||
print("This usually means no boards have been compiled yet.")
|
||||
print("Try running: uv run ci/ci-compile.py uno --examples Blink")
|
||||
return 1
|
||||
|
||||
print(f"Running pio check in {project_dir}")
|
||||
os.chdir(str(project_dir))
|
||||
|
||||
# Run pio check command
|
||||
cp = subprocess.run(
|
||||
[
|
||||
"pio",
|
||||
"check",
|
||||
"--skip-packages",
|
||||
"--src-filters=+<src/>",
|
||||
f"--severity={MINIMUM_REPORT_SEVERTIY}",
|
||||
f"--fail-on-defect={MINIMUM_FAIL_SEVERTIY}",
|
||||
"--flags",
|
||||
"--inline-suppr",
|
||||
],
|
||||
)
|
||||
return cp.returncode
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,6 @@
|
||||
# flake8: noqa
|
||||
# ruff: skip
|
||||
|
||||
Import("env") # type: ignore
|
||||
|
||||
env.Append(CXXFLAGS=["-Wno-register"]) # type: ignore
|
||||
@@ -0,0 +1,271 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run include-what-you-use on the project"
|
||||
)
|
||||
parser.add_argument("board", nargs="?", help="Board to check, optional")
|
||||
parser.add_argument(
|
||||
"--fix",
|
||||
action="store_true",
|
||||
help="Automatically apply suggested fixes using fix_includes",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mapping-file",
|
||||
action="append",
|
||||
help="Additional mapping file to use (can be specified multiple times)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-line-length",
|
||||
type=int,
|
||||
default=100,
|
||||
help="Maximum line length for suggestions (default: 100)",
|
||||
)
|
||||
parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def find_platformio_project_dir(board_dir: Path) -> Path | None:
|
||||
"""Find a directory containing platformio.ini file in the board's build directory.
|
||||
|
||||
With the new pio ci build system, the structure is:
|
||||
.build/uno/Blink/platformio.ini
|
||||
.build/uno/SomeExample/platformio.ini
|
||||
|
||||
We need to find one of these example directories that has a platformio.ini file.
|
||||
"""
|
||||
if not board_dir.exists():
|
||||
return None
|
||||
|
||||
# Look for subdirectories containing platformio.ini
|
||||
for subdir in board_dir.iterdir():
|
||||
if subdir.is_dir():
|
||||
platformio_ini = subdir / "platformio.ini"
|
||||
if platformio_ini.exists():
|
||||
print(f"Found platformio.ini in {subdir}")
|
||||
return subdir
|
||||
|
||||
# Fallback: check if there's a platformio.ini directly in the board directory (old system)
|
||||
if (board_dir / "platformio.ini").exists():
|
||||
print(f"Found platformio.ini directly in {board_dir}")
|
||||
return board_dir
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def check_iwyu_available() -> bool:
|
||||
"""Check if include-what-you-use is available in the system"""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["include-what-you-use", "--version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (
|
||||
subprocess.CalledProcessError,
|
||||
FileNotFoundError,
|
||||
subprocess.TimeoutExpired,
|
||||
):
|
||||
return False
|
||||
|
||||
|
||||
def run_iwyu_on_cpp_tests(args: argparse.Namespace) -> int:
|
||||
"""Run IWYU on the C++ test suite using CMake integration"""
|
||||
here = Path(__file__).parent
|
||||
project_root = here.parent
|
||||
|
||||
print("Running include-what-you-use on C++ test suite...")
|
||||
|
||||
# Use the existing test infrastructure with --check flag
|
||||
cmd = [
|
||||
"uv",
|
||||
"run",
|
||||
"test.py",
|
||||
"--cpp",
|
||||
"--check", # This enables IWYU via CMake
|
||||
"--clang",
|
||||
"--no-interactive",
|
||||
]
|
||||
|
||||
if args.verbose:
|
||||
cmd.append("--verbose")
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd, cwd=project_root)
|
||||
return result.returncode
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"IWYU analysis failed with return code {e.returncode}")
|
||||
return e.returncode
|
||||
|
||||
|
||||
def run_iwyu_on_platformio_project(project_dir: Path, args: argparse.Namespace) -> int:
|
||||
"""Run IWYU on a PlatformIO project"""
|
||||
print(f"Running include-what-you-use in {project_dir}")
|
||||
os.chdir(str(project_dir))
|
||||
|
||||
# Build mapping file arguments
|
||||
mapping_args: List[str] = []
|
||||
|
||||
# Add FastLED mapping files if they exist
|
||||
project_root = project_dir
|
||||
while project_root.parent != project_root: # Find project root
|
||||
if (project_root / "ci" / "iwyu").exists():
|
||||
break
|
||||
project_root = project_root.parent
|
||||
|
||||
fastled_mapping = project_root / "ci" / "iwyu" / "fastled.imp"
|
||||
stdlib_mapping = project_root / "ci" / "iwyu" / "stdlib.imp"
|
||||
|
||||
if fastled_mapping.exists():
|
||||
mapping_args.extend(["--mapping_file", str(fastled_mapping)])
|
||||
|
||||
if stdlib_mapping.exists():
|
||||
mapping_args.extend(["--mapping_file", str(stdlib_mapping)])
|
||||
|
||||
# Add user-specified mapping files
|
||||
if args.mapping_file:
|
||||
for mapping in args.mapping_file:
|
||||
mapping_args.extend(["--mapping_file", mapping])
|
||||
|
||||
# Build IWYU command
|
||||
iwyu_cmd = [
|
||||
"include-what-you-use",
|
||||
f"--max_line_length={args.max_line_length}",
|
||||
"--quoted_includes_first",
|
||||
"--no_comments",
|
||||
]
|
||||
|
||||
if args.verbose:
|
||||
iwyu_cmd.append("--verbose=3")
|
||||
else:
|
||||
iwyu_cmd.append("--verbose=1")
|
||||
|
||||
iwyu_cmd.extend(mapping_args)
|
||||
|
||||
# Run through PlatformIO's check system with IWYU
|
||||
pio_cmd = [
|
||||
"pio",
|
||||
"check",
|
||||
"--skip-packages",
|
||||
"--src-filters=+<src/>",
|
||||
"--tool=include-what-you-use",
|
||||
"--flags",
|
||||
] + iwyu_cmd[1:] # Skip the include-what-you-use binary name
|
||||
|
||||
try:
|
||||
result = subprocess.run(pio_cmd)
|
||||
return result.returncode
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"PlatformIO IWYU check failed with return code {e.returncode}")
|
||||
return e.returncode
|
||||
|
||||
|
||||
def apply_iwyu_fixes(source_dir: Path) -> int:
|
||||
"""Apply IWYU fixes using fix_includes tool"""
|
||||
try:
|
||||
# Check if fix_includes is available
|
||||
subprocess.run(["fix_includes", "--help"], capture_output=True, check=True)
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
print(
|
||||
"Warning: fix_includes tool not found. Install IWYU tools to use --fix option."
|
||||
)
|
||||
return 1
|
||||
|
||||
# Find all .h and .cpp files
|
||||
cpp_files: List[Path] = []
|
||||
for pattern in ["**/*.cpp", "**/*.h", "**/*.hpp"]:
|
||||
cpp_files.extend(source_dir.glob(pattern))
|
||||
|
||||
if not cpp_files:
|
||||
print("No C++ files found to fix")
|
||||
return 0
|
||||
|
||||
print(f"Applying IWYU fixes to {len(cpp_files)} files...")
|
||||
|
||||
# Run fix_includes on the source directory
|
||||
cmd = ["fix_includes", "--update_comments", str(source_dir)]
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd)
|
||||
if result.returncode == 0:
|
||||
print("IWYU fixes applied successfully")
|
||||
else:
|
||||
print(f"fix_includes failed with return code {result.returncode}")
|
||||
return result.returncode
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"Failed to apply IWYU fixes: {e}")
|
||||
return e.returncode
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
here = Path(__file__).parent
|
||||
project_root = here.parent
|
||||
|
||||
# Check if IWYU is available
|
||||
if not check_iwyu_available():
|
||||
print("Error: include-what-you-use not found in PATH")
|
||||
print("Install it with:")
|
||||
print(" Ubuntu/Debian: sudo apt install iwyu")
|
||||
print(" macOS: brew install include-what-you-use")
|
||||
print(" Or build from source: https://include-what-you-use.org/")
|
||||
return 1
|
||||
|
||||
print("Found include-what-you-use")
|
||||
|
||||
# If no board specified, run on C++ test suite
|
||||
if not args.board:
|
||||
print("No board specified, running IWYU on C++ test suite")
|
||||
return run_iwyu_on_cpp_tests(args)
|
||||
|
||||
# Run on specific board
|
||||
build = project_root / ".build"
|
||||
|
||||
if not build.exists():
|
||||
print(f"Build directory {build} not found")
|
||||
print("Run a compilation first: ./compile [board] --examples [example]")
|
||||
return 1
|
||||
|
||||
board_dir = build / args.board
|
||||
if not board_dir.exists():
|
||||
print(f"Board {args.board} not found in {build}")
|
||||
print("Available boards:")
|
||||
for d in build.iterdir():
|
||||
if d.is_dir():
|
||||
print(f" {d.name}")
|
||||
return 1
|
||||
|
||||
project_dir = find_platformio_project_dir(board_dir)
|
||||
if not project_dir:
|
||||
print(f"No platformio.ini found in {board_dir} or its subdirectories")
|
||||
print("This usually means the board hasn't been compiled yet.")
|
||||
print(f"Try running: ./compile {args.board} --examples Blink")
|
||||
return 1
|
||||
|
||||
# Run IWYU on the PlatformIO project
|
||||
result = run_iwyu_on_platformio_project(project_dir, args)
|
||||
|
||||
# Apply fixes if requested and analysis succeeded
|
||||
if args.fix and result == 0:
|
||||
src_dir = project_dir / "src"
|
||||
if src_dir.exists():
|
||||
fix_result = apply_iwyu_fixes(src_dir)
|
||||
if fix_result != 0:
|
||||
result = fix_result
|
||||
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,238 @@
|
||||
"""
|
||||
Fingerprint Cache Feature
|
||||
|
||||
A two-layer file change detection system that efficiently determines if source files
|
||||
have been modified by combining fast modification time checks with slower but accurate
|
||||
MD5 hash verification.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class CacheEntry:
|
||||
"""Cache entry storing file modification time and content hash."""
|
||||
|
||||
modification_time: float
|
||||
md5_hash: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class FingerprintCacheConfig:
|
||||
"""Configuration for fingerprint cache behavior."""
|
||||
|
||||
cache_file: Path
|
||||
hash_algorithm: str
|
||||
ignore_patterns: list[str]
|
||||
max_cache_size: int
|
||||
|
||||
|
||||
class FingerprintCache:
|
||||
"""
|
||||
Two-layer file change detection using modification time and content hashing.
|
||||
|
||||
Provides efficient change detection by:
|
||||
1. Fast modification time comparison (microsecond performance)
|
||||
2. Accurate content verification via MD5 hashing when needed
|
||||
"""
|
||||
|
||||
def __init__(self, cache_file: Path, modtime_only: bool = False):
|
||||
"""
|
||||
Initialize fingerprint cache.
|
||||
|
||||
Args:
|
||||
cache_file: Path to JSON cache file
|
||||
modtime_only: When True, disable hashing and rely solely on modtime.
|
||||
"""
|
||||
self.cache_file = cache_file
|
||||
self._modtime_only: bool = modtime_only
|
||||
self.cache = self._load_cache()
|
||||
|
||||
def _load_cache(self) -> Dict[str, CacheEntry]:
|
||||
"""
|
||||
Load cache from JSON file, return empty dict if file doesn't exist.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping file paths to cache entries
|
||||
"""
|
||||
if not self.cache_file.exists():
|
||||
return {}
|
||||
|
||||
try:
|
||||
with open(self.cache_file, "r") as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Convert JSON dict to CacheEntry objects
|
||||
cache: Dict[str, CacheEntry] = {}
|
||||
for file_path, entry_data in data.items():
|
||||
cache[file_path] = CacheEntry(
|
||||
modification_time=entry_data["modification_time"],
|
||||
md5_hash=entry_data["md5_hash"],
|
||||
)
|
||||
return cache
|
||||
except (json.JSONDecodeError, KeyError, TypeError):
|
||||
# Cache corrupted - start fresh
|
||||
return {}
|
||||
|
||||
def _save_cache(self) -> None:
|
||||
"""Save current cache state to JSON file."""
|
||||
# Ensure cache directory exists
|
||||
self.cache_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Convert CacheEntry objects to JSON-serializable dict
|
||||
data = {}
|
||||
for file_path, entry in self.cache.items():
|
||||
data[file_path] = {
|
||||
"modification_time": entry.modification_time,
|
||||
"md5_hash": entry.md5_hash,
|
||||
}
|
||||
|
||||
with open(self.cache_file, "w") as f:
|
||||
json.dump(data, f, indent=2)
|
||||
|
||||
def _compute_md5(self, file_path: Path) -> str:
|
||||
"""
|
||||
Compute MD5 hash of file content.
|
||||
|
||||
Args:
|
||||
file_path: Path to file to hash
|
||||
|
||||
Returns:
|
||||
MD5 hash as hexadecimal string
|
||||
|
||||
Raises:
|
||||
IOError: If file cannot be read
|
||||
"""
|
||||
try:
|
||||
hasher = hashlib.md5()
|
||||
with open(file_path, "rb") as f:
|
||||
# Read in chunks to handle large files efficiently
|
||||
for chunk in iter(lambda: f.read(8192), b""):
|
||||
hasher.update(chunk)
|
||||
return hasher.hexdigest()
|
||||
except IOError as e:
|
||||
raise IOError(f"Cannot read file {file_path}: {e}")
|
||||
|
||||
def has_changed(self, src_path: Path, previous_modtime: float) -> bool:
|
||||
"""
|
||||
Determine if a source file has changed since the last known modification time.
|
||||
|
||||
Two-layer verification process:
|
||||
1. Fast modification time check - if times match, return False immediately
|
||||
2. Content verification - compute/use cached MD5 hash for accurate comparison
|
||||
|
||||
Args:
|
||||
src_path: Path to the source file to check
|
||||
previous_modtime: Previously known modification time (Unix timestamp)
|
||||
|
||||
Returns:
|
||||
True if the file has changed, False if unchanged
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If source file doesn't exist
|
||||
"""
|
||||
if not src_path.exists():
|
||||
raise FileNotFoundError(f"Source file not found: {src_path}")
|
||||
|
||||
current_modtime = os.path.getmtime(src_path)
|
||||
|
||||
# Optional mode: strictly use modification time only (no hashing)
|
||||
# This is required for toolchains that invalidate PCH on any newer
|
||||
# dependency regardless of content changes (e.g., Clang).
|
||||
if self._modtime_only:
|
||||
# Treat as changed only when file is newer than the reference time
|
||||
# (keeps behavior stable when reference is an external artifact's mtime)
|
||||
return current_modtime > previous_modtime
|
||||
|
||||
# Layer 1: Quick modification time check
|
||||
if current_modtime == previous_modtime:
|
||||
return False # No change detected
|
||||
|
||||
file_key = str(src_path.resolve()) # Use absolute path as key
|
||||
|
||||
# Layer 2: Content verification via hash comparison
|
||||
if file_key in self.cache:
|
||||
cached_entry = self.cache[file_key]
|
||||
previous_hash = (
|
||||
cached_entry.md5_hash
|
||||
) # Store previous hash before potential update
|
||||
|
||||
if current_modtime == cached_entry.modification_time:
|
||||
# File is cached with current modtime - use cached hash
|
||||
current_hash = cached_entry.md5_hash
|
||||
else:
|
||||
# Cache is stale - compute new hash
|
||||
current_hash = self._compute_md5(src_path)
|
||||
self._update_cache_entry(src_path, current_modtime, current_hash)
|
||||
|
||||
# Compare current hash with previous cached hash
|
||||
# If they match, content hasn't actually changed despite modtime difference
|
||||
return current_hash != previous_hash
|
||||
|
||||
else:
|
||||
# File not in cache - compute hash and cache it
|
||||
current_hash = self._compute_md5(src_path)
|
||||
self._update_cache_entry(src_path, current_modtime, current_hash)
|
||||
return True # Assume changed since we have no previous state to compare
|
||||
|
||||
def _update_cache_entry(
|
||||
self, file_path: Path, modification_time: float, md5_hash: str
|
||||
) -> None:
|
||||
"""
|
||||
Update cache with new entry and save to disk.
|
||||
|
||||
Args:
|
||||
file_path: Path to file being cached
|
||||
modification_time: File modification time
|
||||
md5_hash: File content hash
|
||||
"""
|
||||
file_key = str(file_path.resolve())
|
||||
self.cache[file_key] = CacheEntry(
|
||||
modification_time=modification_time, md5_hash=md5_hash
|
||||
)
|
||||
self._save_cache()
|
||||
|
||||
def get_cache_stats(self) -> dict[str, int]:
|
||||
"""
|
||||
Get cache statistics.
|
||||
|
||||
Returns:
|
||||
Dictionary with cache size and other metrics
|
||||
"""
|
||||
return {
|
||||
"total_entries": len(self.cache),
|
||||
"cache_file_exists": self.cache_file.exists(),
|
||||
"cache_file_size_bytes": self.cache_file.stat().st_size
|
||||
if self.cache_file.exists()
|
||||
else 0,
|
||||
}
|
||||
|
||||
def clear_cache(self) -> None:
|
||||
"""Clear all cache entries and remove cache file."""
|
||||
self.cache.clear()
|
||||
if self.cache_file.exists():
|
||||
self.cache_file.unlink()
|
||||
|
||||
|
||||
def has_changed(src_path: Path, previous_modtime: float, cache_file: Path) -> bool:
|
||||
"""
|
||||
Convenience function implementing the main API.
|
||||
|
||||
Args:
|
||||
src_path: Path to the source file to check
|
||||
previous_modtime: Previously known modification time (Unix timestamp)
|
||||
cache_file: Path to the fingerprint cache file
|
||||
|
||||
Returns:
|
||||
True if the file has changed, False if unchanged
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If source file doesn't exist
|
||||
"""
|
||||
cache = FingerprintCache(cache_file)
|
||||
return cache.has_changed(src_path, previous_modtime)
|
||||
@@ -0,0 +1,290 @@
|
||||
#!/usr/bin/env python
|
||||
"""
|
||||
Compile Commands Generator
|
||||
|
||||
Generates a compile_commands.json for clangd/clang-tidy based on explicit
|
||||
tools/flags configuration. See FEATURE.md for usage and rationale.
|
||||
|
||||
Key behaviors:
|
||||
- Reads configuration from ci/build_commands.toml
|
||||
- Enumerates library source files under src/ (respecting optional filters.exclude_dirs)
|
||||
- Emits one entry per translation unit with accurate arguments/command fields
|
||||
|
||||
Notes:
|
||||
- This script intentionally avoids project-unit-test driven flags to keep
|
||||
IntelliSense consistent with the core FastLED library build.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import _thread
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
# Require Python 3.11+ environment for TOML parsing
|
||||
import tomllib
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional, cast
|
||||
|
||||
# Require typeguard; no try-import fallbacks
|
||||
from typeguard import typechecked
|
||||
|
||||
# Use the canonical BuildFlags/Compiler tooling
|
||||
from ci.compiler.clang_compiler import commands_json
|
||||
|
||||
|
||||
PROJECT_ROOT: Path = Path(__file__).resolve().parent.parent
|
||||
# Default to bespoke config for compile-commands generation
|
||||
DEFAULT_CONFIG_PATH: Path = PROJECT_ROOT / "ci" / "build_commands.toml"
|
||||
|
||||
|
||||
@typechecked
|
||||
@dataclass
|
||||
class CliConfig:
|
||||
build_flags_toml: Path
|
||||
exclude_dirs: list[str]
|
||||
|
||||
|
||||
@typechecked
|
||||
@dataclass
|
||||
class Args:
|
||||
output: Path
|
||||
clean: bool
|
||||
verbose: bool
|
||||
config: Path
|
||||
exclude: list[str]
|
||||
|
||||
|
||||
def parse_args(args: Optional[list[str]] = None) -> Args:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Generate compile_commands.json for clangd/clang-tidy based on explicit build configuration."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
dest="output",
|
||||
metavar="PATH",
|
||||
help="Output path for compile_commands.json (default: project root)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--config",
|
||||
dest="config",
|
||||
metavar="PATH",
|
||||
help=(
|
||||
"Path to BuildFlags-style TOML (default: ci/build_example.toml). "
|
||||
"Must be parseable by BuildFlags.parse()."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--clean",
|
||||
dest="clean",
|
||||
action="store_true",
|
||||
help="Rebuild command set from scratch (ignored currently; reserved)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbose",
|
||||
dest="verbose",
|
||||
action="store_true",
|
||||
help="Print collected flags/files",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--exclude-dir",
|
||||
dest="exclude",
|
||||
action="append",
|
||||
default=[],
|
||||
help="Relative directory under project root to exclude (can be used multiple times)",
|
||||
)
|
||||
|
||||
parsed = parser.parse_args(args)
|
||||
|
||||
output_path_str: Optional[str] = None
|
||||
if hasattr(parsed, "output"):
|
||||
output_path_str = parsed.output
|
||||
|
||||
if output_path_str is None or len(output_path_str) == 0:
|
||||
# Default to project root compile_commands.json
|
||||
output_path = PROJECT_ROOT / "compile_commands.json"
|
||||
else:
|
||||
output_path = Path(output_path_str).resolve()
|
||||
|
||||
clean_value: bool = False
|
||||
if hasattr(parsed, "clean"):
|
||||
clean_value = bool(parsed.clean)
|
||||
|
||||
verbose_value: bool = False
|
||||
if hasattr(parsed, "verbose"):
|
||||
verbose_value = bool(parsed.verbose)
|
||||
|
||||
# Config path
|
||||
config_path: Path
|
||||
if getattr(parsed, "config", None):
|
||||
config_path = Path(parsed.config).resolve()
|
||||
else:
|
||||
config_path = DEFAULT_CONFIG_PATH
|
||||
|
||||
# Excluded directories
|
||||
exclude_list: list[str] = list(getattr(parsed, "exclude", []) or [])
|
||||
|
||||
return Args(
|
||||
output=output_path,
|
||||
clean=clean_value,
|
||||
verbose=verbose_value,
|
||||
config=config_path,
|
||||
exclude=exclude_list,
|
||||
)
|
||||
|
||||
|
||||
def _assert_config_exists(config_path: Path) -> None:
|
||||
if not config_path.exists():
|
||||
raise RuntimeError(
|
||||
f"CRITICAL: Required BuildFlags TOML not found at {config_path}. "
|
||||
f"Please provide a valid configuration parseable by BuildFlags.parse()."
|
||||
)
|
||||
|
||||
|
||||
def _create_cli_config(build_flags_toml: Path, exclude: list[str]) -> CliConfig:
|
||||
return CliConfig(build_flags_toml=build_flags_toml, exclude_dirs=exclude)
|
||||
|
||||
|
||||
def _collect_source_files(root: Path, exclude: list[str], verbose: bool) -> list[Path]:
|
||||
source_files: list[Path] = []
|
||||
|
||||
# Compute excluded directories as absolute paths for quick prefix checks
|
||||
excluded_dirs_abs: list[Path] = []
|
||||
for ex in exclude:
|
||||
p = (PROJECT_ROOT / ex).resolve()
|
||||
excluded_dirs_abs.append(p)
|
||||
|
||||
src_root: Path = PROJECT_ROOT / "src"
|
||||
if not src_root.exists():
|
||||
raise RuntimeError(
|
||||
f"CRITICAL: Expected source root at {src_root} not found. "
|
||||
f"This path is required to enumerate translation units."
|
||||
)
|
||||
|
||||
for dirpath, dirnames, filenames in os.walk(src_root):
|
||||
current_dir = Path(dirpath).resolve()
|
||||
|
||||
# Apply directory exclusions
|
||||
skip_dir: bool = False
|
||||
for ex in excluded_dirs_abs:
|
||||
try:
|
||||
current_dir.relative_to(ex)
|
||||
skip_dir = True
|
||||
break
|
||||
except ValueError:
|
||||
# Not under this excluded path
|
||||
pass
|
||||
|
||||
if skip_dir:
|
||||
# Prune search by clearing dirnames to avoid descending further
|
||||
del dirnames[:]
|
||||
continue
|
||||
|
||||
for fname in filenames:
|
||||
if fname.endswith(".cpp") or fname.endswith(".c"):
|
||||
source_files.append(current_dir / fname)
|
||||
|
||||
if verbose:
|
||||
print(f"Discovered {len(source_files)} source files under {src_root}")
|
||||
|
||||
if len(source_files) == 0:
|
||||
raise RuntimeError(
|
||||
"CRITICAL: No source files discovered under 'src/'. "
|
||||
"Please verify the repository layout and filters.exclude_dirs configuration."
|
||||
)
|
||||
|
||||
return source_files
|
||||
|
||||
# No per-file argument composition here; delegated to commands_json()
|
||||
# This function intentionally removed to rely on canonical toolchain
|
||||
|
||||
# Emission handled by commands_json()
|
||||
|
||||
|
||||
def main(entry_args: Optional[list[str]] = None) -> int:
|
||||
args = parse_args(entry_args)
|
||||
|
||||
# Log startup context
|
||||
print("[compile-commands] Starting generation")
|
||||
print(f"[compile-commands] Project root: {PROJECT_ROOT}")
|
||||
print(f"[compile-commands] Config (BuildFlags TOML): {args.config}")
|
||||
print(f"[compile-commands] Output path: {args.output}")
|
||||
if args.exclude:
|
||||
print(f"[compile-commands] Excluding directories: {args.exclude}")
|
||||
else:
|
||||
print("[compile-commands] No excluded directories configured")
|
||||
|
||||
# Determine BuildFlags TOML path
|
||||
_assert_config_exists(args.config)
|
||||
|
||||
# Collect source files
|
||||
source_root = PROJECT_ROOT / "src"
|
||||
print(f"[compile-commands] Source root: {source_root}")
|
||||
sources = _collect_source_files(
|
||||
source_root,
|
||||
exclude=args.exclude,
|
||||
verbose=args.verbose,
|
||||
)
|
||||
print(f"[compile-commands] Discovered {len(sources)} source files")
|
||||
|
||||
# Delegate to canonical commands_json() using BuildFlags toolchain
|
||||
print("[compile-commands] Generating compile_commands.json entries...")
|
||||
commands_json(
|
||||
config_path=args.config,
|
||||
include_root=source_root,
|
||||
sources=sources,
|
||||
output_json=args.output,
|
||||
quick_build=True,
|
||||
strict_mode=False,
|
||||
)
|
||||
|
||||
# Post-generation verification
|
||||
if not args.output.exists():
|
||||
print(
|
||||
f"CRITICAL: Generation completed but output file was not found at {args.output}"
|
||||
)
|
||||
return 2
|
||||
|
||||
try:
|
||||
text = args.output.read_text(encoding="utf-8")
|
||||
# Best-effort count of entries to report status
|
||||
import json as _json
|
||||
from typing import cast
|
||||
|
||||
entries = cast(list[dict[str, Any]], _json.loads(text))
|
||||
num_entries = len(entries) if isinstance(entries, list) else 0
|
||||
print(
|
||||
f"[compile-commands] Wrote {num_entries} entries to {args.output} (size: {args.output.stat().st_size} bytes)"
|
||||
)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(
|
||||
f"WARNING: Output written to {args.output} but could not parse JSON for summary: {e}"
|
||||
)
|
||||
|
||||
if args.verbose:
|
||||
print("[compile-commands] Done.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
sys.exit(main())
|
||||
except KeyboardInterrupt:
|
||||
print("Operation interrupted by user")
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except FileNotFoundError as e:
|
||||
print(f"CRITICAL: Required file not found: {e}")
|
||||
sys.exit(1)
|
||||
except ValueError as e:
|
||||
print(f"CRITICAL: Invalid configuration value: {e}")
|
||||
sys.exit(1)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"CRITICAL: Unexpected error: {e}")
|
||||
raise
|
||||
@@ -0,0 +1,115 @@
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
def _create_board_info(path: Path) -> Dict[str, Any]:
|
||||
build_info = json.loads(path.read_text())
|
||||
assert build_info.keys(), f"No boards found in {build_info}"
|
||||
assert len(build_info.keys()) == 1, (
|
||||
f"Multiple boards found in {build_info}, so correct board should be specified"
|
||||
)
|
||||
return build_info[next(iter(build_info))]
|
||||
|
||||
|
||||
def _find_build_info(board: str) -> Path:
|
||||
# Prefer PIO layout
|
||||
candidate = Path(".build") / "pio" / board / "build_info.json"
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
# Fallback legacy layout
|
||||
candidate = Path(".build") / board / "build_info.json"
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
raise FileNotFoundError(
|
||||
f"build_info.json not found for board '{board}' in .build/pio/{board} or .build/{board}"
|
||||
)
|
||||
|
||||
|
||||
def _run_pio_size(build_dir: Path) -> int | None:
|
||||
try:
|
||||
# Try to compute size without building first
|
||||
result = subprocess.run(
|
||||
["pio", "run", "-d", str(build_dir), "-t", "size"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
output = (result.stdout or "") + "\n" + (result.stderr or "")
|
||||
m = re.search(r"Program:\s*(\d+)\s*bytes", output)
|
||||
if m:
|
||||
return int(m.group(1))
|
||||
# If size target did not yield, try a full build then retry size
|
||||
subprocess.run(
|
||||
["pio", "run", "-d", str(build_dir)], capture_output=False, check=False
|
||||
)
|
||||
result = subprocess.run(
|
||||
["pio", "run", "-d", str(build_dir), "-t", "size"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
output = (result.stdout or "") + "\n" + (result.stderr or "")
|
||||
m = re.search(r"Program:\s*(\d+)\s*bytes", output)
|
||||
if m:
|
||||
return int(m.group(1))
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def check_firmware_size(board: str) -> int:
|
||||
build_info_json = _find_build_info(board)
|
||||
board_info = _create_board_info(build_info_json)
|
||||
assert board_info, f"Board {board} not found in {build_info_json}"
|
||||
prog_path = Path(board_info["prog_path"]) # absolute or relative path
|
||||
base_path = prog_path.parent
|
||||
suffixes = [".bin", ".hex", ".uf2"]
|
||||
firmware: Path | None = None
|
||||
for suffix in suffixes:
|
||||
candidate = base_path / f"firmware{suffix}"
|
||||
if candidate.exists():
|
||||
firmware = candidate
|
||||
break
|
||||
if firmware is None:
|
||||
# As a fallback, if firmware.* doesn't exist, use ELF size (filesize) if present
|
||||
if prog_path.exists():
|
||||
return prog_path.stat().st_size
|
||||
# As a last resort, run PlatformIO size target in the build directory
|
||||
build_dir = build_info_json.parent
|
||||
size = _run_pio_size(build_dir)
|
||||
if size is not None:
|
||||
return size
|
||||
msg = (
|
||||
", ".join([f"firmware{suffix}" for suffix in suffixes])
|
||||
+ f" not found in {base_path}"
|
||||
)
|
||||
raise FileNotFoundError(msg)
|
||||
return firmware.stat().st_size
|
||||
|
||||
|
||||
def main(board: str):
|
||||
try:
|
||||
size = check_firmware_size(board)
|
||||
print(f"Firmware size for {board}: {size} bytes")
|
||||
except FileNotFoundError as e:
|
||||
print(f"Error: {e}")
|
||||
except json.JSONDecodeError:
|
||||
print(f"Error: Unable to parse build_info.json for {board}")
|
||||
except Exception as e:
|
||||
print(f"An unexpected error occurred: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check FastLED firmware size for the specified board."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--board", type=str, required=True, help="Board to check firmware size for"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
main(args.board)
|
||||
@@ -0,0 +1,237 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
import dateutil.parser # type: ignore
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def run_command(command: str):
|
||||
process = subprocess.Popen(
|
||||
command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True
|
||||
)
|
||||
output, error = process.communicate()
|
||||
return output.decode("utf-8"), error.decode("utf-8")
|
||||
|
||||
|
||||
def step_back_commits(steps: int) -> bool:
|
||||
step_back_command = f"git reset --hard HEAD~{steps}"
|
||||
output, error = run_command(step_back_command)
|
||||
if error:
|
||||
print(f"Error stepping back {steps} commit(s): {error}")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def check_firmware_size(board: str) -> int:
|
||||
root_build_dir = Path(".build") / board
|
||||
build_info_json = root_build_dir / "build_info.json"
|
||||
build_info = json.loads(build_info_json.read_text())
|
||||
board_info = build_info.get(board)
|
||||
assert board_info, f"Board {board} not found in {build_info_json}"
|
||||
prog_path = Path(board_info["prog_path"])
|
||||
base_path = prog_path.parent
|
||||
suffixes = [".bin", ".hex", ".uf2"]
|
||||
firmware: Path
|
||||
for suffix in suffixes:
|
||||
firmware = base_path / f"firmware{suffix}"
|
||||
if firmware.exists():
|
||||
break
|
||||
else:
|
||||
msg = (
|
||||
", ".join([f"firmware{suffix}" for suffix in suffixes])
|
||||
+ f" not found in {base_path}"
|
||||
)
|
||||
raise FileNotFoundError(msg)
|
||||
size_command = f"du -b {firmware}"
|
||||
output, error = run_command(size_command)
|
||||
if error:
|
||||
print(f"Error checking firmware size: {error}")
|
||||
return -1
|
||||
size_in_bytes = output.strip().split()[0]
|
||||
return int(size_in_bytes)
|
||||
|
||||
|
||||
def get_commit_hash() -> str | None:
|
||||
hash_command = "git rev-parse HEAD"
|
||||
output, error = run_command(hash_command)
|
||||
if error:
|
||||
print(f"Error getting commit hash: {error}")
|
||||
return None
|
||||
return output.strip()
|
||||
|
||||
|
||||
def get_commit_date(commit_hash: str) -> str | None:
|
||||
date_command = f"git show -s --format=%ci {commit_hash}"
|
||||
output, error = run_command(date_command)
|
||||
if error:
|
||||
print(f"Error getting commit date: {error}")
|
||||
return None
|
||||
return dateutil.parser.parse(output.strip()).isoformat()
|
||||
|
||||
|
||||
def main(
|
||||
board: str,
|
||||
num_commits: int,
|
||||
skip_step: int,
|
||||
start_commit: str | None = None,
|
||||
end_commit: str | None = None,
|
||||
):
|
||||
# change to the script dir
|
||||
os.chdir(str(HERE))
|
||||
# Create tmp directory if it doesn't exist
|
||||
|
||||
if os.path.exists("tmp"):
|
||||
shutil.rmtree("tmp")
|
||||
os.makedirs("tmp", exist_ok=True)
|
||||
|
||||
# Change to the tmp directory
|
||||
os.chdir("tmp")
|
||||
|
||||
# 1. Git clone FastLED repository
|
||||
print("Cloning FastLED repository...")
|
||||
clone_command = "git clone https://github.com/FastLED/FastLED.git"
|
||||
output, error = run_command(clone_command)
|
||||
# if error:
|
||||
# print(f"Error cloning repository: {error}")
|
||||
# os.chdir("..")
|
||||
# return
|
||||
|
||||
# Change to the FastLED directory
|
||||
os.chdir("FastLED")
|
||||
|
||||
# Checkout the latest commit
|
||||
run_command("git checkout master")
|
||||
|
||||
# If end_commit is specified, checkout that commit
|
||||
# if end_commit:
|
||||
# print(f"Checking out end commit: {end_commit}")
|
||||
# checkout_command = f"git checkout {end_commit}"
|
||||
# output, error = run_command(checkout_command)
|
||||
# #if error:
|
||||
# # print(f"Error checking out end commit: {error}")
|
||||
# # return
|
||||
|
||||
# Prepare CSV file
|
||||
csv_filename = "../../firmware_sizes.csv"
|
||||
with open(csv_filename, "w", newline="") as csvfile:
|
||||
csvwriter = csv.writer(csvfile)
|
||||
csvwriter.writerow(["datetime", "commit_hash", "binary_size"])
|
||||
|
||||
commits_checked = 0
|
||||
first_iteration = True
|
||||
while True:
|
||||
current_commit = get_commit_hash()
|
||||
|
||||
if first_iteration and start_commit:
|
||||
first_iteration = False
|
||||
while True:
|
||||
if current_commit == start_commit:
|
||||
break
|
||||
if not step_back_commits(1):
|
||||
break
|
||||
current_commit = get_commit_hash()
|
||||
|
||||
if num_commits and commits_checked >= num_commits:
|
||||
print(f"Checked {num_commits} commits")
|
||||
break
|
||||
|
||||
if end_commit and current_commit == end_commit:
|
||||
print(f"Checked until end commit: {end_commit}")
|
||||
break
|
||||
|
||||
# 2. Run ci-compile.py for current commit
|
||||
print(f"\nChecking commit {commits_checked + 1}")
|
||||
|
||||
# remove .build/esp32dev/pio/build/esp32dev/ directory
|
||||
board_files = Path(".build") / board / ".pio" / "build" / board
|
||||
if board_files.exists():
|
||||
shutil.rmtree(str(board_files), ignore_errors=True)
|
||||
compile_command = f"uv run -m ci.ci-compile {board} --examples Blink"
|
||||
output, error = run_command(compile_command)
|
||||
if error:
|
||||
print(f"Error running ci-compile.py: {error}")
|
||||
if not step_back_commits(skip_step):
|
||||
break
|
||||
continue
|
||||
|
||||
# 3. Check firmware size and get commit hash
|
||||
print("Checking firmware size...")
|
||||
try:
|
||||
size = check_firmware_size(board)
|
||||
except FileNotFoundError as e:
|
||||
print(f"Error checking firmware size: {e}")
|
||||
if not step_back_commits(skip_step):
|
||||
break
|
||||
continue
|
||||
except AssertionError as e:
|
||||
print(f"Error: {e}")
|
||||
if not step_back_commits(skip_step):
|
||||
break
|
||||
continue
|
||||
commit_hash = get_commit_hash()
|
||||
if size and commit_hash:
|
||||
commit_date = get_commit_date(commit_hash)
|
||||
print(f"Firmware size: {size} bytes")
|
||||
|
||||
# Write to CSV incrementally
|
||||
with open(csv_filename, "a", newline="") as csvfile:
|
||||
csvwriter = csv.writer(csvfile)
|
||||
csvwriter.writerow([commit_date, commit_hash, size])
|
||||
|
||||
print(f"Result appended to {csv_filename}")
|
||||
|
||||
commits_checked += 1
|
||||
|
||||
# 4. Step back one commit
|
||||
print("Stepping back 1 commit...")
|
||||
if not step_back_commits(1):
|
||||
break
|
||||
|
||||
# Don't remove the tmp directory
|
||||
print("\nTemporary directory 'tmp' has been left intact for inspection.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Check FastLED firmware size for multiple commits."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-commits", type=int, default=1, help="Number of commits to check"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-step",
|
||||
type=int,
|
||||
default=1,
|
||||
help="Number of commits to skip between checks",
|
||||
)
|
||||
parser.add_argument("--start-commit", type=str, help="Starting commit hash")
|
||||
parser.add_argument("--end-commit", type=str, help="Ending commit hash")
|
||||
parser.add_argument(
|
||||
"--board", type=str, required=True, help="Board to check firmware size for"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.start_commit or args.end_commit:
|
||||
if not (args.start_commit and not args.end_commit):
|
||||
print("Both start commit and end commit must be specified.")
|
||||
exit(1)
|
||||
# if start_commit is specified, end_commit must be specified
|
||||
|
||||
num_commits = args.num_commits
|
||||
if args.start_commit and args.end_commit:
|
||||
if args.start_commit == args.end_commit:
|
||||
print("Start commit and end commit are the same.")
|
||||
exit(1)
|
||||
num_commits = 999999
|
||||
|
||||
main(args.board, num_commits, args.skip_step, args.start_commit, args.end_commit)
|
||||
@@ -0,0 +1,5 @@
|
||||
"""
|
||||
FastLED Compiler Module
|
||||
|
||||
High-performance compilation system built on proven Python compiler API.
|
||||
"""
|
||||
@@ -0,0 +1,196 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Dynamic Library Builder
|
||||
Builds FastLED as a shared/dynamic library for unit testing
|
||||
"""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ci.compiler.clang_compiler import (
|
||||
BuildFlags,
|
||||
Compiler,
|
||||
CompilerOptions,
|
||||
Result,
|
||||
)
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
|
||||
|
||||
def build_fastled_dynamic_library(build_dir: Path) -> Path:
|
||||
"""Build FastLED as a dynamic library"""
|
||||
print("Building FastLED dynamic library...")
|
||||
|
||||
# Define library path with appropriate extension
|
||||
if sys.platform == "win32":
|
||||
fastled_lib_path = build_dir / "fastled.dll"
|
||||
else:
|
||||
fastled_lib_path = build_dir / "libfastled.so"
|
||||
|
||||
# Configure compiler with shared library flags
|
||||
settings = CompilerOptions(
|
||||
include_path=str(Path(PROJECT_ROOT) / "src"),
|
||||
defines=[
|
||||
"STUB_PLATFORM",
|
||||
"ARDUINO=10808",
|
||||
"FASTLED_USE_STUB_ARDUINO",
|
||||
"SKETCH_HAS_LOTS_OF_MEMORY=1",
|
||||
"FASTLED_STUB_IMPL",
|
||||
"FASTLED_FORCE_NAMESPACE=1",
|
||||
"FASTLED_TESTING",
|
||||
"FASTLED_NO_AUTO_NAMESPACE",
|
||||
"FASTLED_NO_PINMAP",
|
||||
"HAS_HARDWARE_PIN_SUPPORT",
|
||||
# "FASTLED_DEBUG_LEVEL=1",
|
||||
# "BUILDING_FASTLED_DLL", # New define for DLL exports
|
||||
# "FASTLED_EXPORT=__declspec(dllexport)" if sys.platform == "win32" else "FASTLED_EXPORT=", # DLL exports for Windows
|
||||
],
|
||||
std_version="c++17",
|
||||
compiler="clang++",
|
||||
compiler_args=[
|
||||
# NOTE: All compiler flags should come from build_unit.toml
|
||||
# Keep only platform-specific include paths and shared library flags
|
||||
"-I" + str(Path(PROJECT_ROOT) / "src/platforms/stub"),
|
||||
"-I" + str(Path(PROJECT_ROOT) / "tests"),
|
||||
# Add shared library flags - platform specific
|
||||
"-shared" if sys.platform != "win32" else "/DLL",
|
||||
"-fPIC" if sys.platform != "win32" else "",
|
||||
],
|
||||
use_pch=True,
|
||||
parallel=True,
|
||||
)
|
||||
|
||||
# Load build flags from TOML
|
||||
build_flags_path = Path(PROJECT_ROOT) / "ci" / "build_unit.toml"
|
||||
build_flags = BuildFlags.parse(
|
||||
build_flags_path, quick_build=False, strict_mode=False
|
||||
)
|
||||
|
||||
compiler = Compiler(settings, build_flags)
|
||||
|
||||
# Compile all FastLED source files
|
||||
fastled_src_dir = Path(PROJECT_ROOT) / "src"
|
||||
all_cpp_files = list(fastled_src_dir.rglob("*.cpp"))
|
||||
|
||||
print(f"Found {len(all_cpp_files)} FastLED source files")
|
||||
|
||||
# Compile to object files
|
||||
object_files: List[Path] = []
|
||||
for src_file in all_cpp_files:
|
||||
relative_path = src_file.relative_to(fastled_src_dir)
|
||||
safe_name = (
|
||||
str(relative_path.with_suffix("")).replace("/", "_").replace("\\", "_")
|
||||
)
|
||||
obj_path = build_dir / f"{safe_name}_fastled.o"
|
||||
|
||||
future = compiler.compile_cpp_file(
|
||||
src_file,
|
||||
output_path=obj_path,
|
||||
additional_flags=[
|
||||
"-c",
|
||||
"-DFASTLED_STUB_IMPL",
|
||||
"-DFASTLED_FORCE_NAMESPACE=1",
|
||||
"-DFASTLED_NO_AUTO_NAMESPACE",
|
||||
"-DFASTLED_NO_PINMAP",
|
||||
"-DPROGMEM=",
|
||||
"-DHAS_HARDWARE_PIN_SUPPORT",
|
||||
"-DFASTLED_ENABLE_JSON=1",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
"-fPIC", # Position Independent Code for shared library
|
||||
],
|
||||
)
|
||||
result: Result = future.result()
|
||||
if not result.ok:
|
||||
print(f"ERROR: Failed to compile {src_file}: {result.stderr}")
|
||||
continue
|
||||
|
||||
object_files.append(obj_path)
|
||||
|
||||
print(f"Linking {len(object_files)} object files into dynamic library...")
|
||||
|
||||
# Link as shared library
|
||||
if sys.platform == "win32":
|
||||
# Use lld-link for Windows DLL creation
|
||||
link_cmd = [
|
||||
"lld-link",
|
||||
"/DLL",
|
||||
"/NOLOGO",
|
||||
"/OUT:" + str(fastled_lib_path),
|
||||
"/LIBPATH:C:/Program Files (x86)/Windows Kits/10/Lib/10.0.19041.0/um/x64",
|
||||
"/LIBPATH:C:/Program Files (x86)/Windows Kits/10/Lib/10.0.19041.0/ucrt/x64",
|
||||
"/LIBPATH:C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/lib/x64",
|
||||
*[str(obj) for obj in object_files],
|
||||
"msvcrt.lib",
|
||||
"legacy_stdio_definitions.lib",
|
||||
"kernel32.lib",
|
||||
"user32.lib",
|
||||
]
|
||||
else:
|
||||
link_cmd = [
|
||||
"clang++",
|
||||
"-shared",
|
||||
"-o",
|
||||
str(fastled_lib_path),
|
||||
*[str(obj) for obj in object_files],
|
||||
]
|
||||
|
||||
try:
|
||||
# Use streaming to prevent buffer overflow
|
||||
process = subprocess.Popen(
|
||||
link_cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
bufsize=1,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
|
||||
stdout_lines: list[str] = []
|
||||
stderr_lines: list[str] = []
|
||||
|
||||
while True:
|
||||
stdout_line = process.stdout.readline() if process.stdout else ""
|
||||
stderr_line = process.stderr.readline() if process.stderr else ""
|
||||
|
||||
if stdout_line:
|
||||
stdout_lines.append(stdout_line.rstrip())
|
||||
if stderr_line:
|
||||
stderr_lines.append(stderr_line.rstrip())
|
||||
|
||||
if process.poll() is not None:
|
||||
remaining_stdout = process.stdout.read() if process.stdout else ""
|
||||
remaining_stderr = process.stderr.read() if process.stderr else ""
|
||||
|
||||
if remaining_stdout:
|
||||
for line in remaining_stdout.splitlines():
|
||||
stdout_lines.append(line.rstrip())
|
||||
if remaining_stderr:
|
||||
for line in remaining_stderr.splitlines():
|
||||
stderr_lines.append(line.rstrip())
|
||||
break
|
||||
|
||||
stderr_result = "\n".join(stderr_lines)
|
||||
if process.returncode != 0:
|
||||
raise Exception(f"Failed to create dynamic library: {stderr_result}")
|
||||
|
||||
print(f"Successfully created dynamic library: {fastled_lib_path}")
|
||||
return fastled_lib_path
|
||||
|
||||
except Exception as e:
|
||||
print(f"ERROR: Exception during library creation: {e}")
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Create build directory in temp
|
||||
import tempfile
|
||||
|
||||
build_dir = Path(tempfile.gettempdir()) / "fastled_test_build"
|
||||
build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Build the dynamic library
|
||||
build_fastled_dynamic_library(build_dir)
|
||||
@@ -0,0 +1,273 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Clean Cache-Enhanced Compilation
|
||||
|
||||
Simple wrapper that adds fingerprint cache to FastLED compilation.
|
||||
Dramatically speeds up incremental builds by skipping unchanged files.
|
||||
"""
|
||||
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Optional, Set
|
||||
|
||||
from ci.ci.fingerprint_cache import FingerprintCache
|
||||
from ci.compiler.clang_compiler import Compiler
|
||||
from ci.compiler.test_example_compilation import CompilationResult
|
||||
|
||||
|
||||
class CacheAwareCompiler:
|
||||
"""Simple wrapper that adds cache checking to compilation."""
|
||||
|
||||
def __init__(self, compiler: Compiler, cache_file: Path, verbose: bool = False):
|
||||
self.compiler = compiler
|
||||
self.cache = FingerprintCache(cache_file)
|
||||
self.verbose = verbose
|
||||
self.stats = {
|
||||
"files_checked": 0,
|
||||
"files_skipped": 0,
|
||||
"files_compiled": 0,
|
||||
"cache_hits": 0,
|
||||
"cache_misses": 0,
|
||||
}
|
||||
|
||||
def should_compile(self, file_path: Path, baseline_time: float) -> bool:
|
||||
"""Check if file needs compilation using cache."""
|
||||
self.stats["files_checked"] += 1
|
||||
|
||||
try:
|
||||
needs_compile = self.cache.has_changed(file_path, baseline_time)
|
||||
|
||||
if needs_compile:
|
||||
self.stats["cache_misses"] += 1
|
||||
self.stats["files_compiled"] += 1
|
||||
return True
|
||||
else:
|
||||
self.stats["cache_hits"] += 1
|
||||
self.stats["files_skipped"] += 1
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Skipping unchanged: {file_path.name}")
|
||||
return False
|
||||
except KeyboardInterrupt:
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Error checking {file_path.name}: {e}")
|
||||
self.stats["cache_misses"] += 1
|
||||
self.stats["files_compiled"] += 1
|
||||
return True
|
||||
|
||||
def _headers_changed(self, baseline_time: float) -> bool:
|
||||
"""Detect if any relevant header dependency has changed since last run.
|
||||
|
||||
Uses the compiler's PCH dependency discovery to collect a conservative
|
||||
set of headers that impact example compilation, then queries the
|
||||
fingerprint cache to see if any actually changed content.
|
||||
"""
|
||||
try:
|
||||
# Reuse the compiler's dependency discovery (covers src/** and platforms/**)
|
||||
dependencies: List[Path] = []
|
||||
if hasattr(self.compiler, "_get_pch_dependencies"):
|
||||
dependencies = self.compiler._get_pch_dependencies() # type: ignore[attr-defined]
|
||||
else:
|
||||
# Fallback: hash all headers under the compiler's include path
|
||||
include_root = Path(self.compiler.settings.include_path)
|
||||
for pattern in ("**/*.h", "**/*.hpp"):
|
||||
dependencies.extend(include_root.glob(pattern))
|
||||
|
||||
changed_any = False
|
||||
for dep in dependencies:
|
||||
try:
|
||||
if self.cache.has_changed(dep, baseline_time):
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Header changed: {dep}")
|
||||
changed_any = True
|
||||
break
|
||||
except FileNotFoundError:
|
||||
# Missing dependency means we must rebuild conservatively
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Header missing (forces rebuild): {dep}")
|
||||
changed_any = True
|
||||
break
|
||||
return changed_any
|
||||
except KeyboardInterrupt:
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
if self.verbose:
|
||||
print(f"[CACHE] Header scan failed, forcing rebuild: {e}")
|
||||
return True
|
||||
|
||||
def compile_with_cache(
|
||||
self,
|
||||
ino_files: List[Path],
|
||||
pch_compatible_files: Set[Path],
|
||||
log_fn: Callable[[str], None],
|
||||
full_compilation: bool,
|
||||
verbose: bool = False,
|
||||
baseline_time: Optional[float] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Compile only files that have changed."""
|
||||
|
||||
if baseline_time is None:
|
||||
baseline_time = time.time() - 3600 # Default: 1 hour ago
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# If any header dependency changed, conservatively rebuild all example files
|
||||
force_recompile_due_to_headers = self._headers_changed(baseline_time)
|
||||
if force_recompile_due_to_headers and self.verbose:
|
||||
print(
|
||||
"[CACHE] Header dependency changes detected - forcing recompilation of example sources"
|
||||
)
|
||||
|
||||
# Check which files need compilation
|
||||
files_to_compile: List[Path] = []
|
||||
for ino_file in ino_files:
|
||||
if force_recompile_due_to_headers or self.should_compile(
|
||||
ino_file, baseline_time
|
||||
):
|
||||
files_to_compile.append(ino_file)
|
||||
|
||||
# Check .cpp files too
|
||||
cpp_files_to_compile: List[Path] = []
|
||||
for ino_file in ino_files:
|
||||
cpp_files = self.compiler.find_cpp_files_for_example(ino_file)
|
||||
for cpp_file in cpp_files:
|
||||
if force_recompile_due_to_headers or self.should_compile(
|
||||
cpp_file, baseline_time
|
||||
):
|
||||
cpp_files_to_compile.append(cpp_file)
|
||||
|
||||
# Log cache results
|
||||
total_files = len(ino_files) + sum(
|
||||
len(self.compiler.find_cpp_files_for_example(f)) for f in ino_files
|
||||
)
|
||||
log_fn(
|
||||
f"[CACHE] {self.stats['files_skipped']}/{total_files} files unchanged, {self.stats['files_compiled']}/{total_files} need compilation"
|
||||
)
|
||||
|
||||
if self.stats["files_skipped"] > 0:
|
||||
time_saved = self.stats["files_skipped"] * 0.5 # Estimate
|
||||
log_fn(f"[CACHE] Estimated time saved: {time_saved:.1f}s")
|
||||
|
||||
# Compile files that changed
|
||||
if files_to_compile or cpp_files_to_compile:
|
||||
result = self._run_actual_compilation(
|
||||
files_to_compile,
|
||||
cpp_files_to_compile,
|
||||
pch_compatible_files,
|
||||
log_fn,
|
||||
full_compilation,
|
||||
verbose,
|
||||
)
|
||||
else:
|
||||
log_fn("[CACHE] All files unchanged - no compilation needed!")
|
||||
result = self._create_success_result(len(ino_files), ino_files)
|
||||
|
||||
# Log final stats
|
||||
if total_files > 0:
|
||||
hit_rate = (
|
||||
(
|
||||
self.stats["cache_hits"]
|
||||
/ (self.stats["cache_hits"] + self.stats["cache_misses"])
|
||||
* 100
|
||||
)
|
||||
if (self.stats["cache_hits"] + self.stats["cache_misses"]) > 0
|
||||
else 0
|
||||
)
|
||||
skip_rate = self.stats["files_skipped"] / total_files * 100
|
||||
log_fn(
|
||||
f"[CACHE] {hit_rate:.1f}% cache hit rate, {skip_rate:.1f}% files skipped"
|
||||
)
|
||||
|
||||
return {
|
||||
"compilation_result": result,
|
||||
"cache_stats": self.stats,
|
||||
"total_time": time.time() - start_time,
|
||||
}
|
||||
|
||||
def _run_actual_compilation(
|
||||
self,
|
||||
ino_files: List[Path],
|
||||
cpp_files_to_compile: List[Path],
|
||||
pch_compatible_files: Set[Path],
|
||||
log_fn: Callable[[str], None],
|
||||
full_compilation: bool,
|
||||
verbose: bool,
|
||||
):
|
||||
"""Run the actual compilation for changed files."""
|
||||
from ci.compiler.test_example_compilation import compile_examples_simple
|
||||
|
||||
# Temporarily filter .cpp files to only compile changed ones
|
||||
original_method = self.compiler.find_cpp_files_for_example
|
||||
|
||||
def filtered_cpp_files(ino_file: Path) -> List[Path]:
|
||||
all_cpp = original_method(ino_file)
|
||||
return [cpp for cpp in all_cpp if cpp in cpp_files_to_compile]
|
||||
|
||||
self.compiler.find_cpp_files_for_example = filtered_cpp_files
|
||||
|
||||
try:
|
||||
return compile_examples_simple(
|
||||
self.compiler,
|
||||
ino_files,
|
||||
pch_compatible_files,
|
||||
log_fn,
|
||||
full_compilation,
|
||||
verbose,
|
||||
)
|
||||
finally:
|
||||
self.compiler.find_cpp_files_for_example = original_method
|
||||
|
||||
def _create_success_result(
|
||||
self, file_count: int, ino_files: Optional[List[Path]] = None
|
||||
) -> CompilationResult:
|
||||
"""Create a successful compilation result for cached files."""
|
||||
|
||||
# For cached files, we need to populate object_file_map with existing object files
|
||||
# so that linking can proceed properly
|
||||
object_file_map: Dict[Path, List[Path]] = {}
|
||||
if ino_files:
|
||||
for ino_file in ino_files:
|
||||
# Find existing object files for this example
|
||||
example_name = ino_file.parent.name
|
||||
build_dir = Path(".build/examples") / example_name
|
||||
|
||||
obj_files: List[Path] = []
|
||||
if build_dir.exists():
|
||||
# Look for the main .ino object file
|
||||
ino_obj = build_dir / f"{ino_file.stem}.o"
|
||||
if ino_obj.exists():
|
||||
obj_files.append(ino_obj)
|
||||
|
||||
# Look for additional .cpp files in the same directory as the .ino
|
||||
cpp_files = self.compiler.find_cpp_files_for_example(ino_file)
|
||||
for cpp_file in cpp_files:
|
||||
cpp_obj = build_dir / f"{cpp_file.stem}.o"
|
||||
if cpp_obj.exists():
|
||||
obj_files.append(cpp_obj)
|
||||
|
||||
if obj_files:
|
||||
object_file_map[ino_file] = obj_files
|
||||
|
||||
return CompilationResult(
|
||||
successful_count=file_count,
|
||||
failed_count=0,
|
||||
compile_time=0.0,
|
||||
failed_examples=[],
|
||||
object_file_map=object_file_map,
|
||||
)
|
||||
|
||||
|
||||
def create_cache_compiler(
|
||||
compiler: Compiler, cache_file: Optional[Path] = None, verbose: bool = False
|
||||
) -> CacheAwareCompiler:
|
||||
"""Create a cache-aware compiler. Simple factory function."""
|
||||
if cache_file is None:
|
||||
cache_file = Path(".build/fingerprint_cache.json")
|
||||
return CacheAwareCompiler(compiler, cache_file, verbose)
|
||||
@@ -0,0 +1,521 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Standalone cache setup script for PlatformIO builds with Python fake compilers.
|
||||
This script is executed by PlatformIO as a post-build extra_script.
|
||||
|
||||
Configuration is passed through environment variables to avoid template string issues.
|
||||
"""
|
||||
|
||||
# ruff: noqa: F405, F821 # Suppress SCons-specific import and undefined name warnings
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
|
||||
|
||||
# Import env and try to import projenv (SCons-specific imports)
|
||||
# These are dynamically available in PlatformIO/SCons environment
|
||||
try:
|
||||
Import("env") # type: ignore[name-defined] # SCons-specific import
|
||||
env: Any # SCons environment object
|
||||
except NameError:
|
||||
env = None # For type checking when not in SCons context
|
||||
|
||||
_VERBOSE = os.environ.get("VERBOSE", "0") in ("1", "true", "True", "yes", "y")
|
||||
|
||||
# Try to import projenv if it exists
|
||||
has_projenv: bool = False
|
||||
projenv: Optional[Any] = None
|
||||
try:
|
||||
Import("projenv") # type: ignore[name-defined] # SCons-specific import
|
||||
# projenv is now available in scope from Import
|
||||
has_projenv = True
|
||||
except (NameError, Exception):
|
||||
has_projenv = False
|
||||
projenv = None
|
||||
|
||||
# Add project root to Python path for fake compiler imports
|
||||
# Note: We're in the build directory, go up to project root
|
||||
current_dir = Path(os.getcwd()).resolve()
|
||||
project_root = current_dir.parent.parent # Go up from .build/pio/board to project root
|
||||
if str(project_root) not in sys.path:
|
||||
sys.path.insert(0, str(project_root))
|
||||
|
||||
# Import cached compiler functions with type stubs
|
||||
create_cached_toolchain: Optional[Any] = None
|
||||
get_platform_packages_paths: Optional[Any] = None
|
||||
cached_compiler_available: bool = False
|
||||
|
||||
try:
|
||||
from ci.util.cached_compiler import (
|
||||
create_cached_toolchain,
|
||||
get_platform_packages_paths,
|
||||
)
|
||||
|
||||
cached_compiler_available = True
|
||||
except ImportError as e:
|
||||
print("WARNING: Could not import cached compiler module: " + str(e))
|
||||
cached_compiler_available = False
|
||||
|
||||
# Debug: Dump the environment state to disk for inspection
|
||||
env_dump: Dict[str, str] = {}
|
||||
try:
|
||||
if env is not None and hasattr(env, "Dictionary"): # type: ignore[has-type]
|
||||
for key in env.Dictionary(): # type: ignore[union-attr]
|
||||
try:
|
||||
value = env[key] # type: ignore[index]
|
||||
# Convert to string to avoid JSON serialization issues
|
||||
env_dump[key] = str(value) # type: ignore[arg-type]
|
||||
except Exception:
|
||||
env_dump[key] = "<error getting value>"
|
||||
else:
|
||||
env_dump["error"] = "env not available or missing Dictionary method"
|
||||
except Exception as e:
|
||||
env_dump["error"] = f"Failed to access env: {e}"
|
||||
|
||||
# Write environment dump to disk
|
||||
env_dump_path = "env_dump.json"
|
||||
with open(env_dump_path, "w") as f:
|
||||
json.dump(env_dump, f, indent=2)
|
||||
print("Environment state dumped to: " + env_dump_path)
|
||||
|
||||
# Also dump projenv if available
|
||||
if has_projenv and projenv is not None:
|
||||
projenv_dump: Dict[str, str] = {}
|
||||
for key in projenv.Dictionary(): # type: ignore
|
||||
try:
|
||||
value = projenv[key] # type: ignore
|
||||
projenv_dump[key] = str(value)
|
||||
except Exception:
|
||||
projenv_dump[key] = "<error getting value>"
|
||||
|
||||
projenv_dump_path = "projenv_dump.json"
|
||||
with open(projenv_dump_path, "w") as f:
|
||||
json.dump(projenv_dump, f, indent=2)
|
||||
print("Projenv state dumped to: " + projenv_dump_path)
|
||||
|
||||
# Read cache configuration from environment variables
|
||||
cache_type = os.environ.get("FASTLED_CACHE_TYPE", "no_cache")
|
||||
cache_executable = os.environ.get("FASTLED_CACHE_EXECUTABLE", "")
|
||||
sccache_path = os.environ.get("FASTLED_SCCACHE_PATH", "")
|
||||
sccache_dir = os.environ.get("FASTLED_SCCACHE_DIR", "")
|
||||
sccache_cache_size = os.environ.get("FASTLED_SCCACHE_CACHE_SIZE", "2G")
|
||||
xcache_path = os.environ.get("FASTLED_XCACHE_PATH", "")
|
||||
debug_enabled = os.environ.get("FASTLED_CACHE_DEBUG", "0") == "1"
|
||||
|
||||
print("Cache configuration from environment:")
|
||||
print(" Cache type: " + cache_type)
|
||||
print(" Cache executable: " + cache_executable)
|
||||
print(" SCCACHE path: " + sccache_path)
|
||||
print(" SCCACHE dir: " + sccache_dir)
|
||||
print(" Debug enabled: " + str(debug_enabled))
|
||||
|
||||
# Set up cache environment variables for subprocess execution
|
||||
try:
|
||||
if env is not None and hasattr(env, "Append"): # type: ignore[has-type]
|
||||
if sccache_dir:
|
||||
env.Append(ENV={"SCCACHE_DIR": sccache_dir}) # type: ignore[union-attr]
|
||||
os.environ["SCCACHE_DIR"] = sccache_dir
|
||||
|
||||
if sccache_cache_size:
|
||||
env.Append(ENV={"SCCACHE_CACHE_SIZE": sccache_cache_size}) # type: ignore[union-attr]
|
||||
os.environ["SCCACHE_CACHE_SIZE"] = sccache_cache_size
|
||||
|
||||
# Ensure sccache binary directory is on PATH so xcache can find it even
|
||||
# inside ESP-IDF's virtual environments where PATH is heavily modified.
|
||||
if sccache_path:
|
||||
sccache_dir_path = str(Path(sccache_path).parent)
|
||||
env.PrependENVPath("PATH", sccache_dir_path) # type: ignore[union-attr]
|
||||
if has_projenv and projenv is not None and hasattr(projenv, "PrependENVPath"):
|
||||
projenv.PrependENVPath("PATH", sccache_dir_path) # type: ignore[union-attr]
|
||||
os.environ["PATH"] = sccache_dir_path + os.pathsep + os.environ.get("PATH", "")
|
||||
|
||||
if debug_enabled:
|
||||
env.Append(ENV={"XCACHE_DEBUG": "1", "SCCACHE_DEBUG": "1"}) # type: ignore[union-attr]
|
||||
if has_projenv and projenv is not None and hasattr(projenv, "Append"):
|
||||
projenv.Append(ENV={"XCACHE_DEBUG": "1", "SCCACHE_DEBUG": "1"}) # type: ignore[union-attr]
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to set up cache environment: {e}")
|
||||
|
||||
# Check if cache is available and cached compiler system can be used
|
||||
USE_CACHE = False
|
||||
|
||||
if cached_compiler_available and cache_executable:
|
||||
if cache_type == "xcache":
|
||||
# For xcache, check if the Python script exists and sccache is available
|
||||
USE_CACHE = (
|
||||
cache_executable
|
||||
and xcache_path
|
||||
and Path(xcache_path).exists()
|
||||
and sccache_path
|
||||
and shutil.which(sccache_path)
|
||||
)
|
||||
|
||||
if USE_CACHE:
|
||||
print("xcache wrapper detected and configured for Python fake compilers")
|
||||
print(" xcache path: " + str(xcache_path))
|
||||
print(" cache executable: " + str(cache_executable))
|
||||
else:
|
||||
# For sccache/ccache, check if executable is in PATH
|
||||
USE_CACHE = shutil.which(cache_executable) is not None
|
||||
|
||||
if USE_CACHE:
|
||||
print(
|
||||
str(cache_type) + " detected and configured for Python cached compilers"
|
||||
)
|
||||
print(" cache executable: " + str(cache_executable))
|
||||
elif not cached_compiler_available:
|
||||
print(
|
||||
"WARNING: Python cached compiler system not available, cache will be disabled"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
"Cache executable not found: "
|
||||
+ str(cache_executable)
|
||||
+ ", cache will be disabled"
|
||||
)
|
||||
|
||||
if USE_CACHE and env is not None and hasattr(env, "get"): # type: ignore[has-type]
|
||||
# Get current compilers from environment
|
||||
original_cc = env.get("CC") # type: ignore[union-attr]
|
||||
original_cxx = env.get("CXX") # type: ignore[union-attr]
|
||||
|
||||
print("DEBUG: Found compilers in env:")
|
||||
# Use repr for safer type conversion
|
||||
cc_str = repr(original_cc)
|
||||
cxx_str = repr(original_cxx)
|
||||
cc_type_str = type(original_cc).__name__
|
||||
cxx_type_str = type(original_cxx).__name__
|
||||
print(" CC: " + cc_str + " (type: " + cc_type_str + ")")
|
||||
print(" CXX: " + cxx_str + " (type: " + cxx_type_str + ")")
|
||||
|
||||
# Extract compiler information for fake compiler generation
|
||||
def extract_compiler_info(compiler_env_var: Any) -> Optional[str]:
|
||||
"""Extract compiler name from environment variable value."""
|
||||
if not compiler_env_var:
|
||||
return None
|
||||
|
||||
if isinstance(compiler_env_var, list):
|
||||
return str(compiler_env_var[0]) if compiler_env_var else None # type: ignore[arg-type]
|
||||
else:
|
||||
# Handle string values like "arm-none-eabi-gcc" or "gcc"
|
||||
return str(compiler_env_var).split()[0] # type: ignore[arg-type]
|
||||
|
||||
cc_name = extract_compiler_info(original_cc) or "gcc"
|
||||
cxx_name = extract_compiler_info(original_cxx) or "g++"
|
||||
|
||||
print("Extracted compiler names:")
|
||||
print(" CC name: " + str(cc_name))
|
||||
print(" CXX name: " + str(cxx_name))
|
||||
|
||||
# Create toolchain info for fake compiler generation
|
||||
toolchain_info: Dict[str, str] = {
|
||||
"CC": cc_name,
|
||||
"CXX": cxx_name,
|
||||
}
|
||||
|
||||
# Create cache config for fake compiler system
|
||||
cache_config: Dict[str, str] = {
|
||||
"CACHE_TYPE": cache_type,
|
||||
"CACHE_EXECUTABLE": cache_executable,
|
||||
"SCCACHE_PATH": sccache_path,
|
||||
"SCCACHE_DIR": sccache_dir,
|
||||
"XCACHE_PATH": xcache_path,
|
||||
}
|
||||
|
||||
# Check if toolset exists and is valid
|
||||
# Create a cache key based on the configuration to invalidate cache when config changes
|
||||
cache_key = f"{cc_name}_{cxx_name}_{cache_type}_{cache_executable}"
|
||||
|
||||
# Cache in the local build directory (e.g., .build/pio/uno/)
|
||||
cache_file = Path(current_dir) / "compiler_cache.json"
|
||||
|
||||
cached_tools: Optional[Dict[str, str]] = None
|
||||
|
||||
# Try to load from persistent cache file
|
||||
if cache_file.exists():
|
||||
try:
|
||||
import json
|
||||
|
||||
with open(cache_file, "r") as f:
|
||||
cache_data = json.load(f)
|
||||
|
||||
cached_real_cc = cache_data.get("real_cc")
|
||||
cached_real_cxx = cache_data.get("real_cxx")
|
||||
cached_fake_cc = cache_data.get("fake_cc")
|
||||
cached_fake_cxx = cache_data.get("fake_cxx")
|
||||
|
||||
if (
|
||||
cached_real_cc
|
||||
and cached_real_cxx
|
||||
and cached_fake_cc
|
||||
and cached_fake_cxx
|
||||
):
|
||||
print("Found local compiler cache:")
|
||||
print(f" Cache file: {cache_file}")
|
||||
print(f" Real CC: {cached_real_cc}")
|
||||
print(f" Real CXX: {cached_real_cxx}")
|
||||
|
||||
# Check if fake compiler scripts still exist, recreate if needed
|
||||
fake_cc_path = Path(cached_fake_cc.replace("python ", ""))
|
||||
fake_cxx_path = Path(cached_fake_cxx.replace("python ", ""))
|
||||
|
||||
if fake_cc_path.exists() and fake_cxx_path.exists():
|
||||
# Fast path: use existing cached compilers
|
||||
# On Windows, ensure we have .cmd shims to avoid 'python script.py @file'
|
||||
# causing Python to treat @file as the script name.
|
||||
import sys as _sys
|
||||
import os as _os
|
||||
|
||||
is_win = (
|
||||
_os.name == "nt"
|
||||
or _sys.platform.startswith("win")
|
||||
or _sys.platform.startswith("cygwin")
|
||||
or _sys.platform.startswith("msys")
|
||||
)
|
||||
|
||||
def _ensure_cmd_wrapper(fake_str: str, real_path: str) -> str:
|
||||
p = Path(fake_str.replace("python ", ""))
|
||||
if is_win and p.suffix.lower() == ".py":
|
||||
cmd = p.with_suffix(".cmd")
|
||||
if not cmd.exists():
|
||||
# Create missing .cmd wrapper now
|
||||
from ci.util.cached_compiler import create_cached_compiler_script
|
||||
out_dir = p.parent
|
||||
created = create_cached_compiler_script(
|
||||
compiler_name=p.stem.split("cached_")[-1],
|
||||
cache_executable=cache_config.get("CACHE_EXECUTABLE", "sccache"),
|
||||
real_compiler_path=real_path,
|
||||
output_dir=out_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
cmd = Path(str(created).replace("python ", ""))
|
||||
if cmd.exists():
|
||||
return str(cmd)
|
||||
return fake_str
|
||||
|
||||
# Create/choose proper wrapper paths
|
||||
new_cc = _ensure_cmd_wrapper(cached_fake_cc, cached_real_cc)
|
||||
new_cxx = _ensure_cmd_wrapper(cached_fake_cxx, cached_real_cxx)
|
||||
|
||||
cached_tools = {"CC": new_cc, "CXX": new_cxx}
|
||||
|
||||
# Update cache file if we switched to .cmd
|
||||
try:
|
||||
if new_cc != cached_fake_cc or new_cxx != cached_fake_cxx:
|
||||
cache_data["fake_cc"] = new_cc
|
||||
cache_data["fake_cxx"] = new_cxx
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump(cache_data, f, indent=2)
|
||||
except KeyboardInterrupt as ke:
|
||||
import _thread
|
||||
_thread.interrupt_main()
|
||||
raise ke
|
||||
except Exception:
|
||||
pass
|
||||
print(
|
||||
"SUCCESS: Using cached compilers (instant, no platform search needed):"
|
||||
)
|
||||
print(f" CC: {cached_tools['CC']}")
|
||||
print(f" CXX: {cached_tools['CXX']}")
|
||||
print(" Platform search skipped - using cached toolset")
|
||||
else:
|
||||
print(
|
||||
"Cached compiler scripts missing, recreating with cached real paths..."
|
||||
)
|
||||
# Recreate cached compilers using cached real paths (fast)
|
||||
cached_compilers_dir = Path(current_dir) / "cached_compilers"
|
||||
cached_compilers_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
from ci.util.cached_compiler import create_cached_compiler_script
|
||||
|
||||
cache_executable = cache_config.get("CACHE_EXECUTABLE", "sccache")
|
||||
|
||||
# Create cached CC script using cached real path
|
||||
cached_cc_script = create_cached_compiler_script(
|
||||
compiler_name="CC",
|
||||
cache_executable=cache_executable,
|
||||
real_compiler_path=cached_real_cc,
|
||||
output_dir=cached_compilers_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
|
||||
# Create cached CXX script using cached real path
|
||||
cached_cxx_script = create_cached_compiler_script(
|
||||
compiler_name="CXX",
|
||||
cache_executable=cache_executable,
|
||||
real_compiler_path=cached_real_cxx,
|
||||
output_dir=cached_compilers_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
|
||||
cached_tools = {
|
||||
"CC": str(cached_cc_script),
|
||||
"CXX": str(cached_cxx_script),
|
||||
}
|
||||
|
||||
# Update cache file with new script paths
|
||||
cache_data["fake_cc"] = cached_tools["CC"]
|
||||
cache_data["fake_cxx"] = cached_tools["CXX"]
|
||||
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump(cache_data, f, indent=2)
|
||||
|
||||
print("Recreated cached compilers using cached real paths:")
|
||||
print(f" CC: {cached_tools['CC']}")
|
||||
print(f" CXX: {cached_tools['CXX']}")
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to load cache file {cache_file}: {e}")
|
||||
# Fall through to full recreation
|
||||
|
||||
# If no valid cache found, create the toolset from scratch
|
||||
if cached_tools is None:
|
||||
print("No valid cache found, creating compiler toolset from scratch...")
|
||||
print(" This is the first compile or configuration changed")
|
||||
|
||||
# Get platform packages paths for toolchain resolution (expensive operation)
|
||||
platform_packages: List[str] = []
|
||||
if get_platform_packages_paths is not None:
|
||||
print("Searching platform packages (this may take ~10 seconds)...")
|
||||
platform_packages = get_platform_packages_paths()
|
||||
print(f"Found {len(platform_packages)} platform package directories")
|
||||
|
||||
if create_cached_toolchain is not None:
|
||||
try:
|
||||
# Find the real compiler paths (expensive operation, done only once)
|
||||
from ci.util.cached_compiler import find_toolchain_compiler
|
||||
|
||||
print("Resolving real compiler paths...")
|
||||
|
||||
real_cc_path = find_toolchain_compiler(cc_name, platform_packages)
|
||||
real_cxx_path = find_toolchain_compiler(cxx_name, platform_packages)
|
||||
|
||||
if not real_cc_path or not real_cxx_path:
|
||||
print(f"ERROR: Could not find real compilers:")
|
||||
print(f" CC '{cc_name}': {real_cc_path}")
|
||||
print(f" CXX '{cxx_name}': {real_cxx_path}")
|
||||
cached_tools = None
|
||||
else:
|
||||
print(f"Found real compilers:")
|
||||
print(f" Real CC: {real_cc_path}")
|
||||
print(f" Real CXX: {real_cxx_path}")
|
||||
|
||||
# Create cached compiler scripts
|
||||
cached_compilers_dir = Path(current_dir) / "cached_compilers"
|
||||
cached_compilers_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
from ci.util.cached_compiler import create_cached_compiler_script
|
||||
|
||||
cache_executable = cache_config.get("CACHE_EXECUTABLE", "sccache")
|
||||
|
||||
# Create cached CC script
|
||||
cached_cc_script = create_cached_compiler_script(
|
||||
compiler_name="CC",
|
||||
cache_executable=cache_executable,
|
||||
real_compiler_path=real_cc_path,
|
||||
output_dir=cached_compilers_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
|
||||
# Create cached CXX script
|
||||
cached_cxx_script = create_cached_compiler_script(
|
||||
compiler_name="CXX",
|
||||
cache_executable=cache_executable,
|
||||
real_compiler_path=real_cxx_path,
|
||||
output_dir=cached_compilers_dir,
|
||||
debug=debug_enabled,
|
||||
)
|
||||
|
||||
cached_tools = {
|
||||
"CC": str(cached_cc_script),
|
||||
"CXX": str(cached_cxx_script),
|
||||
}
|
||||
|
||||
print("Created new compiler toolset:")
|
||||
print(f" CC: {cached_tools['CC']}")
|
||||
print(f" CXX: {cached_tools['CXX']}")
|
||||
|
||||
# Save to local build directory cache file
|
||||
cache_data = {
|
||||
"cache_key": cache_key,
|
||||
"real_cc": real_cc_path,
|
||||
"real_cxx": real_cxx_path,
|
||||
"fake_cc": cached_tools["CC"],
|
||||
"fake_cxx": cached_tools["CXX"],
|
||||
"build_dir": str(current_dir),
|
||||
"platform_packages_count": len(platform_packages),
|
||||
}
|
||||
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump(cache_data, f, indent=2)
|
||||
|
||||
print(f"Saved compiler toolset to local cache: {cache_file}")
|
||||
print(" This cache will persist across builds for this platform")
|
||||
|
||||
except Exception as e:
|
||||
print("ERROR: Toolset creation failed with exception: " + str(e))
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
cached_tools = None
|
||||
else:
|
||||
print("ERROR: create_cached_toolchain function is None")
|
||||
|
||||
if cached_tools:
|
||||
# Use Python cached compilers instead of batch scripts
|
||||
new_cc = cached_tools.get("CC")
|
||||
new_cxx = cached_tools.get("CXX")
|
||||
|
||||
if new_cc and new_cxx:
|
||||
print("Created Python cached compilers:")
|
||||
print(" CC: " + str(new_cc))
|
||||
print(" CXX: " + str(new_cxx))
|
||||
|
||||
# Apply to both environments
|
||||
env.Replace(CC=new_cc, CXX=new_cxx) # type: ignore
|
||||
if has_projenv and projenv is not None:
|
||||
projenv.Replace(CC=new_cc, CXX=new_cxx) # type: ignore
|
||||
print("Applied Python fake compilers to both env and projenv")
|
||||
else:
|
||||
print("Applied Python fake compilers to env (projenv not available)")
|
||||
|
||||
# Apply to library builders (critical for framework caching)
|
||||
try:
|
||||
for lib_builder in env.GetLibBuilders(): # type: ignore
|
||||
lib_builder.env.Replace(CC=new_cc, CXX=new_cxx) # type: ignore
|
||||
if _VERBOSE:
|
||||
print(
|
||||
"Applied Python fake compilers to library builder: "
|
||||
+ str(getattr(lib_builder, "name", "unnamed"))
|
||||
)
|
||||
except KeyboardInterrupt:
|
||||
import _thread
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
print("WARNING: Could not apply to library builders: " + str(e))
|
||||
|
||||
print("Python fake compiler cache enabled: " + str(cache_type))
|
||||
print(" Original CC: " + str(original_cc))
|
||||
print(" Original CXX: " + str(original_cxx))
|
||||
print(" Fake CC: " + str(new_cc))
|
||||
print(" Fake CXX: " + str(new_cxx))
|
||||
else:
|
||||
print(
|
||||
"ERROR: Failed to create Python fake compilers, falling back to no cache"
|
||||
)
|
||||
USE_CACHE = False
|
||||
else:
|
||||
print("ERROR: Python fake compiler creation failed, falling back to no cache")
|
||||
USE_CACHE = False
|
||||
|
||||
if not USE_CACHE:
|
||||
if cache_executable:
|
||||
print("Warning: " + str(cache_type) + " setup failed; using default compilers")
|
||||
else:
|
||||
print("No cache executable configured; using default compilers")
|
||||
|
||||
print("Python fake compiler cache environment configured successfully")
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,261 @@
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
Compilation support for individual boards.
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
from threading import Lock
|
||||
from typing import List
|
||||
|
||||
from ci.boards import Board # type: ignore
|
||||
from ci.util.locked_print import locked_print
|
||||
|
||||
|
||||
ERROR_HAPPENED = False
|
||||
|
||||
|
||||
IS_GITHUB = "GITHUB_ACTIONS" in os.environ
|
||||
FIRST_BUILD_LOCK = Lock()
|
||||
USE_FIRST_BUILD_LOCK = IS_GITHUB
|
||||
|
||||
|
||||
def errors_happened() -> bool:
|
||||
"""Return whether any errors happened during the build."""
|
||||
return ERROR_HAPPENED
|
||||
|
||||
|
||||
def _fastled_js_is_parent_directory(p: Path) -> bool:
|
||||
"""Check if fastled_js is a parent directory of the given path."""
|
||||
# Check if fastled_js is a parent directory of p
|
||||
return "fastled_js" in str(p.absolute())
|
||||
|
||||
|
||||
def compile_for_board_and_example(
|
||||
board: Board,
|
||||
example: Path,
|
||||
build_dir: str | None,
|
||||
verbose_on_failure: bool,
|
||||
libs: list[str] | None,
|
||||
) -> tuple[bool, str]:
|
||||
"""Compile the given example for the given board."""
|
||||
global ERROR_HAPPENED # pylint: disable=global-statement
|
||||
if board.board_name == "web":
|
||||
locked_print(f"Skipping web target for example {example}")
|
||||
return True, ""
|
||||
board_name = board.board_name
|
||||
use_pio_run = board.use_pio_run
|
||||
real_board_name = board.get_real_board_name()
|
||||
libs = libs or []
|
||||
builddir = (
|
||||
Path(build_dir) / board_name if build_dir else Path(".build") / board_name
|
||||
)
|
||||
builddir.mkdir(parents=True, exist_ok=True)
|
||||
srcdir = builddir / "src"
|
||||
# Remove the previous *.ino file if it exists, everything else is recycled
|
||||
# to speed up the next build.
|
||||
if srcdir.exists():
|
||||
shutil.rmtree(srcdir, ignore_errors=False)
|
||||
locked_print(f"*** Building example {example} for board {board_name} ***")
|
||||
cwd: str | None = None
|
||||
shell: bool = False
|
||||
# Copy all files from the example directory to the "src" directory
|
||||
for src_file in example.rglob("*"):
|
||||
if src_file.is_file():
|
||||
if _fastled_js_is_parent_directory(src_file):
|
||||
# Skip the fastled_js folder, it's not needed for the build.
|
||||
continue
|
||||
src_dir = src_file.parent
|
||||
path = src_dir.relative_to(example)
|
||||
dst_dir = srcdir / path
|
||||
os.makedirs(dst_dir, exist_ok=True)
|
||||
locked_print(f"Copying {src_file} to {dst_dir / src_file.name}")
|
||||
os.makedirs(srcdir, exist_ok=True)
|
||||
shutil.copy(src_file, dst_dir / src_file.name)
|
||||
# libs = ["src", "ci"]
|
||||
if use_pio_run:
|
||||
# we have to copy a few folders of pio ci in order to get this to work.
|
||||
for lib in libs:
|
||||
project_libdir = Path(lib)
|
||||
assert project_libdir.exists()
|
||||
build_lib = builddir / "lib" / lib
|
||||
shutil.rmtree(build_lib, ignore_errors=True)
|
||||
shutil.copytree(project_libdir, build_lib)
|
||||
|
||||
cwd = str(builddir)
|
||||
cmd_list = [
|
||||
"pio",
|
||||
"run",
|
||||
]
|
||||
# in this case we need to manually copy the example to the src directory
|
||||
# because platformio doesn't support building a single file.
|
||||
# ino_file = example / f"{example.name}.ino"
|
||||
else:
|
||||
cmd_list = [
|
||||
"pio",
|
||||
"ci",
|
||||
"--board",
|
||||
real_board_name,
|
||||
*[f"--lib={lib}" for lib in libs],
|
||||
"--keep-build-dir",
|
||||
f"--build-dir={builddir.as_posix()}",
|
||||
]
|
||||
cmd_list.append(f"{example.as_posix()}/*ino")
|
||||
cmd_str = subprocess.list2cmdline(cmd_list)
|
||||
msg_lsit = [
|
||||
"\n\n******************************",
|
||||
f"* Running command in cwd: {cwd if cwd else os.getcwd()}",
|
||||
f"* {cmd_str}",
|
||||
"******************************\n",
|
||||
]
|
||||
msg = "\n".join(msg_lsit)
|
||||
locked_print(msg)
|
||||
|
||||
# Start timing for the process
|
||||
start_time = time.time()
|
||||
|
||||
# Run the process with real-time output capture and timing
|
||||
result = subprocess.Popen(
|
||||
cmd_list,
|
||||
cwd=cwd,
|
||||
shell=shell,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
)
|
||||
|
||||
# Capture output lines in real-time with timing
|
||||
stdout_lines: List[str] = []
|
||||
if result.stdout:
|
||||
for line in iter(result.stdout.readline, ""):
|
||||
if line:
|
||||
elapsed = time.time() - start_time
|
||||
# Format timing as seconds with 2 decimal places
|
||||
timing_prefix = f"{elapsed:5.2f} "
|
||||
timed_line = timing_prefix + line.rstrip()
|
||||
stdout_lines.append(
|
||||
line.rstrip()
|
||||
) # Store original line for return value
|
||||
locked_print(timed_line)
|
||||
|
||||
# Wait for process to complete
|
||||
result.wait()
|
||||
|
||||
# Join all stdout lines for the return value
|
||||
stdout = "\n".join(stdout_lines)
|
||||
|
||||
# replace all instances of "lib/src" => "src" so intellisense can find the files
|
||||
# with one click.
|
||||
stdout = stdout.replace("lib/src", "src").replace("lib\\src", "src")
|
||||
if result.returncode != 0:
|
||||
if not verbose_on_failure:
|
||||
ERROR_HAPPENED = True
|
||||
return False, stdout
|
||||
if ERROR_HAPPENED:
|
||||
return False, ""
|
||||
ERROR_HAPPENED = True
|
||||
locked_print(
|
||||
f"*** Error compiling example {example} for board {board_name} ***"
|
||||
)
|
||||
# re-running command with verbose output to see what the defines are.
|
||||
cmd_list.append("-v")
|
||||
cmd_str = subprocess.list2cmdline(cmd_list)
|
||||
msg_lsit = [
|
||||
"\n\n******************************",
|
||||
"* Re-running failed command but with verbose output:",
|
||||
f"* {cmd_str}",
|
||||
"******************************\n",
|
||||
]
|
||||
msg = "\n".join(msg_lsit)
|
||||
locked_print(msg)
|
||||
|
||||
# Start timing for the verbose re-run
|
||||
start_time_verbose = time.time()
|
||||
|
||||
# Run the verbose process with real-time output capture and timing
|
||||
result = subprocess.Popen(
|
||||
cmd_list,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
)
|
||||
|
||||
# Capture output lines in real-time with timing for verbose run
|
||||
stdout_lines_verbose: List[str] = []
|
||||
if result.stdout:
|
||||
for line in iter(result.stdout.readline, ""):
|
||||
if line:
|
||||
elapsed = time.time() - start_time_verbose
|
||||
# Format timing as seconds with 2 decimal places
|
||||
timing_prefix = f"{elapsed:5.2f} "
|
||||
timed_line = timing_prefix + line.rstrip()
|
||||
stdout_lines_verbose.append(
|
||||
line.rstrip()
|
||||
) # Store original line for return value
|
||||
locked_print(timed_line)
|
||||
|
||||
# Wait for verbose process to complete
|
||||
result.wait()
|
||||
|
||||
# Join all stdout lines for the return value
|
||||
stdout = "\n".join(stdout_lines_verbose)
|
||||
stdout = (
|
||||
stdout
|
||||
+ "\n\nThis is a second attempt, but with verbose output, look above for compiler errors.\n"
|
||||
)
|
||||
return False, stdout
|
||||
locked_print(f"*** Finished building example {example} for board {board_name} ***")
|
||||
return True, stdout
|
||||
|
||||
|
||||
# Function to process task queues for each board
|
||||
def compile_examples(
|
||||
board: Board,
|
||||
examples: list[Path],
|
||||
build_dir: str | None,
|
||||
verbose_on_failure: bool,
|
||||
libs: list[str] | None,
|
||||
) -> tuple[bool, str]:
|
||||
"""Process the task queue for the given board."""
|
||||
global ERROR_HAPPENED # pylint: disable=global-statement
|
||||
board_name = board.board_name
|
||||
is_first = True
|
||||
for example in examples:
|
||||
example = example.relative_to(Path(".").resolve())
|
||||
if ERROR_HAPPENED:
|
||||
return True, ""
|
||||
locked_print(f"\n*** Building {example} for board {board_name} ***")
|
||||
if is_first:
|
||||
locked_print(
|
||||
f"*** Building for first example {example} board {board_name} ***"
|
||||
)
|
||||
if is_first and USE_FIRST_BUILD_LOCK:
|
||||
with FIRST_BUILD_LOCK:
|
||||
# Github runners are memory limited and the first job is the most
|
||||
# memory intensive since all the artifacts are being generated in parallel.
|
||||
success, message = compile_for_board_and_example(
|
||||
board=board,
|
||||
example=example,
|
||||
build_dir=build_dir,
|
||||
verbose_on_failure=verbose_on_failure,
|
||||
libs=libs,
|
||||
)
|
||||
else:
|
||||
success, message = compile_for_board_and_example(
|
||||
board=board,
|
||||
example=example,
|
||||
build_dir=build_dir,
|
||||
verbose_on_failure=verbose_on_failure,
|
||||
libs=libs,
|
||||
)
|
||||
is_first = False
|
||||
if not success:
|
||||
ERROR_HAPPENED = True
|
||||
return (
|
||||
False,
|
||||
f"Error building {example} for board {board_name}. stdout:\n{message}",
|
||||
)
|
||||
return True, ""
|
||||
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Abstract Base Class for FastLED Compilers
|
||||
|
||||
Defines the interface that all FastLED compiler implementations must follow.
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from concurrent.futures import Future
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
class CacheType(Enum):
|
||||
"""Compiler cache type options."""
|
||||
|
||||
NO_CACHE = "no_cache"
|
||||
SCCACHE = "sccache"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompilerResult:
|
||||
"""Base result class for compiler operations."""
|
||||
|
||||
success: bool
|
||||
output: str
|
||||
build_dir: Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class InitResult(CompilerResult):
|
||||
"""Result from compiler initialization."""
|
||||
|
||||
@property
|
||||
def platformio_ini(self) -> Path:
|
||||
return self.build_dir / "platformio.ini"
|
||||
|
||||
|
||||
@dataclass
|
||||
class SketchResult(CompilerResult):
|
||||
"""Result from sketch compilation."""
|
||||
|
||||
example: str
|
||||
|
||||
|
||||
class Compiler(ABC):
|
||||
"""Abstract base class defining the interface for FastLED compilers."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
"""Initialize the compiler."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def build(self, examples: list[str]) -> list[Future[SketchResult]]:
|
||||
"""Build a list of examples with proper resource management.
|
||||
|
||||
Args:
|
||||
examples: List of example names or paths to compile
|
||||
|
||||
Returns:
|
||||
List of Future objects containing SketchResult for each example
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def clean(self) -> None:
|
||||
"""Clean build artifacts for this platform."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def clean_all(self) -> None:
|
||||
"""Clean all build artifacts (local and global) for this platform."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def deploy(
|
||||
self, example: str, upload_port: str | None = None, monitor: bool = False
|
||||
) -> SketchResult:
|
||||
"""Deploy (upload) a specific example to the target device.
|
||||
|
||||
Args:
|
||||
example: Name of the example to deploy
|
||||
upload_port: Optional specific port for upload
|
||||
monitor: If True, attach to device monitor after successful upload
|
||||
|
||||
Returns:
|
||||
SketchResult indicating success/failure of deployment
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def cancel_all(self) -> None:
|
||||
"""Cancel all currently running builds."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def check_usb_permissions(self) -> tuple[bool, str]:
|
||||
"""Check if USB device access is properly configured.
|
||||
|
||||
Returns:
|
||||
Tuple of (has_access, status_message)
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def install_usb_permissions(self) -> bool:
|
||||
"""Install platform-specific USB permissions or equivalent.
|
||||
|
||||
Returns:
|
||||
True if installation succeeded, False otherwise
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_cache_stats(self) -> str:
|
||||
"""Get compiler statistics as a formatted string.
|
||||
|
||||
This can include cache statistics, build metrics, performance data,
|
||||
or any other relevant compiler statistics.
|
||||
|
||||
Returns:
|
||||
Formatted string containing compiler statistics, or empty string if none available
|
||||
"""
|
||||
pass
|
||||
@@ -0,0 +1,993 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false, reportReturnType=false, reportMissingParameterType=false
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from concurrent.futures import Future
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Set, Tuple, Union
|
||||
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
from ci.util.running_process import RunningProcess
|
||||
from ci.util.test_args import parse_args as parse_global_test_args
|
||||
|
||||
from .clang_compiler import (
|
||||
BuildFlags,
|
||||
Compiler,
|
||||
CompilerOptions,
|
||||
LinkOptions,
|
||||
test_clang_accessibility,
|
||||
)
|
||||
from .test_example_compilation import create_fastled_compiler
|
||||
|
||||
|
||||
# Configure logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BUILD_DIR = PROJECT_ROOT / "tests" / ".build"
|
||||
BUILD_DIR.mkdir(parents=True, exist_ok=True)
|
||||
CACHE_DIR = PROJECT_ROOT / ".cache"
|
||||
CACHE_DIR.mkdir(parents=True, exist_ok=True)
|
||||
TEST_FILES_LIST = CACHE_DIR / "test_files_list.txt"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# HASH-BASED LINKING CACHE (same optimization as examples)
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def calculate_file_hash(file_path: Path) -> str:
|
||||
"""Calculate SHA256 hash of a file (same as FastLEDTestCompiler)"""
|
||||
if not file_path.exists():
|
||||
return "no_file"
|
||||
|
||||
hash_sha256 = hashlib.sha256()
|
||||
with open(file_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
hash_sha256.update(chunk)
|
||||
return hash_sha256.hexdigest()
|
||||
|
||||
|
||||
def calculate_linker_args_hash(linker_args: list[str]) -> str:
|
||||
"""Calculate SHA256 hash of linker arguments (same as FastLEDTestCompiler)"""
|
||||
args_str = "|".join(sorted(linker_args))
|
||||
hash_sha256 = hashlib.sha256()
|
||||
hash_sha256.update(args_str.encode("utf-8"))
|
||||
return hash_sha256.hexdigest()
|
||||
|
||||
|
||||
def calculate_link_cache_key(
|
||||
object_files: list[str | Path], fastled_lib_path: Path, linker_args: list[str]
|
||||
) -> str:
|
||||
"""Calculate comprehensive cache key for linking (same as examples)"""
|
||||
# Calculate hash for each object file
|
||||
obj_hashes: list[str] = []
|
||||
for obj_file in object_files:
|
||||
obj_hashes.append(calculate_file_hash(Path(obj_file)))
|
||||
|
||||
# Combine object file hashes in a stable way (sorted by path for consistency)
|
||||
sorted_obj_paths = sorted(str(obj) for obj in object_files)
|
||||
sorted_obj_hashes: list[str] = []
|
||||
for obj_path in sorted_obj_paths:
|
||||
obj_file = Path(obj_path)
|
||||
sorted_obj_hashes.append(calculate_file_hash(obj_file))
|
||||
|
||||
combined_obj_hash = hashlib.sha256(
|
||||
"|".join(sorted_obj_hashes).encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
# Calculate other hashes
|
||||
fastled_hash = calculate_file_hash(fastled_lib_path)
|
||||
linker_hash = calculate_linker_args_hash(linker_args)
|
||||
|
||||
# Combine all components (same format as FastLEDTestCompiler)
|
||||
combined = f"fastled:{fastled_hash}|objects:{combined_obj_hash}|flags:{linker_hash}"
|
||||
final_hash = hashlib.sha256(combined.encode("utf-8")).hexdigest()
|
||||
|
||||
return final_hash[:16] # Use first 16 chars for readability
|
||||
|
||||
|
||||
def get_link_cache_dir() -> Path:
|
||||
"""Get the link cache directory (same as examples)"""
|
||||
cache_dir = Path(".build/link_cache")
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_dir
|
||||
|
||||
|
||||
def get_cached_executable(test_name: str, cache_key: str) -> Optional[Path]:
|
||||
"""Check if cached executable exists (same as examples)"""
|
||||
cache_dir = get_link_cache_dir()
|
||||
cached_exe = cache_dir / f"{test_name}_{cache_key}.exe"
|
||||
return cached_exe if cached_exe.exists() else None
|
||||
|
||||
|
||||
def cache_executable(test_name: str, cache_key: str, exe_path: Path) -> None:
|
||||
"""Cache an executable for future use (same as examples)"""
|
||||
cache_dir = get_link_cache_dir()
|
||||
cached_exe = cache_dir / f"{test_name}_{cache_key}.exe"
|
||||
try:
|
||||
shutil.copy2(exe_path, cached_exe)
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to cache {test_name}: {e}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# END HASH-BASED LINKING CACHE
|
||||
# ============================================================================
|
||||
|
||||
|
||||
def get_test_files() -> Set[str]:
|
||||
"""Get a set of all test files"""
|
||||
test_files: Set[str] = set()
|
||||
tests_dir = PROJECT_ROOT / "tests"
|
||||
if tests_dir.exists():
|
||||
for file_path in tests_dir.rglob("test_*.cpp"):
|
||||
# Store relative paths for consistency
|
||||
test_files.add(str(file_path.relative_to(PROJECT_ROOT)))
|
||||
return test_files
|
||||
|
||||
|
||||
def check_test_files_changed() -> bool:
|
||||
"""Check if test files have changed since last run"""
|
||||
try:
|
||||
current_files = get_test_files()
|
||||
|
||||
if TEST_FILES_LIST.exists():
|
||||
# Read previous file list
|
||||
with open(TEST_FILES_LIST, "r") as f:
|
||||
previous_files = set(line.strip() for line in f if line.strip())
|
||||
|
||||
# Compare file sets
|
||||
if current_files == previous_files:
|
||||
return False # No changes
|
||||
else:
|
||||
print("Test files have changed, cleaning build directory...")
|
||||
return True # Files changed
|
||||
else:
|
||||
# No previous file list, need to clean
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Warning: Error checking test file changes: {e}")
|
||||
return True # Default to cleaning on error
|
||||
|
||||
|
||||
def save_test_files_list() -> None:
|
||||
"""Save current test file list"""
|
||||
try:
|
||||
current_files = get_test_files()
|
||||
with open(TEST_FILES_LIST, "w") as f:
|
||||
for file_path in sorted(current_files):
|
||||
f.write(f"{file_path}\n")
|
||||
except KeyboardInterrupt:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to save test file list: {e}")
|
||||
|
||||
|
||||
def clean_build_directory():
|
||||
print("Cleaning build directory...")
|
||||
shutil.rmtree(BUILD_DIR, ignore_errors=True)
|
||||
BUILD_DIR.mkdir(parents=True, exist_ok=True)
|
||||
print("Build directory cleaned.")
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
WASM_BUILD = False
|
||||
USE_ZIG = False
|
||||
USE_CLANG = False
|
||||
|
||||
|
||||
# Legacy CMake helper functions removed - using optimized Python API
|
||||
|
||||
|
||||
def get_unit_test_fastled_sources() -> list[Path]:
|
||||
"""Get essential FastLED .cpp files for unit test library creation (optimized paradigm)."""
|
||||
# Always work from project root
|
||||
project_root = Path(
|
||||
__file__
|
||||
).parent.parent.parent # Go up from ci/compiler/ to project root
|
||||
src_dir = project_root / "src"
|
||||
|
||||
# Core FastLED files that must be included for unit tests
|
||||
core_files: list[Path] = [
|
||||
src_dir / "FastLED.cpp",
|
||||
src_dir / "colorutils.cpp",
|
||||
src_dir / "hsv2rgb.cpp",
|
||||
]
|
||||
|
||||
# Find all .cpp files in key directories
|
||||
additional_sources: list[Path] = []
|
||||
for pattern in ["*.cpp", "lib8tion/*.cpp", "platforms/stub/*.cpp"]:
|
||||
additional_sources.extend(list(src_dir.glob(pattern)))
|
||||
|
||||
# Include essential .cpp files from nested directories
|
||||
additional_sources.extend(list(src_dir.rglob("*.cpp")))
|
||||
|
||||
# Filter out duplicates and ensure files exist
|
||||
all_sources: list[Path] = []
|
||||
seen_files: set[Path] = set()
|
||||
|
||||
for cpp_file in core_files + additional_sources:
|
||||
# Skip stub_main.cpp since unit tests have their own main
|
||||
if cpp_file.name == "stub_main.cpp":
|
||||
continue
|
||||
|
||||
# Skip platform-specific files that aren't needed for unit tests
|
||||
rel_path_str = str(cpp_file)
|
||||
if any(
|
||||
skip in rel_path_str for skip in ["wasm", "esp", "avr", "arm", "teensy"]
|
||||
):
|
||||
continue
|
||||
|
||||
if cpp_file.exists() and cpp_file not in seen_files:
|
||||
all_sources.append(cpp_file)
|
||||
seen_files.add(cpp_file)
|
||||
|
||||
return all_sources
|
||||
|
||||
|
||||
def create_unit_test_fastled_library(
|
||||
clean: bool = False, use_pch: bool = True
|
||||
) -> Path | None:
|
||||
"""Create libfastled.a static library specifically for unit tests with FASTLED_FORCE_NAMESPACE=1.
|
||||
|
||||
CRITICAL: Unit tests need their own separate library from examples because they use
|
||||
different compilation flags. Unit tests require FASTLED_FORCE_NAMESPACE=1 to put
|
||||
all symbols in the fl:: namespace that the tests expect.
|
||||
"""
|
||||
|
||||
# Unit tests get their own separate library directory under .build/fastled/unit/
|
||||
fastled_build_dir = BUILD_DIR.parent / ".build" / "fastled" / "unit"
|
||||
fastled_build_dir.mkdir(parents=True, exist_ok=True)
|
||||
lib_file = fastled_build_dir / "libfastled.a"
|
||||
|
||||
if lib_file.exists() and not clean:
|
||||
print(f"[LIBRARY] Using existing FastLED library: {lib_file}")
|
||||
return lib_file
|
||||
|
||||
print("[LIBRARY] Creating FastLED static library with proper compiler flags...")
|
||||
|
||||
# Create a proper FastLED compiler (NOT unit test compiler) for the library
|
||||
# This ensures the library is compiled without FASTLED_FORCE_NAMESPACE=1
|
||||
print("[LIBRARY] Creating FastLED library compiler (without unit test flags)...")
|
||||
|
||||
# Save current directory and change to project root for create_fastled_compiler
|
||||
import os
|
||||
|
||||
project_root = Path(__file__).parent.parent.parent
|
||||
original_cwd = os.getcwd()
|
||||
os.chdir(str(project_root))
|
||||
|
||||
try:
|
||||
library_compiler = create_fastled_compiler(
|
||||
use_pch=use_pch,
|
||||
parallel=True,
|
||||
)
|
||||
|
||||
# CRITICAL: Add required defines for unit test library compilation
|
||||
if library_compiler.settings.defines is None:
|
||||
library_compiler.settings.defines = []
|
||||
|
||||
# Add FASTLED_FORCE_NAMESPACE=1 to export symbols in fl:: namespace
|
||||
library_compiler.settings.defines.append("FASTLED_FORCE_NAMESPACE=1")
|
||||
print("[LIBRARY] Added FASTLED_FORCE_NAMESPACE=1 to library compiler")
|
||||
|
||||
# Add FASTLED_TESTING=1 to include MockTimeProvider and test utility functions
|
||||
library_compiler.settings.defines.append("FASTLED_TESTING=1")
|
||||
print("[LIBRARY] Added FASTLED_TESTING=1 to library compiler")
|
||||
|
||||
finally:
|
||||
# Restore original working directory
|
||||
os.chdir(original_cwd)
|
||||
|
||||
# Get FastLED sources using optimized selection
|
||||
fastled_sources = get_unit_test_fastled_sources()
|
||||
fastled_objects: list[Path] = []
|
||||
|
||||
obj_dir = fastled_build_dir / "obj"
|
||||
obj_dir.mkdir(exist_ok=True)
|
||||
|
||||
print(f"[LIBRARY] Compiling {len(fastled_sources)} FastLED source files...")
|
||||
|
||||
# Compile each source file with optimized naming (same as examples)
|
||||
futures: List[Tuple[Any, ...]] = []
|
||||
project_root = Path(__file__).parent.parent.parent
|
||||
src_dir = project_root / "src"
|
||||
|
||||
for cpp_file in fastled_sources:
|
||||
# Create unique object file name by including relative path to prevent collisions
|
||||
# Convert path separators to underscores to create valid filename
|
||||
if cpp_file.is_relative_to(src_dir):
|
||||
rel_path = cpp_file.relative_to(src_dir)
|
||||
else:
|
||||
rel_path = cpp_file
|
||||
|
||||
# Replace path separators with underscores for unique object file names
|
||||
obj_name = str(rel_path.with_suffix(".o")).replace("/", "_").replace("\\", "_")
|
||||
obj_file = obj_dir / obj_name
|
||||
future = library_compiler.compile_cpp_file(cpp_file, obj_file)
|
||||
futures.append((future, obj_file, cpp_file))
|
||||
|
||||
# Wait for compilation to complete
|
||||
compiled_count = 0
|
||||
for future, obj_file, cpp_file in futures:
|
||||
try:
|
||||
result = future.result()
|
||||
if result.ok:
|
||||
fastled_objects.append(obj_file)
|
||||
compiled_count += 1
|
||||
else:
|
||||
print(
|
||||
f"[LIBRARY] WARNING: Failed to compile {cpp_file.relative_to(src_dir)}: {result.stderr[:100]}..."
|
||||
)
|
||||
except Exception as e:
|
||||
print(
|
||||
f"[LIBRARY] WARNING: Exception compiling {cpp_file.relative_to(src_dir)}: {e}"
|
||||
)
|
||||
|
||||
print(
|
||||
f"[LIBRARY] Successfully compiled {compiled_count}/{len(fastled_sources)} FastLED sources"
|
||||
)
|
||||
|
||||
if not fastled_objects:
|
||||
print("[LIBRARY] ERROR: No FastLED source files compiled successfully")
|
||||
return None
|
||||
|
||||
# Create static library using the same approach as examples
|
||||
print(f"[LIBRARY] Creating static library: {lib_file}")
|
||||
|
||||
archive_future = library_compiler.create_archive(fastled_objects, lib_file)
|
||||
archive_result = archive_future.result()
|
||||
|
||||
if not archive_result.ok:
|
||||
print(f"[LIBRARY] ERROR: Library creation failed: {archive_result.stderr}")
|
||||
return None
|
||||
|
||||
print(f"[LIBRARY] SUCCESS: FastLED library created: {lib_file}")
|
||||
return lib_file
|
||||
|
||||
|
||||
def create_unit_test_compiler(
|
||||
use_pch: bool = True, enable_static_analysis: bool = False, debug: bool = False
|
||||
) -> Compiler:
|
||||
"""Create compiler optimized for unit test compilation with PCH support."""
|
||||
|
||||
# Always work from the project root, not from ci/compiler
|
||||
project_root = Path(
|
||||
__file__
|
||||
).parent.parent.parent # Go up from ci/compiler/ to project root
|
||||
current_dir = project_root
|
||||
src_path = current_dir / "src"
|
||||
|
||||
# Load build flags configuration
|
||||
build_flags_path = current_dir / "ci" / "build_unit.toml"
|
||||
build_flags = BuildFlags.parse(
|
||||
build_flags_path, quick_build=True, strict_mode=False
|
||||
)
|
||||
|
||||
# Unit test specific defines
|
||||
unit_test_defines = [
|
||||
"FASTLED_UNIT_TEST=1",
|
||||
"FASTLED_FORCE_NAMESPACE=1",
|
||||
"FASTLED_USE_PROGMEM=0",
|
||||
"STUB_PLATFORM",
|
||||
"ARDUINO=10808",
|
||||
"FASTLED_USE_STUB_ARDUINO",
|
||||
"SKETCH_HAS_LOTS_OF_MEMORY=1",
|
||||
"FASTLED_STUB_IMPL",
|
||||
"FASTLED_USE_JSON_UI=1",
|
||||
"FASTLED_TESTING",
|
||||
"FASTLED_NO_AUTO_NAMESPACE",
|
||||
"FASTLED_NO_PINMAP",
|
||||
"HAS_HARDWARE_PIN_SUPPORT",
|
||||
"FASTLED_DEBUG_LEVEL=1",
|
||||
"FASTLED_NO_ATEXIT=1",
|
||||
"DOCTEST_CONFIG_NO_EXCEPTIONS_BUT_WITH_ALL_ASSERTS",
|
||||
"ENABLE_CRASH_HANDLER",
|
||||
"RELEASE=1", # Disable FASTLED_FORCE_DBG to avoid fl::println dependency
|
||||
]
|
||||
|
||||
# Unit test specific compiler args
|
||||
unit_test_args = [
|
||||
"-std=gnu++17",
|
||||
"-fpermissive",
|
||||
"-Wall",
|
||||
"-Wextra",
|
||||
"-Wno-deprecated-register",
|
||||
"-Wno-backslash-newline-escape",
|
||||
"-fno-exceptions",
|
||||
"-fno-rtti",
|
||||
# Optimization/debug controls set below based on debug flag
|
||||
"-fno-omit-frame-pointer",
|
||||
"-fno-inline-functions",
|
||||
"-fno-vectorize",
|
||||
"-fno-unroll-loops",
|
||||
"-fno-strict-aliasing",
|
||||
f"-I{current_dir}",
|
||||
f"-I{src_path}",
|
||||
f"-I{current_dir / 'tests'}",
|
||||
f"-I{src_path / 'platforms' / 'stub'}",
|
||||
]
|
||||
|
||||
# Apply quick vs debug modes
|
||||
if debug:
|
||||
unit_test_args.extend(
|
||||
[
|
||||
"-O0",
|
||||
"-g3",
|
||||
"-fstandalone-debug",
|
||||
]
|
||||
)
|
||||
if os.name == "nt":
|
||||
unit_test_args.extend(
|
||||
["-gdwarf-4"]
|
||||
) # GNU debug info for Windows GNU toolchain
|
||||
else:
|
||||
unit_test_args.extend(
|
||||
[
|
||||
"-O0",
|
||||
"-g0",
|
||||
]
|
||||
)
|
||||
|
||||
# Note: DWARF flags are added above conditionally when debug=True
|
||||
|
||||
# PCH configuration with unit test specific headers
|
||||
pch_output_path = None
|
||||
pch_header_content = None
|
||||
if use_pch:
|
||||
cache_dir = current_dir / ".build" / "cache"
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
pch_output_path = str(cache_dir / "fastled_unit_test_pch.hpp.pch")
|
||||
|
||||
# Unit test specific PCH header content
|
||||
pch_header_content = """// FastLED Unit Test PCH - Common headers for faster test compilation
|
||||
#pragma once
|
||||
|
||||
// Core test framework
|
||||
#include "test.h"
|
||||
|
||||
// Core FastLED headers that are used in nearly all unit tests
|
||||
#include "FastLED.h"
|
||||
|
||||
// Common C++ standard library headers used in tests
|
||||
#include <string>
|
||||
#include <vector>
|
||||
#include <stdio.h>
|
||||
#include <cstdint>
|
||||
#include <cmath>
|
||||
#include <cassert>
|
||||
#include <iostream>
|
||||
#include <memory>
|
||||
|
||||
// Platform headers for stub environment
|
||||
#include "platforms/stub/fastled_stub.h"
|
||||
|
||||
// Commonly tested FastLED components
|
||||
#include "lib8tion.h"
|
||||
#include "colorutils.h"
|
||||
#include "hsv2rgb.h"
|
||||
#include "fl/math.h"
|
||||
#include "fl/vector.h"
|
||||
|
||||
// Using namespace to match test files
|
||||
using namespace fl;
|
||||
"""
|
||||
print(f"[PCH] Unit tests will use precompiled headers: {pch_output_path}")
|
||||
print(
|
||||
f"[PCH] PCH includes: test.h, FastLED.h, lib8tion.h, colorutils.h, and more"
|
||||
)
|
||||
else:
|
||||
print("[PCH] Precompiled headers disabled for unit tests")
|
||||
|
||||
# Determine compiler
|
||||
compiler_cmd = "python -m ziglang c++"
|
||||
if USE_CLANG:
|
||||
compiler_cmd = "clang++"
|
||||
print("USING CLANG COMPILER FOR UNIT TESTS")
|
||||
elif USE_ZIG:
|
||||
print("USING ZIG COMPILER FOR UNIT TESTS")
|
||||
else:
|
||||
print("USING DEFAULT COMPILER FOR UNIT TESTS")
|
||||
|
||||
settings = CompilerOptions(
|
||||
include_path=str(src_path),
|
||||
defines=unit_test_defines,
|
||||
std_version="c++17",
|
||||
compiler=compiler_cmd,
|
||||
compiler_args=unit_test_args,
|
||||
use_pch=use_pch,
|
||||
pch_output_path=pch_output_path,
|
||||
pch_header_content=pch_header_content,
|
||||
parallel=True,
|
||||
)
|
||||
|
||||
return Compiler(settings, build_flags)
|
||||
|
||||
|
||||
def compile_unit_tests_python_api(
|
||||
specific_test: str | None = None,
|
||||
enable_static_analysis: bool = False,
|
||||
use_pch: bool = True,
|
||||
clean: bool = False,
|
||||
debug: bool = False,
|
||||
) -> bool:
|
||||
"""Compile unit tests using the fast Python API instead of CMake.
|
||||
|
||||
Returns:
|
||||
bool: True if all tests compiled and linked successfully, False otherwise
|
||||
"""
|
||||
from .clang_compiler import Compiler, LinkOptions
|
||||
|
||||
print("=" * 60)
|
||||
print("COMPILING UNIT TESTS WITH PYTHON API")
|
||||
print("=" * 60)
|
||||
|
||||
if clean:
|
||||
print("Cleaning build directory...")
|
||||
shutil.rmtree(BUILD_DIR, ignore_errors=True)
|
||||
BUILD_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create optimized compiler for unit tests
|
||||
compiler = create_unit_test_compiler(
|
||||
use_pch=use_pch, enable_static_analysis=enable_static_analysis, debug=debug
|
||||
)
|
||||
|
||||
# Find all test files - work from project root
|
||||
project_root = Path(__file__).parent.parent.parent
|
||||
tests_dir = project_root / "tests"
|
||||
test_files = []
|
||||
|
||||
if specific_test:
|
||||
# Handle specific test with case-insensitive matching
|
||||
test_name = (
|
||||
specific_test
|
||||
if specific_test.startswith("test_")
|
||||
else f"test_{specific_test}"
|
||||
)
|
||||
test_file = tests_dir / f"{test_name}.cpp"
|
||||
|
||||
# First try exact case match
|
||||
if test_file.exists():
|
||||
test_files = [test_file]
|
||||
print(f"Compiling specific test: {test_file.name}")
|
||||
else:
|
||||
# Try case-insensitive matching for all test files
|
||||
found_match = False
|
||||
for existing_file in tests_dir.glob("test_*.cpp"):
|
||||
# Check if the file matches case-insensitively
|
||||
existing_stem = existing_file.stem
|
||||
existing_name = existing_stem.replace("test_", "")
|
||||
|
||||
if (
|
||||
existing_stem.lower() == test_name.lower()
|
||||
or existing_name.lower() == specific_test.lower()
|
||||
):
|
||||
test_files = [existing_file]
|
||||
print(
|
||||
f"Compiling specific test (case-insensitive match): {existing_file.name}"
|
||||
)
|
||||
found_match = True
|
||||
break
|
||||
|
||||
if not found_match:
|
||||
raise RuntimeError(f"Test file not found: {test_file}")
|
||||
else:
|
||||
# Find all test files
|
||||
test_files = list(tests_dir.glob("test_*.cpp"))
|
||||
print(f"Found {len(test_files)} unit test files")
|
||||
|
||||
if not test_files:
|
||||
print("No test files found")
|
||||
return
|
||||
|
||||
# Ensure output directory exists
|
||||
bin_dir = BUILD_DIR / "bin"
|
||||
bin_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create clean execution directory as specified in requirements
|
||||
clean_bin_dir = PROJECT_ROOT / "tests" / "bin"
|
||||
clean_bin_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Step 1: Compile doctest main once
|
||||
print("Compiling doctest main...")
|
||||
doctest_main_path = tests_dir / "doctest_main.cpp"
|
||||
doctest_main_obj = bin_dir / "doctest_main.o"
|
||||
|
||||
if not doctest_main_obj.exists() or clean:
|
||||
doctest_compile_future = compiler.compile_cpp_file(
|
||||
cpp_path=str(doctest_main_path), output_path=str(doctest_main_obj)
|
||||
)
|
||||
doctest_result = doctest_compile_future.result()
|
||||
|
||||
if doctest_result.return_code != 0:
|
||||
raise RuntimeError(
|
||||
f"Failed to compile doctest main: {doctest_result.stderr}"
|
||||
)
|
||||
|
||||
# Step 2: Build FastLED library using optimized examples paradigm
|
||||
print("Building FastLED library...")
|
||||
|
||||
fastled_lib_path = create_unit_test_fastled_library(clean, use_pch=use_pch)
|
||||
|
||||
# Step 3: Compile and link each test (PARALLEL OPTIMIZATION)
|
||||
print(f"Compiling {len(test_files)} tests...")
|
||||
start_time = time.time()
|
||||
|
||||
# Phase 1: Start all compilations in parallel (NON-BLOCKING)
|
||||
print("🚀 Starting parallel compilation...")
|
||||
compile_start = time.time()
|
||||
compile_futures: Dict[str, Future[Any]] = {}
|
||||
test_info: Dict[str, Any] = {}
|
||||
|
||||
for test_file in test_files:
|
||||
test_name = test_file.stem
|
||||
executable_path = bin_dir / test_name
|
||||
object_path = bin_dir / f"{test_name}.o"
|
||||
|
||||
test_info[test_name] = {
|
||||
"test_file": test_file,
|
||||
"executable_path": executable_path,
|
||||
"object_path": object_path,
|
||||
}
|
||||
|
||||
print(f" Compiling {test_name}...")
|
||||
|
||||
# Start compilation (NON-BLOCKING)
|
||||
compile_future = compiler.compile_cpp_file(
|
||||
cpp_path=str(test_file), output_path=str(object_path)
|
||||
)
|
||||
compile_futures[test_name] = compile_future
|
||||
|
||||
# Phase 2: Wait for all compilations to complete and check cache before linking
|
||||
compile_dispatch_time = time.time() - compile_start
|
||||
print(
|
||||
f"⏳ Waiting for compilations to complete... (dispatch took {compile_dispatch_time:.2f}s)"
|
||||
)
|
||||
compile_wait_start = time.time()
|
||||
link_futures: Dict[str, Dict[str, Any]] = {}
|
||||
cache_hits = 0
|
||||
success_count = 0
|
||||
|
||||
for test_name, compile_future in compile_futures.items():
|
||||
try:
|
||||
compile_result = (
|
||||
compile_future.result()
|
||||
) # BLOCKING WAIT for this specific test
|
||||
|
||||
if compile_result.return_code != 0:
|
||||
print(f"❌ Compilation failed for {test_name}: {compile_result.stderr}")
|
||||
continue
|
||||
|
||||
# Prepare linking info (same logic as before)
|
||||
info = test_info[test_name]
|
||||
executable_path = info["executable_path"]
|
||||
object_path = info["object_path"]
|
||||
|
||||
tests_with_own_main = ["test_example_compilation"]
|
||||
if test_name in tests_with_own_main:
|
||||
object_files: list[str | Path] = [object_path]
|
||||
else:
|
||||
object_files: list[str | Path] = [object_path, doctest_main_obj]
|
||||
|
||||
# Platform-specific linker arguments for crash handler support
|
||||
if os.name == "nt": # Windows
|
||||
linker_args = ["-ldbghelp", "-lpsapi"]
|
||||
else: # Linux/macOS
|
||||
linker_args = ["-pthread"]
|
||||
|
||||
# HASH-BASED CACHE CHECK (same as examples)
|
||||
if not fastled_lib_path:
|
||||
print(f"⚠️ No FastLED library found, skipping cache for {test_name}")
|
||||
cache_key = "no_fastled_lib"
|
||||
cached_exe = None
|
||||
else:
|
||||
cache_key = calculate_link_cache_key(
|
||||
object_files, fastled_lib_path, linker_args
|
||||
)
|
||||
cached_exe = get_cached_executable(test_name, cache_key)
|
||||
|
||||
if cached_exe:
|
||||
# Cache hit! Copy cached executable to target location
|
||||
try:
|
||||
shutil.copy2(cached_exe, executable_path)
|
||||
# Also copy to clean execution directory
|
||||
clean_executable_path = clean_bin_dir / f"{test_name}.exe"
|
||||
shutil.copy2(cached_exe, clean_executable_path)
|
||||
cache_hits += 1
|
||||
success_count += 1
|
||||
print(f" ⚡ {test_name}: Using cached executable (cache hit)")
|
||||
continue # Skip linking entirely
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Failed to copy cached {test_name}, will relink: {e}")
|
||||
# Fall through to actual linking
|
||||
|
||||
# Cache miss - proceed with actual linking
|
||||
static_libraries: List[Union[str, Path]] = []
|
||||
if fastled_lib_path and fastled_lib_path.exists():
|
||||
static_libraries.append(fastled_lib_path)
|
||||
|
||||
link_options = LinkOptions(
|
||||
output_executable=str(executable_path),
|
||||
object_files=object_files,
|
||||
static_libraries=static_libraries,
|
||||
linker_args=linker_args,
|
||||
)
|
||||
|
||||
# Start linking (NON-BLOCKING) and store cache info for later
|
||||
link_future = compiler.link_program(link_options)
|
||||
link_futures[test_name] = {
|
||||
"future": link_future,
|
||||
"cache_key": cache_key,
|
||||
"executable_path": executable_path,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ ERROR compiling {test_name}: {e}")
|
||||
continue
|
||||
|
||||
# Phase 3: Wait for all linking to complete and cache successful results
|
||||
compile_wait_time = time.time() - compile_wait_start
|
||||
print(
|
||||
f"🔗 Waiting for linking to complete... (compilation took {compile_wait_time:.2f}s)"
|
||||
)
|
||||
link_start = time.time()
|
||||
cache_misses = 0
|
||||
|
||||
for test_name, link_info in link_futures.items():
|
||||
try:
|
||||
link_future = link_info["future"]
|
||||
cache_key = link_info["cache_key"]
|
||||
executable_path = link_info["executable_path"]
|
||||
|
||||
link_result = link_future.result() # BLOCKING WAIT for this specific test
|
||||
|
||||
if link_result.return_code != 0:
|
||||
print(f"⚠️ Linking failed for {test_name}: {link_result.stderr}")
|
||||
cache_misses += 1
|
||||
continue
|
||||
else:
|
||||
success_count += 1
|
||||
cache_misses += 1 # This was a fresh link, not from cache
|
||||
|
||||
# Cache the successful executable for future use (same as examples)
|
||||
cache_executable(test_name, cache_key, executable_path)
|
||||
|
||||
# Also copy to clean execution directory
|
||||
clean_executable_path = clean_bin_dir / f"{test_name}.exe"
|
||||
shutil.copy2(executable_path, clean_executable_path)
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ ERROR linking {test_name}: {e}")
|
||||
cache_misses += 1
|
||||
continue
|
||||
|
||||
link_time = time.time() - link_start
|
||||
compilation_time = time.time() - start_time
|
||||
|
||||
print(f"✅ Unit test compilation completed in {compilation_time:.2f}s")
|
||||
print(f" 📊 Time breakdown:")
|
||||
print(f" • Dispatch: {compile_dispatch_time:.2f}s")
|
||||
print(f" • Compilation: {compile_wait_time:.2f}s")
|
||||
print(f" • Linking: {link_time:.2f}s")
|
||||
print(f" 🎯 Cache statistics (archive + hash linking optimization):")
|
||||
print(f" • Cache hits: {cache_hits} (skipped linking)")
|
||||
print(f" • Cache misses: {cache_misses} (fresh linking)")
|
||||
print(
|
||||
f" • Cache hit ratio: {cache_hits / max(1, cache_hits + cache_misses) * 100:.1f}%"
|
||||
)
|
||||
print(f" Successfully compiled: {success_count}/{len(test_files)} tests")
|
||||
print(f" Average: {compilation_time / len(test_files):.2f}s per test")
|
||||
print(f" Output directory: {bin_dir}")
|
||||
print(f" Cache directory: {get_link_cache_dir()}")
|
||||
|
||||
# Return success only if ALL tests compiled and linked successfully
|
||||
all_tests_successful = success_count == len(test_files)
|
||||
if not all_tests_successful:
|
||||
failed_count = len(test_files) - success_count
|
||||
print(f"❌ {failed_count} test(s) failed to compile or link")
|
||||
|
||||
return all_tests_successful
|
||||
|
||||
|
||||
def parse_arguments():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compile FastLED library with different compiler options."
|
||||
)
|
||||
parser.add_argument("--use-zig", action="store_true", help="Use Zig compiler")
|
||||
parser.add_argument("--use-clang", action="store_true", help="Use Clang compiler")
|
||||
parser.add_argument("--wasm", action="store_true", help="Build for WebAssembly")
|
||||
parser.add_argument(
|
||||
"--clean",
|
||||
action="store_true",
|
||||
help="Clean the build directory before compiling",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--test",
|
||||
help="Specific test to compile (without test_ prefix)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--check",
|
||||
action="store_true",
|
||||
help="Enable static analysis (IWYU, clang-tidy)",
|
||||
)
|
||||
parser.add_argument("--no-unity", action="store_true", help="Disable unity build")
|
||||
parser.add_argument(
|
||||
"--no-pch", action="store_true", help="Disable precompiled headers (PCH)"
|
||||
)
|
||||
parser.add_argument("--debug", action="store_true", help="Enable debug symbols")
|
||||
|
||||
parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def get_build_info(args: argparse.Namespace) -> dict[str, str | dict[str, str]]:
|
||||
return {
|
||||
"USE_ZIG": str(USE_ZIG),
|
||||
"USE_CLANG": str(USE_CLANG),
|
||||
"WASM_BUILD": str(WASM_BUILD),
|
||||
"CC": os.environ.get("CC", ""),
|
||||
"CXX": os.environ.get("CXX", ""),
|
||||
"AR": os.environ.get("AR", ""),
|
||||
"CFLAGS": os.environ.get("CFLAGS", ""),
|
||||
"CXXFLAGS": os.environ.get("CXXFLAGS", ""),
|
||||
"ARGS": {
|
||||
"use_zig": str(args.use_zig),
|
||||
"use_clang": str(args.use_clang),
|
||||
"wasm": str(args.wasm),
|
||||
"specific_test": str(args.test) if args.test else "all",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def should_clean_build(build_info: dict[str, str | dict[str, str]]) -> bool:
|
||||
build_info_file = BUILD_DIR / "build_info.json"
|
||||
if not build_info_file.exists():
|
||||
return True
|
||||
|
||||
try:
|
||||
with open(build_info_file, "r") as f:
|
||||
old_build_info = json.load(f)
|
||||
except (json.JSONDecodeError, ValueError) as e:
|
||||
# If JSON is corrupted or empty, remove it and force clean build
|
||||
logger.warning(
|
||||
f"Corrupted build_info.json detected ({e}), removing and forcing clean build"
|
||||
)
|
||||
try:
|
||||
build_info_file.unlink()
|
||||
except OSError:
|
||||
pass # Ignore if we can't remove it
|
||||
return True
|
||||
|
||||
# If build parameters have changed, we need to rebuild
|
||||
if old_build_info != build_info:
|
||||
# Check if this is just a change in specific test target
|
||||
old_args_raw = old_build_info.get("ARGS", {})
|
||||
new_args_raw = build_info.get("ARGS", {})
|
||||
|
||||
# Ensure ARGS is a dictionary
|
||||
if not isinstance(old_args_raw, dict) or not isinstance(new_args_raw, dict):
|
||||
return True
|
||||
|
||||
old_args: Dict[str, Any] = old_args_raw # type: ignore
|
||||
new_args: Dict[str, Any] = new_args_raw # type: ignore
|
||||
|
||||
# If only the specific test changed and everything else is the same,
|
||||
# we don't need to clean the build directory
|
||||
old_test = old_args.get("specific_test", "all")
|
||||
new_test = new_args.get("specific_test", "all")
|
||||
|
||||
# Create copies without the specific_test field for comparison
|
||||
old_args_no_test: Dict[str, Any] = {
|
||||
k: v for k, v in old_args.items() if k != "specific_test"
|
||||
}
|
||||
new_args_no_test: Dict[str, Any] = {
|
||||
k: v for k, v in new_args.items() if k != "specific_test"
|
||||
}
|
||||
|
||||
# If only the test target changed, don't clean
|
||||
if (
|
||||
old_args_no_test == new_args_no_test
|
||||
and old_build_info.get("USE_ZIG") == build_info.get("USE_ZIG")
|
||||
and old_build_info.get("USE_CLANG") == build_info.get("USE_CLANG")
|
||||
and old_build_info.get("WASM_BUILD") == build_info.get("WASM_BUILD")
|
||||
and old_build_info.get("CC") == build_info.get("CC")
|
||||
and old_build_info.get("CXX") == build_info.get("CXX")
|
||||
and old_build_info.get("AR") == build_info.get("AR")
|
||||
and old_build_info.get("CFLAGS") == build_info.get("CFLAGS")
|
||||
and old_build_info.get("CXXFLAGS") == build_info.get("CXXFLAGS")
|
||||
):
|
||||
print(
|
||||
f"Build parameters unchanged, only test target changed: {old_test} -> {new_test}"
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def update_build_info(build_info: dict[str, str | dict[str, str]]):
|
||||
build_info_file = BUILD_DIR / "build_info.json"
|
||||
with open(build_info_file, "w") as f:
|
||||
json.dump(build_info, f, indent=2)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
global USE_ZIG, USE_CLANG, WASM_BUILD
|
||||
|
||||
args = parse_arguments()
|
||||
USE_ZIG = args.use_zig # use Zig's clang compiler
|
||||
USE_CLANG = args.use_clang # Use pure Clang for WASM builds
|
||||
WASM_BUILD = args.wasm
|
||||
|
||||
using_gcc = not USE_ZIG and not USE_CLANG and not WASM_BUILD
|
||||
if using_gcc:
|
||||
if not shutil.which("g++"):
|
||||
print(
|
||||
"gcc compiler not found in PATH, falling back zig's built in clang compiler"
|
||||
)
|
||||
USE_ZIG = True
|
||||
USE_CLANG = False
|
||||
|
||||
if USE_CLANG:
|
||||
if not test_clang_accessibility():
|
||||
print(
|
||||
"Clang compiler not found in PATH, falling back to Zig-clang compiler"
|
||||
)
|
||||
USE_ZIG = True
|
||||
USE_CLANG = False
|
||||
|
||||
os.chdir(str(HERE))
|
||||
print(f"Current directory: {Path('.').absolute()}")
|
||||
|
||||
# Auto-detection for --clean based on test file changes
|
||||
need_clean = args.clean
|
||||
if not need_clean:
|
||||
# Only check for changes if --clean wasn't explicitly specified
|
||||
need_clean = check_test_files_changed()
|
||||
|
||||
build_info = get_build_info(args)
|
||||
if need_clean or should_clean_build(build_info):
|
||||
clean_build_directory()
|
||||
# Save the file list after cleaning
|
||||
save_test_files_list()
|
||||
elif args.clean:
|
||||
# If --clean was explicitly specified but not needed according to build info,
|
||||
# still clean and save file list
|
||||
clean_build_directory()
|
||||
save_test_files_list()
|
||||
|
||||
# Unit tests now use optimized Python API by default (same as examples)
|
||||
use_pch = not getattr(
|
||||
args, "no_pch", False
|
||||
) # Default to PCH enabled unless --no-pch specified
|
||||
|
||||
# Use the optimized Python API with PCH optimization (now default for unit tests)
|
||||
compilation_successful = compile_unit_tests_python_api(
|
||||
specific_test=args.test,
|
||||
enable_static_analysis=args.check,
|
||||
use_pch=use_pch,
|
||||
clean=need_clean,
|
||||
debug=args.debug,
|
||||
)
|
||||
|
||||
if compilation_successful:
|
||||
update_build_info(build_info)
|
||||
print("FastLED library compiled successfully.")
|
||||
else:
|
||||
print("❌ FastLED unit test compilation failed!")
|
||||
import sys
|
||||
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,906 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false, reportMissingParameterType=false
|
||||
import argparse
|
||||
import multiprocessing
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time # Added for timing test execution
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from queue import Empty, PriorityQueue
|
||||
from threading import Event, Lock, Thread
|
||||
from typing import List
|
||||
|
||||
import psutil
|
||||
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
|
||||
|
||||
def optimize_python_command(cmd: list[str]) -> list[str]:
|
||||
"""
|
||||
Optimize command list for subprocess execution in uv environment.
|
||||
|
||||
For python commands, we need to use 'uv run python' to ensure access to
|
||||
installed packages like ziglang. Direct sys.executable bypasses uv environment.
|
||||
|
||||
Args:
|
||||
cmd: Command list that may contain 'python' as first element
|
||||
|
||||
Returns:
|
||||
list[str]: Optimized command with 'python' prefixed by 'uv run'
|
||||
"""
|
||||
if cmd and (cmd[0] == "python" or cmd[0] == "python3"):
|
||||
# Use uv run python to ensure access to uv-managed packages
|
||||
optimized_cmd = ["uv", "run", "python"] + cmd[1:]
|
||||
return optimized_cmd
|
||||
return cmd
|
||||
|
||||
|
||||
from ci.util.test_exceptions import (
|
||||
CompilationFailedException,
|
||||
TestExecutionFailedException,
|
||||
TestFailureInfo,
|
||||
TestTimeoutException,
|
||||
)
|
||||
|
||||
|
||||
class OutputBuffer:
|
||||
"""Thread-safe output buffer with ordered output display"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.output_queue: PriorityQueue[tuple[int, int, str]] = PriorityQueue()
|
||||
self.next_sequence: int = 0
|
||||
self.sequence_lock: Lock = Lock()
|
||||
self.stop_event: Event = Event()
|
||||
self.output_thread: Thread = Thread(target=self._output_worker, daemon=True)
|
||||
self.output_thread.start()
|
||||
|
||||
def write(self, test_index: int, message: str) -> None:
|
||||
"""Write a message to the buffer with test index for ordering"""
|
||||
with self.sequence_lock:
|
||||
sequence = self.next_sequence
|
||||
self.next_sequence += 1
|
||||
self.output_queue.put((test_index, sequence, message))
|
||||
|
||||
def _output_worker(self) -> None:
|
||||
"""Worker thread that processes output in order"""
|
||||
while not self.stop_event.is_set() or not self.output_queue.empty():
|
||||
try:
|
||||
item: tuple[int, int, str] = self.output_queue.get(timeout=0.1)
|
||||
_, _, message = item
|
||||
print(message, flush=True)
|
||||
self.output_queue.task_done()
|
||||
except Empty:
|
||||
continue
|
||||
except Exception as e:
|
||||
print(f"Error in output worker: {e}")
|
||||
continue
|
||||
|
||||
def stop(self) -> None:
|
||||
"""Stop the output worker thread"""
|
||||
self.stop_event.set()
|
||||
if self.output_thread.is_alive():
|
||||
self.output_thread.join()
|
||||
|
||||
|
||||
# Configure console for UTF-8 output on Windows
|
||||
if os.name == "nt": # Windows
|
||||
# Try to set console to UTF-8 mode
|
||||
try:
|
||||
# Set stdout and stderr to UTF-8 encoding
|
||||
# Note: reconfigure() was added in Python 3.7
|
||||
if hasattr(sys.stdout, "reconfigure") and callable(
|
||||
getattr(sys.stdout, "reconfigure", None)
|
||||
):
|
||||
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
|
||||
if hasattr(sys.stderr, "reconfigure") and callable(
|
||||
getattr(sys.stderr, "reconfigure", None)
|
||||
):
|
||||
sys.stderr.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
|
||||
except (AttributeError, OSError):
|
||||
# Fallback for older Python versions or if reconfigure fails
|
||||
pass
|
||||
|
||||
|
||||
# Environment flags for backward compatibility
|
||||
_SHOW_COMPILE = os.environ.get("FASTLED_TEST_SHOW_COMPILE", "").lower() in (
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
)
|
||||
_SHOW_LINK = os.environ.get("FASTLED_TEST_SHOW_LINK", "").lower() in (
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class FailedTest:
|
||||
name: str
|
||||
return_code: int
|
||||
stdout: str
|
||||
|
||||
|
||||
def check_iwyu_available() -> bool:
|
||||
"""Check if include-what-you-use is available in the system"""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["include-what-you-use", "--version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (
|
||||
subprocess.CalledProcessError,
|
||||
FileNotFoundError,
|
||||
subprocess.TimeoutExpired,
|
||||
):
|
||||
return False
|
||||
|
||||
|
||||
def run_command(
|
||||
command: str | list[str],
|
||||
use_gdb: bool = False,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> tuple[int, str]:
|
||||
captured_lines: list[str] = []
|
||||
|
||||
# Determine command type
|
||||
is_test_execution = False
|
||||
is_compile = False
|
||||
is_link = False
|
||||
if isinstance(command, str):
|
||||
cmd_lower = command.replace("\\", "/").lower()
|
||||
# Check if running test executable
|
||||
is_test_execution = (
|
||||
"/test_" in cmd_lower
|
||||
or ".build/bin/test_" in cmd_lower
|
||||
or cmd_lower.endswith(".exe")
|
||||
)
|
||||
# Check if compiling
|
||||
is_compile = "-c" in cmd_lower and (".cpp" in cmd_lower or ".c" in cmd_lower)
|
||||
# Check if linking
|
||||
is_link = (
|
||||
not is_compile
|
||||
and not is_test_execution
|
||||
and ("-o" in cmd_lower or "lib" in cmd_lower)
|
||||
)
|
||||
|
||||
if use_gdb:
|
||||
with tempfile.NamedTemporaryFile(mode="w+", delete=False) as gdb_script:
|
||||
gdb_script.write("set pagination off\n")
|
||||
gdb_script.write("run\n")
|
||||
gdb_script.write("bt full\n")
|
||||
gdb_script.write("info registers\n")
|
||||
gdb_script.write("x/16i $pc\n")
|
||||
gdb_script.write("thread apply all bt full\n")
|
||||
gdb_script.write("quit\n")
|
||||
|
||||
gdb_command = (
|
||||
f"gdb -return-child-result -batch -x {gdb_script.name} --args {command}"
|
||||
)
|
||||
process = subprocess.Popen(
|
||||
gdb_command,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, # Merge stderr into stdout
|
||||
shell=True,
|
||||
text=False,
|
||||
)
|
||||
assert process.stdout is not None
|
||||
# Stream and capture output
|
||||
while True:
|
||||
line_bytes = process.stdout.readline()
|
||||
line = line_bytes.decode("utf-8", errors="ignore")
|
||||
if not line and process.poll() is not None:
|
||||
break
|
||||
if line:
|
||||
captured_lines.append(line.rstrip())
|
||||
# Always print GDB output (it's only used for crashes anyway)
|
||||
try:
|
||||
print(line, end="", flush=True)
|
||||
except UnicodeEncodeError:
|
||||
# Fallback: replace problematic characters
|
||||
print(
|
||||
line.encode("utf-8", errors="replace").decode(
|
||||
"utf-8", errors="replace"
|
||||
),
|
||||
end="",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
os.unlink(gdb_script.name)
|
||||
output = "\n".join(captured_lines)
|
||||
return process.returncode, output
|
||||
else:
|
||||
# Optimize list commands to avoid shell overhead
|
||||
if isinstance(command, list):
|
||||
# Optimize python commands and use shell=False for better performance
|
||||
python_exe = optimize_python_command(command)
|
||||
process = subprocess.Popen(
|
||||
python_exe,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, # Merge stderr into stdout
|
||||
shell=False, # Use shell=False for better performance with list commands
|
||||
text=False,
|
||||
)
|
||||
else:
|
||||
# String commands still need shell=True
|
||||
process = subprocess.Popen(
|
||||
command,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, # Merge stderr into stdout
|
||||
shell=True,
|
||||
text=False,
|
||||
)
|
||||
assert process.stdout is not None
|
||||
# Stream and capture output
|
||||
while True:
|
||||
line_bytes = process.stdout.readline()
|
||||
line = line_bytes.decode("utf-8", errors="ignore")
|
||||
if not line and process.poll() is not None:
|
||||
break
|
||||
if line:
|
||||
captured_lines.append(line.rstrip())
|
||||
# Determine if we should print this line
|
||||
should_print = (
|
||||
verbose # Always print in verbose mode
|
||||
or (is_compile and show_compile) # Print compilation if enabled
|
||||
or (is_link and show_link) # Print linking if enabled
|
||||
or (not is_test_execution) # Print non-test output
|
||||
or (
|
||||
is_test_execution and process.returncode != 0
|
||||
) # Print failed test output
|
||||
or (
|
||||
is_test_execution
|
||||
and any(
|
||||
marker in line
|
||||
for marker in [
|
||||
"Running test:",
|
||||
"Test passed",
|
||||
"Test FAILED",
|
||||
"passed with return code",
|
||||
"Test output:",
|
||||
]
|
||||
)
|
||||
) # Print test status
|
||||
)
|
||||
if should_print:
|
||||
try:
|
||||
# Add prefix for compile/link commands
|
||||
if is_compile and show_compile:
|
||||
print("[COMPILE] ", end="", flush=True)
|
||||
elif is_link and show_link:
|
||||
print("[LINK] ", end="", flush=True)
|
||||
print(line, end="", flush=True)
|
||||
except UnicodeEncodeError:
|
||||
# Fallback: replace problematic characters
|
||||
print(
|
||||
line.encode("utf-8", errors="replace").decode(
|
||||
"utf-8", errors="replace"
|
||||
),
|
||||
end="",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
output = "\n".join(captured_lines)
|
||||
return process.returncode, output
|
||||
|
||||
|
||||
def compile_tests(
|
||||
clean: bool = False,
|
||||
unknown_args: list[str] = [],
|
||||
specific_test: str | None = None,
|
||||
quick_build: bool = True,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Compile C++ tests using the Python build system.
|
||||
"""
|
||||
os.chdir(str(PROJECT_ROOT))
|
||||
print("🔧 Compiling tests using Python build system")
|
||||
|
||||
_compile_tests_python(
|
||||
clean,
|
||||
unknown_args,
|
||||
specific_test,
|
||||
quick_build=quick_build,
|
||||
verbose=verbose,
|
||||
show_compile=show_compile,
|
||||
show_link=show_link,
|
||||
)
|
||||
|
||||
|
||||
def _compile_tests_python(
|
||||
clean: bool = False,
|
||||
unknown_args: list[str] = [],
|
||||
specific_test: str | None = None,
|
||||
quick_build: bool = True,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""Python build system with PCH optimization"""
|
||||
# Use the optimized cpp_test_compile system directly
|
||||
import subprocess
|
||||
|
||||
cmd = ["uv", "run", "python", "-m", "ci.compiler.cpp_test_compile"]
|
||||
|
||||
if specific_test:
|
||||
cmd.extend(["--test", specific_test])
|
||||
if clean:
|
||||
cmd.append("--clean")
|
||||
if verbose:
|
||||
cmd.append("--verbose")
|
||||
if "--check" in unknown_args:
|
||||
cmd.append("--check")
|
||||
if "--no-pch" in unknown_args:
|
||||
cmd.append("--no-pch")
|
||||
# Forward debug mode to the compiler when quick_build is disabled
|
||||
if not quick_build:
|
||||
cmd.append("--debug")
|
||||
|
||||
print("🚀 Using Python build system with PCH optimization")
|
||||
result = subprocess.run(cmd)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"Unit test compilation failed with return code {result.returncode}"
|
||||
)
|
||||
|
||||
|
||||
def run_tests(
|
||||
specific_test: str | None = None,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Run compiled tests with GDB crash analysis support.
|
||||
"""
|
||||
_run_tests_python(specific_test)
|
||||
|
||||
|
||||
def _run_tests_python(
|
||||
specific_test: str | None = None,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""Run tests from Python build system"""
|
||||
# Import the test compiler system
|
||||
from ci.compiler.test_compiler import FastLEDTestCompiler
|
||||
|
||||
# Get test executables from Python build system
|
||||
test_compiler = FastLEDTestCompiler.get_existing_instance()
|
||||
if not test_compiler:
|
||||
print("No compiled tests found. Run compilation first.")
|
||||
sys.exit(1)
|
||||
|
||||
test_executables = test_compiler.get_test_executables(specific_test)
|
||||
if not test_executables:
|
||||
test_name = specific_test or "any tests"
|
||||
print(f"No test executables found for: {test_name}")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"Running {len(test_executables)} tests from Python build...")
|
||||
|
||||
# Print list of tests that will be executed
|
||||
print("Tests to execute:")
|
||||
for i, test_exec in enumerate(test_executables, 1):
|
||||
# Convert absolute path to relative for display
|
||||
rel_path = os.path.relpath(test_exec.executable_path)
|
||||
print(f" {i}. {test_exec.name} ({rel_path})")
|
||||
print("")
|
||||
|
||||
failed_tests: list[FailedTest] = []
|
||||
|
||||
# Convert to file list format for compatibility with existing logic
|
||||
files: list[str] = []
|
||||
test_paths: dict[str, str] = {}
|
||||
for test_exec in test_executables:
|
||||
file_name = test_exec.name
|
||||
if os.name == "nt" and not file_name.endswith(".exe"):
|
||||
file_name += ".exe"
|
||||
files.append(file_name)
|
||||
test_paths[file_name] = str(test_exec.executable_path)
|
||||
|
||||
print(f"Starting test execution for {len(files)} test files...")
|
||||
_execute_test_files(files, "", failed_tests, specific_test, test_paths)
|
||||
_handle_test_results(failed_tests)
|
||||
|
||||
|
||||
def _execute_test_files(
|
||||
files: list[str],
|
||||
test_dir: str,
|
||||
failed_tests: list[FailedTest],
|
||||
specific_test: str | None,
|
||||
test_paths: dict[str, str] | None = None,
|
||||
*,
|
||||
verbose: bool = False,
|
||||
show_compile: bool = False,
|
||||
show_link: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Execute test files in parallel with full GDB crash analysis.
|
||||
|
||||
Args:
|
||||
files: List of test file names
|
||||
test_dir: Directory containing tests (for CMake) or empty string (for Python API)
|
||||
failed_tests: List to collect failed tests
|
||||
specific_test: Specific test name if filtering
|
||||
test_paths: Dict mapping file names to full paths (for Python API)
|
||||
"""
|
||||
total_tests = len(files)
|
||||
successful_tests = 0
|
||||
completed_tests = 0
|
||||
|
||||
# Initialize output buffer for ordered output
|
||||
output_buffer = OutputBuffer()
|
||||
output_buffer.write(0, f"Executing {total_tests} test files in parallel...")
|
||||
|
||||
# Determine number of workers based on configuration
|
||||
|
||||
# Get configuration from args
|
||||
args = parse_args()
|
||||
|
||||
# Force sequential execution if NO_PARALLEL is set
|
||||
if os.environ.get("NO_PARALLEL"):
|
||||
max_workers = 1
|
||||
output_buffer.write(
|
||||
0, "NO_PARALLEL environment variable set - forcing sequential execution"
|
||||
)
|
||||
elif args.sequential:
|
||||
max_workers = 1
|
||||
elif args.parallel:
|
||||
max_workers = args.parallel
|
||||
else:
|
||||
max_workers = max(1, multiprocessing.cpu_count() - 1) # Leave one core free
|
||||
|
||||
# Check memory limit
|
||||
if args.max_memory:
|
||||
memory_limit = args.max_memory * 1024 * 1024 # Convert MB to bytes
|
||||
available_memory = psutil.virtual_memory().available
|
||||
if memory_limit > available_memory:
|
||||
output_buffer.write(
|
||||
0,
|
||||
f"Warning: Requested memory limit {args.max_memory}MB exceeds available memory {available_memory / (1024 * 1024):.0f}MB",
|
||||
)
|
||||
output_buffer.write(
|
||||
0, "Reducing number of parallel workers to stay within memory limits"
|
||||
)
|
||||
# Estimate memory per test based on previous runs or default to 100MB
|
||||
memory_per_test = 100 * 1024 * 1024 # 100MB per test
|
||||
max_parallel_by_memory = max(1, memory_limit // memory_per_test)
|
||||
max_workers = min(max_workers, max_parallel_by_memory)
|
||||
|
||||
output_buffer.write(0, f"Using {max_workers} parallel workers")
|
||||
|
||||
# Thread-safe counter
|
||||
counter_lock = Lock()
|
||||
|
||||
def run_single_test(
|
||||
test_file: str, test_index: int
|
||||
) -> tuple[bool, float, str, int]:
|
||||
"""Run a single test and return its results"""
|
||||
nonlocal completed_tests
|
||||
|
||||
if test_paths:
|
||||
test_path = test_paths[test_file]
|
||||
else:
|
||||
test_path = os.path.join(test_dir, test_file)
|
||||
|
||||
# For .cpp files, compile them first
|
||||
if test_path.endswith(".cpp"):
|
||||
# Create a temporary directory for compilation
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Compile the test file
|
||||
output_buffer.write(
|
||||
test_index,
|
||||
f"[{test_index}/{total_tests}] Compiling test: {test_file}",
|
||||
)
|
||||
compile_cmd = [
|
||||
"python",
|
||||
"-m",
|
||||
"ziglang",
|
||||
"c++",
|
||||
"-o",
|
||||
os.path.join(temp_dir, "test.exe"),
|
||||
test_path,
|
||||
"-I",
|
||||
os.path.join(PROJECT_ROOT, "src"),
|
||||
"-I",
|
||||
os.path.join(PROJECT_ROOT, "tests"),
|
||||
# NOTE: Compiler flags now come from build configuration TOML
|
||||
]
|
||||
return_code, stdout = run_command(compile_cmd)
|
||||
if return_code != 0:
|
||||
output_buffer.write(
|
||||
test_index,
|
||||
f"[{test_index}/{total_tests}] ERROR: Failed to compile test: {test_file}",
|
||||
)
|
||||
return False, 0.0, f"Failed to compile test: {stdout}", return_code
|
||||
|
||||
# Update test_path to point to the compiled executable
|
||||
test_path = os.path.join(temp_dir, "test.exe")
|
||||
|
||||
if not (os.path.isfile(test_path) and os.access(test_path, os.X_OK)):
|
||||
output_buffer.write(
|
||||
test_index,
|
||||
f"[{test_index}/{total_tests}] ERROR: Test file not found or not executable: {test_path}",
|
||||
)
|
||||
return False, 0.0, f"Test file not found or not executable: {test_path}", 1
|
||||
|
||||
output_buffer.write(
|
||||
test_index, f"[{test_index}/{total_tests}] Running test: {test_file}"
|
||||
)
|
||||
if verbose:
|
||||
output_buffer.write(test_index, f" Command: {test_path}")
|
||||
|
||||
start_time = time.time()
|
||||
# Pass --minimal flag to doctest when not in verbose mode to suppress output unless tests fail
|
||||
cmd = [test_path]
|
||||
if not verbose:
|
||||
cmd.append("--minimal")
|
||||
return_code, stdout = run_command(cmd)
|
||||
elapsed_time = time.time() - start_time
|
||||
|
||||
output = stdout
|
||||
failure_pattern = re.compile(r"Test .+ failed with return code (\d+)")
|
||||
failure_match = failure_pattern.search(output)
|
||||
is_crash = failure_match is not None
|
||||
|
||||
# Handle crashes with GDB (must be done synchronously)
|
||||
if is_crash:
|
||||
output_buffer.write(
|
||||
test_index, f"Test crashed. Re-running with GDB to get stack trace..."
|
||||
)
|
||||
_, gdb_stdout = run_command(test_path, use_gdb=True)
|
||||
stdout += "\n--- GDB Output ---\n" + gdb_stdout
|
||||
|
||||
# Extract crash information
|
||||
crash_info = extract_crash_info(gdb_stdout)
|
||||
output_buffer.write(
|
||||
test_index, f"Crash occurred at: {crash_info.file}:{crash_info.line}"
|
||||
)
|
||||
output_buffer.write(test_index, f"Cause: {crash_info.cause}")
|
||||
output_buffer.write(test_index, f"Stack: {crash_info.stack}")
|
||||
|
||||
# Print output based on verbosity and status
|
||||
if verbose or return_code != 0:
|
||||
output_buffer.write(test_index, "Test output:")
|
||||
output_buffer.write(test_index, stdout)
|
||||
|
||||
if return_code == 0:
|
||||
output_buffer.write(
|
||||
test_index, f" Test {test_file} passed in {elapsed_time:.2f}s"
|
||||
)
|
||||
else:
|
||||
output_buffer.write(
|
||||
test_index,
|
||||
f" Test {test_file} FAILED with return code {return_code} in {elapsed_time:.2f}s",
|
||||
)
|
||||
|
||||
with counter_lock:
|
||||
completed_tests += 1
|
||||
|
||||
return return_code == 0, elapsed_time, stdout, return_code
|
||||
|
||||
try:
|
||||
# Run tests in parallel using ThreadPoolExecutor
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
# Submit all tests
|
||||
future_to_test = {
|
||||
executor.submit(run_single_test, test_file, i + 1): test_file
|
||||
for i, test_file in enumerate(files)
|
||||
}
|
||||
|
||||
# Process results as they complete
|
||||
for future in as_completed(future_to_test):
|
||||
test_file = future_to_test[future]
|
||||
try:
|
||||
success, _, stdout, return_code = future.result()
|
||||
if success:
|
||||
with counter_lock:
|
||||
successful_tests += 1
|
||||
else:
|
||||
failed_tests.append(
|
||||
FailedTest(
|
||||
name=test_file, return_code=return_code, stdout=stdout
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
output_buffer.write(
|
||||
0, f"ERROR: Test {test_file} failed with exception: {e}"
|
||||
)
|
||||
failed_tests.append(
|
||||
FailedTest(name=test_file, return_code=1, stdout=str(e))
|
||||
)
|
||||
|
||||
# Print final summary
|
||||
output_buffer.write(
|
||||
0,
|
||||
f"Test execution complete: {successful_tests} passed, {len(failed_tests)} failed",
|
||||
)
|
||||
if successful_tests == total_tests:
|
||||
output_buffer.write(0, "All tests passed successfully!")
|
||||
else:
|
||||
output_buffer.write(
|
||||
0, f"Some tests failed ({len(failed_tests)} of {total_tests})"
|
||||
)
|
||||
|
||||
finally:
|
||||
# Ensure output buffer is stopped
|
||||
output_buffer.stop()
|
||||
|
||||
|
||||
def _handle_test_results(
|
||||
failed_tests: list[FailedTest], *, verbose: bool = False
|
||||
) -> None:
|
||||
"""Handle test results and exit appropriately (preserving existing logic)"""
|
||||
if failed_tests:
|
||||
print("Failed tests summary:")
|
||||
failures: List[TestFailureInfo] = []
|
||||
for failed_test in failed_tests:
|
||||
print(
|
||||
f"Test {failed_test.name} failed with return code {failed_test.return_code}"
|
||||
)
|
||||
# Always show output on failure
|
||||
print("Output:")
|
||||
# Show indented output for better readability
|
||||
for line in failed_test.stdout.splitlines():
|
||||
print(f" {line}")
|
||||
print() # Add spacing between failed tests
|
||||
|
||||
failures.append(
|
||||
TestFailureInfo(
|
||||
test_name=failed_test.name,
|
||||
command=f"test_{failed_test.name}",
|
||||
return_code=failed_test.return_code,
|
||||
output=failed_test.stdout,
|
||||
error_type="test_execution_failure",
|
||||
)
|
||||
)
|
||||
|
||||
tests_failed = len(failed_tests)
|
||||
failed_test_names = [test.name for test in failed_tests]
|
||||
print(
|
||||
f"{tests_failed} test{'s' if tests_failed != 1 else ''} failed: {', '.join(failed_test_names)}"
|
||||
)
|
||||
raise TestExecutionFailedException(f"{tests_failed} test(s) failed", failures)
|
||||
if verbose:
|
||||
print("All tests passed.")
|
||||
|
||||
|
||||
@dataclass
|
||||
class CrashInfo:
|
||||
cause: str = "Unknown"
|
||||
stack: str = "Unknown"
|
||||
file: str = "Unknown"
|
||||
line: str = "Unknown"
|
||||
|
||||
|
||||
def extract_crash_info(gdb_output: str) -> CrashInfo:
|
||||
lines = gdb_output.split("\n")
|
||||
crash_info = CrashInfo()
|
||||
|
||||
try:
|
||||
for i, line in enumerate(lines):
|
||||
if line.startswith("Program received signal"):
|
||||
try:
|
||||
crash_info.cause = line.split(":", 1)[1].strip()
|
||||
except IndexError:
|
||||
crash_info.cause = line.strip()
|
||||
elif line.startswith("#0"):
|
||||
crash_info.stack = line
|
||||
for j in range(i, len(lines)):
|
||||
if "at" in lines[j]:
|
||||
try:
|
||||
_, location = lines[j].split("at", 1)
|
||||
location = location.strip()
|
||||
if ":" in location:
|
||||
crash_info.file, crash_info.line = location.rsplit(
|
||||
":", 1
|
||||
)
|
||||
else:
|
||||
crash_info.file = location
|
||||
except ValueError:
|
||||
pass # If split fails, we keep the default values
|
||||
break
|
||||
break
|
||||
except Exception as e:
|
||||
print(f"Error parsing GDB output: {e}")
|
||||
|
||||
return crash_info
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Compile and run C++ tests")
|
||||
parser.add_argument(
|
||||
"--compile-only",
|
||||
action="store_true",
|
||||
help="Only compile the tests without running them",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--run-only",
|
||||
action="store_true",
|
||||
help="Only run the tests without compiling them",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--only-run-failed-test",
|
||||
action="store_true",
|
||||
help="Only run the tests that failed in the previous run",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--clean", action="store_true", help="Clean build before compiling"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--test",
|
||||
help="Specific test to run (without extension)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbose",
|
||||
action="store_true",
|
||||
help="Enable verbose output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--show-compile",
|
||||
action="store_true",
|
||||
help="Show compilation commands and output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--show-link",
|
||||
action="store_true",
|
||||
help="Show linking commands and output",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--parallel",
|
||||
type=int,
|
||||
help="Number of parallel test processes to run (default: CPU count - 1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sequential",
|
||||
action="store_true",
|
||||
help="Run tests sequentially (disables parallel execution)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-memory",
|
||||
type=int,
|
||||
help="Maximum memory usage in MB for parallel test execution",
|
||||
)
|
||||
|
||||
# Create mutually exclusive group for compiler selection
|
||||
compiler_group = parser.add_mutually_exclusive_group()
|
||||
compiler_group.add_argument(
|
||||
"--clang",
|
||||
help="Use Clang compiler",
|
||||
action="store_true",
|
||||
)
|
||||
compiler_group.add_argument(
|
||||
"--gcc",
|
||||
help="Use GCC compiler (default on non-Windows)",
|
||||
action="store_true",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--check",
|
||||
action="store_true",
|
||||
help="Enable static analysis (IWYU, clang-tidy)",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--no-unity",
|
||||
action="store_true",
|
||||
help="Disable unity builds for cpp tests",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-pch",
|
||||
action="store_true",
|
||||
help="Disable precompiled headers (PCH) for unit tests",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--debug",
|
||||
action="store_true",
|
||||
help="Use debug build mode with full debug symbols (default is quick mode with -g0)",
|
||||
)
|
||||
|
||||
args, unknown = parser.parse_known_args()
|
||||
args.unknown = unknown
|
||||
|
||||
return args
|
||||
|
||||
|
||||
def main() -> None:
|
||||
try:
|
||||
args = parse_args()
|
||||
|
||||
# Get verbosity flags from args
|
||||
|
||||
run_only = args.run_only
|
||||
compile_only = args.compile_only
|
||||
specific_test = args.test
|
||||
# only_run_failed_test feature to be implemented in future
|
||||
_ = args.only_run_failed_test
|
||||
use_clang = args.clang
|
||||
|
||||
no_unity = args.no_unity
|
||||
quick_build = (
|
||||
not args.debug
|
||||
) # Default to quick mode unless --debug is specified
|
||||
# use_gcc = args.gcc
|
||||
|
||||
if not run_only:
|
||||
passthrough_args = args.unknown
|
||||
if use_clang:
|
||||
passthrough_args.append("--use-clang")
|
||||
if args.check:
|
||||
passthrough_args.append("--check")
|
||||
if no_unity:
|
||||
passthrough_args.append("--no-unity")
|
||||
if args.no_pch:
|
||||
passthrough_args.append("--no-pch")
|
||||
# Note: --gcc is handled by not passing --use-clang (GCC is the default in compiler/cpp_test_compile.py)
|
||||
compile_tests(
|
||||
clean=args.clean,
|
||||
unknown_args=passthrough_args,
|
||||
specific_test=specific_test,
|
||||
quick_build=quick_build,
|
||||
verbose=args.verbose,
|
||||
show_compile=args.show_compile,
|
||||
show_link=args.show_link,
|
||||
)
|
||||
|
||||
if not compile_only:
|
||||
if specific_test:
|
||||
run_tests(
|
||||
specific_test,
|
||||
verbose=args.verbose,
|
||||
show_compile=args.show_compile,
|
||||
show_link=args.show_link,
|
||||
)
|
||||
else:
|
||||
# Use our own test runner instead of CTest since CTest integration is broken
|
||||
run_tests(
|
||||
None,
|
||||
verbose=args.verbose,
|
||||
show_compile=args.show_compile,
|
||||
show_link=args.show_link,
|
||||
)
|
||||
except (
|
||||
CompilationFailedException,
|
||||
TestExecutionFailedException,
|
||||
TestTimeoutException,
|
||||
) as e:
|
||||
# Print detailed failure information
|
||||
print("\n" + "=" * 60)
|
||||
print("FASTLED TEST FAILURE DETAILS")
|
||||
print("=" * 60)
|
||||
print(e.get_detailed_failure_info())
|
||||
print("=" * 60)
|
||||
|
||||
# Exit with appropriate code
|
||||
if e.failures:
|
||||
# Use the return code from the first failure, or 1 if none available
|
||||
exit_code = (
|
||||
e.failures[0].return_code if e.failures[0].return_code != 0 else 1
|
||||
)
|
||||
else:
|
||||
exit_code = 1
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,784 @@
|
||||
"""
|
||||
Enhanced Arduino Package Index Implementation with Pydantic
|
||||
|
||||
This module provides a robust, production-ready Arduino package management system
|
||||
that fully complies with the Arduino CLI Package Index JSON Specification.
|
||||
|
||||
Key Features:
|
||||
- Pydantic models with comprehensive validation
|
||||
- Multi-source package index support
|
||||
- Caching and persistence
|
||||
- Package installation and dependency resolution
|
||||
- Search and filtering capabilities
|
||||
- Checksum validation and error handling
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import sys
|
||||
import tarfile
|
||||
import zipfile
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Optional, Set, Union
|
||||
from urllib.parse import urlparse
|
||||
from urllib.request import urlopen
|
||||
|
||||
import httpx
|
||||
from pydantic import (
|
||||
BaseModel,
|
||||
EmailStr,
|
||||
Field,
|
||||
HttpUrl,
|
||||
ValidationError,
|
||||
field_validator,
|
||||
model_validator,
|
||||
)
|
||||
|
||||
|
||||
# Custom Exceptions
|
||||
class PackageParsingError(Exception):
|
||||
"""Error parsing package index data"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class PackageInstallationError(Exception):
|
||||
"""Error during package installation"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class PackageValidationError(Exception):
|
||||
"""Error validating package data"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
# Core Pydantic Models with Enhanced Validation
|
||||
|
||||
|
||||
class Help(BaseModel):
|
||||
"""Help information with online resources"""
|
||||
|
||||
online: HttpUrl = Field(description="URL to online help resources")
|
||||
|
||||
class Config:
|
||||
schema_extra = {
|
||||
"example": {"online": "https://github.com/espressif/arduino-esp32"}
|
||||
}
|
||||
|
||||
|
||||
class Board(BaseModel):
|
||||
"""Board information with complete properties"""
|
||||
|
||||
name: str = Field(min_length=1, max_length=200, description="Board display name")
|
||||
properties: Dict[str, Any] = Field(
|
||||
default_factory=dict, description="Board configuration properties"
|
||||
)
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def validate_board_name(cls, v: str) -> str:
|
||||
"""Validate board name format"""
|
||||
if not v.strip():
|
||||
raise ValueError("Board name cannot be empty or whitespace")
|
||||
return v.strip()
|
||||
|
||||
class Config:
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"name": "ESP32 Dev Module",
|
||||
"properties": {
|
||||
"upload.tool": "esptool_py",
|
||||
"upload.maximum_size": "1310720",
|
||||
"build.mcu": "esp32",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class ToolDependency(BaseModel):
|
||||
"""Tool dependency specification with validation"""
|
||||
|
||||
packager: str = Field(
|
||||
min_length=1, max_length=100, description="Tool packager name"
|
||||
)
|
||||
name: str = Field(pattern=r"^[a-zA-Z0-9_.-]+$", description="Tool name")
|
||||
version: str = Field(min_length=1, description="Required tool version")
|
||||
|
||||
@field_validator("version")
|
||||
@classmethod
|
||||
def validate_version_format(cls, v: str) -> str:
|
||||
"""Validate version format - supports semantic versioning and Arduino versioning"""
|
||||
if not v.strip():
|
||||
raise ValueError("Version cannot be empty")
|
||||
# Allow flexible versioning (semantic, date-based, etc.)
|
||||
return v.strip()
|
||||
|
||||
class Config:
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"packager": "esp32",
|
||||
"name": "xtensa-esp32-elf-gcc",
|
||||
"version": "esp-2021r2-patch5-8.4.0",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class Platform(BaseModel):
|
||||
"""Platform specification with comprehensive validation"""
|
||||
|
||||
name: str = Field(min_length=1, max_length=200, description="Platform display name")
|
||||
architecture: str = Field(
|
||||
pattern=r"^[a-zA-Z0-9_-]+$", description="Target architecture"
|
||||
)
|
||||
version: str = Field(description="Platform version")
|
||||
category: str = Field(min_length=1, description="Platform category")
|
||||
url: HttpUrl = Field(description="Download URL for platform archive")
|
||||
archive_filename: str = Field(
|
||||
alias="archiveFileName", description="Archive file name"
|
||||
)
|
||||
checksum: str = Field(
|
||||
pattern=r"^SHA-256:[a-fA-F0-9]{64}$", description="SHA-256 checksum"
|
||||
)
|
||||
size_mb: float = Field(gt=0, alias="size", description="Archive size in megabytes")
|
||||
boards: List[Board] = Field(
|
||||
default_factory=lambda: [], description="Supported boards"
|
||||
)
|
||||
tool_dependencies: List[ToolDependency] = Field(
|
||||
default_factory=lambda: [],
|
||||
alias="toolsDependencies",
|
||||
description="Required tool dependencies",
|
||||
)
|
||||
help: Help = Field(description="Help and documentation links")
|
||||
|
||||
@field_validator("version")
|
||||
@classmethod
|
||||
def validate_version_format(cls, v: str) -> str:
|
||||
"""Validate version format"""
|
||||
if not v.strip():
|
||||
raise ValueError("Version cannot be empty")
|
||||
return v.strip()
|
||||
|
||||
@field_validator("size_mb", mode="before")
|
||||
@classmethod
|
||||
def convert_size_from_bytes(cls, v: Union[str, int, float]) -> float:
|
||||
"""Convert size from bytes to megabytes if needed"""
|
||||
if isinstance(v, str):
|
||||
try:
|
||||
size_bytes = int(v)
|
||||
return size_bytes / (1024 * 1024)
|
||||
except ValueError:
|
||||
raise ValueError(f"Invalid size format: {v}")
|
||||
elif isinstance(v, (int, float)):
|
||||
if v > 1024 * 1024: # Assume bytes if > 1MB
|
||||
return v / (1024 * 1024)
|
||||
return v # Already in MB
|
||||
return v
|
||||
|
||||
@field_validator("archive_filename")
|
||||
@classmethod
|
||||
def validate_archive_filename(cls, v: str) -> str:
|
||||
"""Validate archive filename format"""
|
||||
valid_extensions = [".zip", ".tar.gz", ".tar.bz2", ".tar.xz", ".tar.zst"]
|
||||
if not any(v.lower().endswith(ext) for ext in valid_extensions):
|
||||
raise ValueError(
|
||||
f"Archive must have one of these extensions: {valid_extensions}"
|
||||
)
|
||||
return v
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
schema_extra: Dict[str, Any] = {
|
||||
"example": {
|
||||
"name": "ESP32 Arduino",
|
||||
"architecture": "esp32",
|
||||
"version": "2.0.5",
|
||||
"category": "ESP32",
|
||||
"url": "https://github.com/espressif/arduino-esp32/releases/download/2.0.5/esp32-2.0.5.zip",
|
||||
"archiveFileName": "esp32-2.0.5.zip",
|
||||
"checksum": "SHA-256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
|
||||
"size": "50000000",
|
||||
"boards": [],
|
||||
"toolsDependencies": [],
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class SystemDownload(BaseModel):
|
||||
"""System-specific download information for tools"""
|
||||
|
||||
host: str = Field(min_length=1, description="Target host system identifier")
|
||||
url: HttpUrl = Field(description="Download URL for this system")
|
||||
archive_filename: str = Field(
|
||||
alias="archiveFileName", description="Archive file name"
|
||||
)
|
||||
checksum: str = Field(
|
||||
pattern=r"^SHA-256:[a-fA-F0-9]{64}$", description="SHA-256 checksum"
|
||||
)
|
||||
size_mb: float = Field(gt=0, alias="size", description="Archive size in megabytes")
|
||||
|
||||
@field_validator("size_mb", mode="before")
|
||||
@classmethod
|
||||
def convert_size_from_bytes(cls, v: Union[str, int, float]) -> float:
|
||||
"""Convert size from bytes to megabytes if needed"""
|
||||
if isinstance(v, str):
|
||||
try:
|
||||
size_bytes = int(v)
|
||||
return size_bytes / (1024 * 1024)
|
||||
except ValueError:
|
||||
raise ValueError(f"Invalid size format: {v}")
|
||||
elif isinstance(v, (int, float)):
|
||||
if v > 1024 * 1024: # Assume bytes if > 1MB
|
||||
return v / (1024 * 1024)
|
||||
return v # Already in MB
|
||||
return v
|
||||
|
||||
@field_validator("host")
|
||||
@classmethod
|
||||
def validate_host_format(cls, v: str) -> str:
|
||||
"""Validate host system identifier"""
|
||||
# Common host patterns: i686-pc-linux-gnu, x86_64-apple-darwin, etc.
|
||||
if not v.strip():
|
||||
raise ValueError("Host identifier cannot be empty")
|
||||
return v.strip()
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"host": "x86_64-pc-linux-gnu",
|
||||
"url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
|
||||
"archiveFileName": "xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
|
||||
"checksum": "SHA-256:abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890",
|
||||
"size": "150000000",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class Tool(BaseModel):
|
||||
"""Tool with system-specific downloads and enhanced functionality"""
|
||||
|
||||
name: str = Field(pattern=r"^[a-zA-Z0-9_.-]+$", description="Tool name")
|
||||
version: str = Field(min_length=1, description="Tool version")
|
||||
systems: List[SystemDownload] = Field(
|
||||
min_length=1, description="System-specific downloads"
|
||||
)
|
||||
|
||||
@field_validator("systems")
|
||||
@classmethod
|
||||
def validate_unique_systems(cls, v: List[SystemDownload]) -> List[SystemDownload]:
|
||||
"""Ensure no duplicate host systems"""
|
||||
hosts = [system.host for system in v]
|
||||
if len(hosts) != len(set(hosts)):
|
||||
raise ValueError("Duplicate host systems found in tool downloads")
|
||||
return v
|
||||
|
||||
def get_system_download(self, host_pattern: str) -> Optional[SystemDownload]:
|
||||
"""Get system download matching host pattern"""
|
||||
for system in self.systems:
|
||||
if host_pattern in system.host:
|
||||
return system
|
||||
return None
|
||||
|
||||
def get_compatible_systems(self) -> List[str]:
|
||||
"""Get list of compatible host systems"""
|
||||
return [system.host for system in self.systems]
|
||||
|
||||
class Config:
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"name": "xtensa-esp32-elf-gcc",
|
||||
"version": "esp-2021r2-patch5-8.4.0",
|
||||
"systems": [
|
||||
{
|
||||
"host": "x86_64-pc-linux-gnu",
|
||||
"url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
|
||||
"archiveFileName": "xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
|
||||
"checksum": "SHA-256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
|
||||
"size": "150000000",
|
||||
}
|
||||
],
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class Package(BaseModel):
|
||||
"""Package containing platforms and tools with comprehensive validation"""
|
||||
|
||||
name: str = Field(pattern=r"^[A-Za-z0-9_.-]+$", description="Package identifier")
|
||||
maintainer: str = Field(
|
||||
min_length=1, max_length=200, description="Package maintainer"
|
||||
)
|
||||
website_url: HttpUrl = Field(alias="websiteURL", description="Package website URL")
|
||||
email: EmailStr = Field(description="Maintainer contact email")
|
||||
help: Help = Field(description="Help and documentation links")
|
||||
platforms: List[Platform] = Field(
|
||||
default_factory=lambda: [], description="Available platforms"
|
||||
)
|
||||
tools: List[Tool] = Field(default_factory=lambda: [], description="Available tools")
|
||||
|
||||
@field_validator("platforms")
|
||||
@classmethod
|
||||
def validate_unique_platforms(cls, v: List[Platform]) -> List[Platform]:
|
||||
"""Ensure no duplicate platform architecture/version combinations"""
|
||||
seen: Set[tuple[str, str]] = set()
|
||||
for platform in v:
|
||||
key = (platform.architecture, platform.version)
|
||||
if key in seen:
|
||||
raise ValueError(
|
||||
f"Duplicate platform found: {platform.architecture} v{platform.version}"
|
||||
)
|
||||
seen.add(key)
|
||||
return v
|
||||
|
||||
@field_validator("tools")
|
||||
@classmethod
|
||||
def validate_unique_tools(cls, v: List[Tool]) -> List[Tool]:
|
||||
"""Ensure no duplicate tool name/version combinations"""
|
||||
seen: Set[tuple[str, str]] = set()
|
||||
for tool in v:
|
||||
key = (tool.name, tool.version)
|
||||
if key in seen:
|
||||
raise ValueError(f"Duplicate tool found: {tool.name} v{tool.version}")
|
||||
seen.add(key)
|
||||
return v
|
||||
|
||||
def find_platform(
|
||||
self, architecture: str, version: Optional[str] = None
|
||||
) -> Optional[Platform]:
|
||||
"""Find platform by architecture and optionally version"""
|
||||
for platform in self.platforms:
|
||||
if platform.architecture == architecture:
|
||||
if version is None or platform.version == version:
|
||||
return platform
|
||||
return None
|
||||
|
||||
def find_tool(self, name: str, version: Optional[str] = None) -> Optional[Tool]:
|
||||
"""Find tool by name and optionally version"""
|
||||
for tool in self.tools:
|
||||
if tool.name == name:
|
||||
if version is None or tool.version == version:
|
||||
return tool
|
||||
return None
|
||||
|
||||
def get_latest_platform_version(self, architecture: str) -> Optional[str]:
|
||||
"""Get the latest version for a given architecture"""
|
||||
versions = [p.version for p in self.platforms if p.architecture == architecture]
|
||||
if not versions:
|
||||
return None
|
||||
# Simple version sorting - can be enhanced with proper semver parsing
|
||||
return sorted(versions)[-1]
|
||||
|
||||
class Config:
|
||||
allow_population_by_field_name = True
|
||||
schema_extra: Dict[str, Any] = {
|
||||
"example": {
|
||||
"name": "esp32",
|
||||
"maintainer": "Espressif Systems",
|
||||
"websiteURL": "https://github.com/espressif/arduino-esp32",
|
||||
"email": "hr@espressif.com",
|
||||
"help": {"online": "https://github.com/espressif/arduino-esp32"},
|
||||
"platforms": [],
|
||||
"tools": [],
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class PackageIndex(BaseModel):
|
||||
"""Root package index containing multiple packages"""
|
||||
|
||||
packages: List[Package] = Field(min_length=1, description="Available packages")
|
||||
|
||||
@field_validator("packages")
|
||||
@classmethod
|
||||
def validate_unique_packages(cls, v: List[Package]) -> List[Package]:
|
||||
"""Ensure no duplicate package names"""
|
||||
names = [pkg.name for pkg in v]
|
||||
if len(names) != len(set(names)):
|
||||
raise ValueError("Duplicate package names found in index")
|
||||
return v
|
||||
|
||||
def find_package(self, name: str) -> Optional[Package]:
|
||||
"""Find package by name"""
|
||||
for package in self.packages:
|
||||
if package.name == name:
|
||||
return package
|
||||
return None
|
||||
|
||||
def get_all_platforms(self) -> List[Platform]:
|
||||
"""Get all platforms from all packages"""
|
||||
platforms: List[Platform] = []
|
||||
for package in self.packages:
|
||||
platforms.extend(package.platforms)
|
||||
return platforms
|
||||
|
||||
def get_all_tools(self) -> List[Tool]:
|
||||
"""Get all tools from all packages"""
|
||||
tools: List[Tool] = []
|
||||
for package in self.packages:
|
||||
tools.extend(package.tools)
|
||||
return tools
|
||||
|
||||
|
||||
# Enhanced Parser with Validation and Error Handling
|
||||
|
||||
|
||||
class PackageIndexParser:
|
||||
"""Enhanced parser with comprehensive validation and error handling"""
|
||||
|
||||
def __init__(self, timeout: int = 30):
|
||||
"""Initialize parser with timeout configuration"""
|
||||
self.timeout = timeout
|
||||
|
||||
def parse_package_index(self, json_str: str) -> PackageIndex:
|
||||
"""Parse and validate package index JSON"""
|
||||
try:
|
||||
raw_data = json.loads(json_str)
|
||||
return PackageIndex(**raw_data)
|
||||
except ValidationError as e:
|
||||
raise PackageParsingError(f"Invalid package index format: {e}")
|
||||
except json.JSONDecodeError as e:
|
||||
raise PackageParsingError(f"Invalid JSON format: {e}")
|
||||
|
||||
def parse_from_url(self, url: str) -> PackageIndex:
|
||||
"""Fetch and parse package index from URL with validation"""
|
||||
try:
|
||||
print(f"Fetching package index from: {url}")
|
||||
with urlopen(url, timeout=self.timeout) as response:
|
||||
content = response.read()
|
||||
json_str = content.decode("utf-8")
|
||||
|
||||
return self.parse_package_index(json_str)
|
||||
|
||||
except Exception as e:
|
||||
raise PackageParsingError(f"Error fetching package index from {url}: {e}")
|
||||
|
||||
async def parse_from_url_async(self, url: str) -> PackageIndex:
|
||||
"""Async version of URL parsing"""
|
||||
try:
|
||||
print(f"Fetching package index from: {url}")
|
||||
async with httpx.AsyncClient(timeout=self.timeout) as client:
|
||||
response = await client.get(url)
|
||||
response.raise_for_status()
|
||||
json_str = response.text
|
||||
|
||||
return self.parse_package_index(json_str)
|
||||
|
||||
except Exception as e:
|
||||
raise PackageParsingError(f"Error fetching package index from {url}: {e}")
|
||||
|
||||
|
||||
# Package Manager Configuration
|
||||
|
||||
|
||||
class PackageManagerConfig(BaseModel):
|
||||
"""Configuration for package manager with validation"""
|
||||
|
||||
cache_dir: Path = Field(default_factory=lambda: Path.home() / ".arduino_packages")
|
||||
sources: List[HttpUrl] = Field(
|
||||
default_factory=lambda: [], description="Package index URLs"
|
||||
)
|
||||
timeout: int = Field(
|
||||
default=30, gt=0, le=300, description="Request timeout in seconds"
|
||||
)
|
||||
max_retries: int = Field(
|
||||
default=3, ge=0, le=10, description="Maximum retry attempts"
|
||||
)
|
||||
verify_checksums: bool = Field(
|
||||
default=True, description="Verify download checksums"
|
||||
)
|
||||
allow_insecure: bool = Field(default=False, description="Allow insecure downloads")
|
||||
|
||||
@field_validator("cache_dir")
|
||||
@classmethod
|
||||
def validate_cache_dir(cls, v: Path) -> Path:
|
||||
"""Ensure cache directory is valid"""
|
||||
if v.exists() and not v.is_dir():
|
||||
raise ValueError(f"Cache path exists but is not a directory: {v}")
|
||||
return v
|
||||
|
||||
class Config:
|
||||
validate_assignment = True
|
||||
schema_extra = {
|
||||
"example": {
|
||||
"cache_dir": "~/.arduino_packages",
|
||||
"sources": [
|
||||
"https://espressif.github.io/arduino-esp32/package_esp32_index.json"
|
||||
],
|
||||
"timeout": 30,
|
||||
"max_retries": 3,
|
||||
"verify_checksums": True,
|
||||
"allow_insecure": False,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
# Utility Functions for Enhanced Functionality
|
||||
|
||||
|
||||
def verify_checksum(file_path: Path, expected_checksum: str) -> bool:
|
||||
"""Verify file checksum against expected SHA-256 value"""
|
||||
if not expected_checksum.startswith("SHA-256:"):
|
||||
raise ValueError(f"Invalid checksum format: {expected_checksum}")
|
||||
|
||||
expected_hash = expected_checksum[8:] # Remove 'SHA-256:' prefix
|
||||
|
||||
sha256_hash = hashlib.sha256()
|
||||
with open(file_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
sha256_hash.update(chunk)
|
||||
|
||||
actual_hash = sha256_hash.hexdigest()
|
||||
return actual_hash.lower() == expected_hash.lower()
|
||||
|
||||
|
||||
def extract_archive(archive_path: Path, extract_to: Path) -> bool:
|
||||
"""Extract archive to specified directory with format detection"""
|
||||
try:
|
||||
archive_str = str(archive_path)
|
||||
|
||||
if archive_path.suffix == ".zip":
|
||||
with zipfile.ZipFile(archive_path, "r") as zip_ref:
|
||||
zip_ref.extractall(extract_to)
|
||||
elif archive_str.endswith((".tar.gz", ".tgz")):
|
||||
with tarfile.open(archive_path, "r:gz") as tar_ref:
|
||||
tar_ref.extractall(extract_to)
|
||||
elif archive_str.endswith(".tar.bz2"):
|
||||
with tarfile.open(archive_path, "r:bz2") as tar_ref:
|
||||
tar_ref.extractall(extract_to)
|
||||
elif archive_str.endswith(".tar.xz"):
|
||||
with tarfile.open(archive_path, "r:xz") as tar_ref:
|
||||
tar_ref.extractall(extract_to)
|
||||
else:
|
||||
raise ValueError(f"Unsupported archive format: {archive_path}")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error extracting archive {archive_path}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def format_size(size_mb: float) -> str:
|
||||
"""Format size in a human-readable way"""
|
||||
if size_mb < 1:
|
||||
return f"{size_mb * 1024:.1f} KB"
|
||||
elif size_mb < 1024:
|
||||
return f"{size_mb:.1f} MB"
|
||||
else:
|
||||
return f"{size_mb / 1024:.1f} GB"
|
||||
|
||||
|
||||
# Display Functions with Enhanced Formatting
|
||||
|
||||
|
||||
def display_package_info(package: Package) -> None:
|
||||
"""Display package information with enhanced formatting"""
|
||||
print(f"\n📦 Package: {package.name}")
|
||||
print(f"👤 Maintainer: {package.maintainer}")
|
||||
print(f"🌐 Website: {package.website_url}")
|
||||
print(f"📧 Email: {package.email}")
|
||||
print(f"📚 Help: {package.help.online}")
|
||||
print(f"🛠️ Platforms: {len(package.platforms)}")
|
||||
print(f"🔧 Tools: {len(package.tools)}")
|
||||
|
||||
# Show platform information
|
||||
for i, platform in enumerate(package.platforms[:3]): # Show first 3 platforms
|
||||
print(f"\n 📋 Platform {i + 1}: {platform.name} v{platform.version}")
|
||||
print(f" Architecture: {platform.architecture}")
|
||||
print(f" Category: {platform.category}")
|
||||
print(f" Size: {format_size(platform.size_mb)}")
|
||||
print(f" Archive: {platform.archive_filename}")
|
||||
print(f" Checksum: {platform.checksum[:24]}...")
|
||||
print(f" Help: {platform.help.online}")
|
||||
print(f" Boards: {len(platform.boards)}")
|
||||
|
||||
# Show first 5 boards
|
||||
for board in platform.boards[:5]:
|
||||
print(f" • {board.name}")
|
||||
if len(platform.boards) > 5:
|
||||
print(f" ... and {len(platform.boards) - 5} more")
|
||||
|
||||
print(f" Tool Dependencies: {len(platform.tool_dependencies)}")
|
||||
for dep in platform.tool_dependencies[:3]: # Show first 3 dependencies
|
||||
print(f" • {dep.name} v{dep.version} ({dep.packager})")
|
||||
if len(platform.tool_dependencies) > 3:
|
||||
print(f" ... and {len(platform.tool_dependencies) - 3} more")
|
||||
|
||||
if len(package.platforms) > 3:
|
||||
print(f"\n ... and {len(package.platforms) - 3} more platforms")
|
||||
|
||||
|
||||
def display_validation_summary(package_index: PackageIndex) -> None:
|
||||
"""Display validation summary for the package index"""
|
||||
print(f"\n✅ VALIDATION SUMMARY")
|
||||
print(f" 📦 Total packages: {len(package_index.packages)}")
|
||||
|
||||
total_platforms = sum(len(pkg.platforms) for pkg in package_index.packages)
|
||||
total_tools = sum(len(pkg.tools) for pkg in package_index.packages)
|
||||
total_boards = sum(
|
||||
len(platform.boards)
|
||||
for pkg in package_index.packages
|
||||
for platform in pkg.platforms
|
||||
)
|
||||
|
||||
print(f" 🛠️ Total platforms: {total_platforms}")
|
||||
print(f" 🔧 Total tools: {total_tools}")
|
||||
print(f" 💾 Total boards: {total_boards}")
|
||||
|
||||
# Show architectures
|
||||
architectures: Set[str] = set()
|
||||
for pkg in package_index.packages:
|
||||
for platform in pkg.platforms:
|
||||
architectures.add(platform.architecture)
|
||||
|
||||
print(f" 🏗️ Architectures: {', '.join(sorted(architectures))}")
|
||||
|
||||
|
||||
# Demonstration Functions
|
||||
|
||||
|
||||
def demo_esp32_parsing() -> Optional[PackageIndex]:
|
||||
"""Demonstrate parsing ESP32 package index with enhanced validation"""
|
||||
ESP32_URL = "https://espressif.github.io/arduino-esp32/package_esp32_index.json"
|
||||
|
||||
try:
|
||||
parser = PackageIndexParser(timeout=30)
|
||||
package_index = parser.parse_from_url(ESP32_URL)
|
||||
|
||||
print("🎉 Successfully parsed ESP32 package index with Pydantic validation!")
|
||||
display_validation_summary(package_index)
|
||||
|
||||
# Display first package
|
||||
if package_index.packages:
|
||||
display_package_info(package_index.packages[0])
|
||||
|
||||
return package_index
|
||||
|
||||
except PackageParsingError as e:
|
||||
print(f"❌ Package parsing error: {e}")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"❌ Unexpected error: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def demo_model_validation() -> None:
|
||||
"""Demonstrate Pydantic model validation capabilities"""
|
||||
print("\n🧪 TESTING PYDANTIC MODEL VALIDATION")
|
||||
|
||||
# Define valid platform data
|
||||
valid_platform_data: Dict[str, Any] = {
|
||||
"name": "ESP32 Arduino",
|
||||
"architecture": "esp32",
|
||||
"version": "2.0.5",
|
||||
"category": "ESP32",
|
||||
"url": "https://github.com/espressif/arduino-esp32/releases/download/2.0.5/esp32-2.0.5.zip",
|
||||
"archiveFileName": "esp32-2.0.5.zip",
|
||||
"checksum": "SHA-256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
|
||||
"size": "50000000", # Should convert to MB
|
||||
"boards": [],
|
||||
"toolsDependencies": [],
|
||||
"help": {"online": "https://github.com/espressif/arduino-esp32"},
|
||||
}
|
||||
|
||||
# Test valid platform
|
||||
try:
|
||||
platform = Platform(**valid_platform_data)
|
||||
print(f"✅ Valid platform created: {platform.name} v{platform.version}")
|
||||
print(f" Size converted: {platform.size_mb:.1f} MB")
|
||||
|
||||
except ValidationError as e:
|
||||
print(f"❌ Unexpected validation error: {e}")
|
||||
|
||||
# Test invalid platform
|
||||
try:
|
||||
invalid_platform_data = valid_platform_data.copy()
|
||||
invalid_platform_data["checksum"] = "invalid-checksum-format"
|
||||
|
||||
platform = Platform(**invalid_platform_data)
|
||||
print("❌ Should have failed validation!")
|
||||
|
||||
except ValidationError as e:
|
||||
print(f"✅ Correctly caught invalid checksum: {e.errors()[0]['msg']}")
|
||||
|
||||
# Test invalid URL
|
||||
try:
|
||||
invalid_url_data = valid_platform_data.copy()
|
||||
invalid_url_data["url"] = "not-a-valid-url"
|
||||
|
||||
platform = Platform(**invalid_url_data)
|
||||
print("❌ Should have failed URL validation!")
|
||||
|
||||
except ValidationError as e:
|
||||
print(f"✅ Correctly caught invalid URL: {e.errors()[0]['msg']}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Main function demonstrating enhanced package index functionality"""
|
||||
print("🚀 ENHANCED ARDUINO PACKAGE INDEX WITH PYDANTIC")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
# Demo model validation
|
||||
demo_model_validation()
|
||||
|
||||
# Demo ESP32 parsing
|
||||
package_index = demo_esp32_parsing()
|
||||
|
||||
# Interactive options
|
||||
try:
|
||||
print(f"\n📋 AVAILABLE OPTIONS:")
|
||||
print("1. Show detailed tools information")
|
||||
print("2. Search for specific architecture")
|
||||
print("3. Exit")
|
||||
|
||||
choice = input("\nEnter your choice (1-3): ").strip()
|
||||
|
||||
if choice == "1" and package_index and package_index.packages:
|
||||
# Simple tools info display (without importing original)
|
||||
pkg = package_index.packages[0]
|
||||
print(f"\n🔧 TOOLS INFORMATION for {pkg.name}")
|
||||
print(f"Total tools: {len(pkg.tools)}")
|
||||
for tool in pkg.tools[:3]: # Show first 3 tools
|
||||
print(
|
||||
f" • {tool.name} v{tool.version} ({len(tool.systems)} systems)"
|
||||
)
|
||||
if len(pkg.tools) > 3:
|
||||
print(f" ... and {len(pkg.tools) - 3} more tools")
|
||||
|
||||
elif choice == "2" and package_index:
|
||||
arch = input("Enter architecture to search for: ").strip()
|
||||
platforms = [
|
||||
p
|
||||
for pkg in package_index.packages
|
||||
for p in pkg.platforms
|
||||
if p.architecture == arch
|
||||
]
|
||||
if platforms:
|
||||
print(
|
||||
f"\n🔍 Found {len(platforms)} platforms for architecture '{arch}':"
|
||||
)
|
||||
for platform in platforms:
|
||||
print(
|
||||
f" • {platform.name} v{platform.version} ({format_size(platform.size_mb)})"
|
||||
)
|
||||
else:
|
||||
print(f"❌ No platforms found for architecture '{arch}'")
|
||||
else:
|
||||
print("👋 Goodbye!")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n👋 Interrupted by user")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n👋 Interrupted by user")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,983 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Test Build System - Python Compiler API Integration
|
||||
|
||||
This module provides a high-performance test compilation system built on the proven
|
||||
ci.util.ci.util.clang_compiler API that delivers 8x faster build times compared to CMake.
|
||||
|
||||
Key Features:
|
||||
- Parallel test compilation using ThreadPoolExecutor
|
||||
- 90%+ cache hit rates with proven compiler settings
|
||||
- Integration with build_unit.toml configuration
|
||||
- Support for STUB platform testing without hardware dependencies
|
||||
- GDB-compatible debug symbol generation
|
||||
- Consistent performance across all platforms
|
||||
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import textwrap
|
||||
import time
|
||||
import tomllib
|
||||
from concurrent.futures import Future, as_completed
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from types import TracebackType
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
|
||||
|
||||
def optimize_python_command(cmd: list[str]) -> list[str]:
|
||||
"""
|
||||
Optimize command list for subprocess execution in uv environment.
|
||||
|
||||
For python commands, we need to use 'uv run python' to ensure access to
|
||||
installed packages like ziglang. Direct sys.executable bypasses uv environment.
|
||||
|
||||
Args:
|
||||
cmd: Command list that may contain 'python' as first element
|
||||
|
||||
Returns:
|
||||
list[str]: Optimized command with 'python' prefixed by 'uv run'
|
||||
"""
|
||||
if cmd and (cmd[0] == "python" or cmd[0] == "python3"):
|
||||
# Use uv run python to ensure access to uv-managed packages
|
||||
optimized_cmd = ["uv", "run", "python"] + cmd[1:]
|
||||
return optimized_cmd
|
||||
return cmd
|
||||
|
||||
|
||||
from ci.compiler.clang_compiler import (
|
||||
BuildFlags,
|
||||
BuildTools,
|
||||
Compiler,
|
||||
CompilerOptions,
|
||||
LinkOptions,
|
||||
Result,
|
||||
create_compiler_options_from_toml,
|
||||
link_program_sync,
|
||||
load_build_flags_from_toml,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestExecutable:
|
||||
"""Represents a compiled test executable"""
|
||||
|
||||
name: str
|
||||
executable_path: Path
|
||||
test_source_path: Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompileResult:
|
||||
"""Result of test compilation operation"""
|
||||
|
||||
success: bool
|
||||
compiled_count: int
|
||||
duration: float
|
||||
errors: List["CompileError"] = field(default_factory=lambda: list())
|
||||
|
||||
|
||||
@dataclass
|
||||
class CompileError:
|
||||
"""Compilation error for a specific test"""
|
||||
|
||||
test_name: str
|
||||
message: str
|
||||
|
||||
|
||||
class FastLEDTestCompiler:
|
||||
"""
|
||||
Test compiler built on proven Compiler API for 8x faster builds.
|
||||
|
||||
This class leverages the proven ci.util.ci.util.clang_compiler.Compiler API that has already
|
||||
demonstrated 4x+ performance improvements in example compilation. By using the same
|
||||
proven patterns and configurations, we inherit the optimized build flags, caching,
|
||||
and parallel compilation capabilities.
|
||||
"""
|
||||
|
||||
# Class variable to store existing instance for get_existing_instance()
|
||||
_existing_instance: Optional["FastLEDTestCompiler"] = None
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
compiler: Compiler,
|
||||
build_dir: Path,
|
||||
project_root: Path,
|
||||
quick_build: bool = True,
|
||||
strict_mode: bool = False,
|
||||
no_unity: bool = False,
|
||||
):
|
||||
self.compiler = compiler
|
||||
self.build_dir = build_dir
|
||||
self.project_root = project_root
|
||||
self.quick_build = quick_build
|
||||
self.strict_mode = strict_mode
|
||||
self.no_unity = no_unity
|
||||
self.compiled_tests: List[TestExecutable] = []
|
||||
self.linking_failures: List[str] = []
|
||||
self.build_flags = self._load_build_flags()
|
||||
FastLEDTestCompiler._existing_instance = self
|
||||
|
||||
def _load_build_flags(self) -> BuildFlags:
|
||||
"""Load build flags from TOML configuration"""
|
||||
toml_path = self.project_root / "ci" / "build_unit.toml"
|
||||
return BuildFlags.parse(toml_path, self.quick_build, self.strict_mode)
|
||||
|
||||
@classmethod
|
||||
def create_for_unit_tests(
|
||||
cls,
|
||||
project_root: Path,
|
||||
clean_build: bool = False,
|
||||
enable_static_analysis: bool = False,
|
||||
specific_test: str | None = None,
|
||||
quick_build: bool = True,
|
||||
strict_mode: bool = False,
|
||||
no_unity: bool = False,
|
||||
) -> "FastLEDTestCompiler":
|
||||
"""
|
||||
Create compiler configured for FastLED unit tests using TOML build flags.
|
||||
|
||||
This configuration uses the new TOML-based build flag system:
|
||||
- ci/build_unit.toml for centralized flag configuration
|
||||
- Support for build modes (quick/debug) and strict mode
|
||||
- STUB platform for hardware-free testing
|
||||
- Precompiled headers for faster compilation
|
||||
- Parallel compilation with ThreadPoolExecutor
|
||||
"""
|
||||
|
||||
# Set up build directory in .build/fled/unit for consistency
|
||||
build_dir = project_root / ".build" / "fled" / "unit"
|
||||
if clean_build and build_dir.exists():
|
||||
print("###########################")
|
||||
print("# CLEANING UNIT TEST BUILD DIR #")
|
||||
print("###########################")
|
||||
import errno
|
||||
import os
|
||||
import shutil
|
||||
import stat
|
||||
import time
|
||||
|
||||
def handle_remove_readonly(
|
||||
func: Callable[[str], None],
|
||||
path: str,
|
||||
exc: tuple[type[BaseException], BaseException, TracebackType | None],
|
||||
) -> None:
|
||||
"""Error handler for Windows readonly files"""
|
||||
if hasattr(exc[1], "errno") and exc[1].errno == errno.EACCES: # type: ignore
|
||||
os.chmod(path, stat.S_IWRITE)
|
||||
func(path)
|
||||
else:
|
||||
raise
|
||||
|
||||
# Windows-compatible directory removal with retry
|
||||
max_retries = 3
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
# Use onerror parameter for compatibility across Python versions
|
||||
shutil.rmtree(build_dir, onerror=handle_remove_readonly)
|
||||
break
|
||||
except (OSError, PermissionError) as e:
|
||||
if attempt < max_retries - 1:
|
||||
print(
|
||||
f"Warning: Failed to remove build directory (attempt {attempt + 1}): {e}"
|
||||
)
|
||||
time.sleep(0.01) # Brief pause before retry
|
||||
continue
|
||||
else:
|
||||
print(
|
||||
f"Warning: Could not remove build directory after {max_retries} attempts: {e}"
|
||||
)
|
||||
print("Continuing with existing directory...")
|
||||
break
|
||||
build_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Additional compiler args beyond TOML
|
||||
additional_compiler_args = [
|
||||
f"-I{project_root}/src/platforms/stub", # STUB platform headers
|
||||
]
|
||||
|
||||
# Load build flags using BuildFlags.parse() to get platform-specific configuration
|
||||
toml_path = project_root / "ci" / "build_unit.toml"
|
||||
build_flags = BuildFlags.parse(toml_path, quick_build, strict_mode)
|
||||
|
||||
# Load TOML config for test-specific flags that aren't in BuildFlags
|
||||
with open(toml_path, "rb") as f:
|
||||
config = tomllib.load(f)
|
||||
|
||||
# Get test-specific defines from TOML (not included in BuildFlags.parse)
|
||||
test_defines = config.get("test", {}).get("defines", [])
|
||||
|
||||
# Create compiler args from BuildFlags (includes platform-specific flags)
|
||||
compiler_args: List[str] = []
|
||||
compiler_args.extend(
|
||||
build_flags.compiler_flags
|
||||
) # Platform-aware compiler flags
|
||||
compiler_args.extend(build_flags.include_flags) # Platform-aware include flags
|
||||
|
||||
# Add strict mode flags if enabled (already handled by BuildFlags.parse)
|
||||
|
||||
# Add additional compiler args
|
||||
if additional_compiler_args:
|
||||
compiler_args.extend(additional_compiler_args)
|
||||
|
||||
# Extract defines without the "-D" prefix for CompilerOptions
|
||||
defines: List[str] = []
|
||||
for define in test_defines:
|
||||
if define.startswith("-D"):
|
||||
defines.append(define[2:]) # Remove "-D" prefix
|
||||
else:
|
||||
defines.append(define)
|
||||
|
||||
# Get tools configuration from BuildFlags (respects build_unit.toml)
|
||||
# Use modern command-based approach
|
||||
compiler_command = build_flags.tools.cpp_compiler
|
||||
|
||||
print(f"Using compiler from build_unit.toml: {compiler_command}")
|
||||
|
||||
# Use compiler command from TOML + flags as final compiler args
|
||||
final_compiler_args = compiler_command + compiler_args
|
||||
|
||||
# Create compiler options with TOML-loaded flags and tools
|
||||
settings = CompilerOptions(
|
||||
include_path=str(project_root / "src"),
|
||||
defines=defines,
|
||||
compiler_args=final_compiler_args, # Use cache-aware compiler args
|
||||
# Note: archiver and compiler fields are now handled via command-based approach
|
||||
)
|
||||
|
||||
compiler = Compiler(settings, build_flags)
|
||||
|
||||
# Create final instance with proper compiler and flags
|
||||
instance = cls(
|
||||
compiler, build_dir, project_root, quick_build, strict_mode, no_unity
|
||||
)
|
||||
return instance
|
||||
|
||||
def discover_test_files(self, specific_test: str | None = None) -> List[Path]:
|
||||
"""
|
||||
Discover test_*.cpp files using proven patterns.
|
||||
|
||||
Uses the same file discovery patterns as the existing CMake system
|
||||
to ensure 100% compatibility with current test discovery logic.
|
||||
"""
|
||||
tests_dir = Path(PROJECT_ROOT) / "tests"
|
||||
test_files: List[Path] = []
|
||||
|
||||
# Find all test files in tests directory and subdirectories
|
||||
for test_file in tests_dir.rglob("test_*.cpp"):
|
||||
# Skip the doctest_main.cpp file (not a test file)
|
||||
if test_file.name == "doctest_main.cpp":
|
||||
continue
|
||||
|
||||
# Handle both "test_name" and "name" formats for compatibility
|
||||
test_stem = test_file.stem
|
||||
test_name = test_stem.replace("test_", "")
|
||||
|
||||
# Check if we should do fuzzy matching (if there's a * in the name)
|
||||
if specific_test:
|
||||
if "*" in specific_test:
|
||||
# Convert glob pattern to regex pattern
|
||||
import re
|
||||
|
||||
pattern = specific_test.replace("*", ".*")
|
||||
if re.search(pattern, test_stem) or re.search(pattern, test_name):
|
||||
test_files.append(test_file)
|
||||
else:
|
||||
# Exact matching - either the full name or with test_ prefix (case-insensitive)
|
||||
if (
|
||||
test_stem.lower() == specific_test.lower()
|
||||
or test_stem.lower() == f"test_{specific_test}".lower()
|
||||
or test_name.lower() == specific_test.lower()
|
||||
):
|
||||
test_files.append(test_file)
|
||||
else:
|
||||
test_files.append(test_file)
|
||||
|
||||
return test_files
|
||||
|
||||
def compile_all_tests(self, specific_test: str | None = None) -> CompileResult:
|
||||
"""
|
||||
Compile all tests in parallel using proven API patterns.
|
||||
|
||||
This method uses the same parallel compilation patterns proven in
|
||||
example compilation that deliver 4x+ performance improvements.
|
||||
"""
|
||||
compile_start = time.time()
|
||||
|
||||
# If specific_test is not provided in the method call, use the one from discover_test_files
|
||||
test_files = self.discover_test_files(specific_test)
|
||||
|
||||
if not test_files:
|
||||
return CompileResult(success=True, compiled_count=0, duration=0.0)
|
||||
|
||||
print(f"Compiling {len(test_files)} test files using proven Python API...")
|
||||
# Convert absolute path to relative for display
|
||||
rel_build_dir = os.path.relpath(self.build_dir)
|
||||
print(f"Build directory: {rel_build_dir}")
|
||||
|
||||
# Always show build configuration in unit test mode
|
||||
print("\nFastLED Library Build Configuration:")
|
||||
if self.no_unity:
|
||||
print(" ❌ Unity build disabled - Compiling individual source files")
|
||||
else:
|
||||
print(" ✅ Unity build enabled - Using glob pattern:")
|
||||
print(" src/**/*.cpp")
|
||||
|
||||
print("\nPrecompiled header status:")
|
||||
if self.compiler.settings.use_pch:
|
||||
print(" ✅ PCH enabled - Using precompiled headers for faster compilation")
|
||||
print(" PCH content:")
|
||||
print(textwrap.indent(self.compiler.generate_pch_header(), " "))
|
||||
|
||||
# Create PCH for faster compilation
|
||||
print("\n🔧 Creating precompiled header...")
|
||||
pch_success = self.compiler.create_pch_file()
|
||||
if pch_success:
|
||||
print(" ✅ PCH created successfully")
|
||||
else:
|
||||
print(" ❌ PCH creation failed - continuing with direct compilation")
|
||||
else:
|
||||
print(" ❌ PCH disabled - Not using precompiled headers")
|
||||
|
||||
print("\nCompiler flags:")
|
||||
for flag in self.compiler.get_compiler_args():
|
||||
print(f" {flag}")
|
||||
print("\nLinker flags:")
|
||||
for flag in self._get_platform_linker_args(self.build_dir / "libfastled.lib"):
|
||||
print(f" {flag}")
|
||||
print("")
|
||||
|
||||
# Print list of test files being compiled
|
||||
print("Test files to compile:")
|
||||
for i, test_file in enumerate(test_files, 1):
|
||||
print(f" {i}. {test_file.name}")
|
||||
print("")
|
||||
|
||||
# Submit parallel compilation jobs (proven pattern from example compilation)
|
||||
future_to_test: Dict[Future[Result], Path] = {}
|
||||
|
||||
# Check if we're running in parallel or sequential mode
|
||||
if os.environ.get("NO_PARALLEL"):
|
||||
print("Starting sequential compilation of test files...")
|
||||
else:
|
||||
print("Starting parallel compilation of test files...")
|
||||
for test_file in test_files:
|
||||
# Compile to object file first (since compile_cpp_file uses -c flag)
|
||||
obj_path = self.build_dir / f"{test_file.stem}.o"
|
||||
# Convert absolute path to relative for display
|
||||
rel_obj_path = os.path.relpath(obj_path)
|
||||
print(f"Submitting compilation job for: {test_file.name} -> {rel_obj_path}")
|
||||
# Show compilation command if enabled
|
||||
if os.environ.get("FASTLED_TEST_SHOW_COMPILE", "").lower() in (
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
):
|
||||
# Convert absolute path to relative for display
|
||||
rel_obj_path = os.path.relpath(obj_path)
|
||||
print(f"[COMPILE] {test_file.name} -> {rel_obj_path}")
|
||||
|
||||
compile_future = self.compiler.compile_cpp_file(
|
||||
test_file,
|
||||
output_path=obj_path,
|
||||
additional_flags=[
|
||||
"-I",
|
||||
str(self.project_root / "src"),
|
||||
"-I",
|
||||
str(self.project_root / "tests"),
|
||||
# NOTE: Compiler flags now come from build_unit.toml
|
||||
],
|
||||
)
|
||||
future_to_test[compile_future] = test_file
|
||||
|
||||
# Collect compilation results with progress reporting
|
||||
compiled_objects: List[Path] = []
|
||||
errors: List[CompileError] = []
|
||||
completed = 0
|
||||
|
||||
print(f"Waiting for {len(future_to_test)} compilation jobs to complete...")
|
||||
|
||||
try:
|
||||
for future in as_completed(future_to_test.keys()):
|
||||
test_file = future_to_test[future]
|
||||
result: Result = future.result()
|
||||
completed += 1
|
||||
|
||||
if result.ok:
|
||||
obj_path = self.build_dir / f"{test_file.stem}.o"
|
||||
compiled_objects.append(obj_path)
|
||||
print(f"[{completed}/{len(test_files)}] Compiled {test_file.name}")
|
||||
else:
|
||||
errors.append(
|
||||
CompileError(
|
||||
test_name=test_file.stem,
|
||||
message=result.stderr
|
||||
or result.stdout
|
||||
or "Compilation failed",
|
||||
)
|
||||
)
|
||||
print(
|
||||
f"[{completed}/{len(test_files)}] FAILED {test_file.name}: {result.stderr}"
|
||||
)
|
||||
except KeyboardInterrupt:
|
||||
print("\nKeyboard interrupt detected during compilation")
|
||||
# Clean up any in-progress compilations
|
||||
for future in future_to_test.keys():
|
||||
future.cancel()
|
||||
import _thread
|
||||
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
|
||||
if errors:
|
||||
duration = time.time() - compile_start
|
||||
print(f"Compilation failed with {len(errors)} errors in {duration:.2f}s")
|
||||
return CompileResult(
|
||||
success=False, compiled_count=0, duration=duration, errors=errors
|
||||
)
|
||||
|
||||
print(f"All {len(compiled_objects)} object files compiled successfully")
|
||||
|
||||
# Link each test to executable using proven linking API
|
||||
self.compiled_tests = self._link_tests(compiled_objects)
|
||||
|
||||
# Check for linking failures and add them to errors
|
||||
if hasattr(self, "linking_failures") and self.linking_failures:
|
||||
for failure in self.linking_failures:
|
||||
# Parse the failure string to extract test name and error
|
||||
if ":" in failure:
|
||||
test_name, error_msg = failure.split(":", 1)
|
||||
errors.append(
|
||||
CompileError(
|
||||
test_name=test_name.strip(),
|
||||
message=f"Linking failed: {error_msg.strip()}",
|
||||
)
|
||||
)
|
||||
|
||||
duration = time.time() - compile_start
|
||||
|
||||
# Success requires both compilation AND linking to succeed
|
||||
success = len(errors) == 0 and len(self.compiled_tests) > 0
|
||||
|
||||
if success:
|
||||
print(
|
||||
f"SUCCESS: Compiled and linked {len(self.compiled_tests)} tests in {duration:.2f}s"
|
||||
)
|
||||
# List all the successful test executables
|
||||
print("Test executables created:")
|
||||
for i, test in enumerate(self.compiled_tests, 1):
|
||||
# Convert absolute path to relative for display
|
||||
rel_exe_path = os.path.relpath(test.executable_path)
|
||||
print(f" {i}. {test.name} -> {rel_exe_path}")
|
||||
else:
|
||||
print(f"FAILED: {len(errors)} total failures (compilation + linking)")
|
||||
|
||||
return CompileResult(
|
||||
success=success,
|
||||
compiled_count=len(self.compiled_tests),
|
||||
duration=duration,
|
||||
errors=errors,
|
||||
)
|
||||
|
||||
def _link_tests(self, compiled_objects: List[Path]) -> List[TestExecutable]:
|
||||
"""Link each test to executable using proven linking API"""
|
||||
print(f"Linking {len(compiled_objects)} test executables...")
|
||||
|
||||
# First, build a complete FastLED library similar to CMake approach
|
||||
fastled_lib_path = self._build_fastled_library()
|
||||
|
||||
# Compile doctest_main.cpp once for all tests (provides main function and doctest implementation)
|
||||
doctest_main_path = self.project_root / "tests" / "doctest_main.cpp"
|
||||
doctest_obj_path = self.build_dir / "doctest_main.o"
|
||||
|
||||
if not doctest_obj_path.exists():
|
||||
print("Compiling doctest_main.cpp...")
|
||||
doctest_future = self.compiler.compile_cpp_file(
|
||||
doctest_main_path,
|
||||
output_path=doctest_obj_path,
|
||||
additional_flags=["-c"], # Compile only
|
||||
)
|
||||
doctest_result: Result = doctest_future.result()
|
||||
if not doctest_result.ok:
|
||||
print(
|
||||
f"ERROR: Failed to compile doctest_main.cpp: {doctest_result.stderr}"
|
||||
)
|
||||
return []
|
||||
|
||||
print(f"Compiled FastLED library and doctest_main")
|
||||
|
||||
compiled_tests: List[TestExecutable] = []
|
||||
linking_failures: List[str] = []
|
||||
cache_hits = 0
|
||||
cache_misses = 0
|
||||
|
||||
# Link each test with the FastLED library
|
||||
for obj_path in compiled_objects:
|
||||
# Derive test name from object file
|
||||
test_name = obj_path.stem.replace("test_", "")
|
||||
exe_path = self.build_dir / f"test_{test_name}.exe"
|
||||
|
||||
# Define clean execution path in tests/bin/ (this is where tests will actually run from)
|
||||
tests_bin_dir = self.project_root / "tests" / "bin"
|
||||
tests_bin_dir.mkdir(parents=True, exist_ok=True)
|
||||
clean_exe_path = tests_bin_dir / f"test_{test_name}.exe"
|
||||
|
||||
# Check if this test has its own main() function
|
||||
test_source = self.project_root / "tests" / f"test_{test_name}.cpp"
|
||||
has_own_main = self._test_has_own_main(test_source)
|
||||
|
||||
# Get linker args (needed for cache key calculation)
|
||||
linker_args = self._get_platform_linker_args(fastled_lib_path)
|
||||
|
||||
# Calculate comprehensive cache key based on all linking inputs
|
||||
if has_own_main:
|
||||
# For standalone tests, only include test object file
|
||||
cache_key = self._calculate_link_cache_key(
|
||||
obj_path, fastled_lib_path, linker_args
|
||||
)
|
||||
else:
|
||||
# For doctest tests, include both test object and doctest_main object
|
||||
# Calculate hashes for both object files and combine them
|
||||
test_obj_hash = self._calculate_file_hash(obj_path)
|
||||
doctest_obj_hash = self._calculate_file_hash(doctest_obj_path)
|
||||
combined_obj_hash = hashlib.sha256(
|
||||
f"{test_obj_hash}+{doctest_obj_hash}".encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
# Create a temporary cache key with combined object hash
|
||||
fastled_hash = self._calculate_file_hash(fastled_lib_path)
|
||||
linker_hash = self._calculate_linker_args_hash(linker_args)
|
||||
combined = f"fastled:{fastled_hash}|test:{combined_obj_hash}|flags:{linker_hash}"
|
||||
cache_key = hashlib.sha256(combined.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
# Check for cached executable based on comprehensive cache key
|
||||
cached_exe = self._get_cached_executable(test_name, cache_key)
|
||||
if cached_exe:
|
||||
print(f" {test_name}: Using cached executable (link cache hit)")
|
||||
try:
|
||||
shutil.copy2(cached_exe, exe_path)
|
||||
# Also copy to clean execution location
|
||||
shutil.copy2(cached_exe, clean_exe_path)
|
||||
cache_hits += 1
|
||||
|
||||
test_exe = TestExecutable(
|
||||
name=test_name,
|
||||
executable_path=clean_exe_path, # Point to clean location for execution
|
||||
test_source_path=test_source,
|
||||
)
|
||||
compiled_tests.append(test_exe)
|
||||
continue # Skip linking, we have cached result
|
||||
except Exception as e:
|
||||
print(
|
||||
f" {test_name}: Warning - failed to copy cached executable, will relink: {e}"
|
||||
)
|
||||
# Fall through to linking logic
|
||||
|
||||
if has_own_main:
|
||||
# Link standalone test without doctest_main.o
|
||||
link_options = LinkOptions(
|
||||
object_files=[obj_path], # Only the test object file
|
||||
output_executable=exe_path,
|
||||
linker=f'"{sys.executable}" -m ziglang c++', # Use optimized Python executable for linking
|
||||
linker_args=linker_args, # Use pre-calculated args for consistency
|
||||
)
|
||||
else:
|
||||
# Link with the FastLED library instead of individual objects
|
||||
link_options = LinkOptions(
|
||||
object_files=[
|
||||
obj_path,
|
||||
doctest_obj_path,
|
||||
], # Test object + doctest main
|
||||
output_executable=exe_path,
|
||||
linker=f'"{sys.executable}" -m ziglang c++', # Use optimized Python executable for linking
|
||||
linker_args=linker_args, # Use pre-calculated args for consistency
|
||||
)
|
||||
|
||||
# Show linking command if enabled
|
||||
if os.environ.get("FASTLED_TEST_SHOW_LINK", "").lower() in (
|
||||
"1",
|
||||
"true",
|
||||
"yes",
|
||||
):
|
||||
# Convert absolute path to relative for display
|
||||
rel_exe_path = os.path.relpath(exe_path)
|
||||
print(f"[LINK] {test_name} -> {rel_exe_path}")
|
||||
|
||||
print(f"Linking test: {test_name}")
|
||||
# STRICT: Provide explicit BuildFlags - NO defaults allowed
|
||||
link_result: Result = link_program_sync(
|
||||
link_options, self.compiler.build_flags
|
||||
)
|
||||
|
||||
if not link_result.ok:
|
||||
# Create more detailed error message showing link command context
|
||||
error_msg = f" {test_name}: Linking failed - Linker: {link_options.linker}, Output: {link_options.output_executable}, Objects: {len(link_options.object_files)} files"
|
||||
print(error_msg)
|
||||
print(f" Error details: {link_result.stderr}")
|
||||
linking_failures.append(f"{test_name}: {link_result.stderr}")
|
||||
continue
|
||||
|
||||
print(f" {test_name}: Linking successful")
|
||||
cache_misses += 1
|
||||
|
||||
# Cache the newly linked executable for future use
|
||||
self._cache_executable(test_name, cache_key, exe_path)
|
||||
|
||||
# Copy to clean execution location
|
||||
shutil.copy2(exe_path, clean_exe_path)
|
||||
|
||||
test_exe = TestExecutable(
|
||||
name=test_name,
|
||||
executable_path=clean_exe_path,
|
||||
test_source_path=test_source, # Point to clean location for execution
|
||||
)
|
||||
compiled_tests.append(test_exe)
|
||||
|
||||
if linking_failures:
|
||||
print(f"ERROR: {len(linking_failures)} linking failures:")
|
||||
for failure in linking_failures:
|
||||
print(f" {failure}")
|
||||
# Store linking failures for later reporting
|
||||
self.linking_failures = linking_failures
|
||||
else:
|
||||
self.linking_failures = []
|
||||
|
||||
# Report link cache statistics
|
||||
total_tests = cache_hits + cache_misses
|
||||
if total_tests > 0:
|
||||
hit_rate = (cache_hits / total_tests) * 100
|
||||
print(
|
||||
f"Link cache: {cache_hits} hits, {cache_misses} misses ({hit_rate:.1f}% hit rate) - caching by lib+test+flags"
|
||||
)
|
||||
|
||||
print(f"Successfully linked {len(compiled_tests)} test executables")
|
||||
return compiled_tests
|
||||
|
||||
def _test_has_own_main(self, test_source_path: Path) -> bool:
|
||||
"""Check if a test file defines its own main() function"""
|
||||
try:
|
||||
if not test_source_path.exists():
|
||||
return False
|
||||
|
||||
content = test_source_path.read_text(encoding="utf-8")
|
||||
# Look for main function definition (simple pattern matching)
|
||||
import re
|
||||
|
||||
# Match "int main(" allowing for whitespace and various formats
|
||||
main_pattern = r"\bint\s+main\s*\("
|
||||
return bool(re.search(main_pattern, content))
|
||||
except Exception:
|
||||
# If we can't read the file, assume it doesn't have main
|
||||
return False
|
||||
|
||||
def _build_fastled_library(self) -> Path:
|
||||
"""Build a complete FastLED static library like CMake does"""
|
||||
print("Building FastLED static library...")
|
||||
|
||||
# Define library path
|
||||
fastled_lib_path = self.build_dir / "libfastled.lib"
|
||||
|
||||
# If library already exists, return it (for faster rebuilds)
|
||||
if fastled_lib_path.exists():
|
||||
# Convert absolute path to relative for display
|
||||
rel_lib_path = os.path.relpath(fastled_lib_path)
|
||||
print(f"Using existing FastLED library: {rel_lib_path}")
|
||||
return fastled_lib_path
|
||||
|
||||
# Compile essential FastLED source files for STUB platform
|
||||
# User directive: Just glob ALL .cpp files in src/** - no exclusions needed
|
||||
import glob
|
||||
|
||||
fastled_src_dir = self.project_root / "src"
|
||||
# Include ALL .cpp files recursively - no platform exclusions
|
||||
all_cpp_files: List[str] = glob.glob(
|
||||
str(fastled_src_dir / "**" / "*.cpp"), recursive=True
|
||||
)
|
||||
|
||||
fastled_sources: List[Path] = []
|
||||
for cpp_file in all_cpp_files:
|
||||
# Include ALL files - no filtering needed
|
||||
fastled_sources.append(Path(cpp_file))
|
||||
|
||||
if self.no_unity:
|
||||
print(
|
||||
f"Individual compilation: Found {len(fastled_sources)} FastLED .cpp files for compilation"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f"Unity build mode: Found {len(fastled_sources)} FastLED .cpp files for compilation"
|
||||
)
|
||||
|
||||
# Compile all FastLED source files to object files
|
||||
fastled_objects: List[Path] = []
|
||||
for src_file in fastled_sources:
|
||||
src_path = self.project_root / src_file
|
||||
# Create unique object file name including directory path to avoid collisions
|
||||
# Convert path separators to underscores to create valid filename
|
||||
relative_path = src_path.relative_to(self.project_root / "src")
|
||||
safe_name = (
|
||||
str(relative_path.with_suffix("")).replace("/", "_").replace("\\", "_")
|
||||
)
|
||||
obj_path = self.build_dir / f"{safe_name}_fastled.o"
|
||||
|
||||
if not obj_path.exists():
|
||||
future = self.compiler.compile_cpp_file(
|
||||
src_path,
|
||||
output_path=obj_path,
|
||||
additional_flags=[
|
||||
"-c", # Compile only
|
||||
"-DFASTLED_STUB_IMPL", # Essential for STUB platform
|
||||
# NOTE: All defines and compiler flags now come from build_unit.toml
|
||||
# Remove hardcoded defines - they should be in [test] section
|
||||
],
|
||||
)
|
||||
result: Result = future.result()
|
||||
if not result.ok:
|
||||
print(f"ERROR: Failed to compile {src_file}: {result.stderr}")
|
||||
# Continue with other files rather than failing completely
|
||||
continue
|
||||
|
||||
fastled_objects.append(obj_path)
|
||||
|
||||
print(f"Creating static library from {len(fastled_objects)} object files...")
|
||||
|
||||
# Create static library using ar (archiver)
|
||||
# First try llvm-ar since it's more likely to be available
|
||||
ar_cmd: List[str] = [
|
||||
"llvm-ar", # LLVM archiver tool (works with Clang)
|
||||
"rcs", # Create archive with symbol table
|
||||
str(fastled_lib_path),
|
||||
] + [str(obj) for obj in fastled_objects]
|
||||
|
||||
# Create static library using ziglang ar (archiver)
|
||||
print("Creating static library using ziglang ar...")
|
||||
lib_cmd: List[str] = [
|
||||
"python",
|
||||
"-m",
|
||||
"ziglang",
|
||||
"ar",
|
||||
"rcs", # Create archive with symbol table
|
||||
str(fastled_lib_path),
|
||||
] + [str(obj) for obj in fastled_objects]
|
||||
|
||||
try:
|
||||
# Optimize command to use sys.executable instead of shell 'python' resolution
|
||||
python_exe = optimize_python_command(lib_cmd)
|
||||
|
||||
# Use streaming to prevent buffer overflow
|
||||
process = subprocess.Popen(
|
||||
python_exe,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
bufsize=1,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
cwd=self.build_dir,
|
||||
)
|
||||
|
||||
stdout_lines: list[str] = []
|
||||
stderr_lines: list[str] = []
|
||||
|
||||
while True:
|
||||
stdout_line = process.stdout.readline() if process.stdout else ""
|
||||
stderr_line = process.stderr.readline() if process.stderr else ""
|
||||
|
||||
if stdout_line:
|
||||
stdout_lines.append(stdout_line.rstrip())
|
||||
if stderr_line:
|
||||
stderr_lines.append(stderr_line.rstrip())
|
||||
|
||||
if process.poll() is not None:
|
||||
remaining_stdout = process.stdout.read() if process.stdout else ""
|
||||
remaining_stderr = process.stderr.read() if process.stderr else ""
|
||||
|
||||
if remaining_stdout:
|
||||
for line in remaining_stdout.splitlines():
|
||||
stdout_lines.append(line.rstrip())
|
||||
if remaining_stderr:
|
||||
for line in remaining_stderr.splitlines():
|
||||
stderr_lines.append(line.rstrip())
|
||||
break
|
||||
|
||||
if process.returncode != 0:
|
||||
print(f"ERROR: Failed to create static library: {stderr_lines}")
|
||||
return fastled_lib_path # Return path even if creation failed
|
||||
|
||||
# Convert absolute path to relative for display
|
||||
rel_lib_path = os.path.relpath(fastled_lib_path)
|
||||
print(f"Successfully created FastLED library: {rel_lib_path}")
|
||||
return fastled_lib_path
|
||||
|
||||
except Exception as e:
|
||||
print(f"ERROR: Exception during library creation: {e}")
|
||||
raise
|
||||
|
||||
def get_test_executables(
|
||||
self, specific_test: str | None = None
|
||||
) -> List[TestExecutable]:
|
||||
"""Get compiled test executables with name filtering"""
|
||||
if specific_test:
|
||||
# Check if we should do fuzzy matching (if there's a * in the name)
|
||||
if "*" in specific_test:
|
||||
# Convert glob pattern to regex pattern
|
||||
import re
|
||||
|
||||
pattern = specific_test.replace("*", ".*")
|
||||
return [
|
||||
t
|
||||
for t in self.compiled_tests
|
||||
if re.search(pattern, t.name)
|
||||
or re.search(pattern, f"test_{t.name}")
|
||||
]
|
||||
else:
|
||||
# Exact matching - either the full name or with test_ prefix (case-insensitive)
|
||||
return [
|
||||
t
|
||||
for t in self.compiled_tests
|
||||
if (
|
||||
t.name.lower() == specific_test.lower()
|
||||
or t.name.lower() == f"test_{specific_test}".lower()
|
||||
or t.name.lower() == specific_test.replace("test_", "").lower()
|
||||
)
|
||||
]
|
||||
return self.compiled_tests
|
||||
|
||||
@property
|
||||
def link_cache_dir(self) -> Path:
|
||||
"""Get the link cache directory"""
|
||||
cache_dir = self.build_dir / "link_cache"
|
||||
cache_dir.mkdir(exist_ok=True)
|
||||
return cache_dir
|
||||
|
||||
def _calculate_file_hash(self, file_path: Path) -> str:
|
||||
"""Calculate SHA256 hash of a file"""
|
||||
if not file_path.exists():
|
||||
return "no_file"
|
||||
|
||||
hash_sha256 = hashlib.sha256()
|
||||
with open(file_path, "rb") as f:
|
||||
# Read in chunks to handle large files efficiently
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
hash_sha256.update(chunk)
|
||||
return hash_sha256.hexdigest()
|
||||
|
||||
def _calculate_linker_args_hash(self, linker_args: List[str]) -> str:
|
||||
"""Calculate SHA256 hash of linker arguments"""
|
||||
# Convert linker args to a stable string representation
|
||||
args_str = "|".join(sorted(linker_args))
|
||||
hash_sha256 = hashlib.sha256()
|
||||
hash_sha256.update(args_str.encode("utf-8"))
|
||||
return hash_sha256.hexdigest()
|
||||
|
||||
def _calculate_link_cache_key(
|
||||
self, test_obj_path: Path, fastled_lib_path: Path, linker_args: List[str]
|
||||
) -> str:
|
||||
"""Calculate comprehensive cache key for linking"""
|
||||
# Hash all the factors that affect linking output
|
||||
fastled_hash = self._calculate_file_hash(fastled_lib_path)
|
||||
test_obj_hash = self._calculate_file_hash(test_obj_path)
|
||||
linker_hash = self._calculate_linker_args_hash(linker_args)
|
||||
|
||||
# Combine all hashes into a single cache key
|
||||
combined = f"fastled:{fastled_hash}|test:{test_obj_hash}|flags:{linker_hash}"
|
||||
final_hash = hashlib.sha256(combined.encode("utf-8")).hexdigest()
|
||||
|
||||
return final_hash[:16] # Use first 16 chars for readability
|
||||
|
||||
def _get_cached_executable(self, test_name: str, cache_key: str) -> Optional[Path]:
|
||||
"""Check if a cached executable exists for this test and cache key"""
|
||||
cached_exe = self.link_cache_dir / f"{test_name}_{cache_key}.exe"
|
||||
return cached_exe if cached_exe.exists() else None
|
||||
|
||||
def _cache_executable(self, test_name: str, cache_key: str, exe_path: Path) -> None:
|
||||
"""Cache an executable with the given cache key"""
|
||||
if not exe_path.exists():
|
||||
return
|
||||
|
||||
cached_exe = self.link_cache_dir / f"{test_name}_{cache_key}.exe"
|
||||
try:
|
||||
shutil.copy2(exe_path, cached_exe)
|
||||
print(f" {test_name}: Cached executable for future use")
|
||||
except Exception as e:
|
||||
print(f" {test_name}: Warning - failed to cache executable: {e}")
|
||||
|
||||
def _get_platform_linker_args(self, fastled_lib_path: Path) -> List[str]:
|
||||
"""Get platform-specific linker arguments using BuildFlags"""
|
||||
# Use the build_flags that includes platform-specific configuration
|
||||
args = self.build_flags.link_flags.copy()
|
||||
|
||||
# Load TOML config for test-specific and build mode flags
|
||||
toml_path = self.project_root / "ci" / "build_unit.toml"
|
||||
with open(toml_path, "rb") as f:
|
||||
config = tomllib.load(f)
|
||||
|
||||
# Add test-specific linking flags if available
|
||||
test_link_flags = config.get("linking", {}).get("test", {}).get("flags", [])
|
||||
args.extend(test_link_flags)
|
||||
|
||||
# Add build mode specific flags (use test-specific debug mode)
|
||||
mode = "quick" if self.quick_build else "test_debug"
|
||||
mode_flags = config.get("build_modes", {}).get(mode, {}).get("link_flags", [])
|
||||
args.extend(mode_flags)
|
||||
|
||||
# Add FastLED library
|
||||
args.append(str(fastled_lib_path)) # Link against the FastLED static library
|
||||
|
||||
return args
|
||||
|
||||
@classmethod
|
||||
def get_existing_instance(cls) -> Optional["FastLEDTestCompiler"]:
|
||||
"""Get existing compiler instance for reuse"""
|
||||
return cls._existing_instance
|
||||
|
||||
|
||||
def check_iwyu_available() -> bool:
|
||||
"""Check if include-what-you-use is available (preserving existing logic)"""
|
||||
import shutil
|
||||
|
||||
return (
|
||||
shutil.which("include-what-you-use") is not None
|
||||
or shutil.which("iwyu") is not None
|
||||
)
|
||||
|
||||
|
||||
# For testing this module directly
|
||||
if __name__ == "__main__":
|
||||
print("Testing FastLEDTestCompiler...")
|
||||
|
||||
# Create test compiler
|
||||
test_compiler = FastLEDTestCompiler.create_for_unit_tests(
|
||||
project_root=Path(PROJECT_ROOT), clean_build=True
|
||||
)
|
||||
|
||||
# Discover test files
|
||||
test_files = test_compiler.discover_test_files()
|
||||
print(f"Discovered {len(test_files)} test files")
|
||||
|
||||
# Test compilation of a few files
|
||||
if test_files:
|
||||
limited_files = test_files[:3] # Test first 3 files
|
||||
print(f"Testing compilation of {len(limited_files)} test files...")
|
||||
|
||||
compile_result = test_compiler.compile_all_tests()
|
||||
|
||||
if compile_result.success:
|
||||
print(
|
||||
f"SUCCESS: Compiled {compile_result.compiled_count} tests in {compile_result.duration:.2f}s"
|
||||
)
|
||||
executables = test_compiler.get_test_executables()
|
||||
for exe in executables:
|
||||
print(f" Executable: {exe.name} -> {exe.executable_path}")
|
||||
else:
|
||||
print(f"FAILED: {len(compile_result.errors)} errors")
|
||||
for error in compile_result.errors:
|
||||
print(f" {error.test_name}: {error.message}")
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,171 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from concurrent.futures import Future
|
||||
from pathlib import Path
|
||||
from typing import List, Sequence
|
||||
|
||||
from ci.compiler.clang_compiler import Compiler, CompilerOptions, Result
|
||||
from ci.compiler.test_example_compilation import (
|
||||
create_fastled_compiler,
|
||||
get_fastled_core_sources,
|
||||
)
|
||||
|
||||
|
||||
def _sorted_src_files(files: Sequence[Path]) -> List[Path]:
|
||||
project_root = Path.cwd()
|
||||
|
||||
def key_fn(p: Path) -> str:
|
||||
try:
|
||||
rel = p.relative_to(project_root)
|
||||
except Exception:
|
||||
rel = p
|
||||
return rel.as_posix()
|
||||
|
||||
return sorted(files, key=key_fn)
|
||||
|
||||
|
||||
def _partition(files: List[Path], chunks: int) -> List[List[Path]]:
|
||||
total = len(files)
|
||||
if chunks <= 0:
|
||||
raise ValueError("chunks must be >= 1")
|
||||
chunks = min(chunks, total) if total > 0 else 1
|
||||
base = total // chunks
|
||||
rem = total % chunks
|
||||
partitions: List[List[Path]] = []
|
||||
start = 0
|
||||
for i in range(chunks):
|
||||
size = base + (1 if i < rem else 0)
|
||||
end = start + size
|
||||
if size > 0:
|
||||
partitions.append(files[start:end])
|
||||
start = end
|
||||
return partitions
|
||||
|
||||
|
||||
def build_unity_chunks_and_archive(
|
||||
compiler: Compiler,
|
||||
chunks: int,
|
||||
output_archive: Path,
|
||||
unity_dir: Path,
|
||||
no_parallel: bool,
|
||||
) -> Path:
|
||||
unity_dir.mkdir(parents=True, exist_ok=True)
|
||||
# Gather and sort source files consistently with existing library logic
|
||||
files = get_fastled_core_sources()
|
||||
# Exclude any main-like TU
|
||||
files = [p for p in files if p.name != "stub_main.cpp"]
|
||||
files = _sorted_src_files(files)
|
||||
if not files:
|
||||
raise RuntimeError("No source files found under src/** for unity build")
|
||||
|
||||
partitions = _partition(files, chunks)
|
||||
|
||||
# Prepare per-chunk unity paths
|
||||
unity_cpp_paths: List[Path] = [
|
||||
unity_dir / f"unity{i + 1}.cpp" for i in range(len(partitions))
|
||||
]
|
||||
unity_obj_paths: List[Path] = [p.with_suffix(".o") for p in unity_cpp_paths]
|
||||
|
||||
# Compile chunks
|
||||
futures: List[Future[Result]] = []
|
||||
compile_opts = CompilerOptions(
|
||||
include_path=compiler.settings.include_path,
|
||||
compiler=compiler.settings.compiler,
|
||||
defines=compiler.settings.defines,
|
||||
std_version=compiler.settings.std_version,
|
||||
compiler_args=compiler.settings.compiler_args,
|
||||
use_pch=False,
|
||||
additional_flags=["-c"],
|
||||
parallel=not no_parallel,
|
||||
)
|
||||
|
||||
def submit(idx: int) -> Future[Result]:
|
||||
chunk_files = partitions[idx]
|
||||
unity_cpp = unity_cpp_paths[idx]
|
||||
return compiler.compile_unity(compile_opts, chunk_files, unity_cpp)
|
||||
|
||||
if no_parallel:
|
||||
# Sequential execution
|
||||
for i in range(len(partitions)):
|
||||
result = submit(i).result()
|
||||
if not result.ok:
|
||||
raise RuntimeError(f"Unity chunk {i + 1} failed: {result.stderr}")
|
||||
else:
|
||||
futures = [submit(i) for i in range(len(partitions))]
|
||||
for i, fut in enumerate(futures):
|
||||
result = fut.result()
|
||||
if not result.ok:
|
||||
raise RuntimeError(f"Unity chunk {i + 1} failed: {result.stderr}")
|
||||
|
||||
# Create archive from chunk objects
|
||||
from ci.compiler.clang_compiler import LibarchiveOptions
|
||||
|
||||
output_archive.parent.mkdir(parents=True, exist_ok=True)
|
||||
archive_result = compiler.create_archive(
|
||||
unity_obj_paths, output_archive, LibarchiveOptions()
|
||||
).result()
|
||||
if not archive_result.ok:
|
||||
raise RuntimeError(f"Archive creation failed: {archive_result.stderr}")
|
||||
|
||||
return output_archive
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Chunked UNITY build and archive for FastLED src/**"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--chunks", type=int, default=1, help="Number of unity chunks (1-4 typical)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-parallel", action="store_true", help="Compile unity chunks sequentially"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=str,
|
||||
default=str(Path(".build/fastled/libfastled.a")),
|
||||
help="Path to output archive libfastled.a",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--unity-dir",
|
||||
type=str,
|
||||
default=str(Path(".build/fastled/unity")),
|
||||
help="Directory to place generated unity{N}.cpp/.o files",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--use-pch",
|
||||
action="store_true",
|
||||
help="Enable PCH (not recommended for unity builds)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Anchor to project root (two levels up from this file)
|
||||
here = Path(__file__).resolve()
|
||||
project_root = here.parent.parent.parent
|
||||
os.chdir(project_root)
|
||||
|
||||
try:
|
||||
compiler = create_fastled_compiler(
|
||||
use_pch=args.use_pch, parallel=not args.no_parallel
|
||||
)
|
||||
out = build_unity_chunks_and_archive(
|
||||
compiler=compiler,
|
||||
chunks=args.chunks,
|
||||
output_archive=Path(args.output),
|
||||
unity_dir=Path(args.unity_dir),
|
||||
no_parallel=args.no_parallel,
|
||||
)
|
||||
print(f"[UNITY] Created archive: {out}")
|
||||
return 0
|
||||
except KeyboardInterrupt:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"\x1b[31m[UNITY] FAILED: {e}\x1b[0m")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
Demo script showing how to use the new generic symbol analysis functionality
|
||||
"""
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
|
||||
def run_symbol_analysis(board_name: str):
|
||||
"""Run symbol analysis for a specific board"""
|
||||
print(f"Running symbol analysis for {board_name}...")
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[sys.executable, "ci/util/symbol_analysis.py", "--board", board_name],
|
||||
cwd=".",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
print(f"✅ Symbol analysis completed successfully for {board_name}")
|
||||
return True
|
||||
else:
|
||||
print(f"❌ Symbol analysis failed for {board_name}")
|
||||
print(f"Error: {result.stderr}")
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f"❌ Error running symbol analysis for {board_name}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def load_and_summarize_results(board_name: str):
|
||||
"""Load and display a summary of symbol analysis results"""
|
||||
json_file = Path(f".build/{board_name}_symbol_analysis.json")
|
||||
|
||||
if not json_file.exists():
|
||||
print(f"❌ Results file not found: {json_file}")
|
||||
return
|
||||
|
||||
try:
|
||||
with open(json_file) as f:
|
||||
data = json.load(f)
|
||||
|
||||
summary = data["summary"]
|
||||
print(f"\n📊 SUMMARY FOR {board_name.upper()}:")
|
||||
print(f" Total symbols: {summary['total_symbols']}")
|
||||
print(
|
||||
f" Total size: {summary['total_size']} bytes ({summary['total_size'] / 1024:.1f} KB)"
|
||||
)
|
||||
|
||||
print("\n🔝 TOP 5 LARGEST SYMBOLS:")
|
||||
for i, symbol in enumerate(summary["largest_symbols"][:5], 1):
|
||||
name = symbol.get("demangled_name", symbol["name"])
|
||||
print(
|
||||
f" {i}. {symbol['size']:6d} bytes - {name[:80]}{'...' if len(name) > 80 else ''}"
|
||||
)
|
||||
|
||||
print("\n📁 SYMBOL TYPES:")
|
||||
for sym_type, stats in list(summary["type_breakdown"].items())[:5]:
|
||||
print(f" {sym_type}: {stats['count']} symbols, {stats['size']} bytes")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error loading results for {board_name}: {e}")
|
||||
|
||||
|
||||
def main():
|
||||
"""Demo the symbol analysis functionality"""
|
||||
print("=" * 80)
|
||||
print("FASTLED SYMBOL ANALYSIS DEMO")
|
||||
print("=" * 80)
|
||||
|
||||
# List of boards to analyze (you can add more)
|
||||
boards_to_analyze: List[str] = []
|
||||
|
||||
# Check which boards have build info available
|
||||
build_dir = Path(".build")
|
||||
if build_dir.exists():
|
||||
for item in build_dir.iterdir():
|
||||
if item.is_dir() and (item / "build_info.json").exists():
|
||||
boards_to_analyze.append(item.name)
|
||||
|
||||
if not boards_to_analyze:
|
||||
print("❌ No boards with build_info.json found in .build directory")
|
||||
print(
|
||||
" Please compile some examples first using: uv run -m ci.ci-compile <board> --examples Blink"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
print(
|
||||
f"📋 Found {len(boards_to_analyze)} boards with build info: {', '.join(boards_to_analyze)}"
|
||||
)
|
||||
|
||||
# Run symbol analysis for each board
|
||||
successful_boards: List[str] = []
|
||||
for board in boards_to_analyze:
|
||||
if run_symbol_analysis(board):
|
||||
successful_boards.append(board)
|
||||
|
||||
print(f"\n📈 RESULTS SUMMARY FOR {len(successful_boards)} BOARDS:")
|
||||
print("=" * 80)
|
||||
|
||||
# Load and summarize results
|
||||
for board in successful_boards:
|
||||
load_and_summarize_results(board)
|
||||
print()
|
||||
|
||||
print("🎉 Symbol analysis demo completed!")
|
||||
print("📁 Detailed JSON files saved in .build/ directory")
|
||||
print("💡 Use these files to identify optimization opportunities")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,186 @@
|
||||
"""
|
||||
Work in progress to generate doxygen via a script instead of a GitHub action.
|
||||
"""
|
||||
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from download import download # type: ignore
|
||||
|
||||
|
||||
# Configs
|
||||
DOXYGEN_VERSION = (
|
||||
"1.13.2" # DOXYGEN_AWESOME styler is has certain restrictions with doxygen version
|
||||
)
|
||||
DOXYGEN_AWESOME_VERSION = "2.3.4" # deprecating
|
||||
DOXYFILE_PATH = Path("docs/Doxyfile")
|
||||
HTML_OUTPUT_DIR = Path("docs/html")
|
||||
DOXYGEN_CSS_REPO = "https://github.com/jothepro/doxygen-awesome-css" # deprecating
|
||||
|
||||
|
||||
HERE = Path(__file__).parent.resolve()
|
||||
PROJECT_ROOT = HERE.parent
|
||||
|
||||
DOCS_ROOT = PROJECT_ROOT / "docs"
|
||||
DOCS_TOOL_PATH = PROJECT_ROOT / ".tools_cache"
|
||||
DOCS_OUTPUT_PATH = DOCS_ROOT / "html"
|
||||
|
||||
|
||||
def run(
|
||||
cmd: str,
|
||||
cwd: Optional[str] = None,
|
||||
shell: bool = True,
|
||||
check: bool = True,
|
||||
capture: bool = True,
|
||||
) -> str:
|
||||
print(f"Running: {cmd}")
|
||||
result = subprocess.run(
|
||||
cmd, shell=shell, cwd=cwd, check=False, capture_output=capture, text=False
|
||||
)
|
||||
if capture:
|
||||
stdout = result.stdout.decode("utf-8") if result.stdout else ""
|
||||
stderr = result.stderr.decode("utf-8") if result.stderr else ""
|
||||
else:
|
||||
stdout = ""
|
||||
stderr = ""
|
||||
if result.returncode != 0:
|
||||
msg = f"Command failed with exit code {result.returncode}:\nstdout:\n{stdout}\n\nstderr:\n{stderr}"
|
||||
warnings.warn(msg)
|
||||
if check:
|
||||
raise subprocess.CalledProcessError(
|
||||
result.returncode, cmd, output=result.stdout
|
||||
)
|
||||
return stdout.strip()
|
||||
|
||||
|
||||
def get_git_info() -> Tuple[str, str]:
|
||||
release_tag = os.environ.get("RELEASE_TAG", "")
|
||||
|
||||
try:
|
||||
latest_tag = run("git tag | grep -E '^[0-9]' | sort -V | tail -1")
|
||||
latest_tag = latest_tag if latest_tag else ""
|
||||
except subprocess.CalledProcessError:
|
||||
latest_tag = ""
|
||||
|
||||
git_sha_short = run("git rev-parse --short HEAD")
|
||||
full_sha = run("git rev-parse HEAD")
|
||||
project_number = release_tag or latest_tag or git_sha_short
|
||||
commit_message = (
|
||||
f"{project_number} ({full_sha})"
|
||||
if project_number != git_sha_short
|
||||
else project_number
|
||||
)
|
||||
|
||||
print(f"Project number: {project_number}")
|
||||
print(f"Commit message: {commit_message}")
|
||||
return project_number, commit_message
|
||||
|
||||
|
||||
def install_doxygen_windows() -> Path:
|
||||
print("Installing Doxygen...")
|
||||
doxygen_url = (
|
||||
f"https://www.doxygen.nl/files/doxygen-{DOXYGEN_VERSION}.windows.x64.bin.zip"
|
||||
)
|
||||
zip_path = DOCS_TOOL_PATH / f"doxygen-{DOXYGEN_VERSION}.zip"
|
||||
extract_dir = DOCS_TOOL_PATH / f"doxygen-{DOXYGEN_VERSION}"
|
||||
|
||||
# Create tool path if it doesn't exist
|
||||
DOCS_TOOL_PATH.mkdir(exist_ok=True, parents=True)
|
||||
|
||||
download(doxygen_url, zip_path)
|
||||
shutil.unpack_archive(str(zip_path), extract_dir)
|
||||
bin_path = next(extract_dir.glob("**/doxygen.exe"), None)
|
||||
if not bin_path:
|
||||
raise FileNotFoundError("Doxygen executable not found after extraction.")
|
||||
print(f"Doxygen installed at: {bin_path}")
|
||||
return bin_path
|
||||
|
||||
|
||||
def install_doxygen_unix() -> Path:
|
||||
print("Installing Doxygen...")
|
||||
archive = f"doxygen-{DOXYGEN_VERSION}.linux.bin.tar.gz"
|
||||
url = f"https://www.doxygen.nl/files/{archive}"
|
||||
|
||||
# Create tool path if it doesn't exist
|
||||
DOCS_TOOL_PATH.mkdir(exist_ok=True, parents=True)
|
||||
|
||||
# Change to tool directory for download and extraction
|
||||
original_dir = os.getcwd()
|
||||
os.chdir(str(DOCS_TOOL_PATH))
|
||||
|
||||
try:
|
||||
run(f"wget -q {url}")
|
||||
run(f"tar -xf {archive}")
|
||||
bin_dir = DOCS_TOOL_PATH / f"doxygen-{DOXYGEN_VERSION}"
|
||||
return bin_dir / "bin" / "doxygen"
|
||||
finally:
|
||||
os.chdir(original_dir)
|
||||
|
||||
|
||||
def install_theme() -> Path:
|
||||
print("Installing Doxygen Awesome Theme...")
|
||||
theme_path = DOCS_ROOT / "doxygen-awesome-css"
|
||||
if theme_path.exists():
|
||||
return theme_path
|
||||
run(
|
||||
f"git clone --depth 1 -b v{DOXYGEN_AWESOME_VERSION} {DOXYGEN_CSS_REPO}",
|
||||
cwd=str(DOCS_ROOT),
|
||||
)
|
||||
return theme_path
|
||||
|
||||
|
||||
def generate_docs(doxygen_bin: Path) -> None:
|
||||
print("Generating documentation...")
|
||||
cmd_str = f'"{doxygen_bin}" {DOXYFILE_PATH.name}'
|
||||
run(cmd_str, cwd=str(DOCS_ROOT), capture=False)
|
||||
|
||||
|
||||
# def install_graphviz() -> None:
|
||||
# url: str = get_latest_release_for_platform()
|
||||
# print(url)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
is_windows = platform.system() == "Windows"
|
||||
# is_macos = platform.system() == "Darwin"
|
||||
_, commit_msg = get_git_info()
|
||||
|
||||
if is_windows:
|
||||
doxygen_bin = install_doxygen_windows()
|
||||
# add to path C:\Program Files\Graphviz\bin\
|
||||
os.environ["PATH"] += os.pathsep + r"C:\Program Files\Graphviz\bin"
|
||||
else:
|
||||
doxygen_bin = install_doxygen_unix()
|
||||
|
||||
# install_theme()
|
||||
|
||||
# install_graphviz() # Work in progress
|
||||
|
||||
# Verify Graphviz installation
|
||||
try:
|
||||
dot_version = run("dot -V", check=False)
|
||||
print(f"Graphviz detected: {dot_version}")
|
||||
except Exception:
|
||||
warnings.warn(
|
||||
"Graphviz (dot) not found in PATH. Diagrams may not be generated."
|
||||
)
|
||||
|
||||
# Check it graphviz is installed
|
||||
# if linux
|
||||
if not is_windows:
|
||||
run("dot -Tsvg -Kneato -Grankdir=LR", check=True)
|
||||
|
||||
generate_docs(doxygen_bin=doxygen_bin)
|
||||
|
||||
print(f"\n✅ Docs generated in: {HTML_OUTPUT_DIR}")
|
||||
print(f"📄 Commit message: {commit_msg}")
|
||||
print("✨ You can now manually deploy to GitHub Pages or automate this step.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
ESP32 Symbol Analysis Runner for GitHub Actions
|
||||
Wrapper script to run ESP32 symbol analysis with proper error handling
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from ci.util.esp32_symbol_analysis import main as esp32_symbol_analysis
|
||||
|
||||
|
||||
def is_esp32_board(board_name: str) -> bool:
|
||||
"""Check if the board is ESP32-based"""
|
||||
esp32_boards = [
|
||||
"esp32dev",
|
||||
"esp32",
|
||||
"esp32s2",
|
||||
"esp32s3",
|
||||
"esp32c3",
|
||||
"esp32c6",
|
||||
"esp32h2",
|
||||
"esp32p4",
|
||||
"esp32c2",
|
||||
]
|
||||
return any(esp32_board in board_name.lower() for esp32_board in esp32_boards)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run ESP32 symbol analysis for GitHub Actions"
|
||||
)
|
||||
parser.add_argument("--board", required=True, help="Board name being analyzed")
|
||||
parser.add_argument("--example", default="Blink", help="Example that was compiled")
|
||||
parser.add_argument(
|
||||
"--skip-on-failure",
|
||||
action="store_true",
|
||||
help="Don't fail the build if symbol analysis fails",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
print(
|
||||
f"ESP32 Symbol Analysis Runner - Board: {args.board}, Example: {args.example}"
|
||||
)
|
||||
|
||||
# Only run for ESP32 boards
|
||||
if not is_esp32_board(args.board):
|
||||
print(f"Skipping symbol analysis - {args.board} is not an ESP32-based board")
|
||||
return 0
|
||||
|
||||
# Check if build_info.json exists
|
||||
build_info_path = Path(".build") / args.board / "build_info.json"
|
||||
if not build_info_path.exists():
|
||||
message = (
|
||||
f"Build info not found at {build_info_path}. Skipping symbol analysis."
|
||||
)
|
||||
if args.skip_on_failure:
|
||||
print(f"Warning: {message}")
|
||||
return 0
|
||||
else:
|
||||
print(f"Error: {message}")
|
||||
return 1
|
||||
|
||||
try:
|
||||
# TODO - Remove this.
|
||||
# Change to the ci/util directory since the script expects to be run from there
|
||||
original_cwd = os.getcwd()
|
||||
ci_ci_dir = Path(__file__).parent / "ci"
|
||||
os.chdir(ci_ci_dir)
|
||||
|
||||
# Import and run the ESP32 symbol analysis
|
||||
try:
|
||||
print("Running ESP32 symbol analysis...")
|
||||
esp32_symbol_analysis()
|
||||
print("ESP32 symbol analysis completed successfully!")
|
||||
return 0
|
||||
except ImportError as e:
|
||||
print(f"Failed to import esp32_symbol_analysis module: {e}")
|
||||
if args.skip_on_failure:
|
||||
return 0
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(f"ESP32 symbol analysis failed: {e}")
|
||||
if args.skip_on_failure:
|
||||
return 0
|
||||
return 1
|
||||
finally:
|
||||
os.chdir(original_cwd)
|
||||
|
||||
except Exception as e:
|
||||
print(f"Unexpected error in ESP32 symbol analysis runner: {e}")
|
||||
if args.skip_on_failure:
|
||||
return 0
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,232 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Demonstration of the Fingerprint Cache Feature
|
||||
|
||||
This script shows how to use the fingerprint cache for efficient file change detection
|
||||
in build systems and development workflows.
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ci.ci.fingerprint_cache import FingerprintCache, has_changed
|
||||
|
||||
|
||||
def demo_basic_usage() -> None:
|
||||
"""Demonstrate basic fingerprint cache usage."""
|
||||
print("=== Basic Fingerprint Cache Demo ===")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
cache_file = temp_path / "demo_cache.json"
|
||||
cache = FingerprintCache(cache_file)
|
||||
|
||||
# Create a source file
|
||||
source_file = temp_path / "example.cpp"
|
||||
with open(source_file, "w") as f:
|
||||
f.write("#include <iostream>\nint main() { return 0; }")
|
||||
|
||||
baseline_time = time.time() - 3600 # 1 hour ago
|
||||
current_modtime = os.path.getmtime(source_file)
|
||||
|
||||
print(f"Source file: {source_file}")
|
||||
print(f"Baseline time: {baseline_time}")
|
||||
print(f"Current modtime: {current_modtime}")
|
||||
|
||||
# First check - file is newer than baseline
|
||||
changed = cache.has_changed(source_file, baseline_time)
|
||||
print(f"First check (vs baseline): {changed} (expected: True)")
|
||||
|
||||
# Second check - same modtime
|
||||
changed = cache.has_changed(source_file, current_modtime)
|
||||
print(f"Second check (same modtime): {changed} (expected: False)")
|
||||
|
||||
# Touch file (change modtime but not content)
|
||||
time.sleep(0.01)
|
||||
source_file.touch()
|
||||
|
||||
# Third check - modtime changed but content same
|
||||
changed = cache.has_changed(source_file, current_modtime)
|
||||
print(
|
||||
f"Third check (touched file): {changed} (expected: False - content unchanged)"
|
||||
)
|
||||
|
||||
# Actually modify content
|
||||
time.sleep(0.01)
|
||||
with open(source_file, "w") as f:
|
||||
f.write(
|
||||
'#include <iostream>\nint main() { std::cout << "Hello"; return 0; }'
|
||||
)
|
||||
|
||||
# Fourth check - content actually changed
|
||||
changed = cache.has_changed(source_file, current_modtime)
|
||||
print(f"Fourth check (content changed): {changed} (expected: True)")
|
||||
|
||||
|
||||
def demo_build_system_workflow() -> None:
|
||||
"""Demonstrate build system integration workflow."""
|
||||
print("\n=== Build System Workflow Demo ===")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
cache_file = temp_path / "build_cache.json"
|
||||
cache = FingerprintCache(cache_file)
|
||||
|
||||
# Simulate multiple source files
|
||||
source_files = [
|
||||
temp_path / "main.cpp",
|
||||
temp_path / "utils.cpp",
|
||||
temp_path / "config.h",
|
||||
]
|
||||
|
||||
# Create source files
|
||||
contents = [
|
||||
'#include "config.h"\n#include "utils.h"\nint main() { return 0; }',
|
||||
'#include "utils.h"\nvoid utility_function() {}',
|
||||
'#pragma once\n#define VERSION "1.0"',
|
||||
]
|
||||
|
||||
for src_file, content in zip(source_files, contents):
|
||||
with open(src_file, "w") as f:
|
||||
f.write(content)
|
||||
|
||||
print(f"Created {len(source_files)} source files")
|
||||
|
||||
# Simulate first build
|
||||
print("\n--- First Build (all files new) ---")
|
||||
last_build_time = time.time() - 3600 # 1 hour ago
|
||||
changed_files: List[str] = []
|
||||
|
||||
for src_file in source_files:
|
||||
if cache.has_changed(src_file, last_build_time):
|
||||
changed_files.append(src_file.name)
|
||||
|
||||
print(
|
||||
f"Files to rebuild: {changed_files} ({len(changed_files)}/{len(source_files)})"
|
||||
)
|
||||
|
||||
# Update last build time
|
||||
current_modtimes = [os.path.getmtime(f) for f in source_files]
|
||||
|
||||
# Simulate second build (no changes)
|
||||
print("\n--- Second Build (no changes) ---")
|
||||
changed_files: List[str] = []
|
||||
|
||||
for src_file, modtime in zip(source_files, current_modtimes):
|
||||
if cache.has_changed(src_file, modtime):
|
||||
changed_files.append(src_file.name)
|
||||
|
||||
print(
|
||||
f"Files to rebuild: {changed_files} ({len(changed_files)}/{len(source_files)})"
|
||||
)
|
||||
|
||||
# Modify one file
|
||||
print("\n--- Third Build (one file modified) ---")
|
||||
time.sleep(0.01)
|
||||
with open(source_files[0], "a") as f:
|
||||
f.write("\n// Added comment")
|
||||
|
||||
changed_files: List[str] = []
|
||||
for src_file, modtime in zip(source_files, current_modtimes):
|
||||
if cache.has_changed(src_file, modtime):
|
||||
changed_files.append(src_file.name)
|
||||
|
||||
print(
|
||||
f"Files to rebuild: {changed_files} ({len(changed_files)}/{len(source_files)})"
|
||||
)
|
||||
|
||||
|
||||
def demo_performance() -> None:
|
||||
"""Demonstrate cache performance characteristics."""
|
||||
print("\n=== Performance Demo ===")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
cache_file = temp_path / "perf_cache.json"
|
||||
cache = FingerprintCache(cache_file)
|
||||
|
||||
# Create multiple test files
|
||||
num_files = 20
|
||||
test_files: List[Path] = []
|
||||
|
||||
for i in range(num_files):
|
||||
test_file = temp_path / f"file_{i:03d}.cpp"
|
||||
with open(test_file, "w") as f:
|
||||
f.write(
|
||||
f"// File {i}\n#include <iostream>\nint func_{i}() {{ return {i}; }}"
|
||||
)
|
||||
test_files.append(test_file)
|
||||
|
||||
print(f"Created {num_files} test files")
|
||||
|
||||
# Measure cache miss performance (first check)
|
||||
start_time = time.time()
|
||||
baseline_time = time.time() - 3600
|
||||
|
||||
for test_file in test_files:
|
||||
cache.has_changed(test_file, baseline_time)
|
||||
|
||||
miss_time = time.time() - start_time
|
||||
print(
|
||||
f"Cache miss time: {miss_time * 1000:.1f}ms ({miss_time * 1000 / num_files:.2f}ms per file)"
|
||||
)
|
||||
|
||||
# Measure cache hit performance (second check with same modtime)
|
||||
modtimes = [os.path.getmtime(f) for f in test_files]
|
||||
|
||||
start_time = time.time()
|
||||
for test_file, modtime in zip(test_files, modtimes):
|
||||
cache.has_changed(test_file, modtime)
|
||||
|
||||
hit_time = time.time() - start_time
|
||||
print(
|
||||
f"Cache hit time: {hit_time * 1000:.1f}ms ({hit_time * 1000 / num_files:.2f}ms per file)"
|
||||
)
|
||||
print(f"Speedup: {miss_time / hit_time:.1f}x faster")
|
||||
|
||||
# Show cache statistics
|
||||
stats = cache.get_cache_stats()
|
||||
print(f"Cache entries: {stats['total_entries']}")
|
||||
print(f"Cache file size: {stats['cache_file_size_bytes']} bytes")
|
||||
|
||||
|
||||
def demo_convenience_function() -> None:
|
||||
"""Demonstrate the convenience has_changed function."""
|
||||
print("\n=== Convenience Function Demo ===")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
cache_file = temp_path / "convenience_cache.json"
|
||||
|
||||
# Create test file
|
||||
test_file = temp_path / "test.h"
|
||||
with open(test_file, "w") as f:
|
||||
f.write("#pragma once\n#define API_VERSION 1")
|
||||
|
||||
baseline_time = time.time() - 1800 # 30 minutes ago
|
||||
|
||||
# Use convenience function
|
||||
changed = has_changed(test_file, baseline_time, cache_file)
|
||||
print(f"Convenience function result: {changed}")
|
||||
print(f"Cache file created: {cache_file.exists()}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("Fingerprint Cache Feature Demonstration")
|
||||
print("=" * 50)
|
||||
|
||||
demo_basic_usage()
|
||||
demo_build_system_workflow()
|
||||
demo_performance()
|
||||
demo_convenience_function()
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print("Demo completed successfully!")
|
||||
print("\nKey Benefits:")
|
||||
print("- Fast modification time checks avoid expensive MD5 computation")
|
||||
print("- Content-based verification prevents false positives")
|
||||
print("- Persistent cache survives across tool invocations")
|
||||
print("- Significant speedup for incremental builds and testing")
|
||||
@@ -0,0 +1,205 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
from typing import Any, Dict
|
||||
|
||||
from ci.util.bin_2_elf import bin_to_elf
|
||||
from ci.util.elf import dump_symbol_sizes
|
||||
from ci.util.map_dump import map_dump
|
||||
|
||||
|
||||
def cpp_filt(cpp_filt_path: Path, input_text: str) -> str:
|
||||
"""
|
||||
Demangle C++ symbols using c++filt.
|
||||
|
||||
Args:
|
||||
cpp_filt_path (Path): Path to c++filt executable.
|
||||
input_text (str): Text to demangle.
|
||||
|
||||
Returns:
|
||||
str: Demangled text.
|
||||
"""
|
||||
if not cpp_filt_path.exists():
|
||||
raise FileNotFoundError(f"cppfilt not found at '{cpp_filt_path}'")
|
||||
command = [str(cpp_filt_path), "-t", "-n"]
|
||||
process = subprocess.Popen(
|
||||
command,
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
)
|
||||
stdout, stderr = process.communicate(input=input_text)
|
||||
if process.returncode != 0:
|
||||
raise RuntimeError(f"Error running c++filt: {stderr}")
|
||||
return stdout
|
||||
|
||||
|
||||
def demangle_gnu_linkonce_symbols(cpp_filt_path: Path, map_text: str) -> str:
|
||||
"""
|
||||
Demangle .gnu.linkonce.t symbols in the map file.
|
||||
|
||||
Args:
|
||||
cpp_filt_path (Path): Path to c++filt executable.
|
||||
map_text (str): Content of the map file.
|
||||
|
||||
Returns:
|
||||
str: Map file content with demangled symbols.
|
||||
"""
|
||||
# Extract all .gnu.linkonce.t symbols
|
||||
pattern = r"\.gnu\\.linkonce\\.t\\.(.+?)\\s"
|
||||
matches = re.findall(pattern, map_text)
|
||||
|
||||
if not matches:
|
||||
return map_text
|
||||
|
||||
# Create a block of text with the extracted symbols
|
||||
symbols_block = "\n".join(matches)
|
||||
|
||||
# Demangle the symbols
|
||||
demangled_block = cpp_filt(cpp_filt_path, symbols_block)
|
||||
|
||||
# Create a dictionary of mangled to demangled symbols
|
||||
demangled_dict = dict(zip(matches, demangled_block.strip().split("\n")))
|
||||
|
||||
# Replace the mangled symbols with demangled ones in the original text
|
||||
for mangled, demangled in demangled_dict.items():
|
||||
map_text = map_text.replace(
|
||||
f".gnu.linkonce.t.{mangled}", f".gnu.linkonce.t.{demangled}"
|
||||
)
|
||||
|
||||
return map_text
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Convert a binary file to ELF using map file."
|
||||
)
|
||||
parser.add_argument("--first", action="store_true", help="Inspect the first board")
|
||||
parser.add_argument("--cwd", type=Path, help="Custom working directory")
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def load_build_info(build_info_path: Path) -> Dict[str, Any]:
|
||||
"""
|
||||
Load build information from a JSON file.
|
||||
|
||||
Args:
|
||||
build_info_path (Path): Path to the build_info.json file.
|
||||
|
||||
Returns:
|
||||
dict: Parsed JSON data.
|
||||
"""
|
||||
if not build_info_path.exists():
|
||||
raise FileNotFoundError(f"Build info JSON not found at '{build_info_path}'")
|
||||
return json.loads(build_info_path.read_text())
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
if args.cwd:
|
||||
root_build_dir = args.cwd / ".build"
|
||||
else:
|
||||
root_build_dir = Path(".build")
|
||||
|
||||
# Support nested PlatformIO structure: .build/pio/<board>
|
||||
nested_pio_dir = root_build_dir / "pio"
|
||||
if nested_pio_dir.is_dir():
|
||||
root_build_dir = nested_pio_dir
|
||||
|
||||
board_dirs = [d for d in root_build_dir.iterdir() if d.is_dir()]
|
||||
if not board_dirs:
|
||||
print(f"No board directories found in {root_build_dir.absolute()}")
|
||||
return 1
|
||||
|
||||
print("Available boards:")
|
||||
for i, board_dir in enumerate(board_dirs):
|
||||
print(f"[{i}]: {board_dir.name}")
|
||||
|
||||
which = (
|
||||
0
|
||||
if args.first
|
||||
else int(input("Enter the number of the board you want to inspect: "))
|
||||
)
|
||||
board_dir = board_dirs[which]
|
||||
build_info_json = board_dir / "build_info.json"
|
||||
|
||||
build_info = load_build_info(build_info_json)
|
||||
board = board_dir.name
|
||||
board_info = build_info.get(board) or build_info[next(iter(build_info))]
|
||||
|
||||
# Validate paths from build_info.json
|
||||
elf_path = Path(board_info.get("prog_path", ""))
|
||||
if not elf_path.exists():
|
||||
print(
|
||||
f"Error: ELF path '{elf_path}' does not exist. Check the 'prog_path' in build_info.json."
|
||||
)
|
||||
return 1
|
||||
|
||||
bin_file = elf_path.with_suffix(".bin")
|
||||
if not bin_file.exists():
|
||||
# use .hex or .uf2 if .bin doesn't exist
|
||||
bin_file = elf_path.with_suffix(".hex")
|
||||
if not bin_file.exists():
|
||||
bin_file = elf_path.with_suffix(".uf2")
|
||||
if not bin_file.exists():
|
||||
print(f"Error: Binary file not found for '{elf_path}'")
|
||||
return 1
|
||||
cpp_filt_path = Path(board_info["aliases"]["c++filt"])
|
||||
ld_path = Path(board_info["aliases"]["ld"])
|
||||
as_path = Path(board_info["aliases"]["as"])
|
||||
nm_path = Path(board_info["aliases"]["nm"])
|
||||
objcopy_path = Path(board_info["aliases"]["objcopy"])
|
||||
nm_path = Path(board_info["aliases"]["nm"])
|
||||
map_file = board_dir / "firmware.map"
|
||||
if not map_file.exists():
|
||||
# Search for the map file
|
||||
map_file = bin_file.with_suffix(".map")
|
||||
if not map_file.exists():
|
||||
possible_map_files = list(board_dir.glob("**/firmware.map"))
|
||||
if possible_map_files:
|
||||
map_file = possible_map_files[0]
|
||||
else:
|
||||
print("Error: firmware.map file not found")
|
||||
return 1
|
||||
|
||||
try:
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
temp_dir_path = Path(temp_dir)
|
||||
output_elf = bin_to_elf(
|
||||
bin_file,
|
||||
map_file,
|
||||
as_path,
|
||||
ld_path,
|
||||
objcopy_path,
|
||||
temp_dir_path / "output.elf",
|
||||
)
|
||||
out = dump_symbol_sizes(nm_path, cpp_filt_path, output_elf)
|
||||
print(out)
|
||||
except Exception as e:
|
||||
print(
|
||||
f"Error while converting binary to ELF, binary analysis will not work on this build: {e}"
|
||||
)
|
||||
|
||||
map_dump(map_file)
|
||||
|
||||
# Demangle .gnu.linkonce.t symbols and print map file
|
||||
print("\n##################################################")
|
||||
print("# Map file dump:")
|
||||
print("##################################################\n")
|
||||
map_text = map_file.read_text()
|
||||
demangled_map_text = demangle_gnu_linkonce_symbols(cpp_filt_path, map_text)
|
||||
print(demangled_map_text)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,78 @@
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from ci.util.elf import dump_symbol_sizes
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
PROJECT_ROOT = HERE.parent
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Inspect a compiled binary")
|
||||
parser.add_argument("--first", action="store_true", help="Inspect the first board")
|
||||
parser.add_argument("--cwd", type=str, help="Custom working directory")
|
||||
parser.add_argument("--elf", type=str, help="Path to the ELF file to inspect")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
if args.elf:
|
||||
firmware_path = Path(args.elf)
|
||||
if not firmware_path.exists():
|
||||
print(f"ELF file not found: {firmware_path}")
|
||||
return 1
|
||||
|
||||
if args.cwd:
|
||||
# os.chdir(args.cwd)
|
||||
root_build_dir = Path(args.cwd) / ".build"
|
||||
else:
|
||||
root_build_dir = Path(".build")
|
||||
|
||||
# Support nested PlatformIO structure: .build/pio/<board>
|
||||
nested_pio_dir = root_build_dir / "pio"
|
||||
if nested_pio_dir.is_dir():
|
||||
root_build_dir = nested_pio_dir
|
||||
|
||||
# Find the first board directory
|
||||
board_dirs = [d for d in root_build_dir.iterdir() if d.is_dir()]
|
||||
if not board_dirs:
|
||||
# print("No board directories found in .build")
|
||||
print(f"No board directories found in {root_build_dir.absolute()}")
|
||||
return 1
|
||||
|
||||
# display all the boards to the user and ask them to select which one they want by number
|
||||
print("Available boards:")
|
||||
for i, board_dir in enumerate(board_dirs):
|
||||
print(f"[{i}]: {board_dir.name}")
|
||||
|
||||
if args.first:
|
||||
which = 0
|
||||
else:
|
||||
which = int(input("Enter the number of the board you want to inspect: "))
|
||||
|
||||
board_dir = board_dirs[which]
|
||||
board = board_dir.name
|
||||
|
||||
build_info_json = board_dir / "build_info.json"
|
||||
build_info = json.loads(build_info_json.read_text())
|
||||
board_info = build_info.get(board) or build_info[next(iter(build_info))]
|
||||
|
||||
firmware_path = Path(board_info["prog_path"])
|
||||
cpp_filt_path = Path(board_info["aliases"]["c++filt"])
|
||||
|
||||
print(f"Dumping symbol sizes for {board} firmware: {firmware_path}")
|
||||
try:
|
||||
nm_path = Path(board_info["aliases"]["nm"])
|
||||
symbol_sizes = dump_symbol_sizes(nm_path, cpp_filt_path, firmware_path)
|
||||
print(symbol_sizes)
|
||||
except Exception as e:
|
||||
print(f"Error while dumping symbol sizes: {e}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ci.util.paths import BUILD
|
||||
from ci.util.tools import load_tools
|
||||
|
||||
|
||||
def _list_builds() -> List[Path]:
|
||||
str_paths = os.listdir(BUILD)
|
||||
paths = [BUILD / p for p in str_paths]
|
||||
dirs = [p for p in paths if p.is_dir()]
|
||||
return dirs
|
||||
|
||||
|
||||
def _check_build(build: Path) -> bool:
|
||||
# 1. should contain a build_info.json file
|
||||
# 2. should contain a .pio/build directory
|
||||
has_build_info = (build / "build_info.json").exists()
|
||||
has_pio_build = (build / ".pio" / "build").exists()
|
||||
return has_build_info and has_pio_build
|
||||
|
||||
|
||||
def _prompt_build() -> Path:
|
||||
builds = _list_builds()
|
||||
if not builds:
|
||||
print("Error: No builds found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print("Select a build:")
|
||||
for i, build in enumerate(builds):
|
||||
print(f" [{i}]: {build}")
|
||||
while True:
|
||||
try:
|
||||
which = int(input("Enter the number of the build to use: "))
|
||||
if 0 <= which < len(builds):
|
||||
valid = _check_build(BUILD / builds[which])
|
||||
if valid:
|
||||
return BUILD / builds[which]
|
||||
print("Error: Invalid build", file=sys.stderr)
|
||||
else:
|
||||
print("Error: Invalid selection", file=sys.stderr)
|
||||
continue
|
||||
except ValueError:
|
||||
print("Error: Invalid input", file=sys.stderr)
|
||||
continue
|
||||
|
||||
|
||||
def _prompt_object_file(build: Path) -> Path:
|
||||
# Look for object files in .pio/build directory
|
||||
build_dir = build / ".pio" / "build"
|
||||
object_files: List[Path] = list(build_dir.rglob("*.o"))
|
||||
|
||||
if not object_files:
|
||||
print("Error: No object files found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
print("\nSelect an object file:")
|
||||
for i, obj_file in enumerate(object_files):
|
||||
print(f" [{i}]: {obj_file.relative_to(build_dir)}")
|
||||
|
||||
while True:
|
||||
try:
|
||||
which = int(input("Enter the number of the object file to use: "))
|
||||
if 0 <= which < len(object_files):
|
||||
return object_files[which]
|
||||
print("Error: Invalid selection", file=sys.stderr)
|
||||
except ValueError:
|
||||
print("Error: Invalid input", file=sys.stderr)
|
||||
continue
|
||||
|
||||
|
||||
def cli() -> None:
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Dump object file information using build tools"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"build_path",
|
||||
type=Path,
|
||||
nargs="?",
|
||||
help="Path to build directory containing build info JSON file",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
build_path = args.build_path
|
||||
|
||||
# Check if object file was provided and exists
|
||||
if build_path is None:
|
||||
build_path = _prompt_build()
|
||||
else:
|
||||
if not _check_build(build_path):
|
||||
print("Error: Invalid build directory", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
assert build_path is not None
|
||||
assert build_path
|
||||
|
||||
build_info_path = build_path / "build_info.json"
|
||||
assert build_info_path.exists(), f"File not found: {build_info_path}"
|
||||
|
||||
tools = load_tools(build_info_path)
|
||||
|
||||
object_file = _prompt_object_file(build_path)
|
||||
|
||||
cmd = [str(tools.objdump_path), "--syms", str(object_file)]
|
||||
if sys.platform == "win32":
|
||||
cmd = ["cmd", "/c"] + cmd
|
||||
cmd_str = subprocess.list2cmdline(cmd)
|
||||
subprocess.run(cmd, check=True)
|
||||
print("\nDone. Command used:", cmd_str)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
cli()
|
||||
except KeyboardInterrupt:
|
||||
print("Exiting...")
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,57 @@
|
||||
# FastLED Include What You Use Mapping File
|
||||
# This file tells IWYU about FastLED's specific include patterns and structure
|
||||
|
||||
[
|
||||
# FastLED main headers - these are the public API
|
||||
{ include: ['"FastLED.h"', 'public', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"fastled.h"', 'public', '"FastLED.h"', 'public'] },
|
||||
|
||||
# Core FastLED headers that should be included directly
|
||||
{ include: ['"lib8tion.h"', 'public', '"lib8tion.h"', 'public'] },
|
||||
{ include: ['"pixeltypes.h"', 'public', '"pixeltypes.h"', 'public'] },
|
||||
{ include: ['"hsv2rgb.h"', 'public', '"hsv2rgb.h"', 'public'] },
|
||||
{ include: ['"colorutils.h"', 'public', '"colorutils.h"', 'public'] },
|
||||
{ include: ['"colorpalettes.h"', 'public', '"colorpalettes.h"', 'public'] },
|
||||
|
||||
# Platform-specific headers should not be suggested directly
|
||||
{ include: ['"platforms/*/fastled_*.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"platforms/*/clockless_*.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"platforms/*/*"', 'private', '"FastLED.h"', 'public'] },
|
||||
|
||||
# Internal headers should redirect to public API
|
||||
{ include: ['"controller.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"led_sysdefs.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"fastled_config.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"fastled_delay.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"fastled_progmem.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
|
||||
# fl/ namespace headers - the modern FastLED API
|
||||
{ include: ['"fl/namespace.h"', 'public', '"fl/namespace.h"', 'public'] },
|
||||
{ include: ['"fl/ptr.h"', 'public', '"fl/ptr.h"', 'public'] },
|
||||
{ include: ['"fl/vector.h"', 'public', '"fl/vector.h"', 'public'] },
|
||||
{ include: ['"fl/allocator.h"', 'public', '"fl/allocator.h"', 'public'] },
|
||||
{ include: ['"fl/math.h"', 'public', '"fl/math.h"', 'public'] },
|
||||
{ include: ['"fl/force_inline.h"', 'public', '"fl/force_inline.h"', 'public'] },
|
||||
|
||||
# FX headers - effects system
|
||||
{ include: ['"fx/fx.h"', 'public', '"fx/fx.h"', 'public'] },
|
||||
{ include: ['"fx/fx_engine.h"', 'public', '"fx/fx_engine.h"', 'public'] },
|
||||
{ include: ['"fx/frame.h"', 'public', '"fx/frame.h"', 'public'] },
|
||||
{ include: ['"fx/detail/*"', 'private', '"fx/fx.h"', 'public'] },
|
||||
|
||||
# Third-party headers should not be suggested for user code
|
||||
{ include: ['"third_party/*/*"', 'private', '"FastLED.h"', 'public'] },
|
||||
|
||||
# Arduino-specific headers
|
||||
{ include: ['<Arduino.h>', 'public', '<Arduino.h>', 'public'] },
|
||||
{ include: ['<avr/pgmspace.h>', 'public', '<avr/pgmspace.h>', 'public'] },
|
||||
|
||||
# Chipset-specific headers redirect to main header
|
||||
{ include: ['"chipsets.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"ws2812_controller.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
{ include: ['"apa102_controller.h"', 'private', '"FastLED.h"', 'public'] },
|
||||
|
||||
# Testing headers - keep as-is for test files
|
||||
{ include: ['"test_*.h"', 'public', '"test_*.h"', 'public'] },
|
||||
{ include: ['"doctest.h"', 'public', '"doctest.h"', 'public'] },
|
||||
]
|
||||
@@ -0,0 +1,67 @@
|
||||
# Standard Library Include What You Use Mapping File
|
||||
# This file provides standard library header mappings for better IWYU suggestions
|
||||
|
||||
[
|
||||
# C++ standard library headers
|
||||
{ include: ['<cstdint>', 'public', '<cstdint>', 'public'] },
|
||||
{ include: ['<cstddef>', 'public', '<cstddef>', 'public'] },
|
||||
{ include: ['<cstring>', 'public', '<cstring>', 'public'] },
|
||||
{ include: ['<cstdlib>', 'public', '<cstdlib>', 'public'] },
|
||||
{ include: ['<cmath>', 'public', '<cmath>', 'public'] },
|
||||
{ include: ['<cassert>', 'public', '<cassert>', 'public'] },
|
||||
{ include: ['<climits>', 'public', '<climits>', 'public'] },
|
||||
{ include: ['<cfloat>', 'public', '<cfloat>', 'public'] },
|
||||
|
||||
# C++ containers and algorithms
|
||||
{ include: ['<vector>', 'public', '<vector>', 'public'] },
|
||||
{ include: ['<array>', 'public', '<array>', 'public'] },
|
||||
{ include: ['<string>', 'public', '<string>', 'public'] },
|
||||
{ include: ['<algorithm>', 'public', '<algorithm>', 'public'] },
|
||||
{ include: ['<memory>', 'public', '<memory>', 'public'] },
|
||||
{ include: ['<functional>', 'public', '<functional>', 'public'] },
|
||||
{ include: ['<utility>', 'public', '<utility>', 'public'] },
|
||||
{ include: ['<type_traits>', 'public', '<type_traits>', 'public'] },
|
||||
|
||||
# I/O headers
|
||||
{ include: ['<iostream>', 'public', '<iostream>', 'public'] },
|
||||
{ include: ['<iomanip>', 'public', '<iomanip>', 'public'] },
|
||||
{ include: ['<sstream>', 'public', '<sstream>', 'public'] },
|
||||
{ include: ['<fstream>', 'public', '<fstream>', 'public'] },
|
||||
|
||||
# Common type mappings
|
||||
{ symbol: ['std::uint8_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::uint16_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::uint32_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::uint64_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::int8_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::int16_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::int32_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::int64_t', 'public', '<cstdint>', 'public'] },
|
||||
{ symbol: ['std::size_t', 'public', '<cstddef>', 'public'] },
|
||||
{ symbol: ['std::ptrdiff_t', 'public', '<cstddef>', 'public'] },
|
||||
|
||||
# Common functions
|
||||
{ symbol: ['std::memcpy', 'public', '<cstring>', 'public'] },
|
||||
{ symbol: ['std::memset', 'public', '<cstring>', 'public'] },
|
||||
{ symbol: ['std::strlen', 'public', '<cstring>', 'public'] },
|
||||
{ symbol: ['std::malloc', 'public', '<cstdlib>', 'public'] },
|
||||
{ symbol: ['std::free', 'public', '<cstdlib>', 'public'] },
|
||||
{ symbol: ['std::abs', 'public', '<cmath>', 'public'] },
|
||||
{ symbol: ['std::sin', 'public', '<cmath>', 'public'] },
|
||||
{ symbol: ['std::cos', 'public', '<cmath>', 'public'] },
|
||||
{ symbol: ['std::sqrt', 'public', '<cmath>', 'public'] },
|
||||
|
||||
# C standard library compatibility (prefer C++ versions)
|
||||
{ include: ['<stdint.h>', 'public', '<cstdint>', 'public'] },
|
||||
{ include: ['<stddef.h>', 'public', '<cstddef>', 'public'] },
|
||||
{ include: ['<string.h>', 'public', '<cstring>', 'public'] },
|
||||
{ include: ['<stdlib.h>', 'public', '<cstdlib>', 'public'] },
|
||||
{ include: ['<math.h>', 'public', '<cmath>', 'public'] },
|
||||
{ include: ['<assert.h>', 'public', '<cassert>', 'public'] },
|
||||
{ include: ['<limits.h>', 'public', '<climits>', 'public'] },
|
||||
{ include: ['<float.h>', 'public', '<cfloat>', 'public'] },
|
||||
|
||||
# Platform-specific but common headers
|
||||
{ include: ['<sys/types.h>', 'public', '<sys/types.h>', 'public'] },
|
||||
{ include: ['<unistd.h>', 'public', '<unistd.h>', 'public'] },
|
||||
]
|
||||
@@ -0,0 +1,56 @@
|
||||
# FastLED JavaScript Linting Tools
|
||||
|
||||
This directory contains JavaScript linting tools for the FastLED project, using Node.js + ESLint for **fast** JavaScript validation.
|
||||
|
||||
|
||||
### Via Main Lint Script (Recommended)
|
||||
|
||||
```bash
|
||||
# JavaScript linting is included by default (fast only!)
|
||||
bash lint
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
The fast JavaScript linter is automatically installed during project setup:
|
||||
|
||||
```bash
|
||||
# Installs everything including fast JS linting
|
||||
./install
|
||||
```
|
||||
|
||||
Or manually install just the JavaScript linter:
|
||||
|
||||
```bash
|
||||
uv run ci/setup-js-linting-fast.py
|
||||
```
|
||||
|
||||
This will:
|
||||
- Download Node.js binary to `.js-tools/node/`
|
||||
- Install ESLint to `.js-tools/node_modules/`
|
||||
- Create ESLint configuration in `.js-tools/.eslintrc.js`
|
||||
- Create fast linting script `.cache/js-tools/lint-js-fast`
|
||||
|
||||
## Philosophy: Fast Only
|
||||
|
||||
**JavaScript linting follows a "fast only" policy:**
|
||||
- ✅ **Fast linting available**: Uses Node.js + ESLint (~0.9 seconds)
|
||||
- ⚠️ **Fast linting unavailable**: Skips JavaScript linting entirely
|
||||
- ❌ **No slow fallback**: Fast linting or no linting at all
|
||||
|
||||
## Configuration
|
||||
|
||||
The fast linter only checks for **critical runtime issues**:
|
||||
- `no-debugger` - Prevents debugger statements in production
|
||||
- `no-eval` - Prevents eval() security vulnerabilities
|
||||
|
||||
**No style enforcement** - focuses on critical issues only for maximum speed.
|
||||
|
||||
## Integration
|
||||
|
||||
JavaScript linting is included by default in `bash lint` when fast linting is available.
|
||||
|
||||
## Performance
|
||||
|
||||
- **Fast linting**: ~0.9 seconds (53x faster than previous solution)
|
||||
- **Total lint time**: ~1.6 seconds (Python + C++ + JavaScript)
|
||||
@@ -0,0 +1,16 @@
|
||||
#include "FastLED.h"
|
||||
|
||||
// Let's include a bunch of stuff and see if it breaks the build.
|
||||
#include <WiFi.h>
|
||||
#include <ESPmDNS.h>
|
||||
#include <NetworkUdp.h>
|
||||
#include <ArduinoOTA.h>
|
||||
#include <ArduinoJson.h>
|
||||
|
||||
void setup() {
|
||||
|
||||
}
|
||||
|
||||
void loop() {
|
||||
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
[platformio]
|
||||
src_dir = symlink://../../../src
|
||||
|
||||
[env:dev]
|
||||
platform = https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip
|
||||
board = esp32-s3-devkitc-1
|
||||
framework = arduino
|
||||
lib_deps =
|
||||
https://github.com/dvarrel/AsyncTCP
|
||||
https://github.com/mathieucarbou/ESPAsyncWebServer
|
||||
ArduinoJson
|
||||
FS
|
||||
ArduinoOTA
|
||||
ESPmDNS
|
||||
|
||||
lib_ldf_mode=deep
|
||||
build_flags=-DFASTLED_STUB_MAIN_INCLUDE_INO="../ci/kitchensink/kitchensink.ino.cpp"
|
||||
@@ -0,0 +1,252 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastLED Build System Migration Example
|
||||
|
||||
This shows how to migrate from the old scattered build logic
|
||||
to the new unified BuildAPI.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from ci.build_api import create_example_builder, create_unit_test_builder
|
||||
|
||||
|
||||
def old_way_unit_test_building():
|
||||
"""
|
||||
OLD WAY: Scattered logic with manual PCH and library building
|
||||
|
||||
This is how unit test building worked before the refactor:
|
||||
- Manual PCH building in multiple places
|
||||
- Manual library building with duplicated logic
|
||||
- Mixed build flags between unit tests and examples
|
||||
- Manual directory management
|
||||
"""
|
||||
|
||||
# OLD CODE (before refactor) - DO NOT USE
|
||||
"""
|
||||
# Manual PCH building (scattered in multiple files)
|
||||
from ci.compiler.test_example_compilation import create_fastled_compiler
|
||||
from ci.compiler.cpp_test_compile import create_unit_test_fastled_library
|
||||
|
||||
compiler = create_fastled_compiler(use_pch=True, use_sccache=False, parallel=True)
|
||||
|
||||
# Manual library building (duplicated logic)
|
||||
lib_file = create_unit_test_fastled_library(clean=False, use_pch=True)
|
||||
|
||||
# Manual target compilation loop
|
||||
for test_file in test_files:
|
||||
success = compiler.compile_cpp_file(test_file, output_obj)
|
||||
if success:
|
||||
link_executable(output_obj, lib_file, output_exe)
|
||||
"""
|
||||
|
||||
|
||||
def new_way_unit_test_building():
|
||||
"""
|
||||
NEW WAY: Unified BuildAPI with automatic PCH and library building
|
||||
|
||||
This is how unit test building works after the refactor:
|
||||
- Automatic PCH building (once, reused by all targets)
|
||||
- Automatic library building (once, reused by all targets)
|
||||
- Clean separation of unit test build flags
|
||||
- Automatic directory organization
|
||||
"""
|
||||
|
||||
# NEW CODE (after refactor) - USE THIS
|
||||
unit_builder = create_unit_test_builder(
|
||||
build_dir=".build/unit", use_pch=True, parallel=True
|
||||
)
|
||||
|
||||
test_files = [
|
||||
Path("tests/test_json.cpp"),
|
||||
Path("tests/test_color.cpp"),
|
||||
Path("tests/test_math.cpp"),
|
||||
]
|
||||
|
||||
# Single call handles everything:
|
||||
# - PCH building (automatically, once)
|
||||
# - Library building (automatically, once)
|
||||
# - Individual target compilation and linking
|
||||
results = unit_builder.build_targets(test_files)
|
||||
|
||||
# Clean result handling
|
||||
for result in results:
|
||||
if result.success:
|
||||
print(f"✓ {result.target.name} built successfully")
|
||||
else:
|
||||
print(f"✗ {result.target.name} failed: {result.message}")
|
||||
|
||||
|
||||
def old_way_example_building():
|
||||
"""
|
||||
OLD WAY: Mixed build configuration and manual management
|
||||
|
||||
Examples used the same build_flags.toml as unit tests, causing:
|
||||
- Wrong defines (FASTLED_FORCE_NAMESPACE=1 for examples)
|
||||
- Mixed configuration between unit tests and examples
|
||||
- Manual PCH and library building
|
||||
"""
|
||||
|
||||
# OLD CODE (before refactor) - DO NOT USE
|
||||
"""
|
||||
# Used same build_flags.toml as unit tests (wrong!)
|
||||
compiler = create_fastled_compiler(use_pch=True, use_sccache=False, parallel=True)
|
||||
|
||||
# Manual example compilation with wrong defines
|
||||
for example_file in example_files:
|
||||
success = compiler.compile_ino_file(example_file, output_obj)
|
||||
# ... manual linking logic ...
|
||||
"""
|
||||
|
||||
|
||||
def new_way_example_building():
|
||||
"""
|
||||
NEW WAY: Clean separation with dedicated build configuration
|
||||
|
||||
Examples now have their own build_example.toml with:
|
||||
- Correct defines (no FASTLED_FORCE_NAMESPACE=1)
|
||||
- Arduino compatibility defines
|
||||
- Automatic PCH and library building
|
||||
"""
|
||||
|
||||
# NEW CODE (after refactor) - USE THIS
|
||||
example_builder = create_example_builder(
|
||||
build_dir=".build/examples", use_pch=True, parallel=True
|
||||
)
|
||||
|
||||
example_files = [
|
||||
Path("examples/Blink/Blink.ino"),
|
||||
Path("examples/DemoReel100/DemoReel100.ino"),
|
||||
]
|
||||
|
||||
# Single call handles everything with correct configuration
|
||||
results = example_builder.build_targets(example_files)
|
||||
|
||||
for result in results:
|
||||
if result.success:
|
||||
print(f"✓ {result.target.name} built successfully")
|
||||
else:
|
||||
print(f"✗ {result.target.name} failed: {result.message}")
|
||||
|
||||
|
||||
def migration_benefits():
|
||||
"""
|
||||
Benefits of the new unified BuildAPI:
|
||||
"""
|
||||
|
||||
print("MIGRATION BENEFITS:")
|
||||
print("=" * 50)
|
||||
|
||||
print("1. UNIFIED API:")
|
||||
print(
|
||||
" Before: create_fastled_compiler() + create_unit_test_fastled_library() + manual loops"
|
||||
)
|
||||
print(" After: create_unit_test_builder().build_targets(files)")
|
||||
|
||||
print("\n2. CLEAN SEPARATION:")
|
||||
print(" Before: build_flags.toml used by both unit tests and examples")
|
||||
print(" After: build_unit.toml vs build_example.toml")
|
||||
|
||||
print("\n3. AUTOMATIC ARTIFACT BUILDING:")
|
||||
print(" Before: Manual PCH and library building in callers")
|
||||
print(" After: Automatic PCH and library building in BuildAPI")
|
||||
|
||||
print("\n4. ORGANIZED DIRECTORIES:")
|
||||
print(" Before: Mixed artifacts in .build/")
|
||||
print(" After: Organized .build/unit/ and .build/examples/ with subdirectories")
|
||||
|
||||
print("\n5. MULTIPLE TARGET EFFICIENCY:")
|
||||
print(" Before: PCH and library built for each target")
|
||||
print(" After: PCH and library built once, reused by all targets")
|
||||
|
||||
print("\n6. SIMPLE ERROR HANDLING:")
|
||||
print(" Before: Manual error handling in compilation loops")
|
||||
print(" After: Structured SketchResult objects with clear success/failure")
|
||||
|
||||
|
||||
def migration_checklist():
|
||||
"""
|
||||
Checklist for migrating existing code to the new BuildAPI:
|
||||
"""
|
||||
|
||||
print("\nMIGRATION CHECKLIST:")
|
||||
print("=" * 50)
|
||||
|
||||
print("✓ COMPLETED:")
|
||||
print(" - Split ci/build_flags.toml into build_unit.toml and build_example.toml")
|
||||
print(" - Created unified BuildAPI class")
|
||||
print(" - Implemented automatic PCH and library building")
|
||||
print(" - Implemented organized build directory structure")
|
||||
print(
|
||||
" - Created convenience functions create_unit_test_builder() and create_example_builder()"
|
||||
)
|
||||
|
||||
print("\n⚠ TODO:")
|
||||
print(
|
||||
" - Update existing unit test compilation code to use create_unit_test_builder()"
|
||||
)
|
||||
print(
|
||||
" - Update existing example compilation code to use create_example_builder()"
|
||||
)
|
||||
print(" - Remove old scattered PCH and library building logic")
|
||||
print(" - Update test scripts to use new API")
|
||||
print(" - Deprecate old build_flags.toml (marked as legacy)")
|
||||
|
||||
|
||||
def show_api_simplification():
|
||||
"""
|
||||
Show how the API is dramatically simplified:
|
||||
"""
|
||||
|
||||
print("\nAPI SIMPLIFICATION:")
|
||||
print("=" * 50)
|
||||
|
||||
print("BEFORE (scattered across multiple files):")
|
||||
print("""
|
||||
# In ci/compiler/test_example_compilation.py
|
||||
compiler = create_fastled_compiler(use_pch=True, use_sccache=False, parallel=True)
|
||||
|
||||
# In ci/compiler/cpp_test_compile.py
|
||||
lib_file = create_unit_test_fastled_library(clean=False, use_pch=True)
|
||||
|
||||
# In test runner
|
||||
for test_file in test_files:
|
||||
obj_file = build_dir / f"{test_file.stem}.o"
|
||||
success = compiler.compile_cpp_file(test_file, obj_file)
|
||||
if success:
|
||||
exe_file = build_dir / f"{test_file.stem}.exe"
|
||||
link_success = link_executable(obj_file, lib_file, exe_file)
|
||||
""")
|
||||
|
||||
print("AFTER (unified API):")
|
||||
print("""
|
||||
# Single unified API
|
||||
builder = create_unit_test_builder()
|
||||
results = builder.build_targets(test_files)
|
||||
|
||||
# That's it! Everything else is automatic:
|
||||
# - PCH building
|
||||
# - Library building
|
||||
# - Individual compilation
|
||||
# - Linking
|
||||
# - Directory organization
|
||||
# - Error handling
|
||||
""")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("FastLED Build System Migration Example")
|
||||
print("=" * 60)
|
||||
|
||||
migration_benefits()
|
||||
migration_checklist()
|
||||
show_api_simplification()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("NEXT STEPS:")
|
||||
print("1. Update existing unit test compilation to use create_unit_test_builder()")
|
||||
print("2. Update existing example compilation to use create_example_builder()")
|
||||
print("3. Remove old scattered build logic")
|
||||
print("4. Test the new unified system")
|
||||
print("=" * 60)
|
||||
@@ -0,0 +1,17 @@
|
||||
[platformio]
|
||||
src_dir = symlink://../../../src
|
||||
|
||||
[env:dev]
|
||||
platform = platformio/native
|
||||
|
||||
lib_deps =
|
||||
FastLED
|
||||
|
||||
build_flags =
|
||||
-DFASTLED_STUB_IMPL
|
||||
-DFASTLED_USE_STUB_ARDUINO
|
||||
-DFASTLED_STUB_MAIN_INCLUDE_INO="../examples/Blink/Blink.ino"
|
||||
-std=c++17
|
||||
-I../../src/platforms
|
||||
-isystem ../../src/platforms/stub ; for Arduino.h
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Convert a binary file to ELF using map file."
|
||||
)
|
||||
parser.add_argument("--first", action="store_true", help="Inspect the first board")
|
||||
parser.add_argument("--cwd", type=Path, help="Custom working directory")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
if args.cwd:
|
||||
root_build_dir = args.cwd / ".build"
|
||||
else:
|
||||
root_build_dir = Path(".build")
|
||||
|
||||
# Support nested PlatformIO structure: .build/pio/<board>
|
||||
nested_pio_dir = root_build_dir / "pio"
|
||||
if nested_pio_dir.is_dir():
|
||||
root_build_dir = nested_pio_dir
|
||||
|
||||
board_dirs = [d for d in root_build_dir.iterdir() if d.is_dir()]
|
||||
if not board_dirs:
|
||||
print(f"No board directories found in {root_build_dir.absolute()}")
|
||||
return 1
|
||||
print("Available boards:")
|
||||
for i, board_dir in enumerate(board_dirs):
|
||||
print(f"[{i}]: {board_dir.name}")
|
||||
which = (
|
||||
0
|
||||
if args.first
|
||||
else int(input("Enter the number of the board you want to inspect: "))
|
||||
)
|
||||
board_dir = board_dirs[which]
|
||||
# build_info_json = board_dir / "build_info.json"
|
||||
optimization_report = board_dir / "optimization_report.txt"
|
||||
text = optimization_report.read_text(encoding="utf-8", errors="replace")
|
||||
print(text)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,151 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
Caching wrapper for pyright that only runs if files have changed.
|
||||
This significantly speeds up repeated pyright runs during development.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
|
||||
def get_file_hash(file_path: Path) -> str:
|
||||
"""Get SHA256 hash of a file."""
|
||||
try:
|
||||
with open(file_path, "rb") as f:
|
||||
return hashlib.sha256(f.read()).hexdigest()
|
||||
except (OSError, IOError):
|
||||
return ""
|
||||
|
||||
|
||||
def get_directory_hash(directory: Path, extensions: Optional[List[str]] = None) -> str:
|
||||
"""Get combined hash of all files in a directory."""
|
||||
if extensions is None:
|
||||
extensions = [".py"]
|
||||
|
||||
hashes: List[str] = []
|
||||
if directory.exists():
|
||||
for ext in extensions:
|
||||
for file_path in directory.rglob(f"*{ext}"):
|
||||
if file_path.is_file():
|
||||
hashes.append(
|
||||
f"{file_path.relative_to(directory)}:{get_file_hash(file_path)}"
|
||||
)
|
||||
|
||||
hashes.sort() # Ensure consistent ordering
|
||||
return hashlib.sha256("\n".join(hashes).encode()).hexdigest()
|
||||
|
||||
|
||||
def get_cache_file() -> Path:
|
||||
"""Get the cache file path."""
|
||||
cache_dir = Path(".cache/pyright")
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_dir / "file_hashes.json"
|
||||
|
||||
|
||||
def load_cache() -> Dict[str, str]:
|
||||
"""Load the cache file."""
|
||||
cache_file = get_cache_file()
|
||||
if cache_file.exists():
|
||||
try:
|
||||
with open(cache_file, "r") as f:
|
||||
return json.load(f)
|
||||
except (json.JSONDecodeError, IOError):
|
||||
pass
|
||||
return {}
|
||||
|
||||
|
||||
def save_cache(cache: Dict[str, str]) -> None:
|
||||
"""Save the cache file."""
|
||||
cache_file = get_cache_file()
|
||||
try:
|
||||
with open(cache_file, "w") as f:
|
||||
json.dump(cache, f, indent=2)
|
||||
except IOError:
|
||||
pass # Ignore cache save errors
|
||||
|
||||
|
||||
def has_files_changed() -> bool:
|
||||
"""Check if any relevant files have changed since last run."""
|
||||
cache = load_cache()
|
||||
changed = False
|
||||
|
||||
# Files and directories to check
|
||||
files_to_check = [
|
||||
Path("pyrightconfig.json"),
|
||||
Path(".pyrightconfig.json"),
|
||||
Path("pyproject.toml"),
|
||||
]
|
||||
|
||||
directories_to_check = [
|
||||
Path("ci"),
|
||||
]
|
||||
|
||||
# Check individual files
|
||||
for file_path in files_to_check:
|
||||
if file_path.exists():
|
||||
current_hash = get_file_hash(file_path)
|
||||
if cache.get(str(file_path)) != current_hash:
|
||||
cache[str(file_path)] = current_hash
|
||||
changed = True
|
||||
|
||||
# Check directories
|
||||
for dir_path in directories_to_check:
|
||||
if dir_path.exists():
|
||||
current_hash = get_directory_hash(dir_path)
|
||||
cache_key = f"dir:{dir_path}"
|
||||
if cache.get(cache_key) != current_hash:
|
||||
cache[cache_key] = current_hash
|
||||
changed = True
|
||||
|
||||
# Save cache if any changes were detected
|
||||
if changed:
|
||||
save_cache(cache)
|
||||
|
||||
return changed
|
||||
|
||||
|
||||
def run_pyright(args: List[str]) -> int:
|
||||
"""Run pyright with the given arguments."""
|
||||
env = os.environ.copy()
|
||||
# env["PYRIGHT_PYTHON_FORCE_VERSION"] = "latest"
|
||||
env["PYRIGHT_PYTHON_CACHE_DIR"] = ".cache/pyright"
|
||||
|
||||
cmd = ["uv", "run", "pyright"] + args
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd, env=env)
|
||||
return result.returncode
|
||||
except KeyboardInterrupt:
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(f"Error running pyright: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main entry point."""
|
||||
args = sys.argv[1:] # Remove script name
|
||||
|
||||
# If --force is passed, skip cache check
|
||||
if "--force" in args:
|
||||
args.remove("--force")
|
||||
print("Forcing pyright run (--force flag detected)")
|
||||
return run_pyright(args)
|
||||
|
||||
# Check if files have changed
|
||||
if not has_files_changed():
|
||||
print("No changes detected, skipping pyright (use --force to override)")
|
||||
return 0
|
||||
|
||||
print("Changes detected, running pyright...")
|
||||
return run_pyright(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,435 @@
|
||||
#!/usr/bin/env python
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""A wrapper script around clang-format, suitable for linting multiple files
|
||||
and to use for continuous integration.
|
||||
|
||||
This is an alternative API for the clang-format command line.
|
||||
It runs over multiple files and directories in parallel.
|
||||
A diff output is produced and a sensible exit code is returned.
|
||||
|
||||
"""
|
||||
|
||||
from __future__ import print_function, unicode_literals
|
||||
|
||||
import argparse
|
||||
import codecs
|
||||
import difflib
|
||||
import errno
|
||||
import fnmatch
|
||||
import io
|
||||
import multiprocessing
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import traceback
|
||||
from functools import partial
|
||||
from typing import Any, Dict, Generator, List, Optional, Tuple, Union
|
||||
|
||||
|
||||
try:
|
||||
from subprocess import DEVNULL # py3k
|
||||
except ImportError:
|
||||
DEVNULL = open(os.devnull, "wb") # type: ignore
|
||||
|
||||
|
||||
DEFAULT_EXTENSIONS = "c,h,C,H,cpp,hpp,cc,hh,c++,h++,cxx,hxx"
|
||||
DEFAULT_CLANG_FORMAT_IGNORE = ".clang-format-ignore"
|
||||
|
||||
|
||||
class ExitStatus:
|
||||
SUCCESS = 0
|
||||
DIFF = 1
|
||||
TROUBLE = 2
|
||||
|
||||
|
||||
def excludes_from_file(ignore_file: str) -> List[str]:
|
||||
excludes: List[str] = []
|
||||
try:
|
||||
with io.open(ignore_file, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
if line.startswith("#"):
|
||||
# ignore comments
|
||||
continue
|
||||
pattern = line.rstrip()
|
||||
if not pattern:
|
||||
# allow empty lines
|
||||
continue
|
||||
excludes.append(pattern)
|
||||
except EnvironmentError as e:
|
||||
if e.errno != errno.ENOENT:
|
||||
raise
|
||||
return excludes
|
||||
|
||||
|
||||
def list_files(
|
||||
files: List[str],
|
||||
recursive: bool = False,
|
||||
extensions: Optional[List[str]] = None,
|
||||
exclude: Optional[List[str]] = None,
|
||||
) -> List[str]:
|
||||
if extensions is None:
|
||||
extensions = []
|
||||
if exclude is None:
|
||||
exclude = []
|
||||
|
||||
out: List[str] = []
|
||||
for file in files:
|
||||
if recursive and os.path.isdir(file):
|
||||
for dirpath, dnames, fnames in os.walk(file):
|
||||
fpaths = [os.path.join(dirpath, fname) for fname in fnames]
|
||||
for pattern in exclude:
|
||||
# os.walk() supports trimming down the dnames list
|
||||
# by modifying it in-place,
|
||||
# to avoid unnecessary directory listings.
|
||||
dnames[:] = [
|
||||
x
|
||||
for x in dnames
|
||||
if not fnmatch.fnmatch(os.path.join(dirpath, x), pattern)
|
||||
]
|
||||
fpaths = [x for x in fpaths if not fnmatch.fnmatch(x, pattern)]
|
||||
for f in fpaths:
|
||||
ext = os.path.splitext(f)[1][1:]
|
||||
if ext in extensions:
|
||||
out.append(f)
|
||||
else:
|
||||
out.append(file)
|
||||
return out
|
||||
|
||||
|
||||
def make_diff(file: str, original: List[str], reformatted: List[str]) -> List[str]:
|
||||
return list(
|
||||
difflib.unified_diff(
|
||||
original,
|
||||
reformatted,
|
||||
fromfile="{}\t(original)".format(file),
|
||||
tofile="{}\t(reformatted)".format(file),
|
||||
n=3,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class DiffError(Exception):
|
||||
def __init__(self, message: str, errs: Optional[List[str]] = None):
|
||||
super(DiffError, self).__init__(message)
|
||||
self.errs: List[str] = errs or []
|
||||
|
||||
|
||||
class UnexpectedError(Exception):
|
||||
def __init__(self, message: str, exc: Optional[Exception] = None):
|
||||
super(UnexpectedError, self).__init__(message)
|
||||
self.formatted_traceback = traceback.format_exc()
|
||||
self.exc = exc
|
||||
|
||||
|
||||
def run_clang_format_diff_wrapper(args: Any, file: str) -> Tuple[List[str], List[str]]:
|
||||
try:
|
||||
ret = run_clang_format_diff(args, file)
|
||||
return ret
|
||||
except DiffError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise UnexpectedError("{}: {}: {}".format(file, e.__class__.__name__, e), e)
|
||||
|
||||
|
||||
def run_clang_format_diff(args: Any, file: str) -> Tuple[List[str], List[str]]:
|
||||
try:
|
||||
with io.open(file, "r", encoding="utf-8") as f:
|
||||
original = f.readlines()
|
||||
except IOError as exc:
|
||||
raise DiffError(str(exc))
|
||||
|
||||
if args.in_place:
|
||||
invocation = [args.clang_format_executable, "-i", file]
|
||||
else:
|
||||
invocation = [args.clang_format_executable, file]
|
||||
|
||||
if args.style:
|
||||
invocation.extend(["--style", args.style])
|
||||
|
||||
if args.dry_run:
|
||||
print(" ".join(invocation))
|
||||
return [], []
|
||||
|
||||
# Use of utf-8 to decode the process output.
|
||||
#
|
||||
# Hopefully, this is the correct thing to do.
|
||||
#
|
||||
# It's done due to the following assumptions (which may be incorrect):
|
||||
# - clang-format will returns the bytes read from the files as-is,
|
||||
# without conversion, and it is already assumed that the files use utf-8.
|
||||
# - if the diagnostics were internationalized, they would use utf-8:
|
||||
# > Adding Translations to Clang
|
||||
# >
|
||||
# > Not possible yet!
|
||||
# > Diagnostic strings should be written in UTF-8,
|
||||
# > the client can translate to the relevant code page if needed.
|
||||
# > Each translation completely replaces the format string
|
||||
# > for the diagnostic.
|
||||
# > -- http://clang.llvm.org/docs/InternalsManual.html#internals-diag-translation
|
||||
#
|
||||
# It's not pretty, due to Python 2 & 3 compatibility.
|
||||
encoding_py3: Dict[str, str] = {}
|
||||
if sys.version_info[0] >= 3:
|
||||
encoding_py3["encoding"] = "utf-8"
|
||||
|
||||
try:
|
||||
if sys.version_info[0] >= 3:
|
||||
proc: subprocess.Popen[str] = subprocess.Popen(
|
||||
invocation,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
universal_newlines=True,
|
||||
encoding="utf-8",
|
||||
)
|
||||
else:
|
||||
proc: subprocess.Popen[str] = subprocess.Popen(
|
||||
invocation,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
universal_newlines=True,
|
||||
)
|
||||
except OSError as exc:
|
||||
raise DiffError(
|
||||
"Command '{}' failed to start: {}".format(
|
||||
subprocess.list2cmdline(invocation), exc
|
||||
)
|
||||
)
|
||||
proc_stdout = proc.stdout
|
||||
proc_stderr = proc.stderr
|
||||
assert proc_stdout is not None
|
||||
assert proc_stderr is not None
|
||||
if sys.version_info[0] < 3:
|
||||
# make the pipes compatible with Python 3,
|
||||
# reading lines should output unicode
|
||||
encoding = "utf-8"
|
||||
proc_stdout = codecs.getreader(encoding)(proc_stdout)
|
||||
proc_stderr = codecs.getreader(encoding)(proc_stderr)
|
||||
# hopefully the stderr pipe won't get full and block the process
|
||||
outs = list(proc_stdout.readlines())
|
||||
errs = list(proc_stderr.readlines())
|
||||
proc.wait()
|
||||
if proc.returncode:
|
||||
raise DiffError(
|
||||
"Command '{}' returned non-zero exit status {}".format(
|
||||
subprocess.list2cmdline(invocation), proc.returncode
|
||||
),
|
||||
errs,
|
||||
)
|
||||
if args.in_place:
|
||||
return [], errs
|
||||
return make_diff(file, original, outs), errs
|
||||
|
||||
|
||||
def bold_red(s: str) -> str:
|
||||
return "\x1b[1m\x1b[31m" + s + "\x1b[0m"
|
||||
|
||||
|
||||
def colorize(diff_lines: List[str]) -> Generator[str, None, None]:
|
||||
def bold(s: str) -> str:
|
||||
return "\x1b[1m" + s + "\x1b[0m"
|
||||
|
||||
def cyan(s: str) -> str:
|
||||
return "\x1b[36m" + s + "\x1b[0m"
|
||||
|
||||
def green(s: str) -> str:
|
||||
return "\x1b[32m" + s + "\x1b[0m"
|
||||
|
||||
def red(s: str) -> str:
|
||||
return "\x1b[31m" + s + "\x1b[0m"
|
||||
|
||||
for line in diff_lines:
|
||||
if line[:4] in ["--- ", "+++ "]:
|
||||
yield bold(line)
|
||||
elif line.startswith("@@ "):
|
||||
yield cyan(line)
|
||||
elif line.startswith("+"):
|
||||
yield green(line)
|
||||
elif line.startswith("-"):
|
||||
yield red(line)
|
||||
else:
|
||||
yield line
|
||||
|
||||
|
||||
def print_diff(diff_lines: List[str], use_color: bool) -> None:
|
||||
if use_color:
|
||||
diff_lines = list(colorize(diff_lines))
|
||||
if sys.version_info[0] < 3:
|
||||
sys.stdout.writelines((line.encode("utf-8") for line in diff_lines))
|
||||
else:
|
||||
sys.stdout.writelines(diff_lines)
|
||||
|
||||
|
||||
def print_trouble(prog: str, message: str, use_colors: bool) -> None:
|
||||
error_text = "error:"
|
||||
if use_colors:
|
||||
error_text = bold_red(error_text)
|
||||
print("{}: {} {}".format(prog, error_text, message), file=sys.stderr)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--clang-format-executable",
|
||||
metavar="EXECUTABLE",
|
||||
help="path to the clang-format executable",
|
||||
default="clang-format",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--extensions",
|
||||
help="comma separated list of file extensions (default: {})".format(
|
||||
DEFAULT_EXTENSIONS
|
||||
),
|
||||
default=DEFAULT_EXTENSIONS,
|
||||
)
|
||||
parser.add_argument(
|
||||
"-r",
|
||||
"--recursive",
|
||||
action="store_true",
|
||||
help="run recursively over directories",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-d", "--dry-run", action="store_true", help="just print the list of files"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-i",
|
||||
"--in-place",
|
||||
action="store_true",
|
||||
help="format file instead of printing differences",
|
||||
)
|
||||
parser.add_argument("files", metavar="file", nargs="+")
|
||||
parser.add_argument(
|
||||
"-q",
|
||||
"--quiet",
|
||||
action="store_true",
|
||||
help="disable output, useful for the exit code",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-j",
|
||||
metavar="N",
|
||||
type=int,
|
||||
default=0,
|
||||
help="run N clang-format jobs in parallel (default number of cpus + 1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--color",
|
||||
default="auto",
|
||||
choices=["auto", "always", "never"],
|
||||
help="show colored diff (default: auto)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-e",
|
||||
"--exclude",
|
||||
metavar="PATTERN",
|
||||
action="append",
|
||||
default=[],
|
||||
help="exclude paths matching the given glob-like pattern(s)"
|
||||
" from recursive search",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--style",
|
||||
help="formatting style to apply (LLVM, Google, Chromium, Mozilla, WebKit)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# use default signal handling, like diff return SIGINT value on ^C
|
||||
# https://bugs.python.org/issue14229#msg156446
|
||||
signal.signal(signal.SIGINT, signal.SIG_DFL)
|
||||
|
||||
try:
|
||||
signal.SIGPIPE # type: ignore
|
||||
except AttributeError:
|
||||
# compatibility, SIGPIPE does not exist on Windows
|
||||
pass
|
||||
else:
|
||||
signal.signal(signal.SIGPIPE, signal.SIG_DFL) # type: ignore
|
||||
|
||||
colored_stdout = False
|
||||
colored_stderr = False
|
||||
if args.color == "always":
|
||||
colored_stdout = True
|
||||
colored_stderr = True
|
||||
elif args.color == "auto":
|
||||
colored_stdout = sys.stdout.isatty()
|
||||
colored_stderr = sys.stderr.isatty()
|
||||
|
||||
version_invocation = [args.clang_format_executable, str("--version")]
|
||||
try:
|
||||
subprocess.check_call(version_invocation, stdout=DEVNULL)
|
||||
except subprocess.CalledProcessError as e:
|
||||
print_trouble(parser.prog, str(e), use_colors=colored_stderr)
|
||||
return ExitStatus.TROUBLE
|
||||
except OSError as e:
|
||||
print_trouble(
|
||||
parser.prog,
|
||||
"Command '{}' failed to start: {}".format(
|
||||
subprocess.list2cmdline(version_invocation), e
|
||||
),
|
||||
use_colors=colored_stderr,
|
||||
)
|
||||
return ExitStatus.TROUBLE
|
||||
|
||||
retcode = ExitStatus.SUCCESS
|
||||
|
||||
excludes = excludes_from_file(DEFAULT_CLANG_FORMAT_IGNORE)
|
||||
excludes.extend(args.exclude)
|
||||
|
||||
files = list_files(
|
||||
args.files,
|
||||
recursive=args.recursive,
|
||||
exclude=excludes,
|
||||
extensions=args.extensions.split(","),
|
||||
)
|
||||
|
||||
if not files:
|
||||
return retcode
|
||||
|
||||
njobs = args.j
|
||||
if njobs == 0:
|
||||
njobs = multiprocessing.cpu_count() + 1
|
||||
njobs = min(len(files), njobs)
|
||||
|
||||
if njobs == 1:
|
||||
# execute directly instead of in a pool,
|
||||
# less overhead, simpler stacktraces
|
||||
it = (run_clang_format_diff_wrapper(args, file) for file in files)
|
||||
pool = None
|
||||
else:
|
||||
pool = multiprocessing.Pool(njobs)
|
||||
it = pool.imap_unordered(partial(run_clang_format_diff_wrapper, args), files)
|
||||
pool.close()
|
||||
while True:
|
||||
try:
|
||||
outs, errs = next(it)
|
||||
except StopIteration:
|
||||
break
|
||||
except DiffError as e:
|
||||
print_trouble(parser.prog, str(e), use_colors=colored_stderr)
|
||||
retcode = ExitStatus.TROUBLE
|
||||
sys.stderr.writelines(e.errs)
|
||||
except UnexpectedError as e:
|
||||
print_trouble(parser.prog, str(e), use_colors=colored_stderr)
|
||||
sys.stderr.write(e.formatted_traceback)
|
||||
retcode = ExitStatus.TROUBLE
|
||||
# stop at the first unexpected error,
|
||||
# something could be very wrong,
|
||||
# don't process all files unnecessarily
|
||||
if pool:
|
||||
pool.terminate()
|
||||
break
|
||||
else:
|
||||
sys.stderr.writelines(errs)
|
||||
if outs == []:
|
||||
continue
|
||||
if not args.quiet:
|
||||
print_diff(outs, use_color=colored_stdout)
|
||||
if retcode == ExitStatus.SUCCESS:
|
||||
retcode = ExitStatus.DIFF
|
||||
if pool:
|
||||
pool.join()
|
||||
return retcode
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false
|
||||
"""
|
||||
Unit test file.
|
||||
"""
|
||||
|
||||
import unittest
|
||||
|
||||
|
||||
class ApiTester(unittest.TestCase):
|
||||
"""Main tester class."""
|
||||
|
||||
def test_build_all_examples(self) -> None:
|
||||
"""Test command line interface (CLI)."""
|
||||
from fastled import Api, Test # type: ignore
|
||||
|
||||
with Api.server(auto_updates=True) as server:
|
||||
exception_map = Test.test_examples(host=server)
|
||||
if len(exception_map) > 0:
|
||||
exception: Exception
|
||||
msg: str = ""
|
||||
for example, exception in exception_map.items():
|
||||
msg += f"Failed to compile example: {example}, error: {exception}\n"
|
||||
self.fail(msg)
|
||||
|
||||
# self.assertEqual(0, len(out), f"Failed tests: {out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,285 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Simple test runner for FastLED unit tests.
|
||||
Discovers and runs test executables in parallel with clean output handling.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
from ci.util.running_process import RunningProcess
|
||||
from ci.util.test_exceptions import (
|
||||
TestExecutionFailedException,
|
||||
TestFailureInfo,
|
||||
)
|
||||
|
||||
|
||||
_HERE = Path(__file__).parent
|
||||
_PROJECT_ROOT = _HERE.parent
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestResult:
|
||||
"""Result of a test execution"""
|
||||
|
||||
name: str
|
||||
success: bool
|
||||
duration: float
|
||||
output: str
|
||||
|
||||
|
||||
def _is_test_executable(f: Path) -> bool:
|
||||
"""Check if a file is a valid test executable"""
|
||||
return (
|
||||
f.is_file() and f.suffix not in [".o", ".obj", ".pdb"] and os.access(f, os.X_OK)
|
||||
)
|
||||
|
||||
|
||||
def _get_test_patterns() -> List[str]:
|
||||
"""Get test patterns based on platform"""
|
||||
# On Windows, check both .exe and no extension (Clang generates without .exe)
|
||||
return ["test_*.exe"] if sys.platform == "win32" else ["test_*"]
|
||||
|
||||
|
||||
def discover_tests(build_dir: Path, specific_test: Optional[str] = None) -> List[Path]:
|
||||
"""Find test executables in the build directory"""
|
||||
# Check test directory
|
||||
possible_test_dirs = [
|
||||
_PROJECT_ROOT / "tests" / "bin", # Python build system
|
||||
]
|
||||
|
||||
test_dir = None
|
||||
for possible_dir in possible_test_dirs:
|
||||
if possible_dir.exists():
|
||||
# Check if this directory has actual executable test files
|
||||
executable_tests = [
|
||||
f
|
||||
for pattern in _get_test_patterns()
|
||||
for f in possible_dir.glob(pattern)
|
||||
if _is_test_executable(f)
|
||||
]
|
||||
if executable_tests:
|
||||
test_dir = possible_dir
|
||||
break
|
||||
|
||||
if not test_dir:
|
||||
print(f"Error: No test directory found. Checked: {possible_test_dirs}")
|
||||
sys.exit(1)
|
||||
|
||||
test_files: List[Path] = []
|
||||
for pattern in _get_test_patterns():
|
||||
for test_file in test_dir.glob(pattern):
|
||||
if not _is_test_executable(test_file):
|
||||
continue
|
||||
if specific_test:
|
||||
# Support both "test_name" and "name" formats (case-insensitive)
|
||||
test_stem = test_file.stem
|
||||
test_name = test_stem.replace("test_", "")
|
||||
if (
|
||||
test_stem.lower() == specific_test.lower()
|
||||
or test_name.lower() == specific_test.lower()
|
||||
):
|
||||
test_files.append(test_file)
|
||||
else:
|
||||
test_files.append(test_file)
|
||||
|
||||
return test_files
|
||||
|
||||
|
||||
def run_test(test_file: Path, verbose: bool = False) -> TestResult:
|
||||
"""Run a single test and capture its output"""
|
||||
start_time = time.time()
|
||||
|
||||
# Build command with doctest flags
|
||||
# Convert to absolute path for Windows compatibility
|
||||
test_executable = test_file.resolve()
|
||||
cmd = [str(test_executable)]
|
||||
if not verbose:
|
||||
cmd.append("--minimal") # Only show output for failures
|
||||
|
||||
try:
|
||||
process = RunningProcess(
|
||||
command=cmd,
|
||||
cwd=Path(".").absolute(),
|
||||
check=False,
|
||||
shell=False,
|
||||
auto_run=True,
|
||||
)
|
||||
|
||||
with process.line_iter(timeout=30) as it:
|
||||
for line in it:
|
||||
print(line)
|
||||
|
||||
# Wait for process to complete
|
||||
process.wait()
|
||||
success = process.poll() == 0
|
||||
output = process.stdout
|
||||
|
||||
except Exception as e:
|
||||
success = False
|
||||
output = f"Error running test: {e}"
|
||||
|
||||
duration = time.time() - start_time
|
||||
return TestResult(
|
||||
name=test_file.stem, success=success, duration=duration, output=output
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
try:
|
||||
parser = argparse.ArgumentParser(description="Run FastLED unit tests")
|
||||
parser.add_argument("--test", help="Run specific test (without test_ prefix)")
|
||||
parser.add_argument(
|
||||
"--verbose", "-v", action="store_true", help="Show all test output"
|
||||
)
|
||||
parser.add_argument("--jobs", "-j", type=int, help="Number of parallel jobs")
|
||||
args = parser.parse_args()
|
||||
|
||||
# Find build directory
|
||||
build_dir = Path.cwd() / ".build"
|
||||
if not build_dir.exists():
|
||||
print("Error: Build directory not found. Run compilation first.")
|
||||
sys.exit(1)
|
||||
|
||||
# Discover tests
|
||||
test_files = discover_tests(build_dir, args.test)
|
||||
if not test_files:
|
||||
if args.test:
|
||||
print(f"Error: No test found matching '{args.test}'")
|
||||
else:
|
||||
print("Error: No tests found")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"Running {len(test_files)} tests...")
|
||||
if args.test:
|
||||
print(f"Filter: {args.test}")
|
||||
|
||||
# Determine number of parallel jobs
|
||||
if os.environ.get("NO_PARALLEL"):
|
||||
max_workers = 1
|
||||
print(
|
||||
"NO_PARALLEL environment variable set - forcing sequential test execution"
|
||||
)
|
||||
elif args.jobs:
|
||||
max_workers = args.jobs
|
||||
else:
|
||||
import multiprocessing
|
||||
|
||||
max_workers = max(1, multiprocessing.cpu_count() - 1)
|
||||
|
||||
# Run tests in parallel
|
||||
start_time = time.time()
|
||||
results: List[TestResult] = []
|
||||
failed_tests: List[TestResult] = []
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
future_to_test = {
|
||||
executor.submit(run_test, test_file, args.verbose): test_file
|
||||
for test_file in test_files
|
||||
}
|
||||
|
||||
completed = 0
|
||||
for future in as_completed(future_to_test):
|
||||
test_file = future_to_test[future]
|
||||
completed += 1
|
||||
try:
|
||||
result = future.result()
|
||||
results.append(result)
|
||||
|
||||
# Show progress
|
||||
status = "PASS" if result.success else "FAIL"
|
||||
print(
|
||||
f"[{completed}/{len(test_files)}] {status} {result.name} ({result.duration:.2f}s)"
|
||||
)
|
||||
|
||||
# Show output for failures or in verbose mode
|
||||
if not result.success or args.verbose:
|
||||
if result.output:
|
||||
print(result.output)
|
||||
|
||||
if not result.success:
|
||||
failed_tests.append(result)
|
||||
|
||||
# Early abort after 3 failures
|
||||
if len(failed_tests) >= 3:
|
||||
print(
|
||||
"Reached failure threshold (3). Aborting remaining unit tests."
|
||||
)
|
||||
# Cancel remaining futures and stop accepting new work
|
||||
try:
|
||||
executor.shutdown(wait=False, cancel_futures=True)
|
||||
except TypeError:
|
||||
# Python < 3.9: cancel_futures not available
|
||||
pass
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error running {test_file.name}: {e}")
|
||||
failed_tests.append(
|
||||
TestResult(
|
||||
name=test_file.stem,
|
||||
success=False,
|
||||
duration=0.0,
|
||||
output=str(e),
|
||||
)
|
||||
)
|
||||
if len(failed_tests) >= 3:
|
||||
print(
|
||||
"Reached failure threshold (3) due to errors. Aborting remaining unit tests."
|
||||
)
|
||||
try:
|
||||
executor.shutdown(wait=False, cancel_futures=True)
|
||||
except TypeError:
|
||||
pass
|
||||
break
|
||||
|
||||
# Show summary
|
||||
total_duration = time.time() - start_time
|
||||
success_count = len(results) - len(failed_tests)
|
||||
|
||||
print("\nTest Summary:")
|
||||
print(f"Total tests: {len(results)}")
|
||||
print(f"Passed: {success_count}")
|
||||
print(f"Failed: {len(failed_tests)}")
|
||||
print(f"Total time: {total_duration:.2f}s")
|
||||
|
||||
if failed_tests:
|
||||
print("\nFailed tests:")
|
||||
failures: list[TestFailureInfo] = []
|
||||
for test in failed_tests:
|
||||
print(f" {test.name}")
|
||||
failures.append(
|
||||
TestFailureInfo(
|
||||
test_name=test.name,
|
||||
command=f"test_{test.name}",
|
||||
return_code=1, # Failed tests get return code 1
|
||||
output=test.output,
|
||||
error_type="test_execution_failure",
|
||||
)
|
||||
)
|
||||
|
||||
raise TestExecutionFailedException(
|
||||
f"{len(failed_tests)} test(s) failed", failures
|
||||
)
|
||||
|
||||
except TestExecutionFailedException as e:
|
||||
# Print detailed failure information
|
||||
print("\n" + "=" * 60)
|
||||
print("FASTLED TEST EXECUTION FAILURE DETAILS")
|
||||
print("=" * 60)
|
||||
print(e.get_detailed_failure_info())
|
||||
print("=" * 60)
|
||||
|
||||
# Exit with appropriate code
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,302 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
|
||||
|
||||
"""
|
||||
Fast JavaScript linting setup for FastLED using Node.js + ESLint
|
||||
Downloads Node.js binary and sets up fast linting - much faster than Deno
|
||||
Uses uv run --script for dynamic package management
|
||||
"""
|
||||
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
# Import httpx for HTTP requests (dynamically managed by uv)
|
||||
import httpx
|
||||
|
||||
|
||||
# Force UTF-8 output for Windows consoles
|
||||
if sys.platform.startswith("win"):
|
||||
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
|
||||
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")
|
||||
|
||||
# Configuration
|
||||
NODE_VERSION = "20.11.0" # LTS version
|
||||
TOOLS_DIR = Path(".cache/js-tools")
|
||||
NODE_DIR = TOOLS_DIR / "node"
|
||||
|
||||
|
||||
def get_node_download_info():
|
||||
"""Get Node.js download URL and filename based on platform"""
|
||||
system = platform.system().lower()
|
||||
machine = platform.machine().lower()
|
||||
|
||||
# Map architecture names
|
||||
if machine in ["x86_64", "amd64"]:
|
||||
arch = "x64"
|
||||
elif machine in ["aarch64", "arm64"]:
|
||||
arch = "arm64"
|
||||
else:
|
||||
raise ValueError(f"Unsupported architecture: {machine}")
|
||||
|
||||
if system == "windows":
|
||||
filename = f"node-v{NODE_VERSION}-win-{arch}.zip"
|
||||
is_zip = True
|
||||
elif system == "darwin":
|
||||
filename = f"node-v{NODE_VERSION}-darwin-{arch}.tar.gz"
|
||||
is_zip = False
|
||||
elif system == "linux":
|
||||
filename = f"node-v{NODE_VERSION}-linux-{arch}.tar.xz"
|
||||
is_zip = False
|
||||
else:
|
||||
raise ValueError(f"Unsupported platform: {system}")
|
||||
|
||||
url = f"https://nodejs.org/dist/v{NODE_VERSION}/{filename}"
|
||||
return url, filename, is_zip
|
||||
|
||||
|
||||
def download_and_extract_node():
|
||||
"""Download and extract Node.js binary"""
|
||||
url, filename, is_zip = get_node_download_info()
|
||||
download_path = TOOLS_DIR / filename
|
||||
|
||||
# Get architecture for extraction path
|
||||
machine = platform.machine().lower()
|
||||
if machine in ["x86_64", "amd64"]:
|
||||
arch = "x64"
|
||||
elif machine in ["aarch64", "arm64"]:
|
||||
arch = "arm64"
|
||||
else:
|
||||
raise ValueError(f"Unsupported architecture: {machine}")
|
||||
|
||||
print(f"Downloading Node.js v{NODE_VERSION}...")
|
||||
TOOLS_DIR.mkdir(exist_ok=True)
|
||||
NODE_DIR.mkdir(exist_ok=True)
|
||||
|
||||
if not download_path.exists():
|
||||
print(f"Downloading from: {url}")
|
||||
with httpx.stream("GET", url, follow_redirects=True) as response:
|
||||
response.raise_for_status()
|
||||
with open(download_path, "wb") as f:
|
||||
for chunk in response.iter_bytes(chunk_size=8192):
|
||||
f.write(chunk)
|
||||
print(f"SUCCESS: Downloaded {filename}")
|
||||
|
||||
# Extract Node.js
|
||||
if platform.system() == "Windows":
|
||||
node_exe = NODE_DIR / "node.exe"
|
||||
else:
|
||||
node_exe = NODE_DIR / "bin" / "node"
|
||||
|
||||
if not node_exe.exists():
|
||||
print("Extracting Node.js...")
|
||||
|
||||
if is_zip:
|
||||
with zipfile.ZipFile(download_path, "r") as zip_ref:
|
||||
zip_ref.extractall(NODE_DIR)
|
||||
# Move contents from nested folder to NODE_DIR
|
||||
nested_dir = NODE_DIR / f"node-v{NODE_VERSION}-win-{arch}"
|
||||
if nested_dir.exists():
|
||||
for item in nested_dir.iterdir():
|
||||
if item.is_dir():
|
||||
# For directories, move contents recursively
|
||||
target_dir = NODE_DIR / item.name
|
||||
if target_dir.exists():
|
||||
shutil.rmtree(target_dir)
|
||||
shutil.move(str(item), str(target_dir))
|
||||
else:
|
||||
# For files, move directly
|
||||
target_file = NODE_DIR / item.name
|
||||
if target_file.exists():
|
||||
target_file.unlink()
|
||||
item.rename(target_file)
|
||||
nested_dir.rmdir()
|
||||
else:
|
||||
with tarfile.open(download_path, "r:*") as tar_ref:
|
||||
tar_ref.extractall(NODE_DIR)
|
||||
# Move contents from nested folder to NODE_DIR
|
||||
nested_dir = (
|
||||
NODE_DIR / f"node-v{NODE_VERSION}-{platform.system().lower()}-x64"
|
||||
)
|
||||
if nested_dir.exists():
|
||||
for item in nested_dir.iterdir():
|
||||
item.rename(NODE_DIR / item.name)
|
||||
nested_dir.rmdir()
|
||||
|
||||
print("SUCCESS: Node.js extracted")
|
||||
|
||||
# Clean up download file
|
||||
if download_path.exists():
|
||||
download_path.unlink()
|
||||
|
||||
return node_exe
|
||||
|
||||
|
||||
def setup_eslint():
|
||||
"""Install ESLint and create configuration"""
|
||||
if platform.system() == "Windows":
|
||||
node_exe = NODE_DIR / "node.exe"
|
||||
npm_exe = NODE_DIR / "npm.cmd"
|
||||
else:
|
||||
node_exe = NODE_DIR / "bin" / "node"
|
||||
npm_exe = NODE_DIR / "bin" / "npm"
|
||||
|
||||
print("Installing ESLint...")
|
||||
|
||||
# Create package.json
|
||||
package_json = {
|
||||
"name": "fastled-js-linting",
|
||||
"version": "1.0.0",
|
||||
"private": True,
|
||||
"dependencies": {"eslint": "^8.56.0"},
|
||||
}
|
||||
|
||||
with open(TOOLS_DIR / "package.json", "w") as f:
|
||||
json.dump(package_json, f, indent=2)
|
||||
|
||||
# Install ESLint
|
||||
try:
|
||||
# Use absolute path for npm executable
|
||||
npm_exe_abs = (
|
||||
(TOOLS_DIR / "node" / "npm.cmd").resolve()
|
||||
if platform.system() == "Windows"
|
||||
else (TOOLS_DIR / "node" / "bin" / "npm").resolve()
|
||||
)
|
||||
|
||||
# On Windows, use shell=True to properly execute .cmd files
|
||||
if platform.system() == "Windows":
|
||||
result = subprocess.run(
|
||||
[str(npm_exe_abs), "install"],
|
||||
cwd=TOOLS_DIR,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
shell=True,
|
||||
)
|
||||
else:
|
||||
result = subprocess.run(
|
||||
[str(npm_exe_abs), "install"],
|
||||
cwd=TOOLS_DIR,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
# npm install succeeded
|
||||
except subprocess.CalledProcessError as e:
|
||||
# Show stderr output for debugging
|
||||
print(f"npm install stderr: {e.stderr}")
|
||||
print(f"npm install stdout: {e.stdout}")
|
||||
raise
|
||||
|
||||
# Create minimal .eslintrc.js - FAST ONLY: Critical issues only
|
||||
eslint_config = """module.exports = {
|
||||
env: {
|
||||
browser: true,
|
||||
es2022: true,
|
||||
worker: true
|
||||
},
|
||||
parserOptions: {
|
||||
ecmaVersion: 2022,
|
||||
sourceType: "module"
|
||||
},
|
||||
rules: {
|
||||
// FAST LINTING: Only check for critical runtime issues
|
||||
"no-debugger": "error",
|
||||
"no-eval": "error"
|
||||
}
|
||||
};"""
|
||||
|
||||
with open(TOOLS_DIR / ".eslintrc.js", "w") as f:
|
||||
f.write(eslint_config)
|
||||
|
||||
print("SUCCESS: ESLint configured")
|
||||
|
||||
|
||||
def create_fast_lint_script():
|
||||
"""Create fast linting script"""
|
||||
if platform.system() == "Windows":
|
||||
node_exe = NODE_DIR / "node.exe"
|
||||
eslint_exe = TOOLS_DIR / "node_modules" / ".bin" / "eslint.cmd"
|
||||
else:
|
||||
node_exe = NODE_DIR / "bin" / "node"
|
||||
eslint_exe = TOOLS_DIR / "node_modules" / ".bin" / "eslint"
|
||||
|
||||
script_content = f"""#!/bin/bash
|
||||
# FastLED JavaScript Linting Script (Node.js + ESLint - FAST!)
|
||||
|
||||
# Colors for output
|
||||
RED='\\033[0;31m'
|
||||
GREEN='\\033[0;32m'
|
||||
YELLOW='\\033[1;33m'
|
||||
BLUE='\\033[0;34m'
|
||||
NC='\\033[0m' # No Color
|
||||
|
||||
echo -e "${{BLUE}}FAST FastLED JavaScript Linting (Node.js + ESLint - FAST!)${{NC}}"
|
||||
|
||||
# Check if ESLint is installed
|
||||
if [ ! -f ".cache/js-tools/node_modules/.bin/eslint{".cmd" if platform.system() == "Windows" else ""}" ]; then
|
||||
echo -e "${{RED}}ERROR: ESLint not found. Run: uv run ci/setup-js-linting-fast.py${{NC}}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Find JavaScript files in WASM platform
|
||||
JS_FILES=$(find src/platforms/wasm -name "*.js" -type f 2>/dev/null)
|
||||
|
||||
if [ -z "$JS_FILES" ]; then
|
||||
echo -e "${{YELLOW}}WARNING: No JavaScript files found in src/platforms/wasm/${{NC}}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo -e "${{BLUE}}Found JavaScript files:${{NC}}"
|
||||
echo "$JS_FILES" | sed 's/^/ /'
|
||||
|
||||
# Run ESLint
|
||||
echo -e "${{BLUE}}Running ESLint...${{NC}}"
|
||||
cd .cache/js-tools
|
||||
if "./node_modules/.bin/eslint{".cmd" if platform.system() == "Windows" else ""}" --no-eslintrc --no-inline-config -c .eslintrc.js ../../src/platforms/wasm/compiler/*.js ../../src/platforms/wasm/compiler/modules/*.js; then
|
||||
echo -e "${{GREEN}}SUCCESS: JavaScript linting completed successfully${{NC}}"
|
||||
else
|
||||
echo -e "${{RED}}ERROR: JavaScript linting failed${{NC}}"
|
||||
exit 1
|
||||
fi
|
||||
"""
|
||||
|
||||
script_path = TOOLS_DIR / "lint-js-fast"
|
||||
with open(script_path, "w") as f:
|
||||
f.write(script_content)
|
||||
|
||||
# Make executable on Unix systems
|
||||
if platform.system() != "Windows":
|
||||
os.chmod(script_path, 0o755)
|
||||
|
||||
print("SUCCESS: Fast lint script created")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main setup function"""
|
||||
print("Setting up fast JavaScript linting (Node.js + ESLint)...")
|
||||
|
||||
try:
|
||||
node_exe = download_and_extract_node()
|
||||
setup_eslint()
|
||||
create_fast_lint_script()
|
||||
|
||||
print("\\nFast JavaScript linting setup complete!")
|
||||
print("\\nUsage:")
|
||||
print(" bash .cache/js-tools/lint-js-fast # Fast linting with ESLint")
|
||||
print(" For more info: ci/js/README.md")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Setup failed: {e}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,969 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import _thread
|
||||
import atexit
|
||||
import json
|
||||
import lzma
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Protocol, Tuple, cast
|
||||
|
||||
from ci.util.resumable_downloader import ResumableDownloader
|
||||
|
||||
|
||||
class _SysConf(Protocol):
|
||||
def short_os_name_and_version(self) -> str: ...
|
||||
|
||||
architecture: str
|
||||
|
||||
|
||||
import sys_detection
|
||||
from llvm_installer import LlvmInstaller
|
||||
|
||||
|
||||
_HERE = Path(__file__).parent
|
||||
_PROJECT_ROOT = _HERE.parent
|
||||
|
||||
# Global download directory - lazily initialized
|
||||
_download_dir: Optional[Path] = None
|
||||
|
||||
|
||||
def _get_download_dir() -> Path:
|
||||
"""Get or create the download directory lazily."""
|
||||
global _download_dir
|
||||
if _download_dir is None:
|
||||
_download_dir = _PROJECT_ROOT / ".cache" / "downloads"
|
||||
_download_dir.mkdir(parents=True, exist_ok=True)
|
||||
print(f"Created download directory: {_download_dir}")
|
||||
# Register cleanup on exit as fallback
|
||||
atexit.register(_cleanup_download_dir)
|
||||
return _download_dir
|
||||
|
||||
|
||||
def _cleanup_download_dir() -> None:
|
||||
"""Clean up the download directory."""
|
||||
global _download_dir
|
||||
if _download_dir and _download_dir.exists():
|
||||
try:
|
||||
shutil.rmtree(_download_dir)
|
||||
print(f"Cleaned up download directory: {_download_dir}")
|
||||
except Exception as e:
|
||||
print(
|
||||
f"Warning: Failed to clean up download directory {_download_dir}: {e}"
|
||||
)
|
||||
_download_dir = None
|
||||
|
||||
|
||||
def _download(url: str, filename: str) -> Path:
|
||||
"""Download to the managed download directory.
|
||||
|
||||
Args:
|
||||
url: URL to download
|
||||
filename: Filename to save as (not full path)
|
||||
|
||||
Returns:
|
||||
Path to the downloaded file
|
||||
"""
|
||||
download_dir = _get_download_dir()
|
||||
file_path = download_dir / filename
|
||||
|
||||
print(f"download {url} to {file_path}")
|
||||
|
||||
# Use resumable downloader for large files
|
||||
downloader = ResumableDownloader(chunk_size=1024 * 1024) # 1MB chunks
|
||||
|
||||
try:
|
||||
downloader.download(url, file_path)
|
||||
except Exception as e:
|
||||
# Print download failure banner
|
||||
url_truncated = url[:100]
|
||||
banner_width = max(35, len(url_truncated) + 6)
|
||||
banner_line = "#" * banner_width
|
||||
print(f"\n{banner_line}")
|
||||
print(
|
||||
f"# DOWNLOAD FAILED FOR {url_truncated}{'...' if len(url) > 100 else ''} #"
|
||||
)
|
||||
print(f"{banner_line}")
|
||||
print(f"Error: {e}")
|
||||
print()
|
||||
raise
|
||||
|
||||
return file_path
|
||||
|
||||
|
||||
def is_linux_or_unix() -> Tuple[bool, str]:
|
||||
"""Check if running on Linux or Unix-like system."""
|
||||
system = platform.system().lower()
|
||||
if system == "linux":
|
||||
return True, "linux"
|
||||
elif system in ["darwin", "freebsd", "openbsd", "netbsd"]:
|
||||
return True, system
|
||||
else:
|
||||
return False, system
|
||||
|
||||
|
||||
def get_tool_version(tool_name: str) -> Optional[str]:
|
||||
"""Get version of a tool if it exists on system PATH."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[tool_name, "--version"], capture_output=True, text=True, timeout=5
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return result.stdout.strip()
|
||||
return None
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError, subprocess.SubprocessError):
|
||||
return None
|
||||
|
||||
|
||||
def get_clang_version_number(version_output: str) -> Optional[int]:
|
||||
"""Extract major version number from clang version output."""
|
||||
try:
|
||||
# Look for "clang version X.Y.Z" pattern
|
||||
import re
|
||||
|
||||
match = re.search(r"clang version (\d+)\.", version_output)
|
||||
if match:
|
||||
return int(match.group(1))
|
||||
return None
|
||||
except (ValueError, AttributeError):
|
||||
return None
|
||||
|
||||
|
||||
def find_tool_path(tool_name: str) -> Optional[Path]:
|
||||
"""Find the full path to a tool if it exists."""
|
||||
tool_path = shutil.which(tool_name)
|
||||
return Path(tool_path) if tool_path else None
|
||||
|
||||
|
||||
def check_required_tools() -> Tuple[
|
||||
Dict[str, Optional[Path]], Dict[str, Optional[Path]]
|
||||
]:
|
||||
"""Check for essential and extra LLVM/Clang tools on system PATH.
|
||||
|
||||
On Windows, prefer lld-link as the linker front-end.
|
||||
"""
|
||||
system_name = platform.system().lower()
|
||||
|
||||
essential_tools: List[str] = [
|
||||
"clang",
|
||||
"clang++",
|
||||
"llvm-ar",
|
||||
"llvm-nm",
|
||||
"llvm-objdump",
|
||||
"llvm-addr2line",
|
||||
"llvm-strip",
|
||||
"llvm-objcopy",
|
||||
]
|
||||
|
||||
# Linker tools differ across platforms
|
||||
if system_name == "windows":
|
||||
essential_tools.append("lld-link")
|
||||
else:
|
||||
essential_tools.append("lld")
|
||||
|
||||
# Useful developer tools (not strictly required for building)
|
||||
extra_tools: List[str] = [
|
||||
"clangd",
|
||||
"clang-format",
|
||||
"clang-tidy",
|
||||
"lldb",
|
||||
]
|
||||
|
||||
found_essential: Dict[str, Optional[Path]] = {}
|
||||
for tool in essential_tools:
|
||||
found_essential[tool] = find_tool_path(tool)
|
||||
|
||||
found_extras: Dict[str, Optional[Path]] = {}
|
||||
for tool in extra_tools:
|
||||
found_extras[tool] = find_tool_path(tool)
|
||||
|
||||
return found_essential, found_extras
|
||||
|
||||
|
||||
def create_hard_links(source_tools: Dict[str, Path], target_dir: Path) -> None:
|
||||
"""Create hard links from system tools to target directory."""
|
||||
bin_dir = target_dir / "bin"
|
||||
bin_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
linked_count: int = 0
|
||||
copied_count: int = 0
|
||||
failed_tools: list[str] = []
|
||||
|
||||
for tool_name, source_path in source_tools.items():
|
||||
if source_path and source_path.exists():
|
||||
# Preserve original filename to keep extensions on Windows (e.g., .exe)
|
||||
target_filename = source_path.name
|
||||
target_path = bin_dir / target_filename
|
||||
try:
|
||||
# Remove existing file if it exists
|
||||
if target_path.exists():
|
||||
target_path.unlink()
|
||||
|
||||
# Create hard link
|
||||
target_path.hardlink_to(source_path)
|
||||
print(f"[OK] Linked {tool_name}: {source_path} -> {target_path}")
|
||||
linked_count += 1
|
||||
except (OSError, PermissionError) as e:
|
||||
print(f"[WARN] Failed to link {tool_name}: {e}")
|
||||
# Fallback to copy if hard link fails
|
||||
try:
|
||||
shutil.copy2(source_path, target_path)
|
||||
target_path.chmod(0o755)
|
||||
print(f"[OK] Copied {tool_name}: {source_path} -> {target_path}")
|
||||
copied_count += 1
|
||||
except (OSError, PermissionError):
|
||||
print(f"[ERROR] Failed to copy {tool_name}")
|
||||
failed_tools.append(tool_name)
|
||||
|
||||
total_prepared: int = linked_count + copied_count
|
||||
print(
|
||||
f"Summary: prepared {total_prepared} tools (linked: {linked_count}, copied: {copied_count})"
|
||||
)
|
||||
if failed_tools:
|
||||
print(f"ERROR: Failed to mirror tools: {', '.join(failed_tools)}")
|
||||
|
||||
|
||||
def find_tools_in_directories(
|
||||
tool_names: List[str], directories: List[Path]
|
||||
) -> Dict[str, Optional[Path]]:
|
||||
"""Search for tools across a list of directories, returning first matches.
|
||||
|
||||
On Windows, will match executables with .exe suffix.
|
||||
"""
|
||||
found: Dict[str, Optional[Path]] = {}
|
||||
for tool in tool_names:
|
||||
found[tool] = None
|
||||
|
||||
for directory in directories:
|
||||
if not directory.exists():
|
||||
continue
|
||||
for tool in tool_names:
|
||||
if found[tool] is not None:
|
||||
continue
|
||||
candidate: Path = directory / tool
|
||||
if candidate.exists():
|
||||
found[tool] = candidate
|
||||
continue
|
||||
# Try platform-specific suffix
|
||||
if platform.system().lower() == "windows":
|
||||
exe_candidate: Path = directory / f"{tool}.exe"
|
||||
if exe_candidate.exists():
|
||||
found[tool] = exe_candidate
|
||||
return found
|
||||
|
||||
|
||||
def get_breadcrumb_file(target_dir: Path) -> Path:
|
||||
"""Get the path to the installation completion breadcrumb file."""
|
||||
cache_dir = target_dir / ".cache" / "cc"
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
return cache_dir / "done.txt"
|
||||
|
||||
|
||||
def create_breadcrumb(target_dir: Path) -> None:
|
||||
"""Create breadcrumb file to mark successful installation."""
|
||||
breadcrumb = get_breadcrumb_file(target_dir)
|
||||
breadcrumb.write_text(f"Installation completed at {os.path.basename(target_dir)}\n")
|
||||
print(f"Created installation breadcrumb: {breadcrumb}")
|
||||
|
||||
|
||||
def check_existing_installation(target_dir: Path) -> Tuple[bool, bool]:
|
||||
"""Check if there's an existing installation and if it's complete.
|
||||
|
||||
Returns:
|
||||
(has_artifacts, has_breadcrumb) - bool tuple indicating installation state
|
||||
"""
|
||||
breadcrumb = get_breadcrumb_file(target_dir)
|
||||
has_breadcrumb = breadcrumb.exists()
|
||||
|
||||
# Check for any installation artifacts
|
||||
bin_dir = target_dir / "bin"
|
||||
has_artifacts = bin_dir.exists() and any(bin_dir.iterdir())
|
||||
|
||||
return has_artifacts, has_breadcrumb
|
||||
|
||||
|
||||
def verify_clang_installation(target_dir: Path) -> bool:
|
||||
"""Test if clang is properly installed in target directory."""
|
||||
bin_dir = target_dir / "bin"
|
||||
clang_exe = (
|
||||
bin_dir / "clang.exe"
|
||||
if platform.system().lower() == "windows"
|
||||
else bin_dir / "clang"
|
||||
)
|
||||
|
||||
if not clang_exe.exists():
|
||||
return False
|
||||
|
||||
# Test that clang can run and report version
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[str(clang_exe), "--version"], capture_output=True, text=True, timeout=5
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError, subprocess.SubprocessError):
|
||||
return False
|
||||
|
||||
|
||||
def get_windows_default_llvm_bins() -> List[Path]:
|
||||
"""Return common Windows LLVM bin directories to probe."""
|
||||
roots: List[Path] = []
|
||||
program_files: Optional[str] = os.environ.get("ProgramFiles")
|
||||
program_files_x86: Optional[str] = os.environ.get("ProgramFiles(x86)")
|
||||
chocolatey_lib: Optional[str] = os.environ.get("ChocolateyInstall")
|
||||
scoop: Optional[str] = os.environ.get("SCOOP")
|
||||
|
||||
if program_files:
|
||||
roots.append(Path(program_files) / "LLVM" / "bin")
|
||||
if program_files_x86:
|
||||
roots.append(Path(program_files_x86) / "LLVM" / "bin")
|
||||
if chocolatey_lib:
|
||||
roots.append(Path(chocolatey_lib) / "lib" / "llvm" / "tools" / "llvm" / "bin")
|
||||
if scoop:
|
||||
roots.append(Path(scoop) / "apps" / "llvm" / "current" / "bin")
|
||||
|
||||
# Also consider PATH entries explicitly
|
||||
path_entries: List[str] = os.environ.get("PATH", "").split(os.pathsep)
|
||||
for entry in path_entries:
|
||||
# Only consider entries that look like LLVM bin directories
|
||||
if "llvm" in entry.lower() and "bin" in entry.lower():
|
||||
roots.append(Path(entry))
|
||||
return roots
|
||||
|
||||
|
||||
def _run_subprocess(args: List[str], timeout_seconds: int) -> tuple[int, str, str]:
|
||||
"""Run a subprocess command and return (returncode, stdout, stderr)."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
args,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout_seconds,
|
||||
)
|
||||
return result.returncode, result.stdout, result.stderr
|
||||
except subprocess.TimeoutExpired as e:
|
||||
return 124, "", f"Timeout after {timeout_seconds}s: {e}"
|
||||
except subprocess.SubprocessError as e:
|
||||
return 1, "", f"Subprocess error: {e}"
|
||||
|
||||
|
||||
def install_llvm_windows_direct(target_dir: Path) -> Path:
|
||||
"""Download and extract official LLVM Windows tar.xz archive into target_dir.
|
||||
|
||||
Returns the extracted LLVM bin directory path on success.
|
||||
Fails fast with a descriptive error on failure.
|
||||
"""
|
||||
# Prefer dynamically discovered releases; as fallback, try a small curated list
|
||||
preferred_versions: List[str] = [
|
||||
"20.1.8",
|
||||
"20.1.7",
|
||||
"20.1.6",
|
||||
"19.1.8",
|
||||
"19.1.7",
|
||||
"19.1.6",
|
||||
]
|
||||
|
||||
download_errors: List[str] = []
|
||||
|
||||
# First, try to discover the latest available Windows tar.xz for majors 20 and 19
|
||||
for probe_major in [20, 19]:
|
||||
print(f"Attempting discovery for LLVM major version {probe_major}...")
|
||||
discovered_url = discover_latest_github_llvm_win_zip(probe_major)
|
||||
if discovered_url is not None:
|
||||
print(f"Discovered URL for LLVM {probe_major}: {discovered_url}")
|
||||
# Put discovered URL at the front of attempts
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
extract_root: Path = target_dir
|
||||
|
||||
tmp_archive_path: Optional[Path] = None
|
||||
try:
|
||||
print(f"Attempting to download and extract LLVM {probe_major}...")
|
||||
# Download to managed directory
|
||||
tmp_archive_path = _download(
|
||||
discovered_url, f"llvm-discovered-{probe_major}.tar.xz"
|
||||
)
|
||||
extract_root = target_dir
|
||||
with lzma.open(tmp_archive_path, "rb") as xz_file:
|
||||
with tarfile.open(fileobj=xz_file, mode="r") as tar:
|
||||
tar.extractall(extract_root, filter="data")
|
||||
except KeyboardInterrupt:
|
||||
print("Operation interrupted by user")
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Failed to download/extract LLVM {probe_major}: {e}")
|
||||
download_errors.append(f"discovered-{probe_major}: {e}")
|
||||
finally:
|
||||
# Clean up the downloaded file immediately
|
||||
if tmp_archive_path and tmp_archive_path.exists():
|
||||
try:
|
||||
tmp_archive_path.unlink()
|
||||
print(f"Cleaned up: {tmp_archive_path}")
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to clean up {tmp_archive_path}: {e}")
|
||||
|
||||
extracted_bin: Optional[Path] = None
|
||||
for entry in extract_root.iterdir():
|
||||
if not entry.is_dir():
|
||||
continue
|
||||
candidate = entry / "bin" / "clang.exe"
|
||||
if candidate.exists():
|
||||
extracted_bin = entry / "bin"
|
||||
break
|
||||
|
||||
if extracted_bin is not None:
|
||||
print(
|
||||
f"SUCCESS: LLVM {probe_major} installed successfully via discovery!"
|
||||
)
|
||||
print("OK: Successfully installed LLVM (Windows tar.xz, discovered)")
|
||||
print(f" Installation directory: {extracted_bin.parent}")
|
||||
print(f" Bin directory: {extracted_bin}")
|
||||
create_breadcrumb(target_dir)
|
||||
return extracted_bin
|
||||
else:
|
||||
print(
|
||||
f"WARNING: LLVM {probe_major} downloaded but clang.exe not found in expected location"
|
||||
)
|
||||
else:
|
||||
print(f"No URL discovered for LLVM major version {probe_major}")
|
||||
|
||||
# If discovery did not succeed, try curated versions list
|
||||
print("\n" + "=" * 60)
|
||||
print("Discovery failed for all major versions. Falling back to curated list...")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
for version in preferred_versions:
|
||||
base_url = (
|
||||
"https://github.com/llvm/llvm-project/releases/download/"
|
||||
f"llvmorg-{version}/clang+llvm-{version}-x86_64-pc-windows-msvc.tar.xz"
|
||||
)
|
||||
print(f"Attempting direct LLVM download for Windows: {base_url}")
|
||||
try:
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
tmp_archive_path: Optional[Path] = None
|
||||
try:
|
||||
# Download to managed directory
|
||||
tmp_archive_path = _download(base_url, f"llvm-{version}.tar.xz")
|
||||
except KeyboardInterrupt:
|
||||
print("Operation interrupted by user")
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
download_errors.append(f"{version}: {e}")
|
||||
# Clean up on download error
|
||||
if tmp_archive_path and tmp_archive_path.exists():
|
||||
try:
|
||||
tmp_archive_path.unlink()
|
||||
print(f"Cleaned up after error: {tmp_archive_path}")
|
||||
except Exception:
|
||||
pass
|
||||
continue
|
||||
|
||||
extract_root = target_dir
|
||||
try:
|
||||
with lzma.open(tmp_archive_path, "rb") as xz_file:
|
||||
with tarfile.open(fileobj=xz_file, mode="r") as tar:
|
||||
tar.extractall(extract_root, filter="data")
|
||||
finally:
|
||||
# Clean up the downloaded file immediately
|
||||
if tmp_archive_path and tmp_archive_path.exists():
|
||||
try:
|
||||
tmp_archive_path.unlink()
|
||||
print(f"Cleaned up: {tmp_archive_path}")
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to clean up {tmp_archive_path}: {e}")
|
||||
|
||||
# Find extracted directory that contains bin/clang.exe
|
||||
extracted_bin: Optional[Path] = None
|
||||
for entry in extract_root.iterdir():
|
||||
if not entry.is_dir():
|
||||
continue
|
||||
candidate = entry / "bin" / "clang.exe"
|
||||
if candidate.exists():
|
||||
extracted_bin = entry / "bin"
|
||||
break
|
||||
|
||||
if extracted_bin is None:
|
||||
print(
|
||||
"ERROR: LLVM downloaded but clang.exe not found in expected location"
|
||||
)
|
||||
# Try next version
|
||||
continue
|
||||
|
||||
print("OK: Successfully installed LLVM (Windows tar.xz)")
|
||||
print(f" Installation directory: {extracted_bin.parent}")
|
||||
print(f" Bin directory: {extracted_bin}")
|
||||
create_breadcrumb(target_dir)
|
||||
return extracted_bin
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("Operation interrupted by user")
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
download_errors.append(f"{version}: {e}")
|
||||
continue
|
||||
|
||||
# If we reach here, all attempts failed
|
||||
print("CRITICAL: Failed to download LLVM for Windows from official releases.")
|
||||
if download_errors:
|
||||
for err in download_errors:
|
||||
print(f" - {err}")
|
||||
print("Please ensure internet access is available and retry.")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def discover_latest_github_llvm_win_zip(major: int) -> Optional[str]:
|
||||
"""Find the latest LLVM Windows tar.xz asset for a given major version via GitHub API.
|
||||
|
||||
Returns a browser_download_url string if found, otherwise None.
|
||||
"""
|
||||
try:
|
||||
per_page: int = 50
|
||||
page: int = 1
|
||||
max_pages: int = 3
|
||||
while page <= max_pages:
|
||||
api_url: str = (
|
||||
"https://api.github.com/repos/llvm/llvm-project/releases"
|
||||
f"?per_page={per_page}&page={page}"
|
||||
)
|
||||
req = urllib.request.Request(
|
||||
api_url, headers={"User-Agent": "fastled-setup-llvm"}
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=20) as resp:
|
||||
raw: str = resp.read().decode("utf-8")
|
||||
releases_any: Any = json.loads(raw)
|
||||
releases: list[dict[str, Any]] = []
|
||||
if isinstance(releases_any, list):
|
||||
tmp_list: list[Any] = cast(list[Any], releases_any)
|
||||
filtered: list[dict[str, Any]] = []
|
||||
k: int = 0
|
||||
while k < len(tmp_list):
|
||||
item: Any = tmp_list[k]
|
||||
k += 1
|
||||
if isinstance(item, dict):
|
||||
filtered.append(cast(dict[str, Any], item))
|
||||
releases = filtered
|
||||
|
||||
# Iterate releases
|
||||
idx: int = 0
|
||||
while idx < len(releases):
|
||||
rel: dict[str, Any] = releases[idx]
|
||||
idx += 1
|
||||
tag_val: Any = rel.get("tag_name")
|
||||
tag_name: Optional[str] = tag_val if isinstance(tag_val, str) else None
|
||||
if tag_name is None:
|
||||
continue
|
||||
# Expect tags like llvmorg-19.1.6
|
||||
prefix: str = f"llvmorg-{major}."
|
||||
if not tag_name.startswith(prefix):
|
||||
continue
|
||||
assets_val: Any = rel.get("assets")
|
||||
assets: list[dict[str, Any]] = (
|
||||
cast(list[dict[str, Any]], assets_val)
|
||||
if isinstance(assets_val, list)
|
||||
else []
|
||||
)
|
||||
filtered_assets: list[dict[str, Any]] = [
|
||||
a for a in assets if isinstance(a, dict)
|
||||
]
|
||||
j: int = 0
|
||||
while j < len(filtered_assets):
|
||||
asset: dict[str, Any] = filtered_assets[j]
|
||||
j += 1
|
||||
name_val: Any = asset.get("name")
|
||||
url_val: Any = asset.get("browser_download_url")
|
||||
name: Optional[str] = (
|
||||
name_val if isinstance(name_val, str) else None
|
||||
)
|
||||
url: Optional[str] = url_val if isinstance(url_val, str) else None
|
||||
if name is not None and url is not None:
|
||||
# Match e.g., clang+llvm-19.1.6-x86_64-pc-windows-msvc.tar.xz
|
||||
if (
|
||||
name.startswith(f"clang+llvm-{major}.")
|
||||
and "x86_64-pc-windows-msvc.tar.xz" in name
|
||||
):
|
||||
return url
|
||||
page += 1
|
||||
return None
|
||||
except urllib.error.URLError:
|
||||
return None
|
||||
|
||||
|
||||
def download_and_install_llvm(target_dir: Path) -> None:
|
||||
"""Download and install LLVM using llvm-installer package."""
|
||||
print("Detecting system configuration...")
|
||||
try:
|
||||
sys_conf_obj2: Any = sys_detection.local_sys_conf()
|
||||
local_conf2: _SysConf = cast(_SysConf, sys_conf_obj2)
|
||||
detected_os: str = local_conf2.short_os_name_and_version()
|
||||
architecture: str = local_conf2.architecture
|
||||
print(f"Detected OS: {detected_os}")
|
||||
print(f"Architecture: {architecture}")
|
||||
|
||||
# Use Ubuntu 24.04 as fallback since it's known to work
|
||||
# This package seems to be from YugabyteDB and has limited OS support
|
||||
working_os: str = "ubuntu24.04"
|
||||
if detected_os.startswith("ubuntu24") or detected_os.startswith("ubuntu22"):
|
||||
working_os = detected_os
|
||||
|
||||
print(f"Using OS variant: {working_os}")
|
||||
|
||||
llvm_installer: Any = LlvmInstaller(
|
||||
short_os_name_and_version=working_os, architecture=architecture
|
||||
)
|
||||
|
||||
# Prefer most recent supported major release for linux tarball path
|
||||
llvm_url: str = ""
|
||||
selected_version: Optional[int] = None
|
||||
for version in [20, 19]:
|
||||
try:
|
||||
print(f"Installing LLVM version {version}...")
|
||||
llvm_url = llvm_installer.get_llvm_url(major_llvm_version=version)
|
||||
print(f"Found LLVM {version} at: {llvm_url}")
|
||||
selected_version = version
|
||||
break
|
||||
except KeyboardInterrupt:
|
||||
print("Operation interrupted by user")
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"WARN: Could not resolve LLVM {version}: {e}")
|
||||
|
||||
if not llvm_url:
|
||||
raise RuntimeError(
|
||||
"Could not resolve a suitable LLVM download URL for Linux"
|
||||
)
|
||||
|
||||
# Download and extract
|
||||
print("Downloading LLVM package...")
|
||||
|
||||
# Download to managed directory
|
||||
tmp_archive_path: Path = _download(
|
||||
llvm_url, f"llvm-linux-{selected_version or 'unknown'}.tar.gz"
|
||||
)
|
||||
|
||||
try:
|
||||
print("Extracting LLVM package...")
|
||||
with tarfile.open(tmp_archive_path, "r:gz") as tar:
|
||||
tar.extractall(target_dir, filter="data")
|
||||
finally:
|
||||
# Clean up immediately after extraction
|
||||
try:
|
||||
tmp_archive_path.unlink()
|
||||
print(f"Cleaned up: {tmp_archive_path}")
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to clean up {tmp_archive_path}: {e}")
|
||||
|
||||
# Find the extracted directory and verify clang exists
|
||||
for extracted_dir in target_dir.iterdir():
|
||||
if extracted_dir.is_dir():
|
||||
clang_path = extracted_dir / "bin" / "clang"
|
||||
|
||||
if clang_path.exists():
|
||||
print(
|
||||
f"OK: Successfully installed LLVM {selected_version if selected_version is not None else 'unknown'}"
|
||||
)
|
||||
print(f" Installation directory: {extracted_dir}")
|
||||
print(f" Clang binary: {clang_path}")
|
||||
|
||||
# List available tools (first few) and provide PATH guidance
|
||||
bin_dir = extracted_dir / "bin"
|
||||
if bin_dir.exists():
|
||||
tools = [
|
||||
f.name
|
||||
for f in bin_dir.iterdir()
|
||||
if f.is_file() and not f.name.endswith(".dll")
|
||||
]
|
||||
print(
|
||||
f" Available tools: {', '.join(sorted(tools[:10]))}{'...' if len(tools) > 10 else ''}"
|
||||
)
|
||||
print("")
|
||||
print("Next steps:")
|
||||
print(f" - Add to PATH: {bin_dir}")
|
||||
print(" - Example:")
|
||||
print(f' export PATH="{bin_dir}:$PATH"')
|
||||
print(" - Verify:")
|
||||
print(" clang --version")
|
||||
|
||||
return
|
||||
|
||||
print("ERROR: LLVM downloaded but clang not found in expected location")
|
||||
sys.exit(1)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("Operation interrupted by user")
|
||||
_thread.interrupt_main()
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"Error during installation: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Setup LLVM toolchain - check system first, fallback to package installation."""
|
||||
# Get target directory from command line (optional)
|
||||
if len(sys.argv) > 2:
|
||||
print("Usage: uv run python ci/setup-llvm.py [target_directory]")
|
||||
print(" target_directory: Optional. Defaults to .cache/cc in project root")
|
||||
sys.exit(1)
|
||||
|
||||
if len(sys.argv) == 2:
|
||||
target_dir = Path(sys.argv[1])
|
||||
else:
|
||||
# Default to project root .cache/cc directory
|
||||
target_dir = _PROJECT_ROOT / ".cache" / "cc"
|
||||
print(f"Using default target directory: {target_dir}")
|
||||
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Check platform compatibility
|
||||
is_unix_like, platform_name = is_linux_or_unix()
|
||||
|
||||
if platform_name == "windows":
|
||||
print("Checking for existing LLVM/Clang tools on system (Windows)...")
|
||||
|
||||
# Check for existing installation in target directory
|
||||
has_artifacts, has_breadcrumb = check_existing_installation(target_dir)
|
||||
|
||||
if has_artifacts and not has_breadcrumb:
|
||||
print("WARN: Found installation artifacts without completion marker")
|
||||
print(" This suggests an incomplete or corrupted installation")
|
||||
print(" Re-downloading LLVM to ensure proper installation...")
|
||||
elif has_artifacts and has_breadcrumb:
|
||||
if verify_clang_installation(target_dir):
|
||||
print("OK: Valid LLVM installation found in target directory")
|
||||
print(" Skipping download - clang installation verified")
|
||||
return
|
||||
else:
|
||||
print(
|
||||
"WARN: Installation breadcrumb exists but clang verification failed"
|
||||
)
|
||||
print(" Re-downloading LLVM to fix installation...")
|
||||
|
||||
clang_version_output = get_tool_version("clang")
|
||||
if clang_version_output:
|
||||
clang_version = get_clang_version_number(clang_version_output)
|
||||
print(
|
||||
f"Found system clang: version {clang_version if clang_version else 'unknown'}"
|
||||
)
|
||||
else:
|
||||
clang_version = None
|
||||
|
||||
found_essential, found_extras = check_required_tools()
|
||||
available_essential = {
|
||||
name: path for name, path in found_essential.items() if path
|
||||
}
|
||||
missing_essential = [name for name, path in found_essential.items() if not path]
|
||||
|
||||
if len(available_essential) != len(found_essential):
|
||||
# Probe common install directories (e.g., C:\Program Files\LLVM\bin)
|
||||
probe_dirs = get_windows_default_llvm_bins()
|
||||
print(
|
||||
f"Probing default Windows LLVM locations: {', '.join(str(p) for p in probe_dirs)}"
|
||||
)
|
||||
probed = find_tools_in_directories(list(found_essential.keys()), probe_dirs)
|
||||
for name, p in probed.items():
|
||||
if p is not None and found_essential.get(name) is None:
|
||||
found_essential[name] = p
|
||||
|
||||
available_essential = {
|
||||
name: path for name, path in found_essential.items() if path
|
||||
}
|
||||
missing_essential = [
|
||||
name for name, path in found_essential.items() if not path
|
||||
]
|
||||
|
||||
if len(available_essential) == len(found_essential) and (
|
||||
clang_version is None or clang_version < 19
|
||||
):
|
||||
print(
|
||||
"Note: Detected toolchain but clang < 19. Continuing; ensure compatibility with your build settings."
|
||||
)
|
||||
|
||||
# On Windows, require core tools including llvm-addr2line for stack traces
|
||||
core_required = ["clang", "clang++", "lld-link", "llvm-addr2line"]
|
||||
missing_core = [name for name in core_required if not found_essential.get(name)]
|
||||
|
||||
if not missing_core:
|
||||
print("OK: Core tools present - proceeding with available tools")
|
||||
all_available_tools: Dict[str, Path] = {}
|
||||
for name, path in found_essential.items():
|
||||
if path is not None:
|
||||
all_available_tools[name] = cast(Path, path)
|
||||
for name, path in found_extras.items():
|
||||
if path is not None:
|
||||
all_available_tools[name] = cast(Path, path)
|
||||
|
||||
print(f"Creating hard links in {target_dir}...")
|
||||
create_hard_links(all_available_tools, target_dir)
|
||||
|
||||
version_file = target_dir / "VERSION"
|
||||
version_file.write_text(
|
||||
f"system-clang-{clang_version if clang_version else 'unknown'}\n"
|
||||
)
|
||||
bin_dir = target_dir / "bin"
|
||||
existing = [p for p in bin_dir.iterdir() if p.is_file()]
|
||||
print(f"OK: Tools available in {bin_dir}: {len(existing)} files")
|
||||
create_breadcrumb(target_dir)
|
||||
|
||||
# Warn about missing non-core tools
|
||||
missing_noncore = [
|
||||
name
|
||||
for name in list(found_essential.keys())
|
||||
if name not in core_required and not found_essential.get(name)
|
||||
]
|
||||
if missing_noncore:
|
||||
print(f"WARN: Missing non-core tools: {', '.join(missing_noncore)}")
|
||||
missing_extras = [name for name, path in found_extras.items() if not path]
|
||||
if missing_extras:
|
||||
print(f"WARN: Missing extra tools: {', '.join(missing_extras)}")
|
||||
return
|
||||
|
||||
# Try to auto-install on Windows via direct GitHub download
|
||||
print(f"ERROR: Missing core tools: {', '.join(missing_core)}")
|
||||
print("Downloading LLVM for Windows from GitHub releases...")
|
||||
extracted_bin_dir = install_llvm_windows_direct(target_dir)
|
||||
|
||||
# Collect tools from the extracted bin directory and link/copy them
|
||||
tool_names: List[str] = [
|
||||
"clang",
|
||||
"clang++",
|
||||
"lld-link",
|
||||
"llvm-addr2line",
|
||||
"llvm-ar",
|
||||
"llvm-nm",
|
||||
"llvm-objdump",
|
||||
"llvm-strip",
|
||||
"llvm-objcopy",
|
||||
"clangd",
|
||||
"clang-format",
|
||||
"clang-tidy",
|
||||
"lldb",
|
||||
]
|
||||
discovered: Dict[str, Optional[Path]] = find_tools_in_directories(
|
||||
tool_names, [extracted_bin_dir]
|
||||
)
|
||||
available_tools: Dict[str, Path] = {}
|
||||
for name, p in discovered.items():
|
||||
if p is not None:
|
||||
available_tools[name] = cast(Path, p)
|
||||
|
||||
if not available_tools:
|
||||
print(
|
||||
"CRITICAL: Extracted LLVM bin directory does not contain expected tools."
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
print(f"Creating hard links in {target_dir}...")
|
||||
create_hard_links(available_tools, target_dir)
|
||||
|
||||
version_file = target_dir / "VERSION"
|
||||
version_file.write_text("downloaded-llvm-windows\n")
|
||||
bin_dir = target_dir / "bin"
|
||||
existing = [p for p in bin_dir.iterdir() if p.is_file()]
|
||||
print(f"OK: Tools available in {bin_dir}: {len(existing)} files")
|
||||
create_breadcrumb(target_dir)
|
||||
return
|
||||
|
||||
print("Checking for existing LLVM/Clang tools on system...")
|
||||
|
||||
# Check if clang is available and get its version
|
||||
clang_version_output = get_tool_version("clang")
|
||||
if clang_version_output:
|
||||
clang_version = get_clang_version_number(clang_version_output)
|
||||
print(
|
||||
f"Found system clang: version {clang_version if clang_version else 'unknown'}"
|
||||
)
|
||||
|
||||
if clang_version and clang_version >= 19:
|
||||
print("OK: System clang 19+ detected - checking tools")
|
||||
|
||||
# Check for essential and extra tools
|
||||
found_essential, found_extras = check_required_tools()
|
||||
available_essential = {
|
||||
name: path for name, path in found_essential.items() if path
|
||||
}
|
||||
missing_essential = [
|
||||
name for name, path in found_essential.items() if not path
|
||||
]
|
||||
available_extras = {
|
||||
name: path for name, path in found_extras.items() if path
|
||||
}
|
||||
missing_extras = [name for name, path in found_extras.items() if not path]
|
||||
|
||||
# Check if all essential tools are present
|
||||
if len(available_essential) == len(found_essential):
|
||||
print("OK: All essential tools found - using system tools")
|
||||
|
||||
# Combine available tools for linking
|
||||
all_available_tools = {**available_essential, **available_extras}
|
||||
|
||||
print(f"Creating hard links in {target_dir}...")
|
||||
create_hard_links(all_available_tools, target_dir)
|
||||
|
||||
# Create a simple version file
|
||||
version_file = target_dir / "VERSION"
|
||||
version_file.write_text(f"system-clang-{clang_version}\n")
|
||||
|
||||
print(
|
||||
f"OK: Successfully linked {len(all_available_tools)} system tools"
|
||||
)
|
||||
|
||||
# Warn about missing extras in one line
|
||||
if missing_extras:
|
||||
print(f"WARN: Missing extra tools: {', '.join(missing_extras)}")
|
||||
|
||||
return
|
||||
else:
|
||||
# Missing essential tools - need to install LLVM
|
||||
print(f"ERROR: Missing essential tools: {', '.join(missing_essential)}")
|
||||
if platform_name == "linux":
|
||||
print("Installing LLVM package to provide missing tools...")
|
||||
download_and_install_llvm(target_dir)
|
||||
return
|
||||
else:
|
||||
print(
|
||||
f"WARN: Cannot auto-install on {platform_name} - please install manually"
|
||||
)
|
||||
return
|
||||
else:
|
||||
print(f"System clang version {clang_version} < 19")
|
||||
if platform_name == "linux":
|
||||
print("Installing LLVM 18 package...")
|
||||
download_and_install_llvm(target_dir)
|
||||
return
|
||||
else:
|
||||
print(
|
||||
f"WARN: Cannot auto-install on {platform_name} - please install manually"
|
||||
)
|
||||
return
|
||||
else:
|
||||
print("No system clang found")
|
||||
if platform_name == "linux":
|
||||
print("Installing LLVM package...")
|
||||
download_and_install_llvm(target_dir)
|
||||
else:
|
||||
print(
|
||||
f"WARN: Cannot auto-install on {platform_name} - please install manually"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
finally:
|
||||
# Clean up download directory at the end
|
||||
_cleanup_download_dir()
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Generic Symbol Analysis Runner for GitHub Actions
|
||||
Wrapper script to run symbol analysis with proper error handling for any platform
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from ci.util.symbol_analysis import main as symbol_analysis
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run symbol analysis for any platform with GitHub Actions support"
|
||||
)
|
||||
parser.add_argument("--board", required=True, help="Board name being analyzed")
|
||||
parser.add_argument("--example", default="Blink", help="Example that was compiled")
|
||||
parser.add_argument(
|
||||
"--skip-on-failure",
|
||||
action="store_true",
|
||||
help="Don't fail the build if symbol analysis fails",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
print(f"Symbol Analysis Runner - Board: {args.board}, Example: {args.example}")
|
||||
|
||||
# Check if build_info.json exists
|
||||
build_info_path = Path(".build") / args.board / "build_info.json"
|
||||
if not build_info_path.exists():
|
||||
# Fall back to nested pio structure
|
||||
build_info_path = Path(".build") / "pio" / args.board / "build_info.json"
|
||||
|
||||
if not build_info_path.exists():
|
||||
message = (
|
||||
f"Build info not found at {build_info_path}. Skipping symbol analysis."
|
||||
)
|
||||
if args.skip_on_failure:
|
||||
print(f"Warning: {message}")
|
||||
return 0
|
||||
else:
|
||||
print(f"Error: {message}")
|
||||
return 1
|
||||
|
||||
try:
|
||||
# Import and run the generic symbol analysis
|
||||
print("Running symbol analysis...")
|
||||
|
||||
# Override sys.argv to pass the board argument to the symbol analysis script
|
||||
original_argv = sys.argv
|
||||
sys.argv = ["symbol_analysis.py", "--board", args.board]
|
||||
|
||||
try:
|
||||
symbol_analysis()
|
||||
print("Symbol analysis completed successfully!")
|
||||
return 0
|
||||
finally:
|
||||
sys.argv = original_argv
|
||||
|
||||
except ImportError as e:
|
||||
print(f"Failed to import symbol_analysis module: {e}")
|
||||
if args.skip_on_failure:
|
||||
return 0
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(f"Symbol analysis failed: {e}")
|
||||
if args.skip_on_failure:
|
||||
return 0
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+1155
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,464 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Integration tests for xcache.py using real clang compiler.
|
||||
|
||||
This test suite verifies that xcache works correctly with actual compilation
|
||||
scenarios, including response files and direct execution modes.
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestResults:
|
||||
"""Test results container."""
|
||||
|
||||
passed: int = 0
|
||||
failed: int = 0
|
||||
errors: Optional[list[str]] = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.errors is None:
|
||||
self.errors = []
|
||||
|
||||
|
||||
def find_clang() -> Optional[str]:
|
||||
"""Find clang compiler in PATH."""
|
||||
clang_path = shutil.which("clang")
|
||||
if clang_path:
|
||||
return clang_path
|
||||
|
||||
# Check common locations
|
||||
common_paths = [
|
||||
"/usr/bin/clang",
|
||||
"/usr/local/bin/clang",
|
||||
"/opt/local/bin/clang",
|
||||
"C:/Program Files/LLVM/bin/clang.exe",
|
||||
"C:/msys64/mingw64/bin/clang.exe",
|
||||
]
|
||||
|
||||
for path in common_paths:
|
||||
if os.path.isfile(path) and os.access(path, os.X_OK):
|
||||
return path
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def find_sccache() -> Optional[str]:
|
||||
"""Find sccache in PATH."""
|
||||
return shutil.which("sccache")
|
||||
|
||||
|
||||
def create_test_source_file(temp_dir: Path) -> Path:
|
||||
"""Create a simple C source file for testing."""
|
||||
source_file = temp_dir / "test.c"
|
||||
source_content = """
|
||||
#include <stdio.h>
|
||||
|
||||
int main() {
|
||||
printf("Hello from xcache test!\\n");
|
||||
return 0;
|
||||
}
|
||||
"""
|
||||
source_file.write_text(source_content)
|
||||
return source_file
|
||||
|
||||
|
||||
def create_response_file(temp_dir: Path, args: list[str]) -> Path:
|
||||
"""Create a response file with the given arguments."""
|
||||
response_file = temp_dir / "compile_args.rsp"
|
||||
|
||||
# Write arguments to response file (one per line or space-separated)
|
||||
content = " ".join(args)
|
||||
response_file.write_text(content)
|
||||
|
||||
return response_file
|
||||
|
||||
|
||||
def test_xcache_direct_execution(results: TestResults) -> None:
|
||||
"""Test xcache with direct execution (no response files)."""
|
||||
print("🧪 Testing xcache direct execution...")
|
||||
|
||||
clang_path = find_clang()
|
||||
if not clang_path:
|
||||
results.errors.append(
|
||||
"clang compiler not found - skipping direct execution test"
|
||||
)
|
||||
return
|
||||
|
||||
sccache_path = find_sccache()
|
||||
if not sccache_path:
|
||||
results.errors.append("sccache not found - skipping direct execution test")
|
||||
return
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create test source file
|
||||
source_file = create_test_source_file(temp_path)
|
||||
output_file = temp_path / "test_direct.o"
|
||||
|
||||
# Test xcache with direct arguments (no response files)
|
||||
env = os.environ.copy()
|
||||
env["XCACHE_DEBUG"] = "1"
|
||||
|
||||
cmd = [
|
||||
sys.executable,
|
||||
"ci/util/xcache.py",
|
||||
clang_path,
|
||||
"-c",
|
||||
str(source_file),
|
||||
"-o",
|
||||
str(output_file),
|
||||
]
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, env=env, timeout=30
|
||||
)
|
||||
|
||||
if result.returncode == 0 and output_file.exists():
|
||||
print(" ✅ Direct execution test passed")
|
||||
print(f" Output file created: {output_file.stat().st_size} bytes")
|
||||
results.passed += 1
|
||||
else:
|
||||
print(
|
||||
f" ❌ Direct execution test failed (return code: {result.returncode})"
|
||||
)
|
||||
print(f" Stderr: {result.stderr}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Direct execution failed: {result.stderr}")
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
print(" ❌ Direct execution test timed out")
|
||||
results.failed += 1
|
||||
results.errors.append("Direct execution test timed out")
|
||||
except Exception as e:
|
||||
print(f" ❌ Direct execution test error: {e}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Direct execution error: {e}")
|
||||
|
||||
|
||||
def test_xcache_response_file_execution(results: TestResults) -> None:
|
||||
"""Test xcache with response file execution."""
|
||||
print("\n🧪 Testing xcache response file execution...")
|
||||
|
||||
clang_path = find_clang()
|
||||
if not clang_path:
|
||||
results.errors.append("clang compiler not found - skipping response file test")
|
||||
return
|
||||
|
||||
sccache_path = find_sccache()
|
||||
if not sccache_path:
|
||||
results.errors.append("sccache not found - skipping response file test")
|
||||
return
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create test source file
|
||||
source_file = create_test_source_file(temp_path)
|
||||
output_file = temp_path / "test_response.o"
|
||||
|
||||
# Create response file with compilation arguments
|
||||
compile_args = ["-c", str(source_file), "-o", str(output_file), "-O2", "-Wall"]
|
||||
response_file = create_response_file(temp_path, compile_args)
|
||||
|
||||
# Test xcache with response file
|
||||
env = os.environ.copy()
|
||||
env["XCACHE_DEBUG"] = "1"
|
||||
|
||||
cmd = [sys.executable, "ci/util/xcache.py", clang_path, f"@{response_file}"]
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, env=env, timeout=30
|
||||
)
|
||||
|
||||
if result.returncode == 0 and output_file.exists():
|
||||
print(" ✅ Response file execution test passed")
|
||||
print(f" Output file created: {output_file.stat().st_size} bytes")
|
||||
print(f" Response file handled: @{response_file}")
|
||||
|
||||
# Verify that response file was passed through (not expanded)
|
||||
if f"@{response_file}" in result.stderr:
|
||||
print(" ✅ Response file passed through correctly")
|
||||
else:
|
||||
print(
|
||||
" ⚠️ Response file passthrough not clearly visible in debug output"
|
||||
)
|
||||
|
||||
results.passed += 1
|
||||
else:
|
||||
print(
|
||||
f" ❌ Response file execution test failed (return code: {result.returncode})"
|
||||
)
|
||||
print(f" Stderr: {result.stderr}")
|
||||
results.failed += 1
|
||||
results.errors.append(
|
||||
f"Response file execution failed: {result.stderr}"
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
print(" ❌ Response file execution test timed out")
|
||||
results.failed += 1
|
||||
results.errors.append("Response file execution test timed out")
|
||||
except Exception as e:
|
||||
print(f" ❌ Response file execution test error: {e}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Response file execution error: {e}")
|
||||
|
||||
|
||||
def test_xcache_mixed_arguments(results: TestResults) -> None:
|
||||
"""Test xcache with mixed regular arguments and response files."""
|
||||
print("\n🧪 Testing xcache mixed arguments...")
|
||||
|
||||
clang_path = find_clang()
|
||||
if not clang_path:
|
||||
results.errors.append(
|
||||
"clang compiler not found - skipping mixed arguments test"
|
||||
)
|
||||
return
|
||||
|
||||
sccache_path = find_sccache()
|
||||
if not sccache_path:
|
||||
results.errors.append("sccache not found - skipping mixed arguments test")
|
||||
return
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create test source file
|
||||
source_file = create_test_source_file(temp_path)
|
||||
output_file = temp_path / "test_mixed.o"
|
||||
|
||||
# Create response file with some arguments
|
||||
response_args = ["-O2", "-Wall", "-Wextra"]
|
||||
response_file = create_response_file(temp_path, response_args)
|
||||
|
||||
# Test xcache with mixed arguments
|
||||
env = os.environ.copy()
|
||||
env["XCACHE_DEBUG"] = "1"
|
||||
|
||||
cmd = [
|
||||
sys.executable,
|
||||
"ci/util/xcache.py",
|
||||
clang_path,
|
||||
"-c",
|
||||
str(source_file),
|
||||
f"@{response_file}",
|
||||
"-o",
|
||||
str(output_file),
|
||||
]
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, env=env, timeout=30
|
||||
)
|
||||
|
||||
if result.returncode == 0 and output_file.exists():
|
||||
print(" ✅ Mixed arguments test passed")
|
||||
print(f" Output file created: {output_file.stat().st_size} bytes")
|
||||
print(f" Mixed args: regular + @{response_file}")
|
||||
results.passed += 1
|
||||
else:
|
||||
print(
|
||||
f" ❌ Mixed arguments test failed (return code: {result.returncode})"
|
||||
)
|
||||
print(f" Stderr: {result.stderr}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Mixed arguments failed: {result.stderr}")
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
print(" ❌ Mixed arguments test timed out")
|
||||
results.failed += 1
|
||||
results.errors.append("Mixed arguments test timed out")
|
||||
except Exception as e:
|
||||
print(f" ❌ Mixed arguments test error: {e}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Mixed arguments error: {e}")
|
||||
|
||||
|
||||
def test_xcache_cache_effectiveness(results: TestResults) -> None:
|
||||
"""Test that xcache actually caches compilation results."""
|
||||
print("\n🧪 Testing xcache cache effectiveness...")
|
||||
|
||||
clang_path = find_clang()
|
||||
if not clang_path:
|
||||
results.errors.append(
|
||||
"clang compiler not found - skipping cache effectiveness test"
|
||||
)
|
||||
return
|
||||
|
||||
sccache_path = find_sccache()
|
||||
if not sccache_path:
|
||||
results.errors.append("sccache not found - skipping cache effectiveness test")
|
||||
return
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create test source file
|
||||
source_file = create_test_source_file(temp_path)
|
||||
|
||||
# Set up environment for caching
|
||||
env = os.environ.copy()
|
||||
env["XCACHE_DEBUG"] = "1"
|
||||
cache_dir = temp_path / "sccache"
|
||||
cache_dir.mkdir()
|
||||
env["SCCACHE_DIR"] = str(cache_dir)
|
||||
|
||||
# First compilation (should be cache miss)
|
||||
output_file1 = temp_path / "test_cache1.o"
|
||||
cmd1 = [
|
||||
sys.executable,
|
||||
"ci/util/xcache.py",
|
||||
clang_path,
|
||||
"-c",
|
||||
str(source_file),
|
||||
"-o",
|
||||
str(output_file1),
|
||||
]
|
||||
|
||||
# Second compilation (should be cache hit)
|
||||
output_file2 = temp_path / "test_cache2.o"
|
||||
cmd2 = [
|
||||
sys.executable,
|
||||
"ci/util/xcache.py",
|
||||
clang_path,
|
||||
"-c",
|
||||
str(source_file),
|
||||
"-o",
|
||||
str(output_file2),
|
||||
]
|
||||
|
||||
try:
|
||||
# First compilation
|
||||
result1 = subprocess.run(
|
||||
cmd1, capture_output=True, text=True, env=env, timeout=30
|
||||
)
|
||||
|
||||
# Second compilation
|
||||
result2 = subprocess.run(
|
||||
cmd2, capture_output=True, text=True, env=env, timeout=30
|
||||
)
|
||||
|
||||
if (
|
||||
result1.returncode == 0
|
||||
and output_file1.exists()
|
||||
and result2.returncode == 0
|
||||
and output_file2.exists()
|
||||
):
|
||||
print(" ✅ Cache effectiveness test passed")
|
||||
print(f" First compilation: {output_file1.stat().st_size} bytes")
|
||||
print(f" Second compilation: {output_file2.stat().st_size} bytes")
|
||||
|
||||
# Check if sccache cache directory has content
|
||||
cache_files = list(cache_dir.rglob("*"))
|
||||
if cache_files:
|
||||
print(f" Cache directory has {len(cache_files)} entries")
|
||||
else:
|
||||
print(" ⚠️ Cache directory appears empty")
|
||||
|
||||
results.passed += 1
|
||||
else:
|
||||
print(f" ❌ Cache effectiveness test failed")
|
||||
print(
|
||||
f" First result: {result1.returncode}, {output_file1.exists()}"
|
||||
)
|
||||
print(
|
||||
f" Second result: {result2.returncode}, {output_file2.exists()}"
|
||||
)
|
||||
results.failed += 1
|
||||
results.errors.append("Cache effectiveness test failed")
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
print(" ❌ Cache effectiveness test timed out")
|
||||
results.failed += 1
|
||||
results.errors.append("Cache effectiveness test timed out")
|
||||
except Exception as e:
|
||||
print(f" ❌ Cache effectiveness test error: {e}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Cache effectiveness error: {e}")
|
||||
|
||||
|
||||
def run_all_tests() -> TestResults:
|
||||
"""Run all xcache tests."""
|
||||
print("🚀 XCACHE INTEGRATION TESTS WITH CLANG")
|
||||
print("=" * 60)
|
||||
|
||||
results = TestResults()
|
||||
|
||||
# Check prerequisites
|
||||
clang_path = find_clang()
|
||||
sccache_path = find_sccache()
|
||||
|
||||
print(f"📋 Test Environment:")
|
||||
print(f" Clang: {clang_path or 'NOT FOUND'}")
|
||||
print(f" Sccache: {sccache_path or 'NOT FOUND'}")
|
||||
print(f" xcache: ci/util/xcache.py")
|
||||
print()
|
||||
|
||||
if not clang_path:
|
||||
print("❌ Clang compiler not found - install clang to run tests")
|
||||
results.errors.append("Clang compiler not available")
|
||||
return results
|
||||
|
||||
if not sccache_path:
|
||||
print("❌ Sccache not found - install sccache to run tests")
|
||||
results.errors.append("Sccache not available")
|
||||
return results
|
||||
|
||||
# Run tests
|
||||
test_xcache_direct_execution(results)
|
||||
test_xcache_response_file_execution(results)
|
||||
test_xcache_mixed_arguments(results)
|
||||
test_xcache_cache_effectiveness(results)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main test function."""
|
||||
results = run_all_tests()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("📊 TEST RESULTS SUMMARY")
|
||||
print("=" * 60)
|
||||
|
||||
total_tests = results.passed + results.failed
|
||||
if total_tests > 0:
|
||||
success_rate = (results.passed / total_tests) * 100
|
||||
print(f" Tests passed: {results.passed}")
|
||||
print(f" Tests failed: {results.failed}")
|
||||
print(f" Success rate: {success_rate:.1f}%")
|
||||
else:
|
||||
print(" No tests were executed")
|
||||
|
||||
if results.errors:
|
||||
print(f"\n🚨 Errors encountered:")
|
||||
for i, error in enumerate(results.errors, 1):
|
||||
print(f" {i}. {error}")
|
||||
|
||||
if results.failed == 0 and results.passed > 0:
|
||||
print("\n✅ All xcache tests passed! Ready for ESP32S3 integration.")
|
||||
return 0
|
||||
elif results.passed > 0:
|
||||
print(
|
||||
f"\n⚠️ Some tests passed but {results.failed} failed. Check errors above."
|
||||
)
|
||||
return 1
|
||||
else:
|
||||
print("\n❌ No tests passed. Check prerequisites and errors above.")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+437
@@ -0,0 +1,437 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Standalone tests for xcache.py using clang compiler without requiring sccache server.
|
||||
|
||||
This test suite verifies that xcache works correctly by testing the wrapper
|
||||
functionality and ensuring proper command construction and execution.
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestResults:
|
||||
"""Test results container."""
|
||||
|
||||
passed: int = 0
|
||||
failed: int = 0
|
||||
errors: Optional[list[str]] = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.errors is None:
|
||||
self.errors = []
|
||||
|
||||
|
||||
def find_clang() -> Optional[str]:
|
||||
"""Find clang compiler in PATH."""
|
||||
clang_path = shutil.which("clang")
|
||||
if clang_path:
|
||||
return clang_path
|
||||
|
||||
# Check common locations
|
||||
common_paths = [
|
||||
"/usr/bin/clang",
|
||||
"/usr/local/bin/clang",
|
||||
"/opt/local/bin/clang",
|
||||
"C:/Program Files/LLVM/bin/clang.exe",
|
||||
"C:/msys64/mingw64/bin/clang.exe",
|
||||
]
|
||||
|
||||
for path in common_paths:
|
||||
if os.path.isfile(path) and os.access(path, os.X_OK):
|
||||
return path
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def create_test_source_file(temp_dir: Path) -> Path:
|
||||
"""Create a simple C source file for testing."""
|
||||
source_file = temp_dir / "test.c"
|
||||
source_content = """
|
||||
#include <stdio.h>
|
||||
|
||||
int main() {
|
||||
printf("Hello from xcache test!\\n");
|
||||
return 0;
|
||||
}
|
||||
"""
|
||||
source_file.write_text(source_content)
|
||||
return source_file
|
||||
|
||||
|
||||
def create_response_file(temp_dir: Path, args: list[str]) -> Path:
|
||||
"""Create a response file with the given arguments."""
|
||||
response_file = temp_dir / "compile_args.rsp"
|
||||
|
||||
# Write arguments to response file (space-separated)
|
||||
# Properly quote arguments that contain spaces (Windows paths)
|
||||
# Use forward slashes for Windows paths to avoid backslash escape issues
|
||||
quoted_args: list[str] = []
|
||||
for arg in args:
|
||||
# Convert Windows backslashes to forward slashes for response files
|
||||
normalized_arg = arg.replace("\\", "/")
|
||||
if " " in normalized_arg and not (
|
||||
normalized_arg.startswith('"') and normalized_arg.endswith('"')
|
||||
):
|
||||
quoted_args.append(f'"{normalized_arg}"')
|
||||
else:
|
||||
quoted_args.append(normalized_arg)
|
||||
|
||||
content = " ".join(quoted_args)
|
||||
response_file.write_text(content)
|
||||
|
||||
return response_file
|
||||
|
||||
|
||||
def test_clang_direct_compilation(results: TestResults) -> None:
|
||||
"""Test that clang compilation works directly (baseline test)."""
|
||||
print("🧪 Testing direct clang compilation (baseline)...")
|
||||
|
||||
clang_path = find_clang()
|
||||
if not clang_path:
|
||||
results.errors.append("clang compiler not found - skipping baseline test")
|
||||
return
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create test source file
|
||||
source_file = create_test_source_file(temp_path)
|
||||
output_file = temp_path / "test_baseline.o"
|
||||
|
||||
# Test direct clang compilation
|
||||
cmd = [clang_path, "-c", str(source_file), "-o", str(output_file)]
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
||||
|
||||
if result.returncode == 0 and output_file.exists():
|
||||
print(" ✅ Direct clang compilation works")
|
||||
print(f" Output file created: {output_file.stat().st_size} bytes")
|
||||
results.passed += 1
|
||||
else:
|
||||
print(
|
||||
f" ❌ Direct clang compilation failed (return code: {result.returncode})"
|
||||
)
|
||||
print(f" Stderr: {result.stderr}")
|
||||
results.failed += 1
|
||||
results.errors.append(
|
||||
f"Direct clang compilation failed: {result.stderr}"
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
print(" ❌ Direct clang compilation timed out")
|
||||
results.failed += 1
|
||||
results.errors.append("Direct clang compilation timed out")
|
||||
except Exception as e:
|
||||
print(f" ❌ Direct clang compilation error: {e}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Direct clang compilation error: {e}")
|
||||
|
||||
|
||||
def test_clang_response_file_compilation(results: TestResults) -> None:
|
||||
"""Test that clang can handle response files directly."""
|
||||
print("\n🧪 Testing clang response file compilation...")
|
||||
|
||||
clang_path = find_clang()
|
||||
if not clang_path:
|
||||
results.errors.append("clang compiler not found - skipping response file test")
|
||||
return
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create test source file
|
||||
source_file = create_test_source_file(temp_path)
|
||||
output_file = temp_path / "test_response.o"
|
||||
|
||||
# Create response file with compilation arguments
|
||||
compile_args = ["-c", str(source_file), "-o", str(output_file)]
|
||||
response_file = create_response_file(temp_path, compile_args)
|
||||
|
||||
# Test clang with response file
|
||||
cmd = [clang_path, f"@{response_file}"]
|
||||
|
||||
try:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
||||
|
||||
if result.returncode == 0 and output_file.exists():
|
||||
print(" ✅ Clang response file compilation works")
|
||||
print(f" Output file created: {output_file.stat().st_size} bytes")
|
||||
print(f" Response file: @{response_file}")
|
||||
results.passed += 1
|
||||
else:
|
||||
print(
|
||||
f" ❌ Clang response file compilation failed (return code: {result.returncode})"
|
||||
)
|
||||
print(f" Stderr: {result.stderr}")
|
||||
results.failed += 1
|
||||
results.errors.append(
|
||||
f"Clang response file compilation failed: {result.stderr}"
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
print(" ❌ Clang response file compilation timed out")
|
||||
results.failed += 1
|
||||
results.errors.append("Clang response file compilation timed out")
|
||||
except Exception as e:
|
||||
print(f" ❌ Clang response file compilation error: {e}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Clang response file compilation error: {e}")
|
||||
|
||||
|
||||
def test_xcache_wrapper_script_creation(results: TestResults) -> None:
|
||||
"""Test that xcache creates wrapper scripts correctly."""
|
||||
print("\n🧪 Testing xcache wrapper script creation...")
|
||||
|
||||
clang_path = find_clang()
|
||||
if not clang_path:
|
||||
results.errors.append("clang compiler not found - skipping wrapper script test")
|
||||
return
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create test source file and response file
|
||||
source_file = create_test_source_file(temp_path)
|
||||
compile_args = ["-c", str(source_file), "-o", "test.o"]
|
||||
response_file = create_response_file(temp_path, compile_args)
|
||||
|
||||
# Create a mock sccache script that just echoes its arguments
|
||||
mock_sccache = (
|
||||
temp_path / "sccache.bat" if os.name == "nt" else temp_path / "sccache"
|
||||
)
|
||||
|
||||
if os.name == "nt":
|
||||
mock_content = """@echo off
|
||||
echo MOCK_SCCACHE_CALLED: %*
|
||||
"""
|
||||
else:
|
||||
mock_content = """#!/bin/bash
|
||||
echo "MOCK_SCCACHE_CALLED: $@"
|
||||
"""
|
||||
|
||||
mock_sccache.write_text(mock_content)
|
||||
if os.name != "nt":
|
||||
mock_sccache.chmod(0o755)
|
||||
|
||||
# Set up environment to use mock sccache
|
||||
env = os.environ.copy()
|
||||
env["PATH"] = f"{temp_path}{os.pathsep}{env.get('PATH', '')}"
|
||||
env["XCACHE_DEBUG"] = "1"
|
||||
|
||||
# Test xcache with response file (should create wrapper script)
|
||||
cmd = [sys.executable, "ci/util/xcache.py", clang_path, f"@{response_file}"]
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, env=env, timeout=30
|
||||
)
|
||||
|
||||
# Check if wrapper script was created and executed
|
||||
if "Created compiler wrapper:" in result.stderr:
|
||||
print(" ✅ Wrapper script creation detected")
|
||||
|
||||
# Check if wrapper was executed
|
||||
if "Executing with wrapper:" in result.stderr:
|
||||
print(" ✅ Wrapper script execution detected")
|
||||
|
||||
# Check if mock sccache was called
|
||||
if "MOCK_SCCACHE_CALLED:" in result.stdout:
|
||||
print(" ✅ Mock sccache was called through wrapper")
|
||||
print(f" Sccache called with: {result.stdout.strip()}")
|
||||
results.passed += 1
|
||||
else:
|
||||
print(" ❌ Mock sccache was not called")
|
||||
results.failed += 1
|
||||
results.errors.append(
|
||||
"Mock sccache was not called through wrapper"
|
||||
)
|
||||
else:
|
||||
print(" ❌ Wrapper script was not executed")
|
||||
results.failed += 1
|
||||
results.errors.append("Wrapper script was not executed")
|
||||
else:
|
||||
print(" ❌ Wrapper script was not created")
|
||||
print(f" Stderr: {result.stderr}")
|
||||
results.failed += 1
|
||||
results.errors.append(
|
||||
f"Wrapper script creation failed: {result.stderr}"
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
print(" ❌ Wrapper script test timed out")
|
||||
results.failed += 1
|
||||
results.errors.append("Wrapper script test timed out")
|
||||
except Exception as e:
|
||||
print(f" ❌ Wrapper script test error: {e}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Wrapper script test error: {e}")
|
||||
|
||||
|
||||
def test_xcache_direct_mode(results: TestResults) -> None:
|
||||
"""Test xcache direct mode (no response files)."""
|
||||
print("\n🧪 Testing xcache direct mode...")
|
||||
|
||||
clang_path = find_clang()
|
||||
if not clang_path:
|
||||
results.errors.append("clang compiler not found - skipping direct mode test")
|
||||
return
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create test source file
|
||||
source_file = create_test_source_file(temp_path)
|
||||
|
||||
# Create a mock sccache script that just passes through to real compiler
|
||||
mock_sccache = (
|
||||
temp_path / "sccache.bat" if os.name == "nt" else temp_path / "sccache"
|
||||
)
|
||||
|
||||
if os.name == "nt":
|
||||
mock_content = f'''@echo off
|
||||
echo MOCK_SCCACHE_DIRECT: %*
|
||||
"{clang_path}" %*
|
||||
'''
|
||||
else:
|
||||
mock_content = f'''#!/bin/bash
|
||||
echo "MOCK_SCCACHE_DIRECT: $@"
|
||||
"{clang_path}" "$@"
|
||||
'''
|
||||
|
||||
mock_sccache.write_text(mock_content)
|
||||
if os.name != "nt":
|
||||
mock_sccache.chmod(0o755)
|
||||
|
||||
# Set up environment to use mock sccache
|
||||
env = os.environ.copy()
|
||||
env["PATH"] = f"{temp_path}{os.pathsep}{env.get('PATH', '')}"
|
||||
env["XCACHE_DEBUG"] = "1"
|
||||
|
||||
output_file = temp_path / "test_direct.o"
|
||||
|
||||
# Test xcache without response files (direct mode)
|
||||
cmd = [
|
||||
sys.executable,
|
||||
"ci/util/xcache.py",
|
||||
clang_path,
|
||||
"-c",
|
||||
str(source_file),
|
||||
"-o",
|
||||
str(output_file),
|
||||
]
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, env=env, timeout=30
|
||||
)
|
||||
|
||||
if result.returncode == 0 and output_file.exists():
|
||||
print(" ✅ Direct mode compilation succeeded")
|
||||
print(f" Output file created: {output_file.stat().st_size} bytes")
|
||||
|
||||
# Check if direct execution was used (no wrapper)
|
||||
if "Direct execution:" in result.stderr:
|
||||
print(" ✅ Direct execution mode confirmed")
|
||||
results.passed += 1
|
||||
else:
|
||||
print(" ⚠️ Direct execution mode not clearly indicated")
|
||||
results.passed += 1 # Still pass since compilation worked
|
||||
|
||||
else:
|
||||
print(
|
||||
f" ❌ Direct mode compilation failed (return code: {result.returncode})"
|
||||
)
|
||||
print(f" Stderr: {result.stderr}")
|
||||
results.failed += 1
|
||||
results.errors.append(
|
||||
f"Direct mode compilation failed: {result.stderr}"
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
print(" ❌ Direct mode test timed out")
|
||||
results.failed += 1
|
||||
results.errors.append("Direct mode test timed out")
|
||||
except Exception as e:
|
||||
print(f" ❌ Direct mode test error: {e}")
|
||||
results.failed += 1
|
||||
results.errors.append(f"Direct mode test error: {e}")
|
||||
|
||||
|
||||
def run_all_tests() -> TestResults:
|
||||
"""Run all xcache standalone tests."""
|
||||
print("🚀 XCACHE STANDALONE TESTS WITH CLANG")
|
||||
print("=" * 60)
|
||||
|
||||
results = TestResults()
|
||||
|
||||
# Check prerequisites
|
||||
clang_path = find_clang()
|
||||
|
||||
print(f"📋 Test Environment:")
|
||||
print(f" Clang: {clang_path or 'NOT FOUND'}")
|
||||
print(f" xcache: ci/util/xcache.py")
|
||||
print(f" Note: Using mock sccache for testing wrapper functionality")
|
||||
print()
|
||||
|
||||
if not clang_path:
|
||||
print("❌ Clang compiler not found - install clang to run tests")
|
||||
results.errors.append("Clang compiler not available")
|
||||
return results
|
||||
|
||||
# Run tests
|
||||
test_clang_direct_compilation(results)
|
||||
test_clang_response_file_compilation(results)
|
||||
test_xcache_wrapper_script_creation(results)
|
||||
test_xcache_direct_mode(results)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main test function."""
|
||||
results = run_all_tests()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("📊 TEST RESULTS SUMMARY")
|
||||
print("=" * 60)
|
||||
|
||||
total_tests = results.passed + results.failed
|
||||
if total_tests > 0:
|
||||
success_rate = (results.passed / total_tests) * 100
|
||||
print(f" Tests passed: {results.passed}")
|
||||
print(f" Tests failed: {results.failed}")
|
||||
print(f" Success rate: {success_rate:.1f}%")
|
||||
else:
|
||||
print(" No tests were executed")
|
||||
|
||||
if results.errors:
|
||||
print(f"\n🚨 Errors encountered:")
|
||||
for i, error in enumerate(results.errors, 1):
|
||||
print(f" {i}. {error}")
|
||||
|
||||
if results.failed == 0 and results.passed > 0:
|
||||
print(
|
||||
"\n✅ All xcache standalone tests passed! Wrapper functionality verified."
|
||||
)
|
||||
return 0
|
||||
elif results.passed > 0:
|
||||
print(
|
||||
f"\n⚠️ Some tests passed but {results.failed} failed. Check errors above."
|
||||
)
|
||||
return 1
|
||||
else:
|
||||
print("\n❌ No tests passed. Check prerequisites and errors above.")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to validate the new unified build system.
|
||||
|
||||
This script tests the core functionality of the BuildAPI to ensure
|
||||
the refactored build system works correctly.
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from ci.build_api import BuildType, create_example_builder, create_unit_test_builder
|
||||
|
||||
|
||||
def test_build_api_creation():
|
||||
"""Test that BuildAPI instances can be created successfully"""
|
||||
print("Testing BuildAPI creation...")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Test unit test builder creation
|
||||
unit_builder = create_unit_test_builder(
|
||||
build_dir=temp_path / "unit", use_pch=True, parallel=True, clean=False
|
||||
)
|
||||
|
||||
assert unit_builder.build_type == BuildType.UNIT_TEST
|
||||
assert unit_builder.build_dir == temp_path / "unit"
|
||||
assert unit_builder.use_pch == True
|
||||
|
||||
print("✓ Unit test builder created successfully")
|
||||
|
||||
# Test example builder creation
|
||||
example_builder = create_example_builder(
|
||||
build_dir=temp_path / "examples", use_pch=True, parallel=True, clean=False
|
||||
)
|
||||
|
||||
assert example_builder.build_type == BuildType.EXAMPLE
|
||||
assert example_builder.build_dir == temp_path / "examples"
|
||||
assert example_builder.use_pch == True
|
||||
|
||||
print("✓ Example builder created successfully")
|
||||
|
||||
|
||||
def test_build_configuration_loading():
|
||||
"""Test that build configurations are loaded correctly"""
|
||||
print("\nTesting build configuration loading...")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Test unit test configuration
|
||||
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
|
||||
|
||||
# Check that unit test specific defines are present
|
||||
unit_defines = unit_builder.compiler_settings.defines or []
|
||||
assert "FASTLED_FORCE_NAMESPACE=1" in unit_defines
|
||||
assert "FASTLED_TESTING=1" in unit_defines
|
||||
assert "STUB_PLATFORM" in unit_defines
|
||||
|
||||
print("✓ Unit test configuration loaded correctly")
|
||||
|
||||
# Test example configuration
|
||||
example_builder = create_example_builder(build_dir=temp_path / "examples")
|
||||
|
||||
# Check that example specific defines are present
|
||||
example_defines = example_builder.compiler_settings.defines or []
|
||||
assert "STUB_PLATFORM" in example_defines
|
||||
assert "ARDUINO=10808" in example_defines
|
||||
# Unit test specific defines should NOT be present
|
||||
assert "FASTLED_FORCE_NAMESPACE=1" not in example_defines
|
||||
assert "FASTLED_TESTING=1" not in example_defines
|
||||
|
||||
print("✓ Example configuration loaded correctly")
|
||||
|
||||
|
||||
def test_build_directory_structure():
|
||||
"""Test that build directory structure is created correctly"""
|
||||
print("\nTesting build directory structure...")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
|
||||
|
||||
# Check that all required directories are created
|
||||
assert unit_builder.build_dir.exists()
|
||||
assert unit_builder.artifacts_dir.exists()
|
||||
assert unit_builder.targets_dir.exists()
|
||||
assert unit_builder.cache_dir.exists()
|
||||
|
||||
# Check directory organization
|
||||
expected_structure = [
|
||||
unit_builder.build_dir / "artifacts",
|
||||
unit_builder.build_dir / "targets",
|
||||
unit_builder.build_dir / "cache",
|
||||
]
|
||||
|
||||
for directory in expected_structure:
|
||||
assert directory.exists(), f"Directory {directory} should exist"
|
||||
|
||||
print("✓ Build directory structure created correctly")
|
||||
|
||||
|
||||
def test_build_info():
|
||||
"""Test that build info is returned correctly"""
|
||||
print("\nTesting build info...")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
|
||||
info = unit_builder.get_build_info()
|
||||
|
||||
# Check required fields
|
||||
assert "config_file" in info
|
||||
assert "build_dir" in info
|
||||
assert "build_type" in info
|
||||
assert "use_pch" in info
|
||||
assert "parallel" in info
|
||||
|
||||
# Check values
|
||||
assert info["build_type"] == "unit_test"
|
||||
assert info["use_pch"] == True
|
||||
assert str(temp_path / "unit") in info["build_dir"]
|
||||
assert "build_unit.toml" in info["build_flags_toml"]
|
||||
|
||||
print("✓ Build info returned correctly")
|
||||
|
||||
|
||||
def test_clean_functionality():
|
||||
"""Test that clean functionality works"""
|
||||
print("\nTesting clean functionality...")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
|
||||
|
||||
# Create some fake files to test cleaning
|
||||
test_file = unit_builder.build_dir / "test_file.txt"
|
||||
test_file.write_text("test content")
|
||||
assert test_file.exists()
|
||||
|
||||
# Test clean functionality
|
||||
unit_builder.clean_build_artifacts()
|
||||
|
||||
# Build directory should still exist but be empty (except for subdirs)
|
||||
assert unit_builder.build_dir.exists()
|
||||
assert not test_file.exists()
|
||||
|
||||
# Subdirectories should be recreated
|
||||
assert unit_builder.artifacts_dir.exists()
|
||||
assert unit_builder.targets_dir.exists()
|
||||
assert unit_builder.cache_dir.exists()
|
||||
|
||||
print("✓ Clean functionality works correctly")
|
||||
|
||||
|
||||
def test_configuration_separation():
|
||||
"""Test that unit test and example configurations are properly separated"""
|
||||
print("\nTesting configuration separation...")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
temp_path = Path(temp_dir)
|
||||
|
||||
# Create both builders
|
||||
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
|
||||
example_builder = create_example_builder(build_dir=temp_path / "examples")
|
||||
|
||||
# Get build info for comparison
|
||||
unit_info = unit_builder.get_build_info()
|
||||
example_info = example_builder.get_build_info()
|
||||
|
||||
# Configurations should use different files
|
||||
assert "build_unit.toml" in unit_info["build_flags_toml"]
|
||||
assert "build_example.toml" in example_info["build_flags_toml"]
|
||||
|
||||
# Build types should be different
|
||||
assert unit_info["build_type"] == "unit_test"
|
||||
assert example_info["build_type"] == "example"
|
||||
|
||||
# Build directories should be different
|
||||
assert unit_info["build_dir"] != example_info["build_dir"]
|
||||
|
||||
print("✓ Configuration separation works correctly")
|
||||
|
||||
|
||||
def run_all_tests():
|
||||
"""Run all validation tests"""
|
||||
print("=" * 60)
|
||||
print("FASTLED UNIFIED BUILD SYSTEM VALIDATION")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
test_build_api_creation()
|
||||
test_build_configuration_loading()
|
||||
test_build_directory_structure()
|
||||
test_build_info()
|
||||
test_clean_functionality()
|
||||
test_configuration_separation()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("ALL TESTS PASSED ✓")
|
||||
print("=" * 60)
|
||||
print("The unified build system is working correctly!")
|
||||
print("\nKey features validated:")
|
||||
print("✓ BuildAPI creation for both unit tests and examples")
|
||||
print("✓ Proper configuration loading from separate TOML files")
|
||||
print("✓ Organized build directory structure")
|
||||
print("✓ Clean separation between unit test and example builds")
|
||||
print("✓ Build info reporting")
|
||||
print("✓ Clean functionality")
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ TEST FAILED: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = run_all_tests()
|
||||
exit(0 if success else 1)
|
||||
@@ -0,0 +1,114 @@
|
||||
# pyright: reportUnknownMemberType=false, reportMissingParameterType=false
|
||||
"""Test for checking namespace includes in C++ headers."""
|
||||
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Script to check for includes after namespace declarations in C++ files.
|
||||
This is used in CI/CD to prevent bad code patterns.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
|
||||
def find_includes_after_namespace(file_path: Path) -> List[int]:
|
||||
"""
|
||||
Check if a C++ file has #include directives after namespace declarations.
|
||||
|
||||
Args:
|
||||
file_path (Path): Path to the C++ file to check
|
||||
|
||||
Returns:
|
||||
List[int]: List of line numbers where includes appear after namespaces
|
||||
"""
|
||||
try:
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
content = f.readlines()
|
||||
|
||||
violations: List[int] = []
|
||||
namespace_started = False
|
||||
|
||||
# Basic patterns
|
||||
namespace_pattern = re.compile(r"^\s*namespace\s+\w+\s*\{")
|
||||
include_pattern = re.compile(r"^\s*#\s*include")
|
||||
|
||||
for i, line in enumerate(content, 1):
|
||||
line = line.strip()
|
||||
|
||||
# Skip empty lines and comments
|
||||
if not line or line.startswith("//") or line.startswith("/*"):
|
||||
continue
|
||||
|
||||
# Check for namespace declaration
|
||||
if namespace_pattern.match(line):
|
||||
namespace_started = True
|
||||
|
||||
# Check for #include after namespace started
|
||||
if namespace_started and include_pattern.match(line):
|
||||
violations.append(i)
|
||||
|
||||
return violations
|
||||
except (UnicodeDecodeError, IOError):
|
||||
# Skip files that can't be read
|
||||
return []
|
||||
|
||||
|
||||
def scan_cpp_files(directory: str = ".") -> Dict[str, Any]:
|
||||
"""
|
||||
Scan all C++ files in a directory for includes after namespace declarations.
|
||||
|
||||
Args:
|
||||
directory (str): Directory to scan for C++ files
|
||||
|
||||
Returns:
|
||||
Dict[str, Any]: Dictionary mapping file paths to lists of violation line numbers
|
||||
"""
|
||||
cpp_extensions = [
|
||||
".cpp",
|
||||
".cc",
|
||||
".cxx",
|
||||
".c++",
|
||||
".hpp",
|
||||
".h",
|
||||
".hh",
|
||||
".hxx",
|
||||
".h++",
|
||||
]
|
||||
violations: Dict[str, Any] = {}
|
||||
|
||||
for root, dirs, files in os.walk(directory):
|
||||
for file in files:
|
||||
file_path = os.path.join(root, file)
|
||||
|
||||
# Check if it's a C++ file
|
||||
if any(file.endswith(ext) for ext in cpp_extensions):
|
||||
line_numbers: List[int] = find_includes_after_namespace(Path(file_path))
|
||||
if line_numbers:
|
||||
violations[file_path] = line_numbers
|
||||
|
||||
return violations
|
||||
|
||||
|
||||
def main() -> None:
|
||||
violations: Dict[str, Any] = scan_cpp_files()
|
||||
|
||||
if violations:
|
||||
print("Found #include directives after namespace declarations:")
|
||||
for file_path, line_numbers in violations.items():
|
||||
print(f"\n{file_path}:")
|
||||
for line_num in line_numbers:
|
||||
print(f" Line {line_num}")
|
||||
|
||||
failing: List[str] = list(violations.keys())
|
||||
print(f"\nFailing files: {failing}")
|
||||
sys.exit(1)
|
||||
else:
|
||||
print("No violations found.")
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env python3
|
||||
# pyright: reportUnknownMemberType=false, reportMissingParameterType=false
|
||||
import os
|
||||
import unittest
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from typing import List
|
||||
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
|
||||
|
||||
SRC_ROOT = PROJECT_ROOT / "src"
|
||||
PLATFORMS_DIR = os.path.join(SRC_ROOT, "platforms")
|
||||
|
||||
NUM_WORKERS = 1 if os.environ.get("NO_PARALLEL") else (os.cpu_count() or 1) * 4
|
||||
|
||||
|
||||
class NoUsingNamespaceFlInHeaderTester(unittest.TestCase):
|
||||
def check_file(self, file_path: str) -> List[str]:
|
||||
if "FastLED.h" in file_path:
|
||||
return []
|
||||
failings: List[str] = []
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
for line_number, line in enumerate(f, 1):
|
||||
if line.startswith("//"):
|
||||
continue
|
||||
if "using namespace fl;" in line:
|
||||
failings.append(f"{file_path}:{line_number}: {line.strip()}")
|
||||
return failings
|
||||
|
||||
def test_no_using_namespace(self) -> None:
|
||||
"""Searches through the program files to check for banned headers, excluding src/platforms."""
|
||||
files_to_check: List[str] = []
|
||||
for root, _, files in os.walk(SRC_ROOT):
|
||||
for file in files:
|
||||
if file.endswith(
|
||||
(".h", ".hpp")
|
||||
): # Add or remove file extensions as needed
|
||||
file_path = os.path.join(root, file)
|
||||
files_to_check.append(file_path)
|
||||
|
||||
all_failings: List[str] = []
|
||||
with ThreadPoolExecutor(max_workers=NUM_WORKERS) as executor:
|
||||
futures = [
|
||||
executor.submit(self.check_file, file_path)
|
||||
for file_path in files_to_check
|
||||
]
|
||||
for future in futures:
|
||||
all_failings.extend(future.result())
|
||||
|
||||
if all_failings:
|
||||
msg = (
|
||||
f'Found {len(all_failings)} header file(s) "using namespace fl": \n'
|
||||
+ "\n".join(all_failings)
|
||||
)
|
||||
for failing in all_failings:
|
||||
print(failing)
|
||||
self.fail(msg)
|
||||
else:
|
||||
print("No using namespace fl; found in headers.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,74 @@
|
||||
import subprocess
|
||||
import unittest
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
from ci.util.bin_2_elf import bin_to_elf
|
||||
from ci.util.elf import dump_symbol_sizes
|
||||
from ci.util.paths import PROJECT_ROOT
|
||||
from ci.util.tools import Tools, load_tools
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent.absolute()
|
||||
UNO = HERE / "uno"
|
||||
OUTPUT = HERE / "output"
|
||||
|
||||
|
||||
BUILD_INFO_PATH = PROJECT_ROOT / ".build" / "examples" / "uno" / "build_info.json"
|
||||
BUILD_INFO_PATH2 = (
|
||||
PROJECT_ROOT / ".build" / "fled" / "examples" / "uno" / "build_info.json"
|
||||
)
|
||||
|
||||
DISABLED = True
|
||||
|
||||
|
||||
class TestBinToElf(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
if DISABLED:
|
||||
return
|
||||
uno_build = PROJECT_ROOT / ".build" / "uno"
|
||||
print(f"Checking for Uno build in: {uno_build}")
|
||||
if not uno_build.exists():
|
||||
print("Uno build not found. Running compilation...")
|
||||
try:
|
||||
subprocess.run(
|
||||
"uv run python -m ci.ci-compile uno --examples Blink",
|
||||
shell=True,
|
||||
check=True,
|
||||
)
|
||||
print("Compilation completed successfully.")
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"Error during compilation: {e}")
|
||||
raise
|
||||
|
||||
@unittest.skip("Skip bin to elf conversion test")
|
||||
def test_bin_to_elf_conversion(self) -> None:
|
||||
if DISABLED:
|
||||
return
|
||||
tools: Tools
|
||||
try:
|
||||
tools = load_tools(BUILD_INFO_PATH)
|
||||
except FileNotFoundError as e:
|
||||
warnings.warn(f"Error while loading tools: {e}")
|
||||
tools = load_tools(BUILD_INFO_PATH2)
|
||||
bin_file = UNO / "firmware.hex"
|
||||
map_file = UNO / "firmware.map"
|
||||
output_elf = OUTPUT / "output.elf"
|
||||
try:
|
||||
bin_to_elf(
|
||||
bin_file,
|
||||
map_file,
|
||||
tools.as_path,
|
||||
tools.ld_path,
|
||||
tools.objcopy_path,
|
||||
output_elf,
|
||||
)
|
||||
stdout = dump_symbol_sizes(tools.nm_path, tools.cpp_filt_path, output_elf)
|
||||
print(stdout)
|
||||
except Exception as e:
|
||||
warnings.warn(f"Error while converting binary to ELF: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,53 @@
|
||||
import unittest
|
||||
|
||||
from ci.boards import Board
|
||||
|
||||
|
||||
class TestBoardToPlatformioIni(unittest.TestCase):
|
||||
"""Tests for Board.to_platformio_ini()."""
|
||||
|
||||
def _ini_to_set(self, ini: str) -> set[str]:
|
||||
"""Return a set with each non-empty, stripped line of the ini snippet."""
|
||||
return {line.strip() for line in ini.splitlines() if line.strip()}
|
||||
|
||||
def test_basic_fields(self):
|
||||
board = Board(board_name="uno", platform="atmelavr", framework="arduino")
|
||||
ini = board.to_platformio_ini()
|
||||
lines = self._ini_to_set(ini)
|
||||
expected = {
|
||||
"[env:uno]",
|
||||
"board = uno",
|
||||
"platform = atmelavr",
|
||||
"framework = arduino",
|
||||
}
|
||||
self.assertTrue(expected.issubset(lines))
|
||||
# Should not reference internal attributes
|
||||
self.assertNotIn("platform_needs_install", ini)
|
||||
|
||||
def test_real_board_name(self):
|
||||
board = Board(
|
||||
board_name="esp32c3",
|
||||
real_board_name="esp32-c3-devkitm-1",
|
||||
platform="espressif32",
|
||||
)
|
||||
ini = board.to_platformio_ini()
|
||||
lines = self._ini_to_set(ini)
|
||||
self.assertIn("[env:esp32c3]", lines)
|
||||
self.assertIn("board = esp32-c3-devkitm-1", lines)
|
||||
|
||||
def test_flags_and_unflags(self):
|
||||
board = Board(
|
||||
board_name="custom",
|
||||
defines=["FASTLED_TEST=1"],
|
||||
build_flags=["-O2"],
|
||||
)
|
||||
ini = board.to_platformio_ini()
|
||||
lines = self._ini_to_set(ini)
|
||||
# The build_flags are in multi-line format - check that both flags are present as separate lines
|
||||
self.assertIn("build_flags =", lines)
|
||||
self.assertIn("-DFASTLED_TEST=1", lines)
|
||||
self.assertIn("-O2", lines)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,357 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Unit tests for BuildFlags TOML parsing and serialization functionality.
|
||||
|
||||
Tests the BuildFlags and BuildTools classes for parsing TOML build configuration
|
||||
files and serializing back to TOML format.
|
||||
"""
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from ci.compiler.clang_compiler import ArchiveOptions, BuildFlags, BuildTools
|
||||
|
||||
|
||||
class TestBuildFlagsToml(unittest.TestCase):
|
||||
"""Test BuildFlags TOML parsing and serialization"""
|
||||
|
||||
def setUp(self) -> None:
|
||||
"""Set up test fixtures"""
|
||||
self.temp_dir = Path(tempfile.mkdtemp())
|
||||
|
||||
def tearDown(self) -> None:
|
||||
"""Clean up test fixtures"""
|
||||
# Clean up any test files
|
||||
for file in self.temp_dir.glob("*.toml"):
|
||||
file.unlink()
|
||||
self.temp_dir.rmdir()
|
||||
|
||||
def create_test_toml(self, content: str) -> Path:
|
||||
"""Create a test TOML file with the given content"""
|
||||
test_file = self.temp_dir / "test_build_flags.toml"
|
||||
with open(test_file, "w", encoding="utf-8") as f:
|
||||
f.write(content)
|
||||
return test_file
|
||||
|
||||
def test_build_tools_defaults(self) -> None:
|
||||
"""Test BuildTools default values"""
|
||||
tools = BuildTools(
|
||||
cpp_compiler=[],
|
||||
archiver=[],
|
||||
linker=[],
|
||||
c_compiler=[],
|
||||
objcopy=[],
|
||||
nm=[],
|
||||
strip=[],
|
||||
ranlib=[],
|
||||
)
|
||||
|
||||
# Test modern command-based fields (should be empty by default)
|
||||
self.assertEqual(tools.cpp_compiler, [])
|
||||
self.assertEqual(tools.archiver, [])
|
||||
self.assertEqual(tools.linker, [])
|
||||
|
||||
# Test other important fields
|
||||
self.assertEqual(tools.c_compiler, [])
|
||||
self.assertEqual(tools.objcopy, [])
|
||||
self.assertEqual(tools.nm, [])
|
||||
self.assertEqual(tools.strip, [])
|
||||
self.assertEqual(tools.ranlib, [])
|
||||
|
||||
def test_parse_minimal_toml(self) -> None:
|
||||
"""Test parsing minimal TOML file with required tools section"""
|
||||
toml_content = """
|
||||
[all]
|
||||
defines = ["-DTEST=1"]
|
||||
compiler_flags = ["-Wall"]
|
||||
include_flags = ["-I."]
|
||||
|
||||
[tools]
|
||||
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
linker = ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
c_compiler = ["clang"]
|
||||
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
|
||||
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
|
||||
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
|
||||
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
|
||||
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"]
|
||||
|
||||
[archive]
|
||||
flags = "rcsD"
|
||||
|
||||
[linking.base]
|
||||
flags = ["-pthread"]
|
||||
|
||||
[strict_mode]
|
||||
flags = ["-Werror"]
|
||||
"""
|
||||
|
||||
test_file = self.create_test_toml(toml_content)
|
||||
flags = BuildFlags.parse(test_file, quick_build=False, strict_mode=False)
|
||||
|
||||
# Check basic flags
|
||||
self.assertEqual(flags.defines, ["-DTEST=1"])
|
||||
self.assertEqual(flags.compiler_flags, ["-Wall"])
|
||||
self.assertEqual(flags.include_flags, ["-I."])
|
||||
self.assertEqual(flags.link_flags, ["-pthread"])
|
||||
self.assertEqual(flags.strict_mode_flags, ["-Werror"])
|
||||
|
||||
# Check tools (from [tools] section) - modern command-based approach
|
||||
self.assertEqual(flags.tools.c_compiler, ["clang"])
|
||||
self.assertEqual(
|
||||
flags.tools.linker, ["uv", "run", "python", "-m", "ziglang", "c++"]
|
||||
)
|
||||
self.assertEqual(
|
||||
flags.tools.cpp_compiler,
|
||||
["uv", "run", "python", "-m", "ziglang", "c++"],
|
||||
)
|
||||
self.assertEqual(
|
||||
flags.tools.objcopy, ["uv", "run", "python", "-m", "ziglang", "objcopy"]
|
||||
)
|
||||
self.assertEqual(flags.tools.nm, ["uv", "run", "python", "-m", "ziglang", "nm"])
|
||||
self.assertEqual(
|
||||
flags.tools.strip, ["uv", "run", "python", "-m", "ziglang", "strip"]
|
||||
)
|
||||
self.assertEqual(
|
||||
flags.tools.ranlib, ["uv", "run", "python", "-m", "ziglang", "ranlib"]
|
||||
)
|
||||
self.assertEqual(
|
||||
flags.tools.archiver, ["uv", "run", "python", "-m", "ziglang", "ar"]
|
||||
)
|
||||
|
||||
def test_parse_toml_with_tools(self) -> None:
|
||||
"""Test parsing TOML file with [tools] section"""
|
||||
toml_content = """
|
||||
[all]
|
||||
defines = ["-DTEST=1"]
|
||||
compiler_flags = ["-Wall"]
|
||||
include_flags = ["-I."]
|
||||
|
||||
[tools]
|
||||
cpp_compiler = ["g++"]
|
||||
archiver = ["gcc-ar"]
|
||||
linker = ["ld.gold"]
|
||||
c_compiler = ["gcc"]
|
||||
objcopy = ["arm-objcopy"]
|
||||
nm = ["arm-nm"]
|
||||
strip = ["arm-strip"]
|
||||
ranlib = ["arm-ranlib"]
|
||||
|
||||
[archive]
|
||||
flags = "rcsD"
|
||||
|
||||
[linking.base]
|
||||
flags = ["-pthread"]
|
||||
"""
|
||||
|
||||
test_file = self.create_test_toml(toml_content)
|
||||
flags = BuildFlags.parse(test_file, quick_build=False, strict_mode=False)
|
||||
|
||||
# Check that tools were parsed correctly - modern command-based approach
|
||||
self.assertEqual(flags.tools.cpp_compiler, ["g++"])
|
||||
self.assertEqual(flags.tools.archiver, ["gcc-ar"])
|
||||
self.assertEqual(flags.tools.linker, ["ld.gold"])
|
||||
self.assertEqual(flags.tools.c_compiler, ["gcc"])
|
||||
self.assertEqual(flags.tools.objcopy, ["arm-objcopy"])
|
||||
self.assertEqual(flags.tools.nm, ["arm-nm"])
|
||||
self.assertEqual(flags.tools.strip, ["arm-strip"])
|
||||
self.assertEqual(flags.tools.ranlib, ["arm-ranlib"])
|
||||
|
||||
def test_parse_partial_tools_section(self) -> None:
|
||||
"""Test parsing TOML with all required [tools] section fields"""
|
||||
toml_content = """
|
||||
[all]
|
||||
defines = ["-DTEST=1"]
|
||||
|
||||
[tools]
|
||||
cpp_compiler = ["custom-clang++"]
|
||||
archiver = ["custom-ar"]
|
||||
linker = ["custom-linker"]
|
||||
c_compiler = ["clang"]
|
||||
objcopy = ["custom-objcopy"]
|
||||
nm = ["custom-nm"]
|
||||
strip = ["custom-strip"]
|
||||
ranlib = ["custom-ranlib"]
|
||||
# All tools must be provided - no defaults allowed
|
||||
|
||||
[archive]
|
||||
flags = "rcsD"
|
||||
"""
|
||||
|
||||
test_file = self.create_test_toml(toml_content)
|
||||
flags = BuildFlags.parse(test_file, quick_build=False, strict_mode=False)
|
||||
|
||||
# Check all tools are set as specified - strict validation
|
||||
self.assertEqual(flags.tools.cpp_compiler, ["custom-clang++"])
|
||||
self.assertEqual(flags.tools.archiver, ["custom-ar"])
|
||||
self.assertEqual(flags.tools.linker, ["custom-linker"])
|
||||
self.assertEqual(flags.tools.c_compiler, ["clang"])
|
||||
self.assertEqual(flags.tools.objcopy, ["custom-objcopy"])
|
||||
self.assertEqual(flags.tools.nm, ["custom-nm"])
|
||||
self.assertEqual(flags.tools.strip, ["custom-strip"])
|
||||
self.assertEqual(flags.tools.ranlib, ["custom-ranlib"])
|
||||
|
||||
def test_serialize_build_flags_with_tools(self) -> None:
|
||||
"""Test serializing BuildFlags with tools to TOML"""
|
||||
# Create BuildFlags with custom tools
|
||||
custom_tools = BuildTools(
|
||||
linker=["arm-none-eabi-ld"],
|
||||
c_compiler=["arm-none-eabi-gcc"],
|
||||
objcopy=["arm-none-eabi-objcopy"],
|
||||
nm=["arm-none-eabi-nm"],
|
||||
strip=["arm-none-eabi-strip"],
|
||||
ranlib=["arm-none-eabi-ranlib"],
|
||||
cpp_compiler=["arm-none-eabi-g++"],
|
||||
archiver=["arm-none-eabi-ar"],
|
||||
)
|
||||
|
||||
flags = BuildFlags(
|
||||
defines=["-DARM_BUILD=1"],
|
||||
compiler_flags=["-mcpu=cortex-m4", "-mthumb"],
|
||||
include_flags=["-I.", "-Iarm"],
|
||||
link_flags=["-nostdlib"],
|
||||
strict_mode_flags=["-Werror"],
|
||||
tools=custom_tools,
|
||||
archive=ArchiveOptions(flags="rcsD"),
|
||||
)
|
||||
|
||||
# Serialize to TOML
|
||||
toml_output = flags.serialize()
|
||||
|
||||
# Check that tools section is present with new field names
|
||||
self.assertIn("[tools]", toml_output)
|
||||
self.assertIn("cpp_compiler = ['arm-none-eabi-g++']", toml_output)
|
||||
self.assertIn("archiver = ['arm-none-eabi-ar']", toml_output)
|
||||
self.assertIn("linker = ['arm-none-eabi-ld']", toml_output)
|
||||
self.assertIn("c_compiler = ['arm-none-eabi-gcc']", toml_output)
|
||||
self.assertIn("objcopy = ['arm-none-eabi-objcopy']", toml_output)
|
||||
self.assertIn("nm = ['arm-none-eabi-nm']", toml_output)
|
||||
self.assertIn("strip = ['arm-none-eabi-strip']", toml_output)
|
||||
self.assertIn("ranlib = ['arm-none-eabi-ranlib']", toml_output)
|
||||
|
||||
def test_serialize_with_none_linker(self) -> None:
|
||||
"""Test serializing BuildFlags when linker is None"""
|
||||
flags = BuildFlags(
|
||||
defines=["-DTEST=1"],
|
||||
compiler_flags=[],
|
||||
include_flags=[],
|
||||
link_flags=[],
|
||||
strict_mode_flags=[],
|
||||
tools=BuildTools(
|
||||
linker=[], # Empty list instead of None
|
||||
cpp_compiler=["clang++"],
|
||||
c_compiler=["clang"],
|
||||
archiver=[],
|
||||
objcopy=[],
|
||||
nm=[],
|
||||
strip=[],
|
||||
ranlib=[],
|
||||
),
|
||||
archive=ArchiveOptions(flags="rcsD"),
|
||||
)
|
||||
|
||||
toml_output = flags.serialize()
|
||||
|
||||
# Check that tools section is present but linker is omitted
|
||||
self.assertIn("[tools]", toml_output)
|
||||
self.assertIn("cpp_compiler = ['clang++']", toml_output)
|
||||
self.assertNotIn("linker =", toml_output) # Should be omitted when empty
|
||||
|
||||
def test_round_trip_toml_parsing(self) -> None:
|
||||
"""Test that parse -> serialize -> parse maintains data integrity"""
|
||||
# Create original flags
|
||||
original_tools = BuildTools(
|
||||
linker=["test-ld"],
|
||||
c_compiler=["test-gcc"],
|
||||
cpp_compiler=["test-compiler"],
|
||||
archiver=["test-ar"],
|
||||
objcopy=["test-objcopy"],
|
||||
nm=["test-nm"],
|
||||
strip=["test-strip"],
|
||||
ranlib=["test-ranlib"],
|
||||
)
|
||||
|
||||
original_flags = BuildFlags(
|
||||
defines=["-DROUND_TRIP=1"],
|
||||
compiler_flags=["-Wall", "-O2"],
|
||||
include_flags=["-I.", "-Itest"],
|
||||
link_flags=["-pthread"],
|
||||
strict_mode_flags=["-Werror"],
|
||||
tools=original_tools,
|
||||
archive=ArchiveOptions(flags="rcsD"),
|
||||
)
|
||||
|
||||
# Serialize to TOML
|
||||
toml_content = original_flags.serialize()
|
||||
|
||||
# Write to temporary file
|
||||
temp_file = self.temp_dir / "roundtrip.toml"
|
||||
original_flags.to_toml_file(temp_file)
|
||||
|
||||
# Parse back from file
|
||||
parsed_flags = BuildFlags.parse(temp_file, quick_build=False, strict_mode=False)
|
||||
|
||||
# Check that all data is preserved
|
||||
self.assertEqual(parsed_flags.defines, original_flags.defines)
|
||||
self.assertEqual(parsed_flags.compiler_flags, original_flags.compiler_flags)
|
||||
self.assertEqual(parsed_flags.include_flags, original_flags.include_flags)
|
||||
self.assertEqual(parsed_flags.link_flags, original_flags.link_flags)
|
||||
self.assertEqual(
|
||||
parsed_flags.strict_mode_flags, original_flags.strict_mode_flags
|
||||
)
|
||||
|
||||
# Check tools - modern command-based approach
|
||||
self.assertEqual(
|
||||
parsed_flags.tools.cpp_compiler, original_flags.tools.cpp_compiler
|
||||
)
|
||||
self.assertEqual(parsed_flags.tools.archiver, original_flags.tools.archiver)
|
||||
self.assertEqual(parsed_flags.tools.linker, original_flags.tools.linker)
|
||||
self.assertEqual(parsed_flags.tools.c_compiler, original_flags.tools.c_compiler)
|
||||
|
||||
def test_parse_missing_file(self) -> None:
|
||||
"""Test parsing non-existent TOML file raises FileNotFoundError"""
|
||||
nonexistent_file = self.temp_dir / "does_not_exist.toml"
|
||||
|
||||
# Should raise FileNotFoundError when file is missing
|
||||
with self.assertRaises(FileNotFoundError) as context:
|
||||
BuildFlags.parse(nonexistent_file, quick_build=False, strict_mode=False)
|
||||
|
||||
self.assertIn("Required build_flags.toml not found", str(context.exception))
|
||||
|
||||
def test_from_toml_file_alias(self) -> None:
|
||||
"""Test that from_toml_file() is an alias for parse()"""
|
||||
toml_content = """
|
||||
[all]
|
||||
defines = ["-DALIAS_TEST=1"]
|
||||
|
||||
[tools]
|
||||
cpp_compiler = ["alias-compiler"]
|
||||
linker = ["alias-linker"]
|
||||
c_compiler = ["clang"]
|
||||
objcopy = ["alias-objcopy"]
|
||||
nm = ["alias-nm"]
|
||||
strip = ["alias-strip"]
|
||||
ranlib = ["alias-ranlib"]
|
||||
archiver = ["alias-archiver"]
|
||||
|
||||
[archive]
|
||||
flags = "rcsD"
|
||||
"""
|
||||
|
||||
test_file = self.create_test_toml(toml_content)
|
||||
|
||||
# Parse using both methods
|
||||
flags_parse = BuildFlags.parse(test_file, quick_build=True, strict_mode=False)
|
||||
flags_alias = BuildFlags.from_toml_file(
|
||||
test_file, quick_build=True, strict_mode=False
|
||||
)
|
||||
|
||||
# Should be identical
|
||||
self.assertEqual(flags_parse.defines, flags_alias.defines)
|
||||
self.assertEqual(flags_parse.tools.cpp_compiler, flags_alias.tools.cpp_compiler)
|
||||
self.assertEqual(flags_parse.tools.cpp_compiler, ["alias-compiler"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loaded 100 of 3403 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user