x-cube-n6-camera-capture-nucleo

The STM32N6 is the first STM32 MCU with an integrated NPU (Neural-ART, 600 GOPS, 3 TOPS/W) for on-device AI, on an Arm Cortex-M55 core @ 800 MHz with Helium vector extensions, 4.2 MB of RAM, a dedicated ISP and media/graphics accelerators; it has no internal flash (Load & Run execution from external Octo-SPI NOR). This is the first of a series of projects through which we start testing its capabilities: it begins with the capture→H.264 encode→UVC-over-USB pipeline (Nucleo-N657X0-Q); projects leveraging the NPU for on-device AI will follow. Note: this specific firmware exercises the imaging + hardware H.264 encoder path, not yet the NPU. Content derived from the repository README; the authoritative reference remains GitHub.

Why the STM32N6

The STM32N6 is the first STM32 MCU with an integrated NPU: the Neural-ART accelerator delivers ~600 GOPS at 3 TOPS/W efficiency, enabling on-device AI inference previously reserved for microprocessors. The core is an Arm Cortex-M55 @ 800 MHz (1280 DMIPS / 3360 CoreMark), the first Cortex-M with Helium (MVE) vector extensions — accelerated DSP and ML on the CPU.

The picture is completed by 4.2 MB of contiguous RAM (the largest on STM32), a dedicated ISP, multiple camera interfaces and graphics/multimedia accelerators, with no internal flash (Load & Run execution from Octo-SPI NOR). In this project, concretely: STM32N657X0HxQ, PLL1 at 800 MHz, hardware H.264 encoder (Verisilicon H1 IP), MIPI-CSI2 receiver + ISP, USB High-Speed.

It is the first of a series: here we validate the imaging→encode→USB chain end-to-end on an accessible board (Nucleo), as a concrete base on which to graft the follow-up projects that leverage the NPU for on-device AI.

What the firmware does

At start-up it configures the clock (PLL1 → 800 MHz) and the MPU, brings up the MIPI-CSI2 receiver and the IMX335 sensor through the ST Camera Middleware + ISP library, and captures frames via DCMIPP in YUY2 format.

The luma plane is fed to the STM32N6 hardware H.264 encoder (Verisilicon H1 IP); a static grey chroma plane makes the stream a valid YUV 4:2:0 bitstream. The H.264 NAL units are packed into UVC Frame-Based payloads and sent to the host over USB High-Speed. Everything runs on FreeRTOS with a task dedicated to capture, encode and USB.

Result: a standard UVC webcam with a single stream, default 480×272 H.264 at 30 fps.

The imaging pipeline: from sensor to frame

The IMX335 sensor does not output ready-to-use pixels: over MIPI-CSI2 it sends a RAW Bayer data stream, where each photosite carries a single colour according to the filter matrix. Conversion into a viewable image happens on the STM32N6 itself, in two stages.

The MIPI-CSI2 receiver deserialises the two differential lanes (D-PHY), reassembles the line packets and hands them to the DCMIPP (Digital Camera Memory Interface Pixel Processor), the STM32N6 capture block. Leaning on the ISP (Image Signal Processor) configured by the ST library, the DCMIPP performs demosaicing, white balance, colour correction and colour-space conversion, writing frames to RAM in YUY2 format (packed YUV 4:2:2: full-resolution luma, horizontally subsampled chroma).

Separating luminance (Y) and chrominance (U, V) is not a detail: it is exactly what the next stage needs, because the video encoder works on Y/UV planes and because the human eye is far more sensitive to luminance than to colour — the basis of all modern video compression.

The hardware H.264 encoder

The STM32N6 integrates a hardware H.264 video encoder (Verisilicon H1 IP): a dedicated block that compresses frames without loading the CPU. It takes a YUV 4:2:0 image as input and produces an H.264 bitstream organised into NAL units (Network Abstraction Layer), the same units in which H.264 travels in any file or stream.

In this firmware the encoder receives only the luma plane produced by the DCMIPP; the chroma plane is filled with a static grey, so the bitstream stays a formally valid YUV 4:2:0 while the image is monochrome. It is a deliberate trade-off: without external PSRAM on the Nucleo there is no bandwidth/RAM to carry the colour planes through the full pipeline. Hardware compression greatly reduces the bandwidth needed on USB — by how much, and how much it is really needed at this resolution, is in the arithmetic below.

UVC transport over USB High-Speed

The H.264 NAL units are not saved to a file: they are packed into UVC Frame-Based payloads and pushed to the host over USB High-Speed (480 Mbit/s). The USB Video Class (UVC) is the standard class that makes a camera usable without drivers; the Frame-Based format is the one UVC defines for compressed or non-standard streams such as H.264.

At enumeration the device presents itself as the “STM32 N6 Camera” webcam, exposes its descriptors (format, resolution, frame rate) and negotiates the stream with the host through the Probe/Commit controls. From there on it is a webcam like any other: any UVC application opens it.

Software architecture on FreeRTOS

The firmware runs on FreeRTOS, with USBX (Azure RTOS / ThreadX) for the USB stack. A dedicated task orchestrates the whole capture → encode → send path: it waits for the YUY2 frame from the DCMIPP, passes it to the H.264 encoder, collects the resulting NAL units and hands them to the UVC stack for transmission. Decoupling capture, compression and USB through the RTOS is what keeps the 30 fps stable without one block idly waiting on another.

Hardware

NUCLEO-N657X0-Q board (STM32N657X0HxQ, Cortex-M55, onboard ST-LINK/V3, High-Speed user USB on CN13). External Macronix MX25UM51245G NOR (512 Mbit Octo-SPI, onboard) for the FSBL + signed application.

Sony IMX335 camera (5 MP, 2-lane MIPI) on the MB1854 MIPI-CSI2 daughterboard. The Camera Middleware also supports other sensors (untested on the Nucleo). The Nucleo boot pins must be set to Flash boot for standalone operation.

Boot model (Load & Run)

The Nucleo-N6 has no internal flash: code lives on the external Octo-SPI NOR. The BootROM verifies the FSBL signature and loads it into AXISRAM2; the FSBL brings up the Octo-SPI in memory-mapped mode, copies the signed application into AXISRAM and jumps to it.

FSBL at 0x70000000, application at 0x70100000; both binaries signed with STM32_SigningTool_CLI (dummy keys, development only). The FSBL sub-project is fully vendored in Tools/FSBL/, derived from the ST Template_FSBL_LRUN and converted to a standalone CMake build.

Differences vs the ST example (DK → Nucleo)

It is a port/adaptation of the official ST x-cube-n6-camera-capture example (DK board) to the Nucleo-N657X0-Q: no external PSRAM (all buffers fit in internal AXISRAM/DTCM), no LCD/touch (the only output is the UVC stream), a fully vendored FSBL so it can be programmed and run standalone.

Build with CMake + Ninja + VS Code tasks for build/sign/flash: STM32CubeIDE is not required, only STM32CubeProgrammer. All ST/third-party code is frozen (vendored) in the repository: no external packs or CubeMX projects.

Build, sign and flash

Tested toolchain: arm-none-eabi-gcc 14.3.1 + ST patches, CMake ≥ 3.22, Ninja ≥ 1.10, STM32CubeProgrammer 2.22.0 (required for sign + flash). A cube-cmake wrapper (STM32 VS Code extension) points to the bundled GCC; any arm-none-eabi-gcc on PATH works too.

The VS Code tasks cover: Configure/Build app and FSBL, Sign FSBL/App, Program FSBL (0x70000000) / App (0x70100000), the one-shot “Build + Sign + Program App” and the initial “Full Program (FSBL + App)” provisioning. External loader: MX25UM51245G_STM32N6570-NUCLEO.stldr.

Using it as a webcam

After Program App: reset the board, plug the user USB cable (CN13, USB-C on the Nucleo) into the PC. A “STM32 N6 Camera” UVC device appears; open it with any UVC app — Windows Camera, OBS, VLC, ffplay (e.g. ffplay -f dshow -i video="STM32 N6 Camera").

Troubleshooting

• The device does not appear. Check that the Nucleo boot pins are set to Flash boot and that both the FSBL and the application were programmed (FSBL at 0x70000000, app at 0x70100000). Without a valid FSBL the BootROM starts nothing.
• It appears but the stream won’t start. Connect the cable to the user USB (CN13), not the ST-LINK port: they are two distinct connectors. Some apps open only uncompressed formats — use a player that handles UVC Frame-Based/H.264 (OBS, VLC, ffplay).
• Missing or corrupted image. Make sure the MB1854 MIPI-CSI2 daughterboard with the IMX335 is seated correctly; a different sensor is untested and may need a different ISP configuration.

From capture to on-device AI

This firmware exercises the imaging + H.264 encoder path, not yet the NPU. It is the deliberate first step of a series: validating the sensor → ISP → encode → USB chain end-to-end on an accessible board before grafting on inference. The follow-up projects will leverage the Neural-ART accelerator for on-device AI — detection, classification or pre-processing directly on the stream, without sending frames to a host. The pipeline built here remains the base those developments will rest on.

The chain’s three links, counted

The arithmetic on the whole chain is what settles it, because the bottleneck is not where intuition puts it — and knowing which link it is, is exactly what tells you whether a higher resolution is reachable.

The USB transport: the least loaded of the three

A single isochronous endpoint on USB High-Speed carries at most 3 transactions of 1024 bytes per microframe, that is 196.6 Mbit/s. Against that budget:

The result is counterintuitive and worth stating precisely: at 480×272 the uncompressed stream would fit, taking 32 % of the budget. H.264 here is not there to squeeze the stream into USB — it is there because exercising the hardware encoder is the point of the project, and because it is the only route to the resolutions USB cannot carry: uncompressed blows the budget just above 409,600 pixels, so 854×480 is already out and 720p would ask for 442 Mbit/s.

Memory: the constraint that actually bites

What stops higher resolutions on this board is not USB and not the encoder: it is RAM, and the arithmetic is simple because a YUY2 frame takes exactly two bytes per pixel.

A double-buffered 480×272 fits in half a megabyte; 720p asks for 3.52, seven times as much — and on top of that goes whatever the encoder needs for its reference frames, which H.264 requires for inter-frame prediction. Without external PSRAM the Nucleo has no room for that sum: it is the precise technical reason behind the limitation declared further down, and the “luma only” column also explains the monochrome compromise — giving up chrominance halves the footprint along the whole encoding path, from 255 to 128 kB per frame.

The MIPI link: the most loaded, for a non-obvious reason

The busiest link in the chain is the first one, and the reason is that the DCMIPP scales downstream: what travels on the CSI-2 link is the RAW at whatever resolution the sensor is configured for, not the 480×272 of the output.

At the IMX335’s full resolution the two lanes sit at 75.6 % utilization: the link is the component closest to its own limit in the entire chain, while USB sits at 32 % and the encoder is not straining. It is also why sensor binning, where available, is always worth it: dividing the pixels on the link by four frees bandwidth that no downstream stage can give back. The arithmetic can be redone with the MIPI CSI-2 bandwidth calculator — remembering to add the blanking, which on these figures is worth another 20–30 %.

The chain in one line: MIPI 76 %, encoder comfortable, USB 32 %, RAM exhausted. Anyone wanting to raise the resolution on this board has to start from the last term, not the first three.

Known limitations

Monochrome output: the H.264 encoder requires YUV 4:2:0 input but only the luma plane is filled (static grey chroma) — a RAM/bandwidth trade-off due to the Nucleo having no PSRAM; the DK build of the original example provides colour.

Single stream 480×272 @ 30 fps (higher resolutions would need external PSRAM); the signing keys are the ST dummy development keys — not for production use. Status: working prototype, a personal port published as-is.

Credits and licence

It reuses and adapts code from upstream ST projects, all vendored in the repo under their original licences: the DK camera-capture example, STM32CubeN6 (HAL/CMSIS/BSP/FSBL ref.), the Verisilicon H1 H.264 encoder + EWL, stm32-camera-middleware + ISP, USBX (Azure RTOS / ThreadX), the FreeRTOS kernel and the ST ExtMem Manager.

Application-specific files follow the terms of the ST example they derive from; each third-party component keeps its own licence file alongside its sources.