Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
317 lines (218 loc) · 12.9 KB

DEVELOPMENT.md

File metadata and controls

317 lines (218 loc) · 12.9 KB
Raw

Development guide

This guide explains how to build, test, and lint ONNX Runtime GenAI from source. It is written to be quick to skim for humans and unambiguous for AI coding agents: every section lists the exact command to run.

For end-user install instructions (pip / NuGet), see the README and the official docs.

For the current continuous-batching architecture and transactional paged KV-cache flow, see Paged Attention Engine. The performance and design history behind the batching implementation is documented separately in Engine Batching Design.

1. Prerequisites

Tool Notes
Python ≥ 3.10 Drives build.py; also the runtime for the Python wheel. Official wheels target 3.11–3.14.
CMake ≥ 3.26 Build-system generator. macOS builds require CMake ≥ 3.28.
C++ compiler with C++20 support MSVC (Visual Studio 2022), GCC ≥ 11, or Clang.
Git For cloning the repository.
.NET SDK ≥ 8 Only for the C# bindings (--build_csharp).
JDK ≥ 8 Only for the Java bindings (--build_java). The repository includes the Gradle wrapper, so a separate Gradle installation is not required.
CUDA Toolkit Only for CUDA builds (--use_cuda).

Clone the repository:

git clone https://github.com/microsoft/onnxruntime-genai.git
cd onnxruntime-genai

This repository has no git submodules. CMake-managed native dependencies are fetched automatically at configure time (via FetchContent, driven by cmake/deps.txt), so no submodule initialization is needed. Python, Rust, and Java dependencies use their respective package managers.

Install the common Python build-time dependencies. pybind11 is required when building the Python SDK incrementally with --sdk python:

python -m pip install -r requirements-dev.txt

2. Quick start

build.py is the single cross-platform entry point. build.sh (Linux/macOS) and build.bat (Windows) are thin wrappers around it.

Build the CPU native library, Python wheel, C examples, and native tests with the default RelWithDebInfo configuration:

python build.py

Common everyday build (RelWithDebInfo, parallel, skip tests and examples for a fast inner loop):

python build.py --config RelWithDebInfo --parallel --skip_tests --skip_examples

The Python wheel is built by default. The build tree is created under build/<platform>/<config>/ (for example, build/Windows/RelWithDebInfo/ or build/Linux/RelWithDebInfo/). With a multi-config generator such as Visual Studio, native binaries are in an additional configuration subdirectory such as build/Windows/RelWithDebInfo/RelWithDebInfo/.

3. build.py phases and key flags

build.py runs three phases. If no phase is specified, it requests all three (--update --build --test). For --arm64 and --arm64ec cross-compilation, the test phase is disabled after CMake configuration because the target binaries cannot run on the host. If you pass any phase explicitly, only those phases run.

Phase flag What it does
--update Run CMake to (re)generate the build system.
--build Compile all targets.
--test Run the unit tests.

Example — regenerate CMake, then build, without running tests:

python build.py --update --build

Example — rebuild only (skip CMake regeneration and tests), useful after editing source files:

python build.py --build

Frequently used flags:

Flag Purpose
--config {Debug,Release,RelWithDebInfo,MinSizeRel} Build configuration. Default RelWithDebInfo.
--parallel Parallel compilation.
--skip_tests Skip the test phase.
--skip_wheel Do not build the Python wheel.
--skip_examples Do not build the sample executables.
--build_dir <path> Override the build directory (default build/<platform>).
--use_guidance Enable constrained / grammar-based decoding (guidance).
--ort_home <path> Use a prebuilt ONNX Runtime at this path instead of auto-downloading.
--cmake_extra_defines K=V Pass extra -DK=V options to CMake.

Run python build.py --help for the complete list.

4. Execution-provider (hardware) builds

By default the build targets the CPU execution provider. Enable hardware acceleration with these flags:

# CUDA (reads CUDA_HOME / CUDA_PATH, or pass --cuda_home)
python build.py --use_cuda --config RelWithDebInfo

# DirectML (Windows)
python build.py --use_dml --config RelWithDebInfo

# WinML (Windows; bundles CUDA + DirectML). --winml_sdk_version is optional.
python build.py --use_winml --winml_sdk_version 2.1.1 --config RelWithDebInfo

5. Language bindings

The Python wheel is built by default. Enable the others explicitly:

# C# (requires the .NET SDK)
python build.py --build_csharp --config RelWithDebInfo

# Java (requires a JDK; uses the repository's Gradle wrapper)
python build.py --build_java --config RelWithDebInfo

To skip the Python wheel (e.g. when you only want the native library or C#):

python build.py --skip_wheel --build_csharp

6. Incremental core + SDK builds

The packaging pipeline builds the native core once, installs it, then builds each SDK layer (Python / C# / Java) against that prebuilt core. You can use the same flow locally to iterate on a single binding without rebuilding the core.

Step 1 — build and install only the core to a prefix:

python build.py --config RelWithDebInfo --parallel --skip_tests --skip_examples --skip_wheel --ort_home /path/to/ort --install_dir /path/to/core-install --cmake_extra_defines ENABLE_PYTHON=OFF ENABLE_TESTS=OFF

Step 2 — build a single SDK against the installed core (the core is not rebuilt):

python build.py --sdk python --prebuilt_genai_home /path/to/core-install --ort_home /path/to/ort --config RelWithDebInfo

--sdk accepts python, java, or csharp. Both --prebuilt_genai_home and --ort_home are required in SDK mode.

The core install prefix, including the native CPack archive, uses this layout:

<prefix>/include/                         # Public C and C++ headers
<prefix>/lib/                             # Native GenAI libraries
<prefix>/lib/cmake/onnxruntime-genai/     # CMake package configuration

Use the exported CMake package from a native consumer:

find_package(onnxruntime-genai CONFIG REQUIRED)
target_link_libraries(my_app PRIVATE onnxruntime-genai::onnxruntime-genai)

Configure the consumer with -Donnxruntime-genai_DIR=<prefix>/lib/cmake/onnxruntime-genai. ONNX Runtime is an external runtime dependency and is not included in the core archive.

7. Running tests

C++ unit tests (CTest)

After a build, run the native tests from the build directory:

ctest --test-dir build/<platform>/<config> --build-config <config> --output-on-failure

Example:

ctest --test-dir build/Windows/RelWithDebInfo --build-config RelWithDebInfo --output-on-failure

You can also let build.py run them as part of the --test phase:

python build.py --config RelWithDebInfo --test

Note: build.py automatically disables the native test phase for --arm64 and --arm64ec. Android and iOS disable CMake unit-test targets during configuration; Android emulator tests are available only for an x86-64 Android build with --build_java --android_run_emulator.

Native shared-library exports

On Linux and Android, src/exports.map restricts onnxruntime-genai exports to the public Oga* C API. Apple builds use src/exported-symbols.lst for the same restriction. C++ consumers use the header-only wrappers over that API; GenAI internals and statically linked dependencies, including curl, remain private to prevent symbol collisions with libraries loaded by the host process. This does not hide symbols in separately shipped shared dependencies. Windows retains its explicit OGA_EXPORT exports, and CUDA add-on export policies are unchanged.

ELF links also localize definitions from all static archives with --exclude-libs,ALL, including GCC's implicitly linked libstdc++_nonshared.a compatibility archive on manylinux. Public APIs must remain in direct or OBJECT sources: placing them in STATIC archives would hide them despite the version script.

With ENABLE_TESTS=ON on Linux or macOS, check that every public API declared in src/ort_genai_c.h is exported and no other symbols are exported. The ELF checker inspects externally visible definitions only; local symbols left in the dynamic symbol table by GCC toolset compatibility objects are not exports:

ctest --test-dir build/Linux/Release -R '^SharedLibraryExports$' --output-on-failure

The Linux x64 CPU and CUDA CI jobs run both SharedLibraryExports and TelemetryHelpersTests explicitly, alongside SharedLibraryArchiveExports and SharedLibraryArchiveRuntime, which check strong/weak archive-symbol isolation and runtime calls through the public fixture API. Windows x64 CPU CI runs TelemetryHelpersTests, including redaction and environment-read race regressions. Linux and macOS core release jobs also run the export checker on the installed library against its installed public header, even with ENABLE_TESTS=OFF; macOS checks the signed library.

Python tests (pytest)

Install the Python test dependencies and the wheel produced by your build:

python -m pip install -r test/python/requirements.txt
python -m pip install --force-reinstall <path-to-built-wheel>

The Python tests require model files. From the repository root, point pytest at a models directory with --test_models:

python -m pytest -sv test/python/test_onnxruntime_genai_api.py -k "test_greedy_search" --test_models test/models

Drop the -k filter to run the whole file. See test/python/README.md. Provider-specific tests may require additional dependency files under test/python/<provider>/.

8. Linting and formatting

This project uses lintrunner. Install and initialize once:

pip install -r requirements-lintrunner.txt
lintrunner init

Then, to auto-format your local (changed) files:

lintrunner -a

To format the entire tree:

lintrunner -a --all-files

CI gate: C/C++ formatting is enforced with clang-format 20.1.0 (--dry-run -Werror). Any unformatted changed C/C++ file fails the PR, so run lintrunner -a before pushing.

9. Cross-compilation

# Windows ARM64 (cross-compiled on an x64 host)
python build.py --arm64 --config RelWithDebInfo --skip_tests

# Android (requires ANDROID_HOME / ANDROID_NDK_HOME)
python build.py --android --android_abi arm64-v8a --config RelWithDebInfo --skip_tests

# iOS simulator framework (build on macOS)
python build.py --ios --build_apple_framework --apple_sysroot iphonesimulator --osx_arch arm64 --apple_deploy_target 15.4 --cmake_generator Xcode --config RelWithDebInfo --skip_tests --skip_wheel

For a physical iOS device, use --apple_sysroot iphoneos. Cross-compiled targets cannot run native CTest binaries on the build host, so pass --skip_tests explicitly unless you are using the supported Android emulator-test flow.

10. Repository layout

Path Contents
src/ Core C++ implementation (models, engine, generators, search/sampling, KV cache).
src/python/ Python binding (pybind11) + wheel packaging.
src/csharp/ C# binding.
src/java/ Java binding (Gradle).
src/objectivec/ Objective-C binding.
cmake/ CMake modules (options, ORT resolution, packaging, telemetry) and deps.txt (fetched dependencies).
test/ C++ (test/*.cpp) and Python (test/python/) tests; test models in test/models/.
examples/ Runnable samples (C, C#, Python).
.pipelines/ Azure DevOps packaging pipeline (stages, jobs, steps).
build.py Cross-platform build driver.
VERSION_INFO Package version (e.g. 0.15.0-dev).

11. Troubleshooting

  • Dependency fetch errors during CMake configure — dependencies are downloaded via FetchContent (cmake/deps.txt); ensure network access to the declared URLs, then re-run with --update.
  • Missing Python build dependency — run python -m pip install -r requirements-dev.txt.
  • Stale CMake cache after switching flags (e.g. adding --arm64) — delete the build directory (or CMakeCache.txt inside it) and re-run with --update.
  • onnxruntime.lib / ORT not found — pass --ort_home <path> to point at a prebuilt ONNX Runtime, or omit it to let build.py auto-download a matching build.
  • Clean an existing build — python build.py --clean --skip_examples runs the CMake clean target for the selected configuration; it does not rebuild. Run a normal python build.py command afterward, or delete the build directory for a completely fresh configure.