Skip to main content
Version: 0.10.x [Latest Beta]

Troubleshooting

When reporting a problem, always include:

  1. your OS and architecture
  2. your SDK language and version (Python version, .NET runtime, or C compiler)
  3. example files with which the problem can be reproduced
  4. a minimal reproduction script or project
  5. your hardware details
  6. your execution provider or export target
  7. the full initialization error or stack trace

Enable SDK logs​

import denkflow
denkflow.set_log_level("DEBUG")

Use TRACE only when you need maximum detail.

Enable ONNX runtime logs​

export DENKFLOW_ENABLE_ORT_LOGS=true

Common problems​

Provider not available​

Symptoms:

  • Initialization fails immediately
  • The expected execution provider is missing

Check:

  1. that the denkflow package is installed;
  2. the complete DENKflow pipeline-initialization error and SDK logs;
  3. whether the required host driver is available, for example nvidia-smi for NVIDIA or clinfo -l for an Intel GPU;
  4. whether <data dir>/dependencies/onnxruntime/ contains the managed ONNX Runtime 1.22.1 and the selected provider dependency is either system-resolved or present under <data dir>/dependencies/;
  5. that the data directory is writable and the PAT can download missing dependencies, or the matching offline .denkdependency archives are present.

Do not diagnose this with import onnxruntime: DENKflow manages the native runtime and does not install the ONNX Runtime Python package.

Ambarella pipeline does not initialize​

Ambarella models are compiled for one specific chip and runtime version. Check:

  1. the Hub export target matches the physical chip;
  2. /dev/cavalry exists and is accessible to the application user;
  3. the board vendor's Cavalry driver and firmware are loaded;
  4. the PAT can download runtime dependencies, or the matching offline .denkdependency archive is present;
  5. DENKFLOW_DATA_DIRECTORY is writable and persistent.

DENKflow does not fall back to CPU when an Ambarella model or runtime is incompatible. See Ambarella deployment for the full target mapping and first-run setup.

TensorRT is slow on first run​

This is expected. TensorRT builds an optimized engine on the first run of a new model. Persist the SDK data directory (by default $HOME/.config/denkflow on Linux, e.g. mount a host volume at /root/.config/denkflow in Docker) so later runs can reuse the cache.

Offline license stops working after container restart​

Usually one of these is missing:

  • persistent storage for the SDK data directory (default path on Linux as root: /root/.config/denkflow, or a host mount at whatever path DENKFLOW_DATA_DIRECTORY points to)
  • stable /etc/machine-id
  • reused container volume

OpenVINO runs on the wrong device​

Check the device_id mapping. See Runtime and Device Selection > Runtime specific notes.

OpenVINO fails with "Failed to load shared library"​

This error from ONNX Runtime usually means one of two things:

1. Managed runtime files are unavailable​

DENKflow manages ONNX Runtime 1.22.1 under <data dir>/dependencies/onnxruntime/ and resolves OpenVINO separately from a compatible system install or <data dir>/dependencies/openvino/. Check that the first-use download completed, the manifest verification succeeded, and the data directory is writable. On an offline host, supply both required .denkdependency archives.

You should not need to add managed provider directories to LD_LIBRARY_PATH; DENKflow activates them before creating the session.

2. OpenVINO GPU plugin cannot find the OpenCL runtime​

When using OpenVINO with an Intel GPU (device_id >= 0), the OpenVINO GPU plugin requires:

  • libOpenCL.so.1 — the OpenCL ICD loader
  • Intel compute-runtime — the Intel GPU OpenCL driver, registered via an ICD file

On Ubuntu / Debian:

sudo apt install intel-opencl-icd

The intel-opencl-icd package depends on ocl-icd-libopencl1, so apt installs the OpenCL ICD loader automatically.

On Fedora / RHEL:

sudo dnf install ocl-icd intel-opencl

On NixOS, these are available but not always on the default library path. Set the environment manually:

# Point to the OpenCL ICD loader
export LD_LIBRARY_PATH="/path/to/ocl-icd/lib:$LD_LIBRARY_PATH"

# Tell the ICD loader where to find the Intel GPU driver
export OCL_ICD_VENDORS=/path/to/intel-compute-runtime/etc/OpenCL/vendors

On Windows, the Intel GPU driver installation includes OpenCL support — no extra steps are needed.

To verify OpenCL device visibility:

# Install clinfo if needed, then:
clinfo -l

If no Intel GPU appears, the compute-runtime driver is missing or OCL_ICD_VENDORS is not set correctly.

DirectML QDQ or Intel export does not behave like FP32​

That is expected. Intel export targets (INTEL_CPU, INTEL_GPU, INTEL_NPU) and DirectML QDQ exports include INT8 quantization by design. The SDK does not convert FP32 to QDQ locally during inference.

ONNX Runtime cannot be resolved​

For a standard installation, check:

  1. <data dir>/dependencies/onnxruntime/ exists and is readable;
  2. the data directory and dependencies/manifest.db are writable;
  3. the PAT can reach the Hub on first use, or the ONNX Runtime 1.22.1 .denkdependency archive is present in <data dir>/dependencies/ or the dependency import directory (DENKFLOW_DEPENDENCY_IMPORT_DIRECTORY, which defaults to the process working directory);
  4. DENKFLOW_NONINTERACTIVE=1 is set for unattended startup.

ORT_DYLIB_PATH is not a normal installation step. Use it only when intentionally testing an advanced custom ONNX Runtime build; it can make the managed provider versions incompatible.

Benchmark numbers look worse than expected​

Common reasons:

  • measuring the first TensorRT run instead of steady state
  • not pinning the deployment to the actual target hardware
  • missing GPU driver or runtime dependency, causing CPU fallback
  • not using the export target intended for that hardware