Appearance

Triton Inferencer Node ​

Overview ​

The Triton Inferencer node runs AI inference through NVIDIA Triton Inference Server with TensorRT-optimized models. It uses a custom C++ pipeline backend (rp_pipeline) for detection, segmentation, and classification tasks.

Unlike the standard Inferencer node (which runs a custom Python gRPC server), the Triton Inferencer delegates to the industry-standard Triton server for maximum GPU throughput and TensorRT optimization.

Key Features ​

  • NVIDIA Triton backend: Industry-standard inference server with TensorRT optimization
  • Custom C++ pipeline backend: rp_pipeline for detection, segmentation, classification post-processing
  • Singleton server: Single Triton container shared across all triton-inferencer nodes
  • Lazy start: Server starts on first deploy, stops when no nodes remain
  • LRU model caching: Keep 1-5 models loaded, evict least-recently-used when limit reached
  • Reference-counted loading: Models load/unload with grace periods to avoid thrashing
  • Two transfer modes: Standard gRPC or shared memory (SHM) for large images
  • Firebase model auto-conversion: Download ONNX from Firebase → extract metadata → generate TensorRT config → engine conversion
  • Searchable model dropdown: Unified UI for local and Firebase models with origin badges
  • Dynamic model selection: Choose models at runtime via message properties
  • Promise mode: Asynchronous batch processing
  • Lifecycle status output: Second output emits status messages on state changes

Architecture ​

Components ​

Node-RED                    Docker Container
┌──────────────┐            ┌─────────────────────────┐
│  Triton      │── gRPC ──→ │  NVIDIA Triton Server    │
│  Inferencer  │   or SHM   │  ├── rp_pipeline backend │
│  Node        │            │  ├── TensorRT engines     │
│              │←─ results ─│  └── Model repository     │
└──────────────┘            └─────────────────────────┘
  1. Triton Inferencer Node: Configuration, model management, message handling
  2. Triton Server (singleton): Docker container with Triton + custom rp_pipeline backend
  3. gRPC Protocol: NVIDIA Triton's native KServe gRPC API (256MB message limit)
  4. Shared Memory: /dev/shm for zero-copy image transfer on large inputs
  5. Model Repository: /opt/storage/models/triton-model-dir/ directory

Singleton Server Lifecycle ​

All triton-inferencer nodes share a single Triton server instance:

  1. First node deploys → Container starts, gRPC client connects
  2. Model acquired → Reference count incremented, model loaded in Triton
  3. Model released → 5-second grace period, then unloaded if no references remain
  4. Last node removed → Container shuts down

Model Loading ​

Models use reference counting with LRU eviction:

  • Each node holds a reference to its active model
  • When the LRU cache exceeds maxLoadedModels, the oldest unused model is unloaded
  • A 5-second grace period before unload prevents thrashing during rapid model switches
  • Pipeline models track sub-model dependencies (e.g., pipeline wrapper → TensorRT weights)

Configuration ​

Settings Tab ​

Name ​

  • Type: String
  • Default: triton-inferencer
  • Description: Display name for the node

Firebase Connection ​

  • Type: Node reference
  • Required: No
  • Description: Reference to a firebase-config node. Required for Firebase model source.

Model Source ​

  • Type: Select
  • Options:
    • Select here: Choose from local Triton models or Firebase models (searchable dropdown)
    • Dynamic: Provide model name at runtime via message/flow/global property

Model Name (Static Mode) ​

  • Type: Searchable dropdown
  • Description: Lists available models from the Triton model repository and Firebase
  • Display: Shows model name, task type, format, and origin badge (Local/Firebase)

Model Field (Dynamic Mode) ​

  • Type: TypedInput (msg, flow, global)
  • Default: msg.modelName
  • Description: Property path containing model name at runtime

Transfer Mode ​

  • Type: Select
  • Default: shm
  • Options:
    • Shared Memory (SHM): Zero-copy transfer via /dev/shm — faster for large images
    • gRPC: Standard gRPC byte transfer — simpler, no SHM setup needed

Input Field ​

  • Type: Message property path
  • Default: image
  • Description: Message field containing input image

Output Field ​

  • Type: Message property path
  • Default: inference
  • Description: Where inference results will be stored

Promise Mode ​

  • Type: Checkbox
  • Default: Disabled
  • Description: Return promises for async batch processing

Max Concurrent Predictions ​

  • Type: Number
  • Range: 1-20
  • Default: 5
  • Description: Maximum parallel inference requests

Max Loaded Models ​

  • Type: Number
  • Range: 1-5
  • Default: 1
  • Description: Maximum models kept loaded in Triton simultaneously. Excess models evicted via LRU.

Model Config Tab ​

Task-specific parameters that control inference behavior. Parameter visibility depends on the selected model's task type and format.

Confidence Threshold ​

  • Type: TypedInput (number or message property)
  • Default: 0.25
  • Range: 0.0-1.0
  • Applies to: Detection, Segmentation
  • Description: Minimum confidence score for predictions

IoU Threshold ​

  • Type: TypedInput (number or message property)
  • Default: 0.5
  • Range: 0.0-1.0
  • Applies to: Detection (YOLO), Segmentation (YOLO)
  • Description: Non-maximum suppression IoU threshold

Max Detections ​

  • Type: TypedInput (number or message property)
  • Default: 100
  • Range: 1-10000
  • Applies to: Detection, Segmentation
  • Description: Maximum number of detections per image

Mask Threshold ​

  • Type: TypedInput (number or message property)
  • Default: 0.5
  • Range: 0.0-1.0
  • Applies to: Segmentation
  • Description: Binary threshold for segmentation masks

Contour Epsilon ​

  • Type: TypedInput (number or message property)
  • Default: 0.005
  • Range: 0.0-1.0
  • Applies to: Segmentation
  • Description: Polygon simplification tolerance for contour approximation

Supported Tasks and Formats ​

The FORMAT_PARAMS registry controls which parameters are visible for each task/format combination:

Task/FormatVisible Parameters
detection/yoloconfidenceThreshold, iouThreshold, maxDetections
detection/yolo-e2econfidenceThreshold, maxDetections
detection/rfdetrconfidenceThreshold, maxDetections
segmentation/yoloconfidenceThreshold, iouThreshold, maxDetections, maskThreshold, contourEpsilon
segmentation/yolo-e2econfidenceThreshold, maxDetections, maskThreshold, contourEpsilon
classification/yolo(none — no task-specific parameters)

To add a new format, add one line to the FORMAT_PARAMS object in triton-inferencer.html.

Firebase Model Pipeline ​

When a Firebase model is selected, the node automatically downloads and converts it for Triton:

Firebase Storage          Triton Server Container          Model Repository
┌─────────────┐           ┌───────────────────────┐        ┌──────────────┐
│ ONNX model  │──download→│ Extract ONNX metadata │──────→ │ config.pbtxt │
│ + metadata  │           │ (input/output shapes) │        │ + TensorRT   │
└─────────────┘           │ Generate configs      │        │   engine     │
                          │ TensorRT conversion   │        └──────────────┘
                          └───────────────────────┘

Pipeline Steps ​

  1. Download: ONNX model file downloaded from Firebase Storage
  2. Metadata extraction: Ephemeral container reads ONNX tensor shapes and names
  3. Config generation: Two Triton config files generated:
    • Weights config (config.pbtxt): TensorRT engine configuration with input/output tensor mappings
    • Pipeline config (config.pbtxt): rp_pipeline backend wrapper with task-specific post-processing
  4. TensorRT conversion: Triton converts ONNX to TensorRT engine on first model load
  5. Model ready: Pipeline model available for inference

Config Generation ​

The system generates format-specific preprocessing parameters:

FormatPreprocessingDetection Format
yoloLetterbox padding, pad_value=114, normalized [0,1]yolo
rt-detrNo letterbox, mean/std normalizationrtdetr

Timeout ​

Model preparation (download + conversion) has a 10-minute timeout. Cancel in-progress operations via the REST endpoint or editor button.

Input ​

Basic Input ​

javascript
msg.image = imageBuffer; // JPEG, PNG, or raw bitmap
return msg;

Dynamic Model Selection ​

javascript
msg.modelName = "my-detection-model";
msg.image = imageBuffer;
return msg;

Image Formats ​

The node accepts the same image formats as the standard Inferencer:

  • Rosepetal bitmap: { width, height, data: Buffer, colorSpace: "RGB" }
  • JPEG/PNG buffers: Standard encoded image buffers
  • Arrays: Multiple images for batch processing

Output ​

First Output: Inference Results ​

Results use the same format as the standard Inferencer node (shared convertResultFromProto):

Detection:

javascript
{
  inference: [
    {
      box: { xyxy: [100, 150, 300, 400], confidence: 0.95 },
      class: "person",
      class_id: 0
    }
  ]
}

Segmentation:

javascript
{
  inference: [
    {
      box: { xyxy: [100, 150, 300, 400] },
      class: "person",
      mask: { rle: "...", polygon: [...], bitmap: Buffer }
    }
  ]
}

Classification:

javascript
{
  inference: [
    { class: "cat", class_id: 281, confidence: 0.98 }
  ]
}

Second Output: Lifecycle Status ​

Status messages emitted on state changes:

javascript
{
  topic: "lifecycle",
  status: "ready",          // or "starting", "downloading", "converting", "error"
  timestamp: 1711382400000,
  message: "Ready (my-model) [SHM]"
}

REST Admin Endpoints ​

HTTP endpoints registered at the Node-RED admin API:

MethodEndpointDescription
GET/triton-inferencer/modelsList all models in Triton repository
GET/triton-inferencer/model-config/:modelNameGet Triton config for a specific model
GET/triton-inferencer/pipeline-modelsList rp_pipeline models with task/format
GET/triton-inferencer/firebase-models?firebaseNodeId=<id>List available Firebase models
POST/triton-inferencer/cancel-prepare/:modelIdCancel in-progress model preparation
POST/triton-inferencer/restart?nodeId=<id>Restart Triton server

Status Indicators ​

StatusMeaning
Starting Triton...Docker container starting
Pulling image...Docker image being downloaded
Downloading <model>...Firebase model download in progress
Converting <model>...TensorRT engine conversion in progress
Installing <model>...Model being loaded into Triton
Ready (<model>) [SHM]Model loaded, using shared memory transfer
Ready (<model>) [gRPC]Model loaded, using gRPC transfer
Ready (dynamic) [SHM/gRPC]Dynamic mode, waiting for model name
No model configuredStatic mode with no model selected
Error: <message>Error state with description

Performance Optimization ​

Transfer Mode Selection ​

ModeBest ForTrade-off
SHMLarge images, high throughputRequires /dev/shm mounted, shared memory pool
gRPCSmall images, simpler setupHigher latency for large payloads

SHM uses a pool of shared memory regions (one per concurrent prediction). The first SHM inference logs a performance message for monitoring.

Model Caching ​

  • Set maxLoadedModels based on available GPU memory
  • Models stay loaded between requests for instant switching
  • LRU eviction keeps GPU memory bounded
  • 5-second grace period prevents load/unload thrashing

Concurrent Predictions ​

  • maxConcurrentPredictions controls parallelism (default: 5, max: 20)
  • Excess requests queue until a slot is available
  • Higher values improve throughput but increase GPU memory usage

Troubleshooting ​

Docker Issues ​

"Triton container failed to start" ​

  • Cause: Docker not running, insufficient GPU resources, image not available
  • Solution: Check Docker status, verify NVIDIA drivers, pull image manually

"Pulling image..." takes too long ​

  • Cause: First-time pull of the Triton Docker image (~10GB)
  • Solution: Pre-pull the image: docker pull <image>

GPU Issues ​

"No NVIDIA GPU detected" ​

  • Cause: Missing NVIDIA drivers or Docker GPU runtime
  • Solution: Install nvidia-container-toolkit, verify with nvidia-smi

Model Issues ​

"Model has no RAW_IMAGE input" ​

  • Cause: Model config missing expected input tensor
  • Solution: Verify model's config.pbtxt defines RAW_IMAGE input

"Model preparation timed out" ​

  • Cause: TensorRT conversion exceeding 10-minute timeout
  • Solution: Pre-convert models, or increase timeout in constants

"TensorRT conversion failed" ​

  • Cause: Incompatible ONNX model or unsupported operations
  • Solution: Verify ONNX model compatibility with TensorRT version

Connection Issues ​

"gRPC connection failed" ​

  • Cause: Triton server not ready or port conflict
  • Solution: Wait for startup (30s timeout), check container logs

Adding New Model Formats ​

To support a new model format:

  1. Register format in FORMAT_PARAMS (in triton-inferencer.html):

    javascript
    'detection/myformat': ['confidenceThreshold', 'maxDetections']

  2. Set model_format parameter in the model's config.pbtxt:

    parameters { key: "model_format" value: { string_value: "myformat" } }

  3. Add pipeline template (optional) to engine/pipeline_templates/ in the Triton server

  4. Update Firebase config generation (optional) in firebase-model-manager.js for auto-conversion support

See ADD-NEW-MODELS.md in the triton-inferencer directory for detailed instructions.

Requirements ​

  • NVIDIA GPU: CUDA-capable GPU with drivers installed
  • Docker: With NVIDIA Container Toolkit (nvidia-container-toolkit)
  • Shared Memory: /dev/shm accessible (for SHM transfer mode)
  • Disk Space: ~10GB for Triton Docker image + model storage

See Also ​