Triton Inferencer Node

Overview

The Triton Inferencer node runs AI inference through NVIDIA Triton Inference Server with TensorRT-optimized models. It uses a custom C++ pipeline backend (rp_pipeline) for detection, segmentation, and classification tasks.

Unlike the standard Inferencer node (which runs a custom Python gRPC server), the Triton Inferencer delegates to the industry-standard Triton server for maximum GPU throughput and TensorRT optimization.

Key Features

NVIDIA Triton backend: Industry-standard inference server with TensorRT optimization
Custom C++ pipeline backend: rp_pipeline for detection, segmentation, classification post-processing
Singleton server: Single Triton container shared across all triton-inferencer nodes
Lazy start: Server starts on first deploy, stops when no nodes remain
LRU model caching: Keep 1-5 models loaded, evict least-recently-used when limit reached
Reference-counted loading: Models load/unload with grace periods to avoid thrashing
Two transfer modes: Standard gRPC or shared memory (SHM) for large images
Firebase model auto-conversion: Download ONNX from Firebase → extract metadata → generate TensorRT config → engine conversion
Searchable model dropdown: Unified UI for local and Firebase models with origin badges
Dynamic model selection: Choose models at runtime via message properties
Promise mode: Asynchronous batch processing

Architecture

Components

Node-RED                    Docker Container
┌──────────────┐            ┌─────────────────────────┐
│  Triton      │── gRPC ──→ │  NVIDIA Triton Server    │
│  Inferencer  │   or SHM   │  ├── rp_pipeline backend │
│  Node        │            │  ├── TensorRT engines     │
│              │←─ results ─│  └── Model repository     │
└──────────────┘            └─────────────────────────┘

Triton Inferencer Node: Configuration, model management, message handling
Triton Server (singleton): Docker container with Triton + custom rp_pipeline backend
gRPC Protocol: NVIDIA Triton's native KServe gRPC API (256MB message limit)
Shared Memory: /dev/shm for zero-copy image transfer on large inputs
Model Repository: /opt/storage/models/triton-model-dir/ directory

Singleton Server Lifecycle

All triton-inferencer nodes share a single Triton server instance:

First node deploys → Container starts, gRPC client connects
Model acquired → Reference count incremented, model loaded in Triton
Model released → 5-second grace period, then unloaded if no references remain
Last node removed → Container shuts down

Model Loading

Models use reference counting with LRU eviction:

Each node holds a reference to its active model
When the LRU cache exceeds maxLoadedModels, the oldest unused model is unloaded
A 5-second grace period before unload prevents thrashing during rapid model switches
Pipeline models track sub-model dependencies (e.g., pipeline wrapper → TensorRT weights)

Configuration

Settings Tab

Name

Type: String
Default: triton-inferencer
Description: Display name for the node

Firebase Connection

Type: Node reference
Required: No
Description: Reference to a firebase-config node. Required for Firebase model source.

Model Source

Type: Select
Options:
- Select here: Choose from local Triton models or Firebase models (searchable dropdown)
- Dynamic: Provide model name at runtime via message/flow/global property

Model Name (Static Mode)

Type: Searchable dropdown
Description: Lists available models from the Triton model repository and Firebase
Display: Shows model name, task type, format, and origin badge (Local/Firebase)

Model Field (Dynamic Mode)

Type: TypedInput (msg, flow, global)
Default: msg.modelName
Description: Property path containing model name at runtime

Transfer Mode

Type: Select
Default: shm
Options:
- Shared Memory (SHM): Zero-copy transfer via /dev/shm — faster for large images
- gRPC: Standard gRPC byte transfer — simpler, no SHM setup needed

Input Field

Type: Message property path
Default: image
Description: Message field containing input image

Output Field

Type: Message property path
Default: inference
Description: Where inference results will be stored

Promise Mode

Type: Checkbox
Default: Disabled
Description: Return promises for async batch processing

Max Concurrent Predictions

Type: Number
Range: 1-20
Default: 5
Description: Maximum parallel inference requests

Max Loaded Models

Type: Number
Range: 1-5
Default: 1
Description: Maximum models kept loaded in Triton simultaneously. Excess models evicted via LRU.

Model Config Tab

Task-specific parameters that control inference behavior. Parameter visibility depends on the selected model's task type and format.

Every parameter starts on the default type, which sends no override tensor: the value baked into the model's own config.pbtxt at install time applies. The editor shows that value beside each field (model: 0.5), resolved from the loaded model on the Triton server, from the installed model's config when the server is stopped, or from the model's inference_config.json when it has not been prepared yet (GET /triton-inferencer/model-defaults/<model>).

Do not read the "Default" rows below as values the node sends. They are the range each knob accepts; what a fresh node uses is whatever the selected model carries — 0.5 for an anomaly model (its learned OK/NOK threshold), 0.6 for a PP-OCR detector, the trained task_specific.confidence for a YOLO model. Typing a value overrides the model, and Restore defaults clears every override.

Precision

Type: Select
Default: FP16
Options: FP16, FP32
Description: TensorRT precision of the model's engine, applied when the engine is built (trtexec --fp16); the ONNX stays FP32. Every build is kept: the downloaded ONNX and each engine built from it live in triton-model-dir/.variants/<model>/ (source/, fp16/, fp32/), so switching to a precision built before is a copy on deploy, and a new one is built from the kept ONNX without downloading. A model installed before this store (or by hand) has no kept ONNX: it is downloaded again from Firebase when the node has a connection, otherwise it keeps its engine and the node status says so. A node saved before this option existed keeps whatever engine is installed until the option is set. Precision is never changed automatically: an FP32 build that runs out of GPU memory fails and says to try FP16. One engine serves every node, so two nodes asking different precisions of the same model is an error on the second. OCR presets always build FP32. In dynamic mode the choice applies when a model is installed, not to models already on disk. Beside the selector the editor shows the installed precision and the stored builds.

Confidence Threshold

Type: TypedInput (number or message property)
Default: the model's own value (no override sent)
Range: 0.0-1.0
Applies to: Detection, Segmentation, Anomaly
Description: Minimum confidence score for predictions. For an anomaly model this is the image-level score that separates OK from NOK, so overriding it with a detection-style 0.25 makes the node disagree with the inferencer node on the same image.

IoU Threshold

Type: TypedInput (number or message property)
Default: the model's own value (no override sent)
Range: 0.0-1.0
Applies to: Detection (YOLO), Segmentation (YOLO)
Description: Non-maximum suppression IoU threshold

Max Detections

Type: TypedInput (number or message property)
Default: the model's own value (no override sent)
Range: 1-10000
Applies to: Detection, Segmentation
Description: Maximum number of detections per image

Mask Threshold

Type: TypedInput (number or message property)
Default: the model's own value (no override sent)
Range: 0.0-1.0
Applies to: Segmentation
Description: Binary threshold for segmentation masks

Contour Epsilon

Type: TypedInput (number or message property)
Default: the model's own value (no override sent)
Range: 0.0-1.0
Applies to: Segmentation
Description: Polygon simplification tolerance for contour approximation

Supported Tasks and Formats

The FORMAT_PARAMS registry controls which parameters are visible for each task/format combination:

Task/Format	Visible Parameters
`detection/yolo`	confidenceThreshold, iouThreshold, maxDetections
`detection/yolo-e2e`	confidenceThreshold, maxDetections
`detection/rfdetr`	confidenceThreshold, maxDetections
`detection/deimv2`	confidenceThreshold, maxDetections
`segmentation/yolo`	confidenceThreshold, iouThreshold, maxDetections, maskThreshold, contourEpsilon
`segmentation/yolo-e2e`	confidenceThreshold, maxDetections, maskThreshold, contourEpsilon
`classification/yolo`	(none — no task-specific parameters)
`anomaly/anomalib`	confidenceThreshold (OK/NOK, 0.5 = trained threshold), maskThreshold, contourEpsilon
`ocr_pipeline/paddle_ocr_pipeline`	confidenceThreshold (DB box score), maxDetections, dbThresh, dbUnclipRatio, dbMaxCandidates, dbMinBoxSize, useDocOrientation, useTextlineOrientation

To add a new format, add one line to the FORMAT_PARAMS object in triton-inferencer.html.

Firebase Model Pipeline

When a Firebase model is selected, the node automatically downloads and converts it for Triton:

Firebase Storage          Triton Server Container          Model Repository
┌─────────────┐           ┌───────────────────────┐        ┌──────────────┐
│ ONNX model  │──download→│ Extract ONNX metadata │──────→ │ config.pbtxt │
│ + metadata  │           │ (input/output shapes) │        │ + TensorRT   │
└─────────────┘           │ Generate configs      │        │   engine     │
                          │ TensorRT conversion   │        └──────────────┘
                          └───────────────────────┘

Pipeline Steps

Download: ONNX model file downloaded from Firebase Storage, streamed to disk. An export above protobuf's 2 GB limit (anomaly models with a large ViT encoder, e.g. SuperADD ViT-H+) comes as model.onnx plus a model.onnx.data weights sidecar that the graph references by relative name; both are downloaded into the same directory so metadata extraction and trtexec find the weights.
Metadata extraction: Ephemeral container reads ONNX tensor shapes and names
Config generation: Two Triton config files generated:
- Weights config (config.pbtxt): TensorRT engine configuration with input/output tensor mappings
- Pipeline config (config.pbtxt): rp_pipeline backend wrapper with task-specific post-processing
TensorRT conversion: Triton converts ONNX to TensorRT engine on first model load
Model ready: Pipeline model available for inference

Config Generation

The system generates format-specific preprocessing parameters:

Format	Preprocessing	Detection Format
`yolo`	Letterbox padding, pad_value=114, normalized [0,1]	`yolo`
`rfdetr` / `rt-detr`	No letterbox, mean/std normalization	`detr` (`detr_class_offset=1`)
`deimv2`	No letterbox, normalized [0,1], no mean/std	`detr` (`detr_class_offset=0`)

Timeout

Model preparation (download + conversion) has a 10-minute timeout. Cancel in-progress operations via the REST endpoint or editor button.

Input

Basic Input

javascript

msg.image = imageBuffer; // JPEG, PNG, or raw bitmap
return msg;

Dynamic Model Selection

javascript

msg.modelName = "my-detection-model";
msg.image = imageBuffer;
return msg;

Image Formats

The node accepts the same image formats as the standard Inferencer:

Rosepetal bitmap: { width, height, data: Buffer, colorSpace: "RGB" }
JPEG/PNG buffers: Standard encoded image buffers
Arrays: Multiple images for batch processing

Output

First Output: Inference Results

The same shape as the inferencer node, from the same code — both run convertResultFromProto and then the shared format expansion in lib/result-conversion.js. Swapping one node for the other changes nothing downstream.

Coordinates are relative to the image, 0–1, and area is the fraction of the image covered. Every result carries every box format derived from the same geometry (raw_boxes, 4points, cwh, xywhr, 2points); segmentation and anomaly results also carry polygons, and the box is the polygons' bounding box. There is nothing to configure and no rasterized mask output.

Detection and segmentation return a list, one entry per instance. Anomaly and classification return a single object, one verdict per image. A batch of images returns a list of those, indexed by input order.

Detection:

javascript

{
  inference: [
    {
      tag: "person", confidence: 0.95, classId: 0,
      raw_boxes: [[0.08, 0.14], [0.25, 0.14], [0.25, 0.37], [0.08, 0.37]],
      "2points": { xmin: 0.08, ymin: 0.14, xmax: 0.25, ymax: 0.37 },
      color: { r: 255, g: 0, b: 0 }
      // ...plus cwh, xywhr and 4points
    }
  ]
}

Segmentation:

javascript

{
  inference: [
    {
      tag: "person", confidence: 0.92, classId: 0, area: 0.35,
      polygons: [[[0.1, 0.2], [0.5, 0.2], [0.5, 0.9], [0.1, 0.9]]],
      "2points": { xmin: 0.1, ymin: 0.2, xmax: 0.5, ymax: 0.9 },
      color: { r: 0, g: 255, b: 0 }
    }
  ]
}

Classification:

javascript

{
  inference: { tag: "cat", classId: 281, confidence: 0.98, all_tags: [...] }
}

OCR (PP-OCR presets; coords are normalised to the original image, one entry per detected text line in reading order). The det stage runs at the image's native resolution like the ocr-inferencer (PaddleX limit_side_len=64/min), capped at the engine's 1280 px profile. With Document Orientation on, the page is classified (0/90/180/270), read upright and the coordinates are mapped back to the original frame (metadata.document_angle); with Textline Orientation on, upside-down lines are flipped before recognition (metadata.orientation_angles, 0 = upright / 1 = flipped per line):

javascript

{
  ocr: {
    texts: [ { coords: [{x, y}, {x, y}, {x, y}, {x, y}], text: "L24980 01/06/27", score: 0.97 }, ... ],
    text: "L24980 01/06/27\nEXP 12/2027", text_concatenated: "...",
    text_largest_box: "...", text_highest_score: "...", text_most_centric: "...",
    metadata: { orientation_angles: [0, 0], document_angle: 180 }
  }
}

Anomaly — one object per image, not a list. confidence is the normalised anomaly score, where 0.5 is the trained threshold (an anomalib export is min-max normalised, so 0.5 is what the model learned). polygons outline the anomalous pixels and area is the fraction of the image they cover; both come from thresholding the anomaly map, not from the verdict, so an OK image can still carry a mask:

javascript

{
  inference: {
    confidence: 0.87, tag: "NOK", classId: 1, area: 0.41,
    polygons: [[[0.098, 0.465], [0.091, 0.473], ...]],
    "2points": { xmin: 0.09, ymin: 0.11, xmax: 0.82, ymax: 0.94 },
    color: { r: 224, g: 170, b: 2 }
  }
}

Changed. Anomaly used to come back as a list of one, with polygons and area in pixels of the original image here and in pixels of the model's mask in the inferencer node — the same model, two different answers. Both are relative now. The Triton side of this needs a backend image built after the change; the inferencer side needs a serving image with the matching rosepetal-commons.

REST Admin Endpoints

HTTP endpoints registered at the Node-RED admin API:

Method	Endpoint	Description
GET	`/triton-inferencer/models`	List all models in Triton repository
GET	`/triton-inferencer/model-config/:modelName`	Get Triton config for a specific model
GET	`/triton-inferencer/pipeline-models`	List `rp_pipeline` models with task/format
GET	`/triton-inferencer/firebase-models?firebaseNodeId=<id>`	List available Firebase models
POST	`/triton-inferencer/cancel-prepare/:modelId`	Cancel in-progress model preparation
POST	`/triton-inferencer/restart?nodeId=<id>`	Restart Triton server

Status Indicators

Status	Meaning
`Starting Triton...`	Docker container starting
`Pulling image...`	Docker image being downloaded
`Downloading <model>...`	Firebase model download in progress
`Converting <model>...`	TensorRT engine conversion in progress
`Installing <model>...`	Model being loaded into Triton
`Ready (<model>) [SHM]`	Model loaded, using shared memory transfer
`Ready (<model>) [gRPC]`	Model loaded, using gRPC transfer
`Ready (dynamic) [SHM/gRPC]`	Dynamic mode, waiting for model name
`No model configured`	Static mode with no model selected
`Error: <message>`	Error state with description

Performance Optimization

Transfer Mode Selection

Mode	Best For	Trade-off
SHM	Large images, high throughput	Requires `/dev/shm` mounted, shared memory pool
gRPC	Small images, simpler setup	Higher latency for large payloads

SHM uses a pool of shared memory regions (one per concurrent prediction). The first SHM inference logs a performance message for monitoring.

Model Caching

Set maxLoadedModels based on available GPU memory
Models stay loaded between requests for instant switching
LRU eviction keeps GPU memory bounded
5-second grace period prevents load/unload thrashing

Concurrent Predictions

maxConcurrentPredictions controls parallelism (default: 5, max: 20)
Excess requests queue until a slot is available
Higher values improve throughput but increase GPU memory usage

Troubleshooting

Docker Issues

"Triton container failed to start"

Cause: Docker not running, insufficient GPU resources, image not available
Solution: Check Docker status, verify NVIDIA drivers, pull image manually

"Pulling image..." takes too long

Cause: First-time pull of the Triton Docker image (~10GB)
Solution: Pre-pull the image: docker pull <image>

GPU Issues

"No NVIDIA GPU detected"

Cause: Missing NVIDIA drivers or Docker GPU runtime
Solution: Install nvidia-container-toolkit, verify with nvidia-smi

Model Issues

"Model has no RAW_IMAGE input"

Cause: Model config missing expected input tensor
Solution: Verify model's config.pbtxt defines RAW_IMAGE input

"Model preparation timed out"

Cause: TensorRT conversion exceeding 10-minute timeout
Solution: Pre-convert models, or increase timeout in constants

"TensorRT conversion failed"

Cause: Incompatible ONNX model or unsupported operations
Solution: Verify ONNX model compatibility with TensorRT version

Connection Issues

"gRPC connection failed"

Cause: Triton server not ready or port conflict
Solution: Wait for startup (30s timeout), check container logs

Adding New Model Formats

Recommended: the model declares its recipe in its config.json (common.preprocess / common.postprocess, written by rpdet). With the primitives the backend already has, the node needs no change at all: no FORMAT_PARAMS line, no model_format branch. RP-OD works this way. Details in ADD-NEW-MODELS.md.

For a model that predates the blocks:

Register format in FORMAT_PARAMS (in triton-inferencer.html):
javascript
```
'detection/myformat': ['confidenceThreshold', 'maxDetections']
```
1

Set model_format parameter in the model's config.pbtxt:

parameters { key: "model_format" value: { string_value: "myformat" } }

Add pipeline template (optional) to engine/pipeline_templates/ in the Triton server
Update Firebase config generation (optional) in firebase-model-manager.js for auto-conversion support

See ADD-NEW-MODELS.md in the triton-inferencer directory for detailed instructions.

Requirements

NVIDIA GPU: CUDA-capable GPU with drivers installed
Docker: With NVIDIA Container Toolkit (nvidia-container-toolkit)
Shared Memory: /dev/shm accessible (for SHM transfer mode)
Disk Space: ~10GB for Triton Docker image + model storage

Triton Inferencer Node ​

Overview ​

Key Features ​

Architecture ​

Components ​

Singleton Server Lifecycle ​

Model Loading ​

Configuration ​

Settings Tab ​

Name ​

Firebase Connection ​

Model Source ​

Model Name (Static Mode) ​

Model Field (Dynamic Mode) ​

Transfer Mode ​

Input Field ​

Output Field ​

Promise Mode ​

Max Concurrent Predictions ​

Max Loaded Models ​

Model Config Tab ​

Precision ​

Confidence Threshold ​

IoU Threshold ​

Max Detections ​

Mask Threshold ​

Contour Epsilon ​

Supported Tasks and Formats ​

Firebase Model Pipeline ​

Pipeline Steps ​

Config Generation ​

Timeout ​

Input ​

Basic Input ​

Dynamic Model Selection ​

Image Formats ​

Output ​

First Output: Inference Results ​

REST Admin Endpoints ​

Status Indicators ​

Performance Optimization ​

Transfer Mode Selection ​

Model Caching ​

Concurrent Predictions ​

Troubleshooting ​

Docker Issues ​

"Triton container failed to start" ​

"Pulling image..." takes too long ​

GPU Issues ​

"No NVIDIA GPU detected" ​

Model Issues ​

"Model has no RAW_IMAGE input" ​

"Model preparation timed out" ​

"TensorRT conversion failed" ​

Connection Issues ​

"gRPC connection failed" ​

Adding New Model Formats ​

Requirements ​

See Also ​

Triton Inferencer Node

Overview

Key Features

Architecture

Components

Singleton Server Lifecycle

Model Loading

Configuration

Settings Tab

Name

Firebase Connection

Model Source

Model Name (Static Mode)

Model Field (Dynamic Mode)

Transfer Mode

Input Field

Output Field

Promise Mode

Max Concurrent Predictions

Max Loaded Models

Model Config Tab

Precision

Confidence Threshold

IoU Threshold

Max Detections

Mask Threshold

Contour Epsilon

Supported Tasks and Formats

Firebase Model Pipeline

Pipeline Steps

Config Generation

Timeout

Input

Basic Input

Dynamic Model Selection

Image Formats

Output

First Output: Inference Results

REST Admin Endpoints

Status Indicators

Performance Optimization

Transfer Mode Selection

Model Caching

Concurrent Predictions

Troubleshooting

Docker Issues

"Triton container failed to start"

"Pulling image..." takes too long

GPU Issues

"No NVIDIA GPU detected"

Model Issues

"Model has no RAW_IMAGE input"

"Model preparation timed out"

"TensorRT conversion failed"

Connection Issues

"gRPC connection failed"

Adding New Model Formats

Requirements

See Also