Appearance

Inferencer Node ​

Overview ​

The Inferencer node provides AI-powered computer vision inference by running machine learning models in Docker containers. It acts as a bridge between Node-RED flows and Python-based ML models, supporting detection, classification, segmentation, anomaly detection, and OCR tasks.

The node uses a modular 7-stage inference pipeline and communicates with the inference server via gRPC.

Key Features ​

  • Multiple ML frameworks: YOLO, PaddlePaddle, RF-DETR, TensorRT, Anomalib
  • Task support: Object detection, image classification, instance/semantic segmentation, anomaly detection, OCR, recognition
  • Docker-based execution: Isolated Python containers with automatic lifecycle management
  • Dynamic worker scaling: Scale inference workers without restarting containers
  • Dynamic models: Load models at runtime or configure statically
  • Always-on warmup: Server-side synthetic image generation for zero cold-start latency
  • Firebase integration: Automatic model download from Firebase storage with unified model dropdown
  • Promise mode: Asynchronous processing for high-throughput batch workflows
  • Config prefabs: 13 pre-built model type configurations for quick setup
  • Output customization: Configure detection boxes, masks, and result formats
  • Class mapping: Rename or filter model output classes
  • Debug visualization: Display processed images on Node-RED canvas
  • Performance tracking: Built-in metrics at msg.performance.inference.{node_name}

Architecture ​

Inference Pipeline ​

The node processes each request through a 7-stage modular pipeline:

validate-request → resolve-model → ensure-model → process-images → execute-inference → convert-results → assemble-response
  1. validate-request: Validates input message structure and image format
  2. resolve-model: Determines target model (static config or dynamic from message)
  3. ensure-model: Downloads model from Firebase if needed, verifies model availability
  4. process-images: Reads and preprocesses images (format conversion, color space)
  5. execute-inference: Sends gRPC request to inference server, handles concurrency
  6. convert-results: Transforms raw protobuf results into output format
  7. assemble-response: Builds final message with results and performance metrics

Components ​

  1. Node-RED Node: Configuration, message routing, and pipeline orchestration
  2. Docker Containers: Python inference servers (1-10 worker instances)
  3. gRPC Protocol: Communication between Node-RED and containers
  4. Model Storage: /opt/storage/models/ directory
  5. Firebase: Optional model download source

Data Flow ​

Input Images → Pipeline → gRPC → Docker Container → ML Model → Predictions → Output
                                       ↓
                              Dynamic Worker Scaling

Supported Model Types ​

FrameworkTaskConfig Prefab
YOLODetectionyolo_detection
YOLOClassificationyolo_classification
YOLOSegmentationyolo_segmentation
PaddlePaddleDetectionpaddle_detection
PaddlePaddleOCRpaddle_ocr
PaddlePaddleRecognitionpaddle_recognition
PaddlePaddleDocument Rotationpaddle_document_rotation
PipelineOCRpipeline_ocr
RF-DETRDetectionrfdetr_detection
TensorRTClassificationtensorrt_classification
TensorRTDetectiontensorrt_detection
TensorRTSegmentationtensorrt_segmentation
AnomalibAnomalyanomaly_anomaly

Configuration ​

Settings Tab ​

Name ​

  • Type: String
  • Optional: Yes
  • Description: Display name for the node

Input Field ​

  • Type: Message property path
  • Default: payload
  • Description: Message field containing input images (single image or array)

Output Field ​

  • Type: Message property path
  • Default: payload
  • Description: Where inference results will be stored
  • Note: Performance stats saved to msg.performance.inference.{node_name}

Model Source ​

  • Type: Select
  • Options:
    • Select here: Choose from available models in /opt/storage/models or from Firebase (unified dropdown)
    • Dynamic: Provide model name at runtime via message/flow/global property

Model Name (Static Mode) ​

  • Type: Dropdown (unified Firebase + local models)
  • Description: Select from available models — shows model name, task type, framework, and source (local/Firebase)

Model Field (Dynamic Mode) ​

  • Type: TypedInput (msg, flow, global)
  • Example: msg.modelName, flow.currentModel, global.activeModel
  • Description: Property path containing model name at runtime
  • Note: Enables automatic model download if Firebase is configured

Promise Mode ​

  • Type: Checkbox
  • Default: Disabled
  • Description: Return promises instead of waiting for results
  • Use Case: High-throughput batch processing — resolve promises in a downstream function node using Promise.all()

Promises Field ​

  • Type: Message property path
  • Default: promises
  • Visible: When Promise Mode enabled
  • Description: Array field to store pending promises

Number of Workers ​

  • Type: Number
  • Range: 1-10
  • Default: 1
  • Description: Inference worker instances for load balancing
  • Note: Workers scale dynamically without container restart (since v1.2.1)

Maximum Concurrent Predictions ​

  • Type: Number
  • Range: 1-20
  • Default: 5
  • Description: Parallel requests per server
  • Behavior: Excess requests queue until slot available

Show Debug Image ​

  • Type: Checkbox
  • Default: Disabled
  • Description: Display processed images on Node-RED canvas

Debug Interval ​

  • Type: Number
  • Default: 1
  • Visible: When debug enabled
  • Description: Show every Nth image (1 = all images)

Debug Image Width ​

  • Type: Number (pixels)
  • Default: 200
  • Visible: When debug enabled
  • Description: Display width for debug images

JSON Config Tab ​

Advanced model configuration in JSON format. Structure varies by model type. Each model type has a corresponding config prefab that provides sensible defaults.

Common Fields (All Models) ​

json
{
  "common": {
    "model_name": "my-model",
    "config_type": "predict",
    "task": "Detection",
    "device": "auto",
    "image_shape": {
      "width": 640,
      "height": 640
    },
    "verbose": false,
    "max_batch": 100
  }
}

Fields:

  • model_name: Model identifier
  • config_type: Always "predict"
  • task: Detection, Classification, Segmentation, Anomaly, OCR, Recognition
  • device: "auto", "cpu", "cuda", "mps"
  • image_shape: Target dimensions for resizing
  • verbose: Enable detailed logging
  • max_batch: Maximum batch size

Detection Models (YOLO, RF-DETR, PaddlePaddle, TensorRT) ​

json
{
  "common": { },
  "task_specific": {
    "conf_threshold": 0.5,
    "nms_iou_threshold": 0.7,
    "max_det": 300
  }
}

Classification Models ​

json
{
  "common": { },
  "task_specific": {
    "top_k": 5
  }
}

Segmentation Models (YOLO, TensorRT) ​

json
{
  "common": { },
  "task_specific": {
    "conf_threshold": 0.5,
    "retina_masks": true
  }
}

Anomaly Detection ​

json
{
  "common": { },
  "task_specific": {
    "threshold": 0.5,
    "normalize": true
  },
  "model_specific": {
    "optional": {
      "image_threshold": 0.5,
      "pixel_threshold": 0.5
    }
  }
}

TensorRT Batch Configuration ​

TensorRT models support batch size tuning via model_specific:

json
{
  "model_specific": {
    "optional": {
      "half": true,
      "min_batch": 1,
      "opt_batch": 4,
      "max_batch": 16
    }
  }
}

Constraint: min_batch <= opt_batch <= max_batch

Output Formats Tab ​

Configure detection and segmentation output formats.

Detection Output Formats ​

  • boxes_xyxy: Bounding boxes [x1, y1, x2, y2] (top-left, bottom-right)
  • boxes_xywh: Bounding boxes [x_center, y_center, width, height]
  • boxes_tlwh: Bounding boxes [x_top_left, y_top_left, width, height]
  • boxes_corners: Four corners [[x1,y1], [x2,y2], [x3,y3], [x4,y4]]

Segmentation Output Formats ​

  • masks_rle: Run-length encoding (compact format)
  • masks_polygon: Polygon contours
  • masks_bitmap: Binary pixel masks

Classes Tab ​

Map or filter model output classes.

Features:

  • Rename classes to custom labels
  • Filter results by remapping to empty string
  • Per-model configuration storage

Example:

Original: "person" → Custom: "human"
Original: "car" → Custom: "vehicle"
Original: "background" → Custom: "" (filtered out)

Image Format ​

The node accepts multiple image formats:

1. Rosepetal Bitmap Format ​

javascript
{
  width: 1920,
  height: 1080,
  data: Buffer,           // Raw pixel data
  colorSpace: "RGB",      // "GRAY", "RGB", "RGBA", "BGR", "BGRA"
  channels: 3,            // Auto-inferred if omitted
  dtype: "uint8"          // Currently only uint8 supported
}

Color space mapping:

  • GRAY: 1 channel
  • RGB / BGR: 3 channels
  • RGBA / BGRA: 4 channels

2. JPEG/PNG Buffers ​

javascript
msg.payload = fs.readFileSync('image.jpg');

The node automatically decodes standard image formats.

3. Array of Images ​

javascript
msg.payload = [image1, image2, image3];

Processes multiple images in batch for improved performance.

Input ​

Basic Input ​

javascript
msg.payload = imageBuffer;
return msg;

Dynamic Model Selection ​

javascript
// Configure node with Model Source: Dynamic, Model Field: msg.modelName
msg.modelName = "yolo-detection-v8";
msg.payload = imageBuffer;
return msg;

Warmup Request ​

javascript
msg.warmup = true;
return msg;

Triggers model warmup. The server generates a synthetic image internally — no need to provide one.

Inferencer Request (Control Messages) ​

javascript
// Load a specific model
msg.inferencerRequest = { type: "loadModel", modelName: "my-model" };

// Unload the current model
msg.inferencerRequest = { type: "unloadModel" };

// Trigger warmup
msg.inferencerRequest = { type: "warmup" };

Output ​

Detection Results ​

javascript
{
  payload: [
    {
      box: {
        xyxy: [100, 150, 300, 400],
        xywh: [200, 275, 200, 250],
        confidence: 0.95
      },
      class: "person",
      class_id: 0
    }
  ],
  performance: {
    inference: {
      "my-inferencer": {
        inferenceTime: 45.2,
        preprocessTime: 5.1,
        postprocessTime: 3.8,
        totalTime: 54.1
      }
    }
  }
}

Classification Results ​

javascript
{
  payload: [
    { class: "cat", class_id: 281, confidence: 0.98 },
    { class: "dog", class_id: 179, confidence: 0.01 }
  ]
}

Segmentation Results ​

javascript
{
  payload: [
    {
      box: { xyxy: [100, 150, 300, 400] },
      class: "person",
      mask: {
        rle: "...",
        polygon: [[x1, y1], [x2, y2], ...],
        bitmap: Buffer
      }
    }
  ]
}

Promise Mode Output ​

javascript
{
  promises: [
    Promise { <pending> },
    Promise { <pending> }
  ]
}

Resolve in a function node:

javascript
msg.results = await Promise.all(msg.promises);
return msg;

Usage Examples ​

Example 1: Simple Object Detection ​

javascript
// Function node: Load image
msg.payload = {
  width: 640, height: 480,
  data: imageBuffer,
  colorSpace: "RGB"
};
return msg;

Inferencer configuration:

  • Model: yolo-v8-detection
  • Input: payload
  • Output: detections
javascript
// Function node: Filter high confidence
msg.payload = msg.detections.filter(det => det.box.confidence > 0.8);
return msg;

Example 2: Batch Processing ​

javascript
// Function node: Prepare batch
msg.payload = [
  { width: 640, height: 480, data: buffer1, colorSpace: "RGB" },
  { width: 640, height: 480, data: buffer2, colorSpace: "RGB" },
  { width: 640, height: 480, data: buffer3, colorSpace: "RGB" }
];
return msg;

Output:

javascript
msg.payload = [
  [detection1a, detection1b],  // Results from image 1
  [detection2a],               // Results from image 2
  []                           // No detections in image 3
];

Example 3: Dynamic Model Selection ​

javascript
// Function node: Choose model based on input
if (msg.topic === "quality-check") {
  msg.modelName = "defect-detection-model";
} else if (msg.topic === "classification") {
  msg.modelName = "product-classifier";
}
msg.payload = imageData;
return msg;

Example 4: Multi-Model Pipeline ​

[Camera] → [Inferencer: Detection] → [Function: Crop] → [Inferencer: Classification] → [Output]

Detection step:

javascript
msg.detections = msg.payload;
msg.payload = cropDetections(msg.payload); // Extract regions
return msg;

Classification step:

javascript
msg.classifications = msg.payload;
return msg;

Performance Optimization ​

Concurrency Settings ​

Low throughput (sporadic requests):

  • Workers: 1
  • Max concurrent: 5

High throughput (production):

  • Workers: 3-10
  • Max concurrent: 10-20

Batch Processing ​

Optimal batch size:

  • Small models (YOLO-nano): 10-20 images
  • Medium models (YOLO-v8): 5-10 images
  • Large models (Segmentation): 2-5 images

Model Warmup ​

Warmup is always enabled (since v1.2.1). The server generates a synthetic image internally on startup, eliminating cold-start latency on the first real request.

Docker Container Management ​

Container Lifecycle ​

  1. Startup: Node creates Docker container with inference server
  2. Health Check: Waits for gRPC handshake
  3. Warmup: Server-side synthetic image warmup
  4. Running: Process inference requests
  5. Shutdown: Clean container removal on deploy/close

Container Image ​

europe-southwest1-docker.pkg.dev/rosepetal-artifact/node-red-vp/rosepetal-detection:<tag>

Container Logs ​

bash
# View container logs
docker logs <container-id>

# Find inferencer containers
docker ps | grep rosepetal-serving

Memory Usage ​

Per container (approximate):

  • YOLO-nano: 500MB-1GB
  • YOLO-v8: 2GB-4GB
  • Segmentation: 4GB-8GB
  • PaddlePaddle OCR: 2GB-3GB

Worker scaling multiplies memory usage proportionally.

Model Management ​

Model Directory Structure ​

/opt/storage/models/
├── yolo-v8-detection/
│   ├── model.pt
│   └── config.json
├── product-classifier/
│   ├── model.onnx
│   └── config.json
└── segmentation-model/
    ├── model.pt
    └── config.json

Adding Models Manually ​

  1. Create directory: /opt/storage/models/<model-name>/
  2. Copy model file: model.pt, model.onnx, etc.
  3. Create config.json (optional, for auto-detection)
  4. Refresh Node-RED editor

Firebase Model Download ​

Requirements:

  • Firebase config node connected
  • Model exists in Firebase storage
  • Dynamic mode enabled or Firebase model selected in unified dropdown

Behavior:

  • Checks local storage first
  • Downloads if missing
  • Caches for future use
  • Shows download progress in status

Error Handling ​

Common Errors ​

"Docker image not available" ​

  • Cause: Inference Docker image not pulled
  • Solution: docker pull <image-name>

"Model not found" ​

  • Cause: Model doesn't exist in /opt/storage/models
  • Solution: Verify model name, check directory, download if needed

"Container failed to start" ​

  • Cause: Port conflict, insufficient memory, corrupted model
  • Solution: Check Docker logs, verify system resources

"gRPC connection failed" ​

  • Cause: Container not ready, network issue
  • Solution: Wait for container startup, check Docker status

"Invalid image format" ​

  • Cause: Missing required fields, wrong data type
  • Solution: Validate image object structure

"CUDA out of memory" ​

  • Cause: Batch too large, concurrent requests exceeded capacity
  • Solution: Reduce batch size, lower concurrency, use CPU

"TensorRT batch config invalid" ​

  • Cause: min_batch, opt_batch, max_batch constraint violated
  • Solution: Ensure min_batch <= opt_batch <= max_batch

Debugging ​

Enable verbose logging:

json
{
  "common": {
    "verbose": true
  }
}

Check container logs:

bash
docker logs <container-id>

Enable debug visualization:

  • Check "Show debug image"
  • Set debug interval: 1
  • Verify images display correctly

Best Practices ​

Model Selection ​

  1. Match task to model: Detection vs Classification vs Segmentation
  2. Consider speed/accuracy tradeoff: nano (fast) vs large (accurate)
  3. Use TensorRT for production: Optimized inference on NVIDIA GPUs
  4. Test on representative data: Validate before production
  5. Version models: Track changes, enable rollback

Configuration ​

  1. Use config prefabs: Start from built-in configurations for each model type
  2. Document class mappings: Clear naming conventions
  3. Set appropriate thresholds: Balance false positives/negatives

Maintenance ​

  1. Clean up unused containers: docker system prune
  2. Monitor disk space: Models can be large
  3. Update Docker images: Stay current with improvements
  4. Log performance metrics: Track degradation over time

Troubleshooting ​

Slow Inference ​

Possible causes:

  • Model too large for hardware
  • CPU inference on GPU model
  • High concurrent load

Solutions:

  • Use smaller model variant or TensorRT optimization
  • Enable GPU if available
  • Reduce concurrency
  • Cache models locally

High Memory Usage ​

Possible causes:

  • Too many workers
  • Large batch sizes
  • Memory leak in model

Solutions:

  • Reduce number of workers
  • Process smaller batches
  • Restart containers periodically
  • Update to latest image

Inconsistent Results ​

Possible causes:

  • Wrong color space
  • Incorrect image dimensions
  • Threshold too sensitive

Solutions:

  • Validate input format
  • Check preprocessing
  • Adjust confidence thresholds
  • Enable debug visualization

See Also ​