Model Manager Node

Overview

The Model Manager node manages what a shared inference server holds, for the whole flow.

The inference nodes stay deliberately simple: each holds one model, loads it on demand, and releases it when it moves on or is removed. A model stays resident while any node uses it. This node exists for the cases that simple rule cannot express:

Pre-loading a set so the first prediction is not a cold start
Seeing what is loaded and why — which node holds what, and how much memory is left
Freeing everything before a memory-heavy phase

It does not run inference. It sits beside the inferencer, ocr-inferencer and triton-inferencer nodes and controls the server they share.

Key Features

Pinned residency: models it loads are held under this node's id and cannot be evicted by another node switching models
Request/reply: one input, one reply per request, on output 1
Residency events: load, unload (with reason), server ready, and evictions the server made on its own, on output 2
Automatic fetch: with a Firebase connection, a model not yet on the controller is downloaded (and TensorRT-converted for Triton) before loading
Safe ordering: fetching happens before any unload, so a failed download never leaves the server changed
Merged view: loadedModels reports residency, pins, per-slot detail and free memory in one reply

Pins

A model this node loads is pinned: held under this node's id, exactly like an inference node's reference. That is what stops a node switching models from unloading something you asked for, and it is why a pin is released when this node is removed or redeployed — it dies with the node that created it, like any other reference.

A model is resident while any inference node uses it or any manager pins it. When neither holds it, it is unloaded after a short grace period.

One manager per backend is enough. Several are allowed, and each holds its own pins.

Configuration

Properties

Name

Type: String
Default: ""
Description: Optional node label. When empty the node shows models (<backend>).

Backend

Type: Select
Default: inference
Options:
- inference — the container shared by inferencer and ocr-inferencer nodes. Residency is bounded by measured free RAM and VRAM; there is no model count limit.
- triton — the container shared by triton-inferencer nodes. Residency is bounded only by claims.
Description: Which shared server this node manages.

Routing across both backends

The runtime also accepts backend: "both", which routes each model to whichever backend has it on disk. It is implemented and tested but not offered in the editor; re-add the <option> in model-manager.html to expose it. With both backends active, a name present in both repositories is refused rather than guessed, and so is one present in neither — set backend on the request or list entry to disambiguate.

Firebase Config

Type: Node reference (firebase-config)
Required: No
Description: With a connection, a model that is not on the controller yet is fetched before loading — a download for the inference server; download + TensorRT conversion + install for Triton (minutes the first time, with the phase reported in the node status). Without one, a missing model is a clear error rather than a silent "not found" from the server.

Poll (s)

Type: Number (0–3600)
Default: 5
Description: How often the node reads what the server holds and emits a residencyChanged event when it differs from the last look. This is how evictions the server decided on its own (memory pressure, idle TTL) become visible — the server never announces them. Set to 0 to disable polling.

Input

msg.payload is either a type string or an object:

javascript

{ "type": "loadedModels" }
{ "type": "loadModel",   "model": "detector-a" }
{ "type": "unloadModel", "model": "detector-a" }
{ "type": "unloadAll" }
{ "type": "setModels",   "models": ["detector-a", "detector-b"] }

A bare string works as shorthand: msg.payload = "loadedModels".

Request Types

Type	Aliases	Purpose
`loadedModels`	`listModels`, `list`	What is resident, and why
`loadModel`	`load`, `pin`	Load if absent, and pin
`unloadModel`	`unload`, `unpin`	Drop this node's pin and unload now
`unloadAll`	—	Unload everything now, dropping all pins
`setModels`	`set`	Make this node's pinned set exactly the list

Type matching is case-insensitive and trimmed.

Request Fields

Field	Type	Applies to	Description
`type`	String	all	Request type (required)
`model`	String	`loadModel`, `unloadModel`	Model name. `modelName` is accepted as an alias
`models`	Array \| String	`setModels`	Array of names, a comma-separated string, or objects `{ model, variant, config }`
`variant`	String	`loadModel`, `unloadModel`	Slot variant, used by OCR slots
`config`	Object	`loadModel`	Per-model load configuration
`backend`	String	all	`inference` or `triton`; overrides routing
`exclusive`	Boolean	`setModels`	Unload everything not in the list

Behaviour Notes

unloadModel — inference nodes still using the model reload it on their next message; the reply names them.
unloadAll — inference-node references are kept, so their models come back on demand.
setModels — loads one at a time and replies only when every load has finished. Models held by inference nodes are left alone unless exclusive: true, which unloads everything else as well (those nodes reload on demand). A name that fails to load is reported in failed without aborting the rest; a name that cannot be routed aborts the whole request before any server is touched, so a set never half-applies.

Output 1 — Reply

The same msg, with topic set to the outcome and payload replaced by the reply. _msgid and any other properties are preserved.

Request	`msg.topic`
`loadedModels`	`loadedModels`
`loadModel`	`modelLoaded`
`unloadModel`	`modelUnloaded`
`unloadAll`	`unloadedAll`
`setModels`	`modelsSet`

`loadedModels` Reply

javascript

{
  "message": "2 model(s) loaded (1 pinned), 38279 MiB RAM / 21610 MiB VRAM free",
  "loadedCount": 2,
  "models": ["detector-a", "detector-b"],
  "pinned": ["detector-a"],
  "slots": [
    { "model": "detector-a", "variant": "", "idle_seconds": 12.4, "loading": false,
      "pinned": true,  "pinnedBy": ["<this node>"], "referencedBy": [] },
    { "model": "detector-b", "variant": "", "idle_seconds": 0.8,  "loading": false,
      "pinned": false, "pinnedBy": [],              "referencedBy": ["a54203ad47c57fd8"] }
  ],
  "memory": { "ram_free_mib": 38279, "vram_free_mib": 21610 },
  "pendingReload": []
}

Triton replies carry dependencies (sub-models it loaded implicitly — ensembles, TensorRT weights) and residentCount instead of memory. Dependencies are never listed in models: they are not something you asked for.

pendingReload lists models that are claimed but not resident — the server evicted them, or they were unloaded explicitly. Inference nodes bring theirs back on the next message; a pin does not reload by itself, because a server that evicted under memory pressure should not be asked for the same model again automatically.

Other Replies

javascript

// modelLoaded
{ "model": "detector-a", "variant": "", "backend": "inference", "pinned": true,
  "message": "Model 'detector-a' loaded and pinned (inference)" }

// modelUnloaded
{ "model": "detector-a", "variant": "", "backend": "inference",
  "stillReferencedBy": ["a54203ad47c57fd8"],
  "message": "Model 'detector-a' unloaded (1 node(s) will reload it on demand)" }

// unloadedAll
{ "evicted": ["detector-a", "detector-b"], "failed": [],
  "message": "Unloaded 2 model(s)" }

// modelsSet
{ "loaded": ["detector-a", "detector-b"], "failed": [], "evicted": [],
  "exclusive": false, "message": "Loaded 2 model(s)" }

Output 2 — Events

Unsolicited messages with topic: "event", a backend naming the source, and a timestamp.

`event`	Payload	Meaning
`modelLoaded`	`{ model, by }`	`by` is the node id that caused it, or `"restart"`
`modelUnloaded`	`{ model, reason }`	`reason` is `released` (last claim gone), `explicit` (a request), or `dependency` (went with its parent)
`serverReady`	—	The shared container came up or was reconnected
`residencyChanged`	`{ added, removed, loadedCount }`	From the poll: what changed without this process doing it, i.e. the server's own evictions

javascript

{
  "topic": "event",
  "event": "residencyChanged",
  "backend": "inference",
  "timestamp": 1789593159425,
  "added": [],
  "removed": ["detector-b"],
  "loadedCount": 1
}

Wire this output to a logger or a NATS-out node for fleet-wide traceability of what is loaded where.

Node Status

Status	Meaning
⚪ `connecting`	Attaching to the shared server(s)
🟢 `N loaded`	Current residency, refreshed on every poll
🟡 `Downloading <model>…`	Fetching from Firebase (also `Converting`, `Installing`)
🟡 `N server(s) unreachable`	A poll could not reach a managed server
🔴 message	The last request failed; the text is the error

Examples

Pre-load a set at startup

An inject node set to fire once after a short delay, wired into the manager:

javascript

msg.payload = {
  type: "setModels",
  models: ["detector-a", "detector-b", "classifier-main"]
};
return msg;

The reply on output 1 arrives only when every model is resident, so the first real prediction is never a cold start.

Free everything before a heavy phase

javascript

msg.payload = { type: "unloadAll" };
return msg;

Inference nodes keep their references, so their models come back on demand afterwards.

Hold exactly one model for a recipe change

javascript

msg.payload = {
  type: "setModels",
  models: ["recipe-b-detector"],
  exclusive: true
};
return msg;

exclusive: true unloads everything else, including models held by inference nodes — those nodes reload theirs on their next message.

Log evictions the server made on its own

Wire output 2 to a switch node on msg.event, route residencyChanged to a debug or NATS-out node, and you get a record of every eviction caused by memory pressure or an idle TTL.

Notes

The node does not run inference; it only manages residency.
Pins are released on redeploy. A full deploy therefore drops every pin and re-creates it as the node starts again.
Polling uses an unreferenced timer, so it never keeps the Node-RED process alive on its own.
With poll set to 0, residencyChanged events stop; the other three event types still fire, since they are pushed by the server.

inferencer — detection, classification, segmentation and anomaly inference
ocr-inferencer — PaddleOCR text detection and recognition
triton-inferencer — TensorRT inference via NVIDIA Triton
list-model — browse the models available in Firebase
firebase-config — connection used to fetch missing models

Model Manager Node ​

Overview ​

Key Features ​

Pins ​

Configuration ​

Properties ​

Name ​

Backend ​

Firebase Config ​

Poll (s) ​

Input ​

Request Types ​

Request Fields ​

Behaviour Notes ​

Output 1 — Reply ​

loadedModels Reply ​

Other Replies ​

Output 2 — Events ​

Node Status ​

Examples ​

Pre-load a set at startup ​

Free everything before a heavy phase ​

Hold exactly one model for a recipe change ​

Log evictions the server made on its own ​

Notes ​

Related Nodes ​

Model Manager Node

Overview

Key Features

Pins

Configuration

Properties

Name

Backend

Firebase Config

Poll (s)

Input

Request Types

Request Fields

Behaviour Notes

Output 1 — Reply

`loadedModels` Reply

Other Replies

Output 2 — Events

Node Status

Examples

Pre-load a set at startup

Free everything before a heavy phase

Hold exactly one model for a recipe change

Log evictions the server made on its own

Notes

Related Nodes