HawkeyeVision/docs/RECOGNITION_CONTROL.md

Recognition Control System

The recognition control system manages the image processing lifecycle: acquiring images, running them through a processing pipeline, and emitting results as events. Each camera gets its own IRecognitionControl instance, registered in the per-camera Ninject child kernel.

Architecture Overview

IRecognitionControl (interface)
    ^
    |
BaseRecognitionControl (abstract base - lifecycle, threading, events)
    ^
    |--- HawkeyeRecognitionControl (workflow-based pipeline processing)
    |--- CandyboxImageProcessingControl (histogram-based processing)
    |--- TestImageProcessingControl (minimal test stub)
    |--- [Your custom implementation]

BaseRecognitionControlSettings (abstract base - shared config)
    ^
    |--- HawkeyeRecognitionSettings
    |--- CandyboxRecognitionControlSettings
    |--- TestRecognitionControlSettings
    |--- [Your custom settings]

IRecognitionControl Interface

Defined in VisionBuilder.UI.Common/Processing/IRecognitionControl.cs.

public interface IRecognitionControl
{
    bool IsRunning { get; }
    List<RecipeData> GetRecipesData();
    void SetRecipe(RecipeData recipe);
    void Start();
    void Stop();
    void Pause();
    void Resume();
    event Action<ImageProcessedEvent> ImageProcessed;
    event Action<SessionStartedEvent> SessionStarted;
    event Action<SessionEndedEvent> SessionEnded;
    event Action<ErrorsInSequenceEvent> ErrorsInSequenceAlarm;
    void HotReload();
    event Action RecipeFileChanged;
}

Member	Description
IsRunning	Whether the processing loop is active.
GetRecipesData()	Returns available recipes (name + optional preview image).
SetRecipe(recipe)	Selects a recipe. Throws if IsRunning is true.
Start()	Initializes the recipe, warms up, then starts the processing loop on a background task.
Stop()	Cancels the processing loop, waits for it to finish, runs cleanup.
Pause() / Resume()	Temporarily suspends/resumes image processing without stopping the session.
HotReload()	Reloads recipe parameters without stopping the session.
ImageProcessed	Fired after every processed image with full result data.
SessionStarted / SessionEnded	Fired when a session starts/stops.
ErrorsInSequenceAlarm	Fired when consecutive errors reach the configured threshold.
RecipeFileChanged	Fired when the recipe file on disk changes (for hot-reload UI).

BaseRecognitionControl

Defined in VisionBuilder.UI.Common/Processing/BaseRecognitionControl.cs. This abstract class implements all of IRecognitionControl and provides the processing loop, threading, pause logic, error-in-sequence tracking, and event plumbing. Subclasses only need to implement four methods.

Abstract Methods to Implement

// Load/prepare resources for the given recipe
protected abstract void Initialize(RecipeData currentRecipe);

// Run a throwaway processing pass to warm up caches, JIT, models, etc.
protected abstract void WarmUp();

// Acquire and process one image. Return null to stop the loop.
protected abstract (
    Mat originalImage,
    Mat analysisImage,
    TimeSpan processingTime,
    TimeSpan acquisitionTime,
    string[] errorNames
)? ProcessImage(CancellationToken token);

// Return the list of available recipes
public abstract List<RecipeData> GetRecipesData();

Virtual Methods (Optional Overrides)

// Release resources when the session stops (called from Stop())
protected virtual void Cleanup() { }

// Reload recipe parameters without restarting the session
public virtual void HotReload() { }

Protected Helper

// Call this to raise the RecipeFileChanged event (e.g. from a file watcher)
protected void RaiseRecipeFileChanged();

Processing Loop Behavior

The Start() method:

1. Validates that a recipe is set and the control is not already running.
2. Shows a loading indicator via ILoadingService.
3. Calls Initialize(currentRecipe).
4. Calls WarmUp() (errors are logged, not thrown).
5. Launches Loop() on Task.Run with a CancellationToken.
6. Fires SessionStarted.
7. Hides the loading indicator.

The Loop() runs continuously until cancellation:

1. If paused, sleeps 100ms and continues.
2. Calls ProcessImage(token). If it returns null, the loop exits.
3. If processing time is below MinimumProcessingTime, sleeps the difference.
4. Tracks consecutive errors. If they reach ErrorsInSequenceAlarm, fires the alarm event.
5. Builds an ImageProcessedEvent and fires it.

The Stop() method:

1. Cancels the token, waits for the loop task to finish.
2. Calls Cleanup().
3. Fires SessionEnded.

Thread Safety

All public lifecycle methods (Start, Stop, Pause, Resume, SetRecipe) are protected by a shared lock. The processing loop runs on a background Task. ProcessImage receives a CancellationToken and should respect it for timely shutdown.

BaseRecognitionControlSettings

Defined in VisionBuilder.UI.Common/Processing/BaseRecognitionControlSettings.cs. Provides the shared configuration for all recognition controls.

public abstract class BaseRecognitionControlSettings(string cameraName) : ISettings
{
    public string CameraName { get; }                    // Injected camera identifier
    public int MinimumProcessingTime { get; set; }       // Floor for loop iteration (ms)
    public string RecipeNameFilter { get; set; } = "*";  // Glob filter for recipe discovery
    public int ErrorsInSequenceAlarm { get; set; } = 5;  // Consecutive errors before alarm
}

All properties are registered in the Inspectron.Settings system under {CameraName}/Recognition by the RegisterSettings method. Override it to add custom settings (always call base.RegisterSettings first).

Events

All event classes are in VisionBuilder.UI.Common/Commands/.

ImageProcessedEvent

Fired after every image. Contains everything needed to update the UI and record statistics.

public class ImageProcessedEvent : IEvent
{
    public DateTime SessionStart { get; set; }
    public string ImageSource { get; set; }       // Camera name
    public string RecipeName { get; set; }
    public Mat ImageOriginal { get; set; }        // Raw camera image
    public Mat ImageAnalysis { get; set; }        // Annotated/processed image
    public bool HasError { get; set; }
    public List<string> ErrorNames { get; set; }  // Names of failed operations
    public TimeSpan AnalysisTime { get; set; }    // Processing minus acquisition
    public TimeSpan AcquisitionTime { get; set; } // Camera capture time
    public TimeSpan SleepTime { get; set; }       // MinimumProcessingTime padding
}

SessionStartedEvent / SessionEndedEvent

public class SessionStartedEvent : IEvent
{
    public string RecipeName { get; set; }
    public DateTime SessionStarted { get; set; }
}

public class SessionEndedEvent : IEvent
{
    public DateTime SessionEnded { get; set; }
}

ErrorsInSequenceEvent

public class ErrorsInSequenceEvent
{
    public int Errors { get; set; }  // Current consecutive error count
}

RecipeData

Defined in VisionBuilder.UI.Common/ViewModel/Classes/RecipeData.cs.

public class RecipeData
{
    public Mat? Image { get; set; }        // Optional preview thumbnail
    public string RecipeName { get; set; } // Recipe identifier (typically filename without extension)
}

ILoadingService

Defined in VisionBuilder.UI.Common/Processing/ILoadingService.cs. Called by BaseRecognitionControl during Start() and Stop() to show/hide a loading overlay in the UI.

public interface ILoadingService
{
    void StartLoading(string title);
    void StopLoading(string title);
}

The loading service is injected into BaseRecognitionControl via the constructor and exposed as the LoadingService protected property, so subclasses can use it in GetRecipesData() or elsewhere.

How to Implement a Custom Recognition Control

Step 1: Create Settings

Extend BaseRecognitionControlSettings with any additional configuration your processing needs.

using Inspectron.Settings;
using VisionBuilder.UI.Common.Processing;

public class MyRecognitionSettings(string cameraName) : BaseRecognitionControlSettings(cameraName)
{
    public double Threshold { get; set; } = 0.5;

    public override void RegisterSettings(InspectronSettings settings)
    {
        base.RegisterSettings(settings);
        settings.RegisterSimple(this, () => Threshold,
            CameraName + "/Recognition", nameof(Threshold));
    }
}

The RegisterSettings call makes the property editable in the application's settings UI. Always call base.RegisterSettings(settings) first to register the base properties (MinimumProcessingTime, RecipeNameFilter, ErrorsInSequenceAlarm).

Step 2: Create the Recognition Control

Extend BaseRecognitionControl and implement the four abstract members.

using System.Diagnostics;
using OpenCvSharp;
using VisionBuilder.UI.Common.Processing;
using VisionBuilder.UI.Common.ViewModel.Classes;

public class MyRecognitionControl : BaseRecognitionControl
{
    private readonly MyRecognitionSettings _settings;
    private readonly IImageSource _imageSource;
    private MyProcessor _processor;  // your processing logic

    public MyRecognitionControl(
        MyRecognitionSettings settings,
        IImageSource imageSource,
        ILoadingService loadingService)
        : base(settings, loadingService)
    {
        _settings = settings;
        _imageSource = imageSource;
    }

    public override List<RecipeData> GetRecipesData()
    {
        // Return available recipes. Use _settings.RecipeNameFilter if applicable.
        var recipes = new List<RecipeData>();
        foreach (var file in Directory.GetFiles(@"..\Data\Recipes", "*.myrecipe"))
        {
            recipes.Add(new RecipeData
            {
                RecipeName = Path.GetFileNameWithoutExtension(file),
                Image = null  // or load a preview thumbnail
            });
        }
        return recipes;
    }

    protected override void Initialize(RecipeData currentRecipe)
    {
        // Load the recipe and prepare your processing pipeline.
        // Called once when Start() is invoked.
        var path = Path.Combine(@"..\Data\Recipes", currentRecipe.RecipeName + ".myrecipe");
        _processor = MyProcessor.Load(path, _settings.Threshold);
    }

    protected override void WarmUp()
    {
        // Optional: run a throwaway pass to warm up JIT, GPU, ONNX models, etc.
        // Errors here are caught and logged, they won't prevent startup.
        var dummy = new Mat(100, 100, MatType.CV_8UC3, Scalar.All(0));
        _processor.Process(dummy);
        dummy.Dispose();
    }

    protected override (Mat originalImage, Mat analysisImage,
        TimeSpan processingTime, TimeSpan acquisitionTime,
        string[] errorNames)?
        ProcessImage(CancellationToken token)
    {
        // 1. Acquire image
        var swTotal = Stopwatch.StartNew();
        var swAcquire = Stopwatch.StartNew();
        var image = _imageSource.GetImage(token).Result;
        swAcquire.Stop();

        if (image == null)
            return null;  // null signals "stop the loop"

        // 2. Process
        var result = _processor.Process(image);
        swTotal.Stop();

        // 3. Build annotated image
        var annotated = image.Clone();
        // ... draw overlays on annotated ...

        // 4. Collect error names (empty array = pass)
        string[] errors = result.Defects
            .Select(d => d.Name)
            .ToArray();

        return (image, annotated, swTotal.Elapsed, swAcquire.Elapsed, errors);
    }

    protected override void Cleanup()
    {
        // Release resources when the session stops.
        _processor?.Dispose();
        _processor = null;
    }
}

Key implementation notes for ProcessImage

• Return null to signal the loop should exit (e.g. when the image source is exhausted or cancelled).
• Respect the CancellationToken — pass it to blocking calls like GetImage(). Catch OperationCanceledException if you need cleanup before returning null.
• errorNames drives the pass/fail logic. An empty array means the image passed. Each string becomes a named error in statistics and UI.
• originalImage is the raw camera image; analysisImage is the annotated version shown in the preview UI.
• processingTime is the total wall-clock time including acquisition; acquisitionTime is just the camera capture. The base class computes AnalysisTime = processingTime - acquisitionTime for the event.

Step 3: Register via DI

Register your settings and control in the per-camera Ninject child kernel. This is typically done in a plugin's RegisterCameraModules or in an extension method.

Option A: In a plugin

public class MyPlugin : IPlugin
{
    public void RegisterGlobalModules(IKernel kernel) { }

    public void RegisterCameraModules(IKernel kernel, string cameraName)
    {
        kernel.Bind<MyRecognitionSettings, BaseRecognitionControlSettings, ISettings>()
            .ToConstant(new MyRecognitionSettings(cameraName));

        kernel.Rebind<IRecognitionControl>()
            .To<MyRecognitionControl>()
            .InSingletonScope();
    }
}

Option B: As an extension method (like HawkeyeRecipe)

public static class ModuleExtensions
{
    public static IChildKernel UseMyRecipes(this IChildKernel self, string cameraName)
    {
        self.Bind<MyRecognitionSettings, BaseRecognitionControlSettings, ISettings>()
            .ToConstant(new MyRecognitionSettings(cameraName));
        self.Bind<IRecognitionControl>()
            .To<MyRecognitionControl>()
            .InSingletonScope();
        return self;
    }
}

Important DI details:

• Bind your settings class to three types: itself, BaseRecognitionControlSettings, and ISettings. The ISettings binding makes RegisterSettings get called by the settings system. The BaseRecognitionControlSettings binding lets the base class constructor receive it.
• Use Rebind<IRecognitionControl> (not Bind) if replacing an existing binding from another recipe provider.
• Always use InSingletonScope() — there should be exactly one recognition control per camera.

Step 4: Hot Reload (Optional)

To support live recipe parameter updates without stopping:

1. Set up a FileSystemWatcher in Initialize() to monitor the recipe file.
2. When the file changes, call RaiseRecipeFileChanged() to notify the UI.
3. Override HotReload() to apply the new parameters on the next loop iteration.

Example from HawkeyeRecognitionControl:

private FileSystemWatcher? _fileWatcher;
private System.Timers.Timer? _debounceTimer;
private string? _pendingHotReloadFile;
private string? _currentRecipeFilePath;

protected override void Initialize(RecipeData currentRecipe)
{
    _currentRecipeFilePath = Path.GetFullPath(/* recipe path */);
    _workflow = LoadRecipe(_currentRecipeFilePath);
    StartFileWatcher(_currentRecipeFilePath);
}

private void StartFileWatcher(string filePath)
{
    var directory = Path.GetDirectoryName(filePath)!;
    var fileName = Path.GetFileName(filePath);

    _debounceTimer = new System.Timers.Timer(500) { AutoReset = false };
    _debounceTimer.Elapsed += (_, _) => RaiseRecipeFileChanged();

    _fileWatcher = new FileSystemWatcher(directory, fileName)
    {
        NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.Size,
        EnableRaisingEvents = true
    };
    _fileWatcher.Changed += (_, _) =>
    {
        _debounceTimer.Stop();
        _debounceTimer.Start();  // debounce to 500ms
    };
}

public override void HotReload()
{
    // Store path; ProcessImage picks it up on the next iteration
    Interlocked.Exchange(ref _pendingHotReloadFile, _currentRecipeFilePath);
}

protected override void Cleanup()
{
    _fileWatcher?.Dispose();
    _debounceTimer?.Dispose();
}

The UI flow: RecipeFileChanged event -> SingleCameraVM sets IsHotReloadAvailable = true -> user clicks hot-reload button -> calls HotReload() -> next ProcessImage iteration applies new parameters.

How the UI Consumes IRecognitionControl

SingleCameraVM (in VisionBuilder.UI.Common/ViewModel/SingleCameraVM.cs) is the primary consumer:

Commands:

// Start/Stop run on background tasks to avoid blocking the UI
await Task.Run(() => _recognitionControl.Start());
await Task.Run(() => _recognitionControl.Stop());

// Pause/Resume are synchronous (just flip a flag)
_recognitionControl.Pause();
_recognitionControl.Resume();

// Recipe selection
_recognitionControl.SetRecipe(selectedRecipe);

// Hot reload
_recognitionControl.HotReload();

Event subscriptions (in constructor):

recognitionControl.ImageProcessed += Handle;         // -> PreviewVm, StatisticsVm, ErrorsVm
recognitionControl.SessionStarted += StatisticsVm.Handle;
recognitionControl.SessionStarted += ErrorsVm.Handle;
recognitionControl.RecipeFileChanged += () =>
    SynchronizationContext?.Post(_ => IsHotReloadAvailable = true, null);

Other modules subscribing to events:

• VisionBuilderStatistics — writes CSV logs per session, tracks pass/fail counts.
• VisionBuilderRingbuffer — saves images to ringbuffers (separate good/bad).
• IOCommanderCameraModule — emits GPIO signals on session start/end, image result, and error alarms.
• Plugin modules (e.g. PralinenModule) — custom PLC integration based on results.

Existing Implementations Reference

HawkeyeRecognitionControl

Project: VisionBuilder.UI.Recipes.HawkeyeRecipe

The primary implementation for workflow-based image processing.

• GetRecipesData: Loads .jhrcp files from Data/Recipes/, deserializes each to extract recipe name and preview image.
• Initialize: Loads the workflow JSON file, starts a file watcher for hot-reload.
• WarmUp: Executes the workflow once with an EmptyImageSource.
• ProcessImage: Runs the full workflow pipeline, collects errors from operations where Result == false, draws graphics overlays on the analysis image using OpenCVCanvas.
• HotReload: Uses Interlocked.Exchange to pass the file path; ProcessImage checks for pending reload via _workflow.HotReloadParameters().
• Cleanup: Disposes the file watcher and debounce timer.
• Settings: HawkeyeRecognitionSettings adds AlwaysError (bool) for testing.

CandyboxImageProcessingControl

Project: Plugins/CandyboxPlugin

Histogram-based image processing for candy box quality control.

• GetRecipesData: Loads .json files from Data/Recipes/, finds matching sample .bmp images for previews.
• Initialize: Creates a HistoRecipe, applies camera width settings to HawkEye hardware.
• WarmUp: No-op.
• ProcessImage: Acquires image, processes via HistoRecipe, collects errors from ErrorPoints and ErrorReason.
• Settings: CandyboxRecognitionControlSettings adds AlwaysError with [SettingDescription] attribute.
• DI: Plugin replaces the existing BaseRecognitionControlSettings binding before adding its own.

TestImageProcessingControl

Project: Plugins/TestPlugin

Minimal stub for testing and development.

• GetRecipesData: Returns two hardcoded recipes ("Hello", "World") with red 100x100 Mat images.
• Initialize / WarmUp: No-op.
• ProcessImage: Returns a fixed red image with "TestError".
• Settings: TestRecognitionControlSettings with no extra properties.

Checklist for New Implementations

1. Create a settings class extending BaseRecognitionControlSettings.
2. Override RegisterSettings and call base.RegisterSettings(settings) first.
3. Create a recognition control class extending BaseRecognitionControl.
4. Implement GetRecipesData(), Initialize(), WarmUp(), ProcessImage().
5. Override Cleanup() to release resources.
6. (Optional) Override HotReload() and set up file watching with RaiseRecipeFileChanged().
7. Register both classes in the DI container (bind settings to 3 types, bind control as singleton).
8. Ensure ProcessImage respects CancellationToken and returns null when cancelled/exhausted.