Imagine that there are several cameras around your house. The smart-house application can see what your mood is at any time. By knowing this, it can utilize the mood to better predict your needs.

We are going to add web-camera capabilities to our application. If you do not have a web camera, you can follow along, but load images using the techniques we have already seen.

First we need to add a NuGet package to our smart-house application. Search for OpenCvSharp3-AnyCPU and install the package by shimat. This is a package that allows for the processing of images, and is utilized by the next dependency we are going to add.

In the example code provided, there is a project called VideoFrameAnalyzer. This is a project written by Microsoft that allows us to grab frame-by-frame images from a web camera. Using this, we are able to analyze emotions in our application. The use case we will execute is as follows:

In our HomeView.xaml file, add two new buttons. One will be to start the web camera while the other will be to stop it.

In the corresponding View model, add two ICommand properties for each of the buttons. Also add the following private members:

    private FrameGrabber<CameraResult> _frameGrabber;
    private static readonly ImageEncodingParam[] s_jpegParams = {
        new ImageEncodingParam(ImwriteFlags.JpegQuality, 60)
    };

The first one is a FrameGrabber object, which is from the VideoFrameAnalyzer project. The static member is an array of parameters for images, and is used when fetching web camera images. Additionally, we need to add a CameraResult class, which should be within the ViewModel file.

We initialize the EmotionScores to null, as shown in the following code. This is done so that new emotion scores always will be assigned from the most resent analysis result:

    internal class CameraResult {
        public EmotionScores EmotionScores { get; set; } = null;
    }

Add an initialization of the _frameGrabber member in the constructor and add the following in the Initialization function:

    _frameGrabber.NewFrameProvided += OnNewFrameProvided;

Each time a new frame is provided from the camera, an event is raised.

When we receive new frames, we want to create a BitmapImage from it to show it in the UI. To do so requires us to invoke the action from the current dispatcher, as the event is triggered from a background thread, as shown in the following code:

    private void OnNewFrameProvided(object sender, FrameGrabber<CameraResult>.NewFrameEventArgs e) {          
        Application.Current.Dispatcher.Invoke(() => {
            BitmapSource bitmapSource = e.Frame.Image.ToBitmapSource();

            JpegBitmapEncoder encoder = new JpegBitmapEncoder();
            MemoryStream memoryStream = new MemoryStream();
            BitmapImage image = new BitmapImage();

We get the BitmapSource of the Frame and create some required variables.

Using the encoder we created, we add the bitmapSource and save it to the memoryStream, as follows:

    encoder.Frames.Add(BitmapFrame.Create(bitmapSource));
    encoder.Save(memoryStream);

This memoryStream is then assigned to the BitmapImage we created, as shown in the following code. This is in turn assigned to the ImageSource, which will show the frame in the UI:

    memoryStream.Position = 0;
    image.BeginInit(); 
    image.CacheOption = BitmapCacheOption.OnLoad;
    image.StreamSource = memoryStream;
    image.EndInit();

    memoryStream.Close(); 
    ImageSource = image;

As this event will be triggered a lot, we will get a fluent stream in the UI, and it will seem like it is a direct video feed.

In our Initialization function, we will also need to create our ICommand for the buttons, as follows:

    StopCameraCommand = new DelegateCommand(StopCamera);
    StartCameraCommand = new DelegateCommand(StartCamera, CanStartCamera);

To be able to start the camera, we need to have selected a person group, and we need to have at least one camera available:

    private bool CanStartCamera(object obj) {
        return _frameGrabber.GetNumCameras() > 0 && SelectedPersonGroup != null;
    }

To start a camera, we need to specify which camera to use and how often we want to trigger an analysis using the following code:

    private async void StartCamera(object obj) {
        _frameGrabber.TriggerAnalysisOnInterval(TimeSpan.FromSeconds(5));
        await _frameGrabber.StartProcessingCameraAsync();
    }

If no camera is specified in StartProcessingCameraAsync, the first one available is chosen by default.

We will get back to the analysis part of this process soon.

To stop the camera, we run the following command:

    private async void StopCamera(object obj) {
        await _frameGrabber.StopProcessingAsync();
    }