Recognizing celebrities using domain models

One of the features of the Computer Vision API is the ability to recognize domain-specific content. At the time of writing, the API only supports celebrity recognition, where it is able to recognize around 200,000 celebrities.

For this example, we choose to use an image from the internet. The UI will then need a textbox to input the URL. It will need a button to load the image and perform the domain analysis. There should be an image element to see the image and a textbox to output the result.

The corresponding ViewModel should have two string properties for the URL and the analysis result. It should have a BitmapImage property for the image and an ICommand property for our button.

Add a private variable for the IVisionServiceClient type at the start of the ViewModel, as follows:

    private IVisionServiceClient _visionClient;

This should be assigned in the constructor, which will take a parameter of the IVisionServiceClient type.

As we need a URL to fetch an image from the internet, we need to initialize the Icommand property with both an action and a predicate. The latter checks whether the URL property is set or not, as shown in the following code:

    public CelebrityViewModel(IVisionServiceClient visionClient) {
        _visionClient = visionClient;
        LoadAndFindCelebrityCommand = new DelegateCommand(LoadAndFindCelebrity, CanFindCelebrity);
    }

The LoadAndFindCelebrity load creates a Uri with the given URL. Using this, it creates a BitmapImage and assigns this to ImageSource, the BitmapImage property, as shown in the following code. The image should be visible in the UI:

    private async void LoadAndFindCelebrity(object obj) {
        UrifileUri = new Uri(ImageUrl);
        BitmapImage image = new BitmapImage(fileUri);

        image.CacheOption = BitmapCacheOption.None;
        image.UriSource = fileUri;

        ImageSource = image;

We call the AnalyzeImageInDomainAsync type with the given URL, as shown in the following code. The first parameter we pass in is the image URL. Alternatively, this could have been an image that was opened as a Stream type:

    try {
        AnalysisInDomainResultcelebrityResult = await _visionClient.AnalyzeImageInDomainAsync(ImageUrl, "celebrities");

        if (celebrityResult != null)
            Celebrity = celebrityResult.Result.ToString();
    }

The second parameter is the domain model name, which is in a string format. As an alternative, we could have used a specific Model object, which can be retrieved by calling the following:

    VisionClient.ListModelsAsync();

This would return an array of Models, which we can display and select from. As there is only one available at this time, there is no point in doing so.

The result from AnalyzeImageInDomainAsync is an object of the AnalysisInDomainResult type. This object will contain the request ID, metadata of the image, and the result, containing an array of celebrities. In our case, we simply output the entire result array. Each item in this array will contain the name of the celebrity, the confidence of a match, and the face rectangle in the image. Do try it in the example code provided.

Utilizing optical character recognition

For some tasks, optical character recognition (OCR) can be very useful. Say that you took a photo of a receipt. Using OCR, you can read the amount from the photo itself and have it automatically added to accounting.

OCR will detect text in images and extract machine-readable characters. It will automatically detect language. Optionally, the API will detect image orientation and correct it before reading the text.

To specify a language, you need to use the BCP-47 language code. At the time of writing, the following languages are supported: simplified Chinese, traditional Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Turkish, Arabic, Romanian, Cyrillic Serbian, Latin Serbian, and Slovak.

In the code example, the UI will have an image element. It will also have a button to load the image and detect text. The result will be printed to a textbox element.

The ViewModel will need a string property for the result, a BitmapImage property for the image, and an ICommand property for the button.

Add a private variable to the ViewModel for the Computer Vision API, as follows:

    private IVisionServiceClient _visionClient;

The constructor should have one parameter of the IVisionServiceClient type, which should be assigned to the preceding variable.

Create a function as a command for our button. Call it BrowseAndAnalyze and have it accept object as the parameter. Then, open a file browser and find an image to analyze. With the image selected, we run the OCR analysis, as follows:

    using (StreamfileStream = File.OpenRead(filePath)) {
        OcrResultsanalysisResult = await _visionClient.RecognizeTextAsync (fileStream);

        if(analysisResult != null)
            OcrResult = PrintOcrResult(analysisResult);
    }

With the image opened as a Stream type, we call the RecognizeTextAsync method. In this case, we pass on the image as a Stream type, but we could just as easily have passed on a URL to an image.

Two more parameters may be specified in this call. First, you can specify the language of the text. The default is unknown, which means that the API will try to detect the language automatically. Second, you can specify whether or not the API should detect the orientation of the image. The default is set to false.

If the call succeeds, it will return data in the form of an OcrResults object. We send this result to a function, the PrintOcrResult function, where we will parse it and print the text, as follows:

    private string PrintOcrResult(OcrResultsocrResult)
    {
        StringBuilder result = new StringBuilder();

        result.AppendFormat("Language is {0}\n", ocrResult.Language);
        result.Append("The words are:\n\n");

First, we create a StringBuilder object, which will hold all the text. The first content we add to it is the language of the text in the image, as follows:

        foreach(var region in ocrResult.Regions) { 
            foreach(var line in region.Lines) { 
                foreach(var text in line.Words) { 
                    result.AppendFormat("{0} ", text.Text);
                }
                result.Append("\n");
            }
            result.Append("\n\n");
        }

The result has an array, which contains the Regions property. Each item represents recognized text, and each region contains multiple lines. The line variables are arrays, where each item represents recognized text. Each line contains an array of the Words property. Each item in this array represents a recognized word.

With all the words appended to the StringBuilder function, we return it as a string. This will then be printed in the UI, as shown in the following screenshot:

The result also contains the orientation and angle of the text. Combining this with the bounding box, also included, you can mark each word in the original image.

Generating image thumbnails

In today's world, we, as developers, have to consider different screen sizes when displaying images. The Computer Vision API offers some help with this by providing the ability to generate thumbnails.

Thumbnail generation, in itself, is not that big a deal. What makes the API clever is that it analyzes the image and determines the region of interest.

It will also generate smart cropping coordinates. This means that if the specified aspect ratio differs from the original, it will crop the image, with a focus on the interesting regions.

In the example code, the UI consists of two image elements and one button. The first image is the image in its original size. The second is for the generated thumbnail, which we specify to be 250 x 250 pixels in size.

The View model will need the corresponding properties, two BitmapImages methods to act as image sources, and one ICommand property for our button command.

Define a private variable in the ViewModel, as follows:

    private IVisionServiceClient _visionClient;

This will be our API access point. The constructor should accept an IVisionServiceClient object, which should be assigned to the preceding variable.

For the ICommand property, we create a function, BrowseAndAnalyze, accepting an object parameter. We do not need to check whether we can execute the command. We will browse for an image each time.

In the BrowseAndAnalyze function, we open a file dialog and select an image. When we have the image file path, we can generate our thumbnail, as follows:

    using (StreamfileStream = File.OpenRead(filePath))
    {
        byte[] thumbnailResult = await _visionClient.GetThumbnailAsync(fileStream, 250, 250);

        if(thumbnailResult != null &&thumbnailResult.Length != 0)
            CreateThumbnail(thumbnailResult);
    }

We open the image file so that we have a Stream type. This stream is the first parameter in our call to the GetThumbnailAsync method. The next two parameters indicate the width and height that we want for our thumbnail.

By default, the API call will use smart cropping, so we do not have to specify it. If we have a case where we do not want smart cropping, we could add a bool variable as the fourth parameter.

If the call succeeds, we get a byte array back. This is the image data. If it contains data, we pass it on to a new function, CreateThumbnail, to create a BitmapImage object from it, as follows:

    private void CreateThumbnail(byte[] thumbnailResult)
    {
        try {
            MemoryStreamms = new MemoryStream(thumbnailResult);
            ms.Seek(0, SeekOrigin.Begin);

To create an image from a byte array, we create a MemoryStream object from it. We make sure that we start at the beginning of the array.

Next, we create a BitmapImage object and begin to initialize it. We specify the CacheOption and set the StreamSource to the MemoryStream variables we created earlier. Finally, we stop the BitmapImage initialization and assign the image to our Thumbnail property, as shown in the following code:

        BitmapImage image = new BitmapImage();
        image.BeginInit();
        image.CacheOption = BitmapCacheOption.None;
        image.StreamSource = ms;
        image.EndInit();

        Thumbnail = image;   

Close up the try clause and add the corresponding catch clause. You should now be able to generate thumbnails.