Better face suggestions

Better face suggestions

Face recognition is actually a combination of face detection and face recognition.   These two tasks work together.   Suggestions result from the (second) face recognition step.   

In this post we are discussing face recognition as a separate step.  Refer to the post on "Face detection considerations" for a discussion of factors impacting face detection.

At the present time (Oct 2021) we are testing a new machine learning algorithm that is more accurate (>20% improvement) than our current method.  The downside is that it is slower and takes up lots of memory... so we are trying to minimize the downside while keeping the benefits before we roll it into production.  We hope to do this by end of Nov 2021.  We utilize face detection and recognition technology from Applied Recognition Corp.  They are doing the advanced R&D to continuously improve the algorithms that ultimately improve the accuracy of Tag That Photo.

How recognition works - we extract the pixels around an identified face that meets a certain quality threshold.  The identified face is passed to the recognition module from the face detection module. The pixels extracted from the original image are sent through a deep learning algorithm that generates a sequence of digital numbers (technically a vector of approx. 100 floating point numbers).  This sequence of digital numbers is also known as a face template in face recognition jargon.  The template is then used for two purposes;
  1. To compare with all other known people (faces) to see if there is a match.  The best match turns into a suggestion.  There can be more than one match but that is rare. 
  2. If no matches are found, then the template is compared with all unknown faces to potentially find an existing cluster of unknown faces that look similar.  
The time consuming part in the process is; a) face detection setup on images that can contain millions of pixels, and b) generation of the face template.  The actual comparison task between two face templates is very fast - typically 3ms (or 3 thousandths of a second) on current computers.  So once the software has a new template, it can quickly compare that template with hundreds of thousands of known and unknown face templates.

The factors affecting face recognition accuracy include; lighting, pose, resolution, expression, occlusion (stuff blocking the face), weight, age, facial hair, and glasses.  As you can see there are a LOT of things getting in the way of the recognition algorithm identifying a face as a match for another face.

A match implies that the mathematical distance between two face templates is short.  The template comparison module actually returns a score - and that score is higher for closer templates.  The score is based on a statistical scale - so that the higher the score, the higher the probability that the faces are from the same person. 

One thing that can "confuse" the recognition algorithm is having too many faces of a person that are not good representations of that person for some of the reasons listed above.  For example, they have sunglasses on and there is extreme sunlight washing out the image.  The pixels for that face and the resulting template could be relatively distant from a "good" face for that person.  When Tag That Photo does a face comparison, the entire group of tagged faces for a person (excluding faces tagged manually) are compared using something similar to an average.  The more variety of faces for a person, the less unique (in general) that Person becomes.

We are considering limiting the total number of "known" faces for a person to a max of 50, and choosing those based on quality scores during face detection.  We may allow users to over-ride those faces with other "good" faces that perhaps best represent that person in terms of pose, expression, lighting, etc.  This will help the recognition accuracy on a large database with hundreds of known people, and potentially tens of thousands of tagged faces.

We also are considering sorting the suggestions in order of popularity - so your close friends and family are first.  And, making better use of screen real estate to show more thumbnails will help you with tagging efficiency.

    • Related Articles

    • Face detection considerations

      Face recognition is actually a combination of face detection and face recognition.   These two tasks work together.    In this post we are discussing face detection as a separate step.  Refer to the post on "Better face suggestions" for a discussion ...
    • Add an "are you sure" request on the face thumbnail regeneration option

      Some users select many, many faces while tagging...using Shift-Click or Ctrl-Click. in the majority of cases to delete the faces.  But, if they select a large group of faces, right-click on one of the faces, and want to choose "Delete", but ...
    • Exporting faces

      We currently offer the ability to export photos selected via a complex search.  Those can be used for slideshows, books or other purposes. But, some users would like to export just the face thumbnails themselves.  So this would be feature added to ...
    • Better button layout to reduce mouse travel

      It was suggested from one of our users, Ralf, to better organize the popular button click sequences to reduce the need for mouse travel. We will also try to make more common tasks tied to a keyboard shortcut.  Remember that Enter will also mimick a ...
    • Enhancing images

      Do you have some poor resolution images? Perhaps they came from scans of bad images or the photographer (we are not pointing fingers at you) wasn't very good on that day? There is an online service for enhancing photos that uses artificial ...