What does it take to turn AI glasses into your very own drawing teaching tool? We integrated Meta Ray-Ban AI glasses into our hand-drawing teaching app, in order to provide for the app’s users a complete, drawing-focused timelapse of their sketching process from their own POV. This turns a standard Computer Vision task into an interesting challenge by removing multiple common simplifying assumptions: track a feature-poor, partially-occluded, moving object from a moving perspective, at low FPS, as it is being created from scratch, all while a highly-similar but unwanted target - the app’s reference drawing - is in view. In this session we'll discuss integrating the new AI Glasses technology into our app, and solving this novel problem via semantic segmentation models, a custom processing and stabilization pipeline, and strongly in-domain data.
Senior AI research engineer @ Simply