What does it take to turn AI glasses into your very own drawing teaching tool? As it turns out - more than you'd think. We integrated Meta's Ray-Ban AI glasses into our hand-drawing teaching app, in order to provide for the app’s users a complete, drawing-focused timelapse of their sketching process from their own POV. With a tight launch deadline looming, what appeared as a standard Computer Vision task turned into a considerable challenge by removing multiple common simplifying assumptions: track a feature-poor, partially-occluded, moving object from a moving perspective, at low FPS, as it is being created from scratch, all while a highly-similar but unwanted target - the app’s reference drawing - is in view. In this session we'll discuss solving this novel problem via semantic segmentation models, a custom processing and stabilization pipeline, and iteratively revealing the in-domain data distribution for improved results.
Senior AI research engineer @ Simply