Phone down
Published March 30 - 4 min read
Had really fun working on a new feature for @GrapicApp recently!
— Nikolaus West (@NikolausWest) March 30, 2021
Looking at metrics and visualizations is always a large part of shipping new algoritms. This time I put them all directly into the live app which made the whole process super playful and fun. pic.twitter.com/VpUzco9tmK
The background is that @GrapicApp is used for remote whiteboarding by live-streaming the paper you draw on. It uses AR to keep things stable if you want to just hold your phone for a quick sketch. But that means you end up putting your phone down a lot.
— Nikolaus West (@NikolausWest) March 30, 2021
Before, that would often lead to the tracking drifting so bad that you had to start over each time you picked it up again.
— Nikolaus West (@NikolausWest) March 30, 2021
So the idea was to add some special handling for the "phone down" case to make sure the AR map wouldn't be destroyed and spare the battery a bit as a bonus.
I started out with a hidden "experiment UI" to use as playground pic.twitter.com/Ds7L4ClJGf
— Nikolaus West (@NikolausWest) March 30, 2021
I figured I needed to know that the camera was still, horizontal, and with a covered camera (still needs to work with a phone mount etc).
— Nikolaus West (@NikolausWest) March 30, 2021
Apple's CoreMotion gives both debiased motion and orientation metrics that are easy enough to use. pic.twitter.com/CVVd0m16cK
Fun fact 🔎: it turns out that CoreMotion gives a higher base / noise level acceleration when the phone is placed screen down than screen up.
— Nikolaus West (@NikolausWest) March 30, 2021
Once I found some good thresholds for the basic case, I noticed I needed more smoothing and maybe some outlier removal when testing with a tripod. pic.twitter.com/C2CXnxrnWM
— Nikolaus West (@NikolausWest) March 30, 2021
For the purpose of this feature, it should be ok to just assume that any sufficiently non-sharp image is due to the camera being covered. For a sharpness score, the "Variance of Laplacian" seemed reasonable and easy to implement. pic.twitter.com/FIM0smiZkW
— Nikolaus West (@NikolausWest) March 30, 2021
After tuning the parameters on a few different devices, detecting a covered camera seems to work pretty well. pic.twitter.com/fIZjSQbyTy
— Nikolaus West (@NikolausWest) March 30, 2021
I'm usually the first to insist on setting up stringent repeatable evaluation on big datasets, but that can be unfeasible early on for a small team. Plus, when estimation and interaction are tightly linked, you need to be able to play around and tinker.
— Nikolaus West (@NikolausWest) March 30, 2021
I usually end up just printing metrics, sometimes with some helpful ✅❌📉📈 symbols, and try to follow along in the logs, which is difficult, always one-off, and not that fun.
— Nikolaus West (@NikolausWest) March 30, 2021
When visualizing live metrics is easy, I just do it all the time to understand basic behaviors better. Like how long does it takes to save an AR map? pic.twitter.com/EW7JMSKDgK
— Nikolaus West (@NikolausWest) March 30, 2021
In the past I've often invested in really detailed after-the-fact visualizations and analytics for datasets and recordings but going completely live has definitely been the most fun way yet. Will be doing a lot more of it in the future!
— Nikolaus West (@NikolausWest) March 30, 2021
This is how the final interaction ended up. Shipping in the next release =) pic.twitter.com/K6I2NPzuOT
— Nikolaus West (@NikolausWest) March 30, 2021