Post

A for-fun Android project: one flat photo in, a tilt-reactive 2.5D depth card out. Depth estimation, subject cutout, background inpaint, per-pixel parallax, all on-device, mostly CPU. Kotlin+Jetpack Compose, tested on a 2019 Xiaomi Mi 9. Thread, with the how-it-works. 🧵

The whole thing, coarse. Two small models run once per photo, depth (how far is every pixel) and the subject mask, then classic image processing: refine the mask edge, cut the subject out, fill the hole behind it. Per frame it's just pixels: the warp, plus a glare and rim light riding the tilt.

Stage 1 — depth, and how a model actually runs on a phone. The .tflite is mmap'd straight from APK assets, runs on the GPU delegate (CPU fallback). In/out are direct ByteBuffers, allocated once and reused native memory the runtime reads in place, no JNI copy per run. Left: the real output.

Stage 2 — cut out the subject, rebuild the background. BiRefNet gives the mask; a guided filter refines it only in a thin band around the edge, full-frame refining adds halo. The hole is filled with a push-pull pyramid: average known colors down to 1×1, refill on the way up. No model, runs once.

The payoff. Per frame, the inpainted plate is backward-warped by depth, pivoted at the subject's depth: the subject plane holds still, far pixels drift one way, near ones the other. The cutout stays rigid on top, warping a flat person in 3D reads uncanny; moving only the background reads natural.

The loading FX reuse the same depth field. One idea: a front travels through the scene far->near, and each pixel compares its depth to it. Left: a gaussian glow around the current depth slice. Right: a smoothstep dissolve in depth order; the subject mask decides what fades last.

Bonus: the depth map read as a relief. The slope at each pixel tells which way that spot faces, that's a normal map. Tilt moves a virtual light: spots facing the light catch a sheen, spots leaning away go dark(Blinn-Phong). Full-frame it read as hammered metal, so it stayed a sandbox.