Try-On, No Server Attached

The trade-off nobody talks about

Most virtual try-on products today ship a server-side diffusion model. The renders look great — and every single preview costs money, adds latency, and requires the shopper to upload a photo or video of themselves to someone else's GPU. For a lot of people, that last part is a non-starter.

This project sits at the other end of that trade-off. Instead of one big generative model on a server, it uses three small, fast pieces on the client: a segmentation model, a pose model, and a thin-plate-spline garment warp — all running in the browser via LiteRT.js on WebGPU. The preview is instant, and serving it costs exactly what a static site costs: nothing. There's a clear path to a paid, photorealistic server-side tier later, but the free tier is genuinely free — for me and for the user.

See it: original vs. try-on

Drag the seam to compare the input frame with the composited render.

How the pipeline is cut

webcam/photo ──► [Worker: LiteRT.js]
                   ├─ segmenter ──► person mask ──────┐
                   └─ pose ──► 17 keypoints ─► TPS ───┤
                                                      ▼
main thread ◄─────────────────── compositor (canvas 2D)

1. Segmentation. MediaPipe Selfie Segmenter produces a per-pixel person-confidence mask.
2. Pose estimation. MoveNet SinglePose Lightning finds 17 body keypoints per frame.
3. The warp. Six garment anchors — shoulders, waist, hem — are mapped from garment-pixel space onto the detected body with a thin-plate-spline transform, rendered as a coarse mesh of affine-textured triangles. Canvas 2D can't do a true nonlinear warp, so each triangle approximates one locally.
4. Compositing. The warped garment is clipped to the feathered person mask and drawn over the frame. A capsule-shaped clip restores the original arm pixels on top of the fabric, so a hand-on-hip pose still reads correctly.

Both models run inside a Web Worker. Only mask and keypoint results — plus transferred ImageBitmaps — ever cross back to the main thread. The video frame itself never touches the network layer, which turns the "privacy-first" pitch from marketing copy into an architectural fact.

WebGPU is the fast path, not the only path

On a WebGPU-capable browser (Chrome or Edge 113+), LiteRT.js runs both models on the GPU. Everywhere else, the app falls back to a Wasm/XNNPack CPU path — and the UI lets you toggle between the two live, which turns the whole app into its own benchmark.

The unglamorous parts that made it work

Real garments, not illustrations. The four garments shipped in the demo — two floral dresses, an embroidered kurti, a wrap dress — are real product photos. A tooling script removes their backgrounds with a flood-fill color-distance key: a plain per-pixel key would punch transparent holes through any light print motif that resembles the backdrop, so the fill only treats a pixel as background if it's actually connected to the image border.

Anchors need a human eye. The same script suggests the six anchor points, but the heuristic can mistake a puff sleeve's bulge for a shoulder line. Every new garment gets rendered against a test photo and hand-corrected before its anchors are trusted.

Framework-free math. The pipeline layer has no React in it. The TPS solver, keypoint smoothing, and letterbox math are all unit-tested with Vitest, independent of any UI — which is exactly what made iterating on the warp bearable.

Deployment is a footnote. npm run build produces a static dist/; GitHub Actions pushes it to GitHub Pages on every commit to main. Any static host would work identically, because there is simply no server component to provision.

What this architecture buys you

Zero marginal cost per try-on. Zero user data leaving the device. Instant previews with no round trip. And a stack — Vite, React, TypeScript strict, @litertjs/core — that any web team can read without an ML PhD. The photorealistic diffusion render still has its place, but as a paid upgrade for the shopper who wants a print-quality image, not as the toll booth in front of every single preview.