Reciply – Recipe Scraper
Situation
Recipe content lives in social-media reels, but the actual recipe is trapped in a caption you have to scroll, screenshot and retype by hand. No tool turned a video URL into a saved, structured recipe you actually own — the goal was one URL in, one structured recipe out, kept in your own library.
Task
Build a full-stack app that takes an Instagram/Facebook reel URL and produces a clean, editable, persisted recipe with user accounts, images and optional community sharing — without polluting the database with unconfirmed AI output, and hardened enough to deploy. The core design constraint: extraction must not persist. Parsing returns an ephemeral draft the user edits and only then saves, so the DB never fills with AI guesses.
Action
The pipeline is URL → scraper → AI parser → editable preview → save. A caption-first scraper shells out to yt-dlp (--dump-json, no browser), covering ~80% of reels; Whisper transcription and headless-browser fallbacks are documented but deliberately deferred to keep v1 small. Captions go to Gemini with a responseSchema for structured output, and every result is Zod-validated before it reaches the client. Auth is JWT with short-lived access tokens (15 min) plus rotating refresh tokens (7 days, stored as a SHA-256 hash, delivered as an httpOnly cookie) with reuse detection, bcrypt (cost 12) for passwords, and per-userId ownership scoping on every read, write and delete (404 instead of leaking existence). Images upload to Cloudinary with the public_id derived deterministically from the stored URL, and a community feed lets users publish and "add to my library" as an independent snapshot; recipe identity is the sourceUrl, so duplicates return a 409 via app-level guards rather than a risky unique index. Expensive routes (extract, community) are cached in memory, rate limiters cover the global API plus extract, image-upload and refresh, env is Zod-validated at boot, errors use one { message } envelope, and Express 5 async auto-forwarding removed ~16 redundant try/catch blocks.
Result
v1 runs end-to-end: URL → yt-dlp → Gemini → editable preview → MongoDB, with auth, per-user libraries, favorites, image uploads and a public community feed with dedup. It is deploy-hardened — in-memory caching, full rate-limiter coverage, boot-time env validation and strict ownership isolation. The scope is deliberately held to caption-based extraction with documented upgrade paths, so the codebase stays small enough to reason about.
Key Takeaways
Scoping to the caption-first ladder and writing down the Whisper/headless fallbacks instead of building them kept v1 shippable without closing any doors. Keeping extract and save separate turned out to be the key architectural call — the database only ever holds recipes a human confirmed. App-level 409 dedup on sourceUrl avoided a unique index and its migration risk on existing data, and Express 5’s async auto-forwarding quietly deleted a whole class of boilerplate. The obvious next step is swapping the in-memory caches for Redis, already marked at the call site.