// Full-Stack · Multimodal AI · AR · 2025
A full-stack, multimodal augmented-reality web app that turns a walk through Milan's Piazza del Duomo, Palazzo Reale and Galleria Vittorio Emanuele into an interactive experience — a grounded LLM guide, a WebXR treasure hunt, and a vision model that checks whether you really found the landmark.
01 — The idea
Heritage sites are surrounded by information that tourists rarely reach — it lives in plaques, archives and guidebooks, not in the moment you're looking at the façade. HeritEdge puts that knowledge in your hand: point your phone at a monument and get an accurate, grounded answer, then turn the visit into a game.
The brief was deliberately multimodal: combine text (an LLM assistant), vision (recognizing what the camera sees), and spatial interaction (AR in the browser) into one coherent web app that runs on an ordinary phone with no install.
02 — What it does
An LLM chat grounded in a Chroma vector database of curated heritage facts, so answers stay tied to real sources instead of the model's imagination.
An augmented-reality hunt driven by camera and device-orientation input — players follow clues and find AR markers placed around each site.
A vision model checks a player's photo against the target landmark to confirm they actually reached it before unlocking the next step.
Everything runs in a mobile browser via WebXR, so visitors join instantly by opening a link — no app store, no download.
03 — Under the hood
The frontend is a React/TypeScript single-page app; a Node.js/Express backend serves the RAG endpoint and verification calls. Retrieval runs against a Chroma vector store of heritage facts, and the whole stack is containerized and shipped to AWS.
I owned the cloud path end to end: multi-stage Docker builds to keep images lean, then deployment to AWS across EC2, ECR, S3 and CloudFront — configuring IAM roles, security groups, and CDN-to-origin routing so the static frontend and the API served cleanly behind one domain.
04 — Stack