How we built a new type of video search

2 points

2 years ago

Hey HN, we recently launched Remy, an AI agent designed to take the pain out of video search, and wanted to give the HN community a technical deep dive on how it works under the hood.

The Problem: There’s a ton of valuable content on the internet, but finding the “best parts” of long videos is frustratingly inefficient. Current video search methods haven’t evolved much since the early days of YouTube and aren’t designed for today’s massive volume and variety of information. Instead, we’re left scrubbing through long videos or, worse, missing valuable content entirely due to decision paralysis. Remy’s goal is to offer a smarter way to surface exactly what you’re looking for, in a fraction of the time.

The Solution: In less than a minute, Remy delivers custom playlists that isolate the best video moments from across the internet. It finds, clips, and organises segments to get you exactly what you need — without the endless search and skip game.

How it works:

Remy is powered by a stack of LLMs (and some non-LLM magic) designed for fast, focused video search and transcript processing. Here’s the pipeline:

1. Request Analysis When you send a message, the system decides whether to provide an immediate response, search the web, or start assembling video clips. If video is the best option, Remy generates a playlist outline with concise titles and detailed descriptions tailored to your query. For temporal queries, Remy automatically adjusts context to absolute dates (e.g., “tomorrow’s NBA games” → “November 15th NBA Games”).

2. Content Retrieval Using a ‘wide net’ approach, Remy generates a large set of targeted queries and searches the web for videos that could match your needs.

3. Multi-Step Filtration & Processing Each video goes through:

- Transcript Pull: Extracts YouTube transcripts.

- Non-LLM Filter: Filters out low-quality or AI-generated content based on YouTube stats, creator channels, release date, and other parameters.

- Punctuation Restoration (BERT): Restores punctuation for better LLM comprehension.

- Clipping: Uses an LLM to locate and clip the most relevant segments of the transcript based on your request.

- Evaluation: Uses an LLM to score clips on relevance, completeness, and interest level. Only the best make it to the final playlist.

4. Reordering & Overview A final LLM gets fed the top 16 clips for each topic, filters them down to the best 4 (at most), and arranges them for maximum uniqueness. Each section gets a brief overview, with added context pulled and cited from the web to give you additional context.

The result? A playlist tailored to your exact query, delivered in under a minute, without unnecessary noise.

We’d love to hear your thoughts and feedback. If there’s something you’re curious to try or any edge cases that it's not handling, let us know. And if you come across any bad clips, please use the report button to flag them! Thanks

https://www.useremy.com/