← Projects

AI Race News

Data Engineering

Scrapes 110+ AI/ML sources every 15 minutes in production, then removes duplicates in three layers (URL, content hash, title similarity). Articles are auto-tagged across 12 categories and served as 5 audience-specific feeds.

Screenshot of AI Race News: The AI Race News homepage.
The AI Race News homepage.

Highlights

The full story

A production pipeline that ingests 110+ AI/ML sources every 15 minutes, deduplicates and auto-tags articles, and serves audience-specific feeds through a public FastAPI REST API and a Next.js frontend.

A production platform that ingests 110+ AI/ML sources every 15 minutes via APScheduler, deduplicates with a 3-layer pipeline (URL, content hash, title similarity), auto-tags across 12 categories, and serves 5 audience-specific feeds through a public FastAPI REST API and a Next.js frontend.

Built with

FastAPI · BeautifulSoup · APScheduler · Docker