arxiv-sanity-preserver
Web interface for browsing, search and filtering recent arxiv submissions
Arxiv Sanity Preserver is a web app by AI researcher Andrej Karpathy that helps you search, filter, and get personalized recommendations from the daily flood of new machine learning papers posted on Arxiv.
Arxiv is a website where researchers post scientific papers before they go through formal publishing, and it receives hundreds of new submissions every day across many fields. Arxiv Sanity Preserver is a web application built by AI researcher Andrej Karpathy to make that flood of papers easier to navigate, particularly for machine learning.
The application works by regularly fetching new paper listings from Arxiv, downloading the PDFs, extracting the text, and converting each paper into a numerical representation based on the words it contains. Those representations allow the system to calculate how similar any two papers are to each other. The result is a searchable, filterable interface where you can find papers by keyword, sort results by similarity to a paper you already know, see which papers are currently popular, and save papers to a personal reading list. A personalized recommendation system learns from papers you have liked and surfaces other papers it thinks you would find relevant.
Running your own instance requires setting up a pipeline of scripts that fetch, download, parse, and index papers. This is not a one-click deployment; the README walks through each script in sequence and notes that the process takes several hours on a machine with 25,000 papers. The web interface is built with Flask, and a separate script can optionally pull Twitter mentions of papers to gauge social interest.
The author notes in the README that this original version has been superseded by a rewrite called arxiv-sanity-lite, which is simpler, more reliable, and easier to run. The newer version also supports email digests of new relevant papers and multiple topic tags per account. New users are pointed toward that project instead.
The original site was running live and serving over 25,000 papers from machine learning categories at the time the README was last updated.
Where it fits
- Find machine learning papers similar to a paper you already know and like, ranked by content similarity.
- Build a personal reading list of saved papers and get recommendations based on what you have liked before.
- Search the Arxiv ML paper archive by keyword with a filterable web interface instead of the default Arxiv site.