neuraltalk2

NeuralTalk2 is a research project that automatically generates text captions for images. You give it a photo and it produces a sentence describing what is in the picture, for example "a dog sitting on a grass field" or "two people riding bicycles on a street." This kind of task is called image captioning.

The system works by combining two components. A convolutional neural network looks at the image and extracts a compact summary of its visual content. A recurrent neural network then takes that summary and generates a sentence word by word. The two components are trained together on large datasets of images paired with human-written captions.

This release was written by Andrej Karpathy and is significantly faster than an earlier version called NeuralTalk. It runs on a graphics card, processes images in batches, and can fine-tune the image recognition component during training. According to the README, good models can be trained in two to three days with this setup, and the pretrained model scored competitively on a public image captioning benchmark.

The code is written in Lua using a framework called Torch, which was a popular research tool at the time. It requires a compatible NVIDIA graphics card for training. A pretrained model is available for people who want to run captions on their own images without training from scratch. The author notes that Google Brain later released a similar model that performs better, and suggests using that for production purposes. This repository is kept up for educational reference and as a Torch implementation of the approach.