Transformers Benchmarks

This repository helps you understand how fast you can train large AI language models on different GPUs. If you're planning to train a big model like BERT or GPT-2, you need to know: which GPU should you buy, how many GPUs do you need, and how long will training actually take? This project gives you real-world answers to those questions by measuring actual performance on popular hardware.

The core idea is simple: instead of relying on theoretical specifications from GPU manufacturers, the project runs real training workloads and measures how many calculations per second the GPU can actually handle (measured in TeraFLOPS—trillions of floating-point operations per second). The README includes a table showing results for expensive data-center GPUs like the A100 down to consumer GPUs like the RTX 4090, along with their prices and memory specs. This lets you compare cost versus performance and make informed decisions.

The project works by providing Jupyter notebooks that you can run on your own hardware. Two main notebooks let you benchmark different scenarios: one focuses on understanding how individual Transformer layers perform in isolation, and another trains actual BERT and GPT-2 models to see end-to-end training speed. Since real performance varies based on your exact setup—cooling, CUDA version, batch sizes, and more—you can run these notebooks on your machines to get numbers tailored to your situation.

This is useful for anyone training large language models: researchers deciding whether to use a single powerful GPU or multiple cheaper ones, startups estimating cloud compute costs before training a custom model, or engineers picking hardware for a new lab setup. Rather than guessing from marketing materials, you get concrete data showing which hardware-to-cost ratio makes sense for your workload. The README doesn't go into detail about how to interpret the results, but the notebooks themselves guide you through the setup and measurement process.