Но дадут ли нобелевку по литературе за Deep Learning Book

Soumith Chintala (создатель pytorch) выдаёт базу о том как тренироваться на 10К GPU
x.com/soumithchintala/status/1841498799652708712

Оч короткий TL;DR (всем рекомендую прочитать оригинал, он не длинный)

1. Maximize batch size and GPU utilization: 3D parallelism + gradient checkpointing
2. Overlap communication, e.g. while N-1th layer is computing backward, all GPUs with an Nth layer can all-reduce

3. Optimize for your GPU cluster network topology

4. Failure recovery, at 10k GPU scale, things fail all the time -- GPUs, NICs, cables, etc

5. At 10K scale bit flips actually become a problem and can cause loss explosions. Save your model state as frequently and as quickly as you can. To speed it up save it in shards and to CPU memory first and then in a seaprate thread write to disk

O1 mini inference scaling experiments

Прикольное саммари экспериментов одного чела. Коротко: если убедить модель дольше думать (что пока что непросто) pass@1 реально будет расти лог-линейно. При этом это скорее всего не majority voting или self consistency тк эти методы упираются в потолок

Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
arxiv.org/abs/2404.15758

We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens.

arXiv.org

Let's Think Dot by Dot: Hidden Computation in Transformer...

Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task...