MULTI-TOKEN Prediction Promises and results

Next Token Prediction (NTP) has long been the dominant training method for LLMs. However, the Multi-Token Prediction (MTP) approach is gaining increasing traction among researchers, driven by two main reasons: However, a key challenge of MTP remains the assumption of token independence during parallel generation. Tokens are most commonly generated in parallel by different transformer […]
HoTPP & DETPP

A review of two papers from our colleagues at Sber AI Lab: DeTPP & HoTPP. HoTPP In HoTPP, the authors articulate all these problems and propose a new benchmark along with a metric called T-mAP for comparing such temporal distributions. The T-mAP Metric The metric combines three ideas: The metric ends up with two hyperparameters: […]