TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters

The paper introduces Tokenformer. The architecture leverages the attention mechanism to facilitate not only inter-token computations but also interactions between tokens and model parameters. The authors replace all linear projection layers in the Transformer with Pattention layers, allowing for efficient incremental scaling without the need for retraining from scratch.

Future work:
- Extending the Mixture-of-Experts Paradigm
- Advancing Parameter-Efficient Tuning
- Integrating Vision and Language Models
- Device-Cloud Collaboration
- Enhancing Model Interpretability

Code: https://github.com/Haiyang-W/TokenFormer

The Artificial Inflation (AI) of Artificial Intelligence (AI)—or AI^2 Bursts its Bubble, Bringing Down the Hype of the AI Threat

While some of the promises of AI have come true, and technology (like ChatGPT and its plugins) will continue to impress with its capabilities, AI-based technologies have largely failed to live up to the mountainous hype. In 2025, the authors expect the industry to pull back on the promises, investment, and hype of new AI capabilities and settle down into what is real versus marketing noise.