Nvidia CEO Jensen Huang recently unveiled a unique test for evaluating the worth of an engineer: a token budget. Speaking at the close of GTC 2026, Huang stated that if an engineer’s annual AI token consumption falls below half their salary, it’s a cause for significant concern. He confirmed that Nvidia is strategically aiming for a $2 billion annual token expenditure for its engineering workforce.
This move highlights a broader trend across the industry, where many companies are shifting substantial financial resources from human capital to AI-powered token consumption. The four largest hyperscalers are projected to spend a combined $700 billion on capital expenditure in 2026, nearly doubling last year’s figures. Concurrently, data indicates that AI is the leading reason for U.S. job cuts, marking the fourth consecutive month this trend has been observed.
An internal Meta memo revealed that the company’s recent workforce reductions were aimed at offsetting significant AI investments, even as revenue grew by 33% in the same quarter. This suggests that layoffs in many tech giants are not driven by immediate financial distress but rather by a strategic reallocation of capital.
However, this shift in investment has not consistently delivered the promised returns. A survey by Gartner of 350 executives at companies with over $1 billion in revenue, all of whom are deploying AI agents or automation, found that approximately 80% had reduced headcount without seeing a corresponding improvement in returns. As analyst Helen Poitevin aptly put it, “Workforce reductions may create budget room, but they do not create return.”
Uber’s experience with AI coding tools offers a stark example of the challenges associated with token expenditure. After equipping 5,000 engineers with these tools in December, the company depleted its entire 2026 AI budget by April. Despite 70% of committed code being AI-generated, Chief Operating Officer Andrew Macdonald admitted that the link to tangible customer benefits remains elusive: “That link is not there yet.”
The juxtaposition of these two scenarios—layoffs as financing and costly token consumption without clear ROI—illuminates the core issue. Companies have often treated their token budgets as fixed costs and their workforce as variable, when the reverse is proving to be more accurate. Headcount reductions are permanent, leading to the loss of invaluable institutional knowledge. In contrast, token budgets, with careful engineering, offer significant flexibility.
### Where the Token Budget Bends
The most accessible and cost-effective solution lies in optimizing token usage by avoiding redundant processing. Prompt caching, now a standard feature across major API providers, can reduce the cost of repeated inputs by up to 90%. This is because static elements like system instructions and reference documents are processed once and then retrieved efficiently.
Security firm ProjectDiscovery demonstrated the power of prompt restructuring, increasing its cache hit rate from 7% to 84%. This optimization led to a 59% to 70% reduction in total LLM spend, while successfully serving 9.8 billion tokens from cache. This single engineering effort yielded more budget savings than many AI-driven layoff initiatives.
Another key strategy involves directing workloads to appropriately sized models. Flagship models can cost up to five times more per token than their smaller counterparts, yet many routine tasks like classification and summarization are still being sent to the most expensive tiers. Furthermore, batch processing, which offers a 50% discount for non-real-time tasks, is often underutilized.
Retrieval-augmented generation addresses the problem by providing models with only the relevant context from a knowledge base, rather than the entire dataset. Prompt compression further refines this by trimming unnecessary examples in each query. The use of open-weight models can also significantly reduce costs for teams willing to manage the associated infrastructure, handling routine tasks at a fraction of the price of proprietary APIs.
These technical optimizations are akin to basic energy conservation, such as turning off lights in empty rooms. Uber’s subsequent implementation of a $1,500 monthly cap per engineer, following their budget overrun, serves as an early indicator that spending discipline eventually becomes a necessity. Proactive companies, however, are choosing to implement such controls before financial constraints force their hand.
### The Other Half of the Fix is Human
The true value of optimizing token expenditure is realized when savings are reinvested productively, and the strongest evidence points towards investing in people. Research indicates that organizations achieving better ROI are those that leverage AI to augment their workforce, rather than replace it.
Klarna’s experiment with replacing approximately 700 customer service roles with an OpenAI-powered assistant resulted in a decline in customer satisfaction. CEO Sebastian Siemiatkowski candidly stated, “The result was lower quality, and that’s not sustainable.”
The fintech company has since adopted a hybrid model, where AI handles high-volume, routine queries, while human agents manage tasks requiring judgment and nuanced interaction. Industry analysts anticipate this trend will become more widespread, with predictions that half of the companies that replaced customer service staff with AI will rehire them by 2027.
A crucial investment that optimization logic makes urgent, rather than optional, is in nurturing future talent. Research from Stanford University’s Institute for Human-Centered AI found a nearly 20% decline in employment for software developers aged 22 to 25 from 2024 levels, even as older cohorts saw growth. This trend suggests that companies are inadvertently dismantling the training grounds for the senior engineers who will be essential for managing complex AI systems in the future.
Companies that have achieved substantial savings on their token bills have the financial latitude to continue hiring entry-level talent. The decision to do so, however, rests on leadership vision rather than purely financial considerations.
Jensen Huang’s provocative challenge will undoubtedly resonate through future earnings calls, and capital expenditure figures will continue to rise. The companies that will ultimately thrive will not be those that simply spend the most on tokens or drastically cut headcount to fund them. Instead, they will be the organizations that recognize the inherent flexibility of token budgets, engineer smarter usage rather than reducing personnel, and strategically reinvest the resulting savings into the human talent that makes AI truly valuable.
Original article, Author: Samuel Thompson. If you wish to reprint this article, please indicate the source:https://aicnbc.com/23597.html
