On August 7, the Financial Times reported that ByteDance — the company behind TikTok — has begun pre-training an AI model with up to 10 trillion parameters across a cluster of roughly 30,000 GPUs. The model, developed by the company's internal Seed team, is explicitly targeting Anthropic's Mythos 5 — widely considered the current US frontier. In raw scale, ByteDance's system would be more than three times larger than Moonshot AI's Kimi K3, currently the largest publicly deployed Chinese model at 2.8 trillion parameters.
That number gets headlines. But if you've ever actually tried to coordinate thousands of accelerators through a multi-month training run, you know that the headline is almost never where the story lives. The story here isn't the parameter count. It's the hardware.
Mixture of Experts Is Doing a Lot of Work in That Number
Let's establish the technical baseline first. The 10 trillion figure is total model parameters — not active parameters per inference. ByteDance is using a Mixture of Experts (MoE) architecture, which routes each input token through only a subset of specialized expert sub-networks rather than activating the full model for every pass. In practice, a 10 trillion parameter MoE model might activate somewhere between 200 and 400 billion parameters per token — large by any reasonable measure, but not 10x the compute cost of a 1 trillion parameter dense model.
MoE buys you real capacity gains: you can scale total knowledge without proportionally scaling inference compute. But it creates coordination problems during training that dense architectures don't have. Routing decisions across experts must be load-balanced — you can't let one expert absorb all the traffic while others idle — and maintaining that balance across 30,000 chips over months of training is not a solved problem. ByteDance published their MegaScale systems research in early 2024, specifically addressing distributed training at 10,000+ GPU scale. That's the infrastructure pedigree they're drawing on. But going from 10,000 to 30,000 is not a linear extrapolation.
The Hardware Situation Is the Actual Story
This is what makes the ByteDance announcement fundamentally different from any comparable US lab run: they're doing it on domestic Chinese hardware, with no NVIDIA in the loop.
The April 2025 H20 ban — which eliminated the last NVIDIA product that could legally ship to China — forced every Chinese AI lab to pivot hard to domestic alternatives. ByteDance responded by placing a $5.6 billion procurement order for Huawei Ascend chips in 2026, the largest known non-NVIDIA AI hardware deal in history. The primary chips in question are the Ascend 910C for training — delivering roughly 60% of an H100's compute performance per chip — and the Ascend 950PR for inference, which Huawei claims achieves 2.8x the H20's FP4 throughput.
Sixty percent per chip sounds manageable until you run the math. If ByteDance is running 30,000 Ascend 910C chips in this training cluster, the effective H100-equivalent compute is roughly 18,000 H100s. That's a credible frontier-class training cluster, but not dominant by current standards. China's GPU cloud has been rapidly consolidating around Huawei and domestic suppliers as the only viable path — and ByteDance is now the most aggressive buyer in that ecosystem.
But the more difficult problem is software, not raw performance numbers. NVIDIA's CUDA ecosystem represents roughly two decades of optimization work: custom kernels, numerics libraries, collective communication primitives tuned to GPU microarchitecture (NCCL), and deep framework integrations with PyTorch and JAX that have been battle-tested at scale. Huawei's software stack — CANN (Compute Architecture of Neural Networks) and MindSpore — is capable, but it is not CUDA-equivalent in the long tail of operations. Every custom training kernel ByteDance has written for their previous NVIDIA-based clusters needs to be rewritten or translated. Every assumption about interconnect topology, collective operation latency, and numerics precision needs to be re-validated from scratch.
Zhang Yiming's explicit instruction to the Seed team was to pursue world-leading capabilities through independent development, ruling out the rival-copying shortcut. That independence is admirable on a competitive strategy level. It carries a real cost in engineering hours.
What Thirty Thousand GPUs Actually Looks Like to Run
I've managed infrastructure at a fraction of this scale, and even then, failure stops being a rare event and starts being a statistical constant. At 30,000 accelerators running for three to six months of continuous pre-training, hardware failures measured in per-day events are not a worst-case scenario — they're the expected operating condition. Modern AI accelerators have mean-time-between-failure figures measured in months per chip. At 30,000 units, you're looking at multiple chip failures per day across the cluster.
A single chip failure in a tightly coupled training run — depending on architecture — can stall or corrupt the entire job unless you've built robust checkpointing and fast restart logic. ByteDance's MegaScale paper addressed exactly this problem: efficient checkpoint cadence, online repair without full-cluster restarts, fault isolation at pod boundaries. But those solutions were validated on NVIDIA hardware with known failure modes and known interconnect behavior. Whether they translate cleanly to Ascend chips is an open question, because the failure characteristics of Huawei's HBM implementation and die packaging may differ in ways that aren't publicly documented.
The networking layer deserves its own paragraph. NVIDIA's NVLink and InfiniBand have well-understood performance profiles and years of kernel-level tuning for large-scale collective operations — AllReduce in particular, which is the backbone of distributed gradient synchronization. Huawei's High-Speed Interconnect is less publicly documented and hasn't been stress-tested at 30,000-node scale in any peer-reviewed work. Getting the cluster topology right — how you spread 30,000 chips across racks, pods, and data-center segments while minimizing hop count for all-reduce traffic — is a configuration problem that can cost 20 to 30% of your effective cluster throughput if you get it wrong. And you typically don't find out you got it wrong until you're six weeks into a training run.
Why This Run Matters Regardless of the Outcome
I'm not writing this to dismiss what ByteDance is attempting. The opposite, actually. They are running a direct empirical test of a proposition that the US export control strategy depends on: that cutting off NVIDIA access creates a hardware gap wide enough and durable enough to matter.
If the Seed team completes this run — even if the resulting model scores meaningfully below Mythos 5 on every benchmark — the world will have learned that you can execute a frontier-scale MoE training run on non-NVIDIA hardware. That proof point alone reshapes the competitive landscape. It tells every other Chinese lab, and every country currently squeezed by export controls, that the path exists. It tells hardware startups that the CUDA moat has a practical floor. It tells policymakers that the embargo strategy is buying time, not permanent capability suppression.
For people running production AI infrastructure in the West, the more immediate question is what this means for pricing. ByteDance already operates Doubao, China's most widely-used AI assistant, with hundreds of millions of users. If their new model trains successfully and delivers something close to frontier performance, they'll be in a position to deploy it at Chinese consumer price points — which historically run well below US equivalents on a per-token basis. That will create downward pricing pressure on frontier API access regardless of where you're running your workloads. We've seen this movie before with Alibaba's Qwen series and DeepSeek; a successful ByteDance frontier model is that dynamic at much larger scale.
The Signal I'm Actually Watching For
ByteDance hasn't committed to a public release timeline. The 3-to-6-month pre-training estimate doesn't account for alignment, safety evaluation, or the post-training work that now typically adds months to a frontier model launch. My working estimate: late 2026 or early 2027 before this model is public, if the training run completes cleanly — which, for the infrastructure reasons I've outlined, is genuinely uncertain.
What I'll be tracking most closely isn't the eventual benchmark numbers. It's whether ByteDance publishes a systems paper about this run, the way they did with MegaScale. That's where the real value transfers to the rest of the industry: the lessons from actually executing the largest MoE training job ever attempted on domestic Chinese hardware. What interconnect configurations worked, where the checkpointing overhead landed, how they handled expert load balancing at this scale. That's infrastructure knowledge the whole field needs, regardless of which side of an export control line you happen to be on.
Ten trillion parameters on thirty thousand Huawei GPUs. I don't know if it's going to work. But I'm paying close attention to find out.