Large Language Models (LLMs) like GPT-OSS, Mixtral, and DeepSeek-V3/R1 use Mixture-of-Experts (MoE) layers to massively expand model capacity without increasing inference cost. But the mechanism behind this is often confusing — especially when comparing dense vs. MoE feed-forward networks (FFNs).

This article explains the concept simply, concretely, and concisely, using an example where both dense and MoE versions have equal total parameters — and then explains why real MoE models have far more parameters overall.

1. Background: FFNs Are the Largest Part of a Transformer Block

In a typical transformer block:

• The multi-head attention (MHA) is relatively small.

• The feed-forward network (FFN) contains the majority of the model’s parameters.

• This block is repeated many times (e.g., DeepSeek-V3 uses 61 layers).

So modifying the FFN architecture has a huge impact on total model size and compute.

2. Dense Architecture (Baseline)

In a dense transformer:

• There is one FFN per block.

• Every token goes through all the FFN parameters every time.

• Compute and memory scale with the full hidden dimension, every token, every layer.

Example:
A dense FFN has 308M parameters → all 308M are active per forward pass.

3. MoE Architecture (Sparse)

Instead of one big FFN, MoE replaces it with many experts, each of which is its own FFN.

Example:

• 8 experts

• Each expert has 38.5M parameters

• Total ≈ 308M parameters (same as dense)

But MoE adds a router that chooses only top_k experts per token.

For example:

num_experts = 8 top_k = 2

Only 2 out of 8 experts are activated for each token.

That means:

• Total parameters: ~308M

• Active parameters per token: ~77M

• The rest stay idle!

This sparsity is why MoE is called sparse.

4. Why MoE Saves Memory and Compute

Even if total parameters equal the dense case, MoE saves compute because only a small fraction is used.

In Dense:

• 100% of 308M active per token

• Full activations must be stored

• Full FLOPs needed

In MoE:

• Only 2 experts × 38.5M = 77M active parameters

• Activations only stored for active experts

• FLOPs reduced by top_k / num_experts ratio
  (in this example, 2/8 = 25% of dense compute)

This gives significantly lower memory and compute cost per token.

5. Then Why Do People Say “MoE Increases Total Parameters”?

This is the key point that confuses many readers.

👉 In theoretical demos or scripts, MoE is configured to have the same total parameters as the dense FFN.
This makes comparisons fair when measuring memory savings.

But…

6. In Real MoE Models, Capacity Is Greatly Increased

Real MoE models use many more experts, each larger, because we can afford it — we only activate a small subset.

Instead of 8 experts, a real MoE could have:

• 64 experts

• 128 experts

• 256 experts (as in DeepSeek-V3)

• 1024 experts

This explodes the total parameter count without exploding inference cost.

Example (realistic):

• Dense FFN: 300M parameters

• MoE FFN with 256 experts:

  • Total capacity = 300M × 256 = 76.8B parameters

• top_k = 8 → only 8 experts active

  • Active parameters = 8 × 300M = 2.4B per token

Thus:

🟢 Massive total capacity

🔴 Small active compute

This is the magic of MoE.

7. Why This Works (One-Sentence Summary)

MoE increases total capacity (many experts) but reduces active compute because each token uses only a few experts selected by the router.

8. Visual Analogy (Very Simple)

Think of a workshop:

Dense model = one giant machine

Every job must use the same giant machine, even if unnecessary.

MoE model = 256 specialized machines

For each job:

• A router picks the best 2 machines.

• The rest stay powered off.

Total machine count = huge
Power used per job = very small

9. Concrete Example (DeepSeek-V3)

• Total model parameters: 671B

• Experts per MoE layer: 256

• Active experts per token: 1 shared + 8 routed = 9

• Each expert: ~4.1B parameters.

• Active params per token: ~37B

• Total params: 671B

Thus DeepSeek-V3 has extremely high capacity, but inference cost like a much smaller dense model.

10. Summary for Retrieval Practice

Memorize these short bullets:

• MoE replaces 1 FFN with many FFNs (experts).

• Router picks top_k experts per token.

• Only those experts are active → low compute.

• Total parameters can be massive → high capacity.

• Real MoE models have far more parameters than dense models.

• But active parameters are small → efficient inference.

• Memory savings come from reduced activations and fewer FFN computations.