AI Compute Landscape: GPUs, TPUs, Custom Silicon, and the Economics of Scale

Compute is the binding constraint of modern AI. The choice of hardware -- and how it is provisioned -- determines what models can be trained, at what cost, and how quickly inference can be served. Understanding the hardware landscape is no longer optional for AI strategy.

Hardware Comparison Table

Cost-Per-Token Estimates (Inference, 2026)

Approximate Cost per Million Output Tokens (Llama-3 70B class)
===============================================================

Hardware         $/M tokens    Relative Cost
─────────────    ──────────    ─────────────
H100 (cloud)       $0.80         1.0x (baseline)
H200 (cloud)       $0.60         0.75x
B200 (cloud)       $0.35         0.44x
TPU v5p            $0.55         0.69x
Trainium2          $0.40         0.50x
Inferentia2        $0.25         0.31x (inference only)
MI300X             $0.65         0.81x
Gaudi 3            $0.50         0.63x

Note: Estimates vary significantly by workload, batch size,
      quantization level, and cloud provider pricing.

Compute Scaling Laws

Scaling Laws Relationship Diagram
===================================

Performance (loss) improves as power law of three factors:

  Loss ~ f(Compute, Data, Parameters)

  Compute (C) ──────┐
                     │
  Data tokens (D) ───┼──► Loss = L(C, D, N)
                     │
  Parameters (N) ────┘

Chinchilla-optimal ratio (Hoffmann et al. 2022):
  D ≈ 20 × N  (tokens ≈ 20x parameters)

  Model Size    Optimal Tokens    Compute (FLOPs)
  ──────────    ──────────────    ───────────────
  1B            20B               3.6 × 10^19
  7B            140B              1.8 × 10^21
  70B           1.4T              1.8 × 10^23
  400B          8T                5.8 × 10^24
  1T            20T               3.6 × 10^25

Post-Chinchilla insight (LLama, Gemma):
  Over-training smaller models on more data improves
  inference efficiency at the cost of training efficiency.

Cloud GPU Pricing Comparison (On-Demand, per hour, 2026)

Note: GPU cloud pricing is highly dynamic. Spot/preemptible pricing can be 50-70% lower. Reserved instances offer 30-50% discounts.

Training vs Inference: Different Hardware Strategies

Training vs Inference Requirements
====================================

              Training                    Inference
              ────────                    ─────────
Memory:       Massive (model +           Model weights +
              optimizer + gradients)      KV cache only

Compute:      Sustained, high            Bursty, latency-
              throughput                  sensitive

Interconnect: Critical (multi-node       Less critical
              all-reduce)                (single node often)

Precision:    BF16/FP16, some FP32       INT8/INT4/FP8
                                         (quantized)

Cost driver:  GPU-hours ×                Tokens served ×
              number of GPUs             latency SLA

Optimal HW:   H100/H200/B200/TPU v5p    Inferentia2/L4/
              (high bandwidth)            quantized on B200

Strategic Considerations

The hardware landscape favors organizations that can plan their compute strategy across three horizons: immediate (which cloud GPUs to reserve now), medium-term (which custom silicon pipelines to evaluate), and long-term (which architectural bets -- such as inference-optimized chips or optical computing -- to track). The 10x cost differences between providers and chip generations make compute strategy a board-level decision.

Resources

• NVIDIA Blackwell Architecture Whitepaper
• Hoffmann et al. - Training Compute-Optimal Large Language Models (Chinchilla)
• Google Cloud TPU Documentation
• AWS Trainium and Inferentia :::