Training and serving energy
Turn a model's size, its training tokens, and the hardware into operations, GPU hours, days, and energy. Then add the prompts it answers.
What was reported for GPT-3 175B
OpenAI reported 3.14 × 10²³ operations. Patterson and colleagues (2021) estimated 1,287 MWh over 14.8 days, from OpenAI's measured speed (24.6 trillion operations a second per V100) and power (330 W per GPU with its share of the server), a building overhead of 1.10, and 10,000 GPUs, a number they took from NVIDIA.
This setup: 19.7% of the V100's peak, 1.17 GWh, about 109 years of a U.S. home's electricity.
0.24 Wh per prompt, measured
Measured across Google's serving fleet, counting idle machines, host CPUs and memory, and the building. Counting only busy AI chips, in Google's most efficient data centers, gives 0.10 Wh (Elsworth and colleagues, 2025).
Challenges
0 of 4 complete
- With Llama 3.1 405B, set the utilization so the run takes Meta's reported 30.84 million GPU hours (within 5%). (not yet)
- Finish GPT-3's training in a single day on the original V100s. Watch the energy while you do it. (not yet)
- Train GPT-3 again with at most a fifth of the 1,287 MWh estimated for the original run. (not yet)
- With Llama 3.1 405B and Google's measured 0.24 Wh per prompt, find how many prompts a day make a year of serving use more energy than the training run. (not yet)
Try this
- Start with GPT-3 on its V100s. Switch to H100 and raise the utilization to 40%. Same operations: watch the energy.
- Drag GPUs up and down. The time changes; the energy does not.
- Choose Llama 3.1 405B and lower the utilization until the GPU hours match Meta’s reported 30.84 million.
- Under serving, compare Google’s measured 0.24 Wh with the 100,000-token input, then raise the prompts a day until a year of serving passes the training run.
Training and serving energy in a few linesPython with the standard library
# Training: operations -> GPU hours -> days -> energy.
PUE = 1.1 # building overhead: cooling, power conversion
def train(params, tokens, peak, utilization, gpus, watts):
ops = 6 * params * tokens # 2 forward + 4 backward per parameter per token
gpu_hours = ops / (peak * utilization) / 3600
days = gpu_hours / gpus / 24
mwh = gpu_hours * watts * PUE / 1e6 # watt-hours to megawatt-hours
return ops, gpu_hours, days, mwh
# GPT-3 on 10,000 V100s at OpenAI's measured 24.6 of 125 trillion operations a second.
ops, hours, days, mwh = train(174.6e9, 300e9, 125e12, 24.6 / 125, 10_000, 300)
print(f"GPT-3: {ops:.3g} operations, {days:.1f} days, {mwh:,.0f} MWh at 300 W x {PUE}")
print(f" at the measured 330 W x 1.10: {hours * 330 * 1.10 / 1e6:,.0f} MWh")
# Llama 3.1 405B: what share of the H100's peak do Meta's GPU hours imply?
ops = 6 * 405.85e9 * 15.6e12
print(f"Llama 3.1 405B: {ops:.2g} operations, {ops / (30.84e6 * 3600 * 989.5e12):.1%} of peak")
# Serving: Epoch AI's estimate for one GPT-4o query, from its stated assumptions.
query_ops = 2 * 100e9 * 500 # 2 per active parameter, 500 output tokens
seconds = query_ops / (989.5e12 * 0.10) # an H100 at 10% utilization
print(f"one query: {seconds * 1500 * 0.7 / 3600:.2f} Wh") # 1,500 W all in, 70% drawnGPT-3: 3.14e+23 operations, 14.8 days, 1,171 MWh at 300 W x 1.1
at the measured 330 W x 1.10: 1,288 MWh
Llama 3.1 405B: 3.8e+25 operations, 34.6% of peak
one query: 0.29 WhEvery preset shows what its developers or later studies reported, so you can see how close the simple model gets and what each published figure counts. The per-prompt figures differ by more than a hundred times because they measure different things: busy chips only or the whole data center, a short prompt or a long document, a 2022 demo or a 2025 fleet. The lesson walks through each source.