1
2
5
10
2
5
100
2
5
0.01
2
5
0.1
2
5
1
2
5
10
2
5
kv comms time (ms)
params comms time (ms)
flops time (ms)
total latency (ms)
2048 context
2048 context
4096 context
8192 context
16384 context
32768 context
65536 context
Latency Breakdown for LLAMA 3-70B
Batch Size
Time (s)
plotly-logomark