blintz baseA KV cache calculator
I made a small KV cache calculator for some current open-weight models, based on Neil Movva’s.
KV Cache Calculator
at 85% of GPU memory usable for weights + KV cache
| Model | Max Batch Size |
|---|
I made a small KV cache calculator for some current open-weight models, based on Neil Movva’s.
| Model | Max Batch Size |
|---|