Deployment Configuration
Model Parameters and Quantization
Model Size
Quantization Type
Runtime Parameters
GPU VRAM Distribution
GPU 0
GPU 1
GPU 2
GPU 3
GPU 4
GPU 5
GPU 6
GPU 7
0.0GB6.9GB13.8GB20.7GB27.6GB34.5GB
Framework Overhead (1.00 GB) Model Parameters (8.75 GB) Activations (2.19 GB) Output Tensor (15.66 GB)
Model Details
Hidden Size10240
Layers80
Attention Heads96
KV Heads20
Max Position Embeddings32768
Vocabulary Size128,000
Parameters Per Layer0.88 B
Total Parameters70.00 B
Attention Dimension160
FFN Expansion Ratio3.5x
GQA Ratio4:1
Hardware Configuration
Select Available GPU
Selected GPU
Current Quantization: FP8No GPU selected. All supported GPUs in the current category will be considered.
CPU-
Memory-
Network-
Storage-
Optimization Plan-
GPU Compatibility CheckNo incompatibility detected for the current GPU selection.