LongCat-2.0 on Limited Hardware

The official Hugging Face recipe targets 16× H20 GPUs with tensor + expert parallelism. Most developers do not have that — here are realistic paths and what to measure.

HF: LongCat-2.0-FP8 API Cost Calculator

Official Baseline (from Hugging Face)

The model card for meituan-longcat/LongCat-2.0-FP8 recommends SGLang with 16× H20, --tp 16, --ep 16, across 2 nodes. Weights are ~1.8T parameters with FP8 tensors — this is datacenter-scale serving, not a single-workstation install. 

python -m sglang.launch_server \
  --model meituan-longcat/LongCat-2.0-FP8 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 13423 \
  --tp 16 \
  --ep 16 \
  --max-running-requests 64 \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 2048 \
  --nsa-prefill-backend fa3 \
  --kv-cache-dtype bfloat16 \
  --nnodes 2 \
  --node-rank 0 \
  --dist-init-addr $MASTER_IP:20000

Build a patched sgl-kernel first — see the full commands on the Hugging Face model card. Hierarchical LSA indexing is disabled in this deployment path for simplicity.

Why 16 GPUs Is the Default

  • MoE at 1.6T scale — even with sparsity, expert weights and KV cache for 1M context dominate VRAM
  • ~48B active params/token — activation is large compared to ultra-sparse models; FP8 helps but does not magically fit one consumer GPU
  • LongCat Sparse Attention — LSA adds indexing overhead; official GPU path uses FlashAttention-3 backend for prefill
  • Expert parallelism — --ep 16 spreads MoE experts across devices; cutting EP without understanding routing risks OOM or severe slowdown

Four Realistic Paths for Individuals & Small Teams

Path A — API / routed inference (recommended default)

For agent workflows (Claude Code, OpenClaw, custom harnesses), routing to longcat.ai, OpenRouter, or LongCat API (Anthropic-compatible) avoids capex entirely. Use the token cost calculator to compare monthly spend vs renting 16 GPUs.

Path B — Short-term cloud burst (16-GPU rental)

Rent a 2×8-GPU node block for evaluation windows (benchmark your repo, tune prompts). Breakeven vs API depends on utilization — heavy 24/7 agent fleets sometimes favor self-host; intermittent dev usage rarely does.

Path C — Experimental reduced parallelism (advanced)

Some teams attempt lower --tp / --ep on 8× H100/H800 with aggressive limits:

  • Reduce --max-running-requests and context length for initial smoke tests
  • Use --chunked-prefill-size to cap prefill peaks
  • Monitor OOM during MoE expert swaps — first failure mode on undersized clusters
  • Experimental only — document your flags if you publish results

An 8-GPU configuration is not documented by the official release. Check Hugging Face for community AWQ/GPTQ/INT4 variants before committing engineering time.

Path D — Local dev with smaller LongCat models

Use LongCat-Flash-Lite or Flash-Chat locally for harness development; route production agent traffic to LongCat-2.0 via API. This separates integration testing from frontier model quality.

What to Benchmark When You Do Run It

Whether on 16× H20 or an experimental smaller setup, record reproducible numbers:

Metric How to measure Why it matters for agents
Prefill tokens/s Large system prompt + codebase snapshot Agent sessions start with heavy context ingestion
Decode tokens/s 1K–4K output after tool loop Multi-turn agents are output-heavy
Max stable context Increase until OOM or quality cliff 1M is nominal; real limit depends on KV cache budget
Tool-call JSON validity Fixed harness, 50+ tool schemas Quantization can hurt structured output before chat quality drops
SWE-style task pass rate Your repo's issue subset Validate SWE-bench scores on your own codebase
Cost per successful task Tokens × price (self-host amortized GPU hours) Infrastructure decision metric — see agent workflow eval

FP8 vs INT8 / Community Quants

The official checkpoint is LongCat-2.0-FP8 (F8_E4M3 tensors). INT4/INT8 community quants may appear on the Hugging Face model tree over time — each requires validating tool-call reliability, not just perplexity.

  • Start from the official FP8 weights before third-party quants
  • Compare tool-use pass rate before and after quantization on the same harness
  • Log VRAM at peak prefill — agent workloads spike during context load, not steady decode
SGLang (Flash models) Agent Workflow Decision All Docs