Qwen3.5-397B-A17B (W8A8)
qwen3_5_moe-architecture model (Qwen3_5MoeForConditionalGeneration) — a 512-expert
MoE (10 experts per token + 1 shared expert) with a hybrid attention stack: of its
60 layers only 15 carry a full-attention KV cache, the other 45 are linear-attention
(one full-attention layer every 4). It ships a native MTP head and is served as
W8A8 (~392 GB weights: 373 GB experts + 18 GB non-expert). 397B total /
~17B activated per token.
It was validated on Ascend 910B3 (64 GB/card) in two topologies on the same 16
cards — one 16-card cross-node aggregation and one 8-card single-node aggregation —
so that per-scenario placement can be chosen on measured data rather than assumed. Load
is a replay of real agent traces (Exgentic v2, claude_code harness), swept over
concurrent agent sessions, not synthetic fixed-length prompts.
In short — split the 16 cards into two independent 8-card services for chat and coding, and keep one 16-card service only for RAG.
On identical hardware and at the same total number of concurrent sessions, two 8-card
islands served chat and coding at 1.7–2.1× lower inter-token latency and up to 89 %
higher throughput than a single 16-card cross-node service — but only while each
island's KV cache has headroom. RAG must stay on the 16-card service (24 % of its
requests exceed the window an 8-card deployment can hold), and coding should move back
to it above ~16 total sessions, where an island reaches 86 % KV and latency quadruples.
Also: do not hand-write cudagraph_capture_sizes — see the caution below.
TOC
Model identityValidated hardware × stackModel configurationBenchmark methodBenchmark resultschatcodingRAGRecommended topology per scenarioSessions sustained at an absolute ITL budget (measured rungs only)Head-to-head at equal total sessionsVerdictDeployCaveatsModel identity
Only 15 of the 60 layers hold a KV cache. Sizing this model as if all 60 layers were
full-attention overstates per-request KV by 4× and understates how many sessions fit
by the same factor. The KV figures below use the 15 full-attention layers, read from
layer_types in config.json.
Validated hardware × stack
Model configuration
Do not hand-write cudagraph_capture_sizes. Left alone, this engine sizes the capture
list to cover exactly the largest decode step the admission gate can produce —
max_num_seqs × (1 + num_speculative_tokens) tokens. Measured on both topologies here:
gate 64 → captures [8, 16, …, 128]; gate 16 → [8, 16, 24, 32]. The default has no
cliff by construction. A hand-written list silently overrides it, and any decode step
whose token count exceeds the largest captured size falls back to eager dispatch for the
whole step.
The unit is tokens per step, not requests: with MTP (k = 1) each running sequence
contributes 2 tokens per step, so a captured maximum of 64 covers only 32 sequences per
rank. An earlier configuration of this model pinned [1,2,4,8,16,32,64] against a gate of
64 — half the coverage it needed. A controlled probe on that deployment measured
506–675 ms per step above the boundary versus 145 ms below it, and one rank crossing
it stalls the whole DP group.
Removing the list costs ~9 min of extra graph capture at startup on the 16-card topology (749 s vs 322 s) and nothing in KV cache — the 8-card engine reported the same 375,260-token cache either way.
Benchmark method
Closed-loop aiperf 0.12.0, replaying real agent sessions from the Exgentic v2
corpus (harness=claude_code) rather than synthetic prompts. The x-axis is concurrent
agent sessions: one session has at most one request in flight, and finished sessions
are immediately replaced, so a rung holds its session count for the whole run.
Each rung sends a fixed request count scaled with concurrency
(max(200, 25 × concurrency)) so every rung drives each session to the same depth — a
fixed per-rung request count would let high-concurrency rungs spread the same work over
more, shallower sessions and flatter the numbers.
Every figure below is measured over the second half of its rung — the steady state.
Closed-loop replay starts all N sessions at turn 0 at once, so a rung opens with every session doing a full cold prefill simultaneously; those prefill steps crowd out decode and inflate ITL until the prefix cache catches up. On this model that alone accounts for a 38–50 % fall in ITL from the first to the last fifth of a rung, on an already-warm engine. Production traffic is a mixture of session depths, not a synchronized cold start, so the settled half is the representative operating point and the whole-rung average is pessimistic. That window is itself re-checked for drift (≤ 25 % change in median ITL across it); rungs that still drift are marked ⚠️ and excluded from the conclusions.
Engine warm-up is a separate, one-time effect — a fresh process is several times slower for its first ~600 long-context requests while it compiles shapes — and is absorbed by a corpus warm-up phase before the sweep.
Benchmark results
chat
Customer-support agent (tau2_retail), ISL ~22k
16-card — DP2 × TP=8 (2 nodes)
8-card — TP=8 (1 node, per island)
⚠️ 64 sessions (16-card, -41 %) — the measured window is not flat to within 25 %. Shown for completeness, excluded from the capacity conclusions.
coding
Coding agent (swebench), ISL ~30k
16-card — DP2 × TP=8 (2 nodes)
8-card — TP=8 (1 node, per island)
⚠️ 16 sessions (8-card, +27 %), 8 sessions (8-card, -25 %) — the measured window is not flat to within 25 %. Shown for completeness, excluded from the capacity conclusions.
RAG
Retrieval-heavy browsing agent (browsecompplus), ISL ~50k
16-card — DP2 × TP=8 (2 nodes)
⚠️ 16 sessions (16-card, +26 %) — the measured window is not flat to within 25 %. Shown for completeness, excluded from the capacity conclusions.
Achieved sessions is the mean number of requests in flight over the measured window (time-integrated); it is below the nominal session count because a replayed session is idle between turns, exactly as recorded. How far below is a property of the workload: chat and coding pause a median 0.3 s between turns against a ~28 s request (83–93 % of nominal in flight), while RAG — a browsing agent — pauses a median 5.3 s against a ~14 s request (~53 %). So a RAG deployment sized in sessions holds roughly twice as many as the concurrent-request figure suggests. KV peak is the highest KV-cache occupancy either rank reached; there were no preemptions in any rung.
Recommended topology per scenario
Sessions sustained at an absolute ITL budget (measured rungs only)
+ = the highest rung tested still met the budget, so the real limit is above it.
< N = even the lowest rung tested exceeded the budget (censored — not a measured capacity).
Head-to-head at equal total sessions
One 16-card service vs. two 8-card islands. An island carries half the total sessions, so island rung N is compared with 16-card rung 2N. ITL and throughput are comparable; TTFT is not.
Excluded (steady window still drifting — rung too short for the session ramp): chat/agg-dp2tp8@64 (-41%), rag/agg-dp2tp8@16 (26%), coding/agg-tp8dp1@16 (27%), coding/agg-tp8dp1@8 (-25%)
Verdict
Why the split wins where it wins. Both options use the same silicon, so the gain is not extra compute — it is that a cross-node DP2 service couples its two ranks. Every prefill step runs eagerly and blocks decode on both ranks, so one island's prefill bursts cannot disturb the other once they are separate services. Splitting also halves the blast radius of a node failure and lets each island be tuned for its own scenario.
Why it stops winning. An island has one eighth of the cluster's KV cache
(375,260 tokens vs 1,416,463) and admits at most max-num-seqs = 16 sequences. Chat
sessions (~23k tokens) fit 16-deep at 60 % KV; coding sessions grow past 28k and reach
86 % KV, at which point latency roughly quadruples over the rung and throughput halves.
RAG sessions (~52k, p90 101k) do not fit at all. The deciding variable is context
length per session, not the scenario label — size an island by
KV tokens ÷ expected session context, and keep a margin: the degradation is gradual and
begins well before the cache is full.
If you deploy the split, raise max-num-seqs per island only together with KV
headroom, and re-check vllm:kv_cache_usage_perc under your own traffic — the 16-session
coding rung above shows what running close to the limit costs.
How to read the comparison. Both options use the same 16 cards, so the fair question is not "which service is faster" but "for a given number of concurrent sessions, is it better to run one 16-card service or two independent 8-card islands?" Two islands split the load, so an island carries half the total sessions — island rung N is therefore compared against 16-card rung 2N. The decision rule applied below:
- Does the scenario fit on 8 cards at all? The 8-card deployment is KV-bound to a 65536-token window; a scenario whose sessions exceed that is 16-card only, regardless of latency.
- At matched total sessions, which side has lower ITL? ITL drives how fast tokens stream back to a user and is comparable across the two topologies. (TTFT is not — the two run different prefill chunk sizes.)
- What does each option cost operationally? Two islands halve the blast radius of a
node failure and can be tuned per scenario, but each island admits at most
max-num-seqssequences, so a burst on one island queues instead of borrowing capacity from the other. One 16-card service pools all capacity but fails as a unit.
Deploy
Two self-contained InferNex manifests — pick the one matching the topology you chose in Recommended topology per scenario:
The engine image is pinned by digest (vllm-ascend@sha256:6c183df…, self-reported
vLLM 0.23.0) for traceability — mirror that exact build, or substitute the equivalent
quay.io/ascend/vllm-ascend:v0.23.0-openeuler release tag and re-verify before relying
on the numbers below.
Both manifests deliberately omit cudagraph_capture_sizes and keep only
{"cudagraph_mode":"FULL_DECODE_ONLY"}. Leave it that way unless you also change
max-num-seqs — see the caution in Model configuration.
Caveats
- Numbers are steady-state, not cold-start. See Benchmark method: each rung is reported over its second half, once replayed sessions have reached realistic depth and the prefix cache is warm. A freshly started engine serves its first ~600 long-context requests several times slower; size for steady state, but expect the first minutes after a rollout to be worse.
- TTFT is not comparable across the two topologies. The 8-card deployment runs
max-num-batched-tokens4096 against the 16-card's 8192, so prefill is chunked differently. ITL, throughput and session capacity are comparable; TTFT is not. - The 8-card topology cannot serve the RAG scenario. Measured RAG input lengths are mean 47.6k / p90 91.6k / max 126k tokens, so 24 % of RAG requests exceed the 8-card's 65536-token window — they cannot be served at all, not merely served slowly. Its 375,260-token KV cache would also hold only ~7 full RAG contexts. The 16-card window (180224) fits 100 % of the corpus. RAG requires the 16-card topology.
- 🔴 The coding result does not prove a 64k window is enough for coding. No coding
request overflowed the 8-card's 65536-token window in this run, but that is a property
of the sample, not of the scenario. The corpus threshold for coding is 80k, set from
the p95 peak session context over all 495
swebenchsessions (75,935 tokens). Each rung here drew only 20 sessions (4 % of the corpus) and sampling is sequential, so the long tail was never exercised — the deepest context reached was 40,291 tokens. The 16-card topology, whose window is 180224, likewise peaked at 39,306. Size coding deployments against the 80k threshold, not against the 40k observed here. - Sessions, not requests. The x-axis is concurrent agent sessions; one session has at most one request in flight, and replayed sessions idle between turns exactly as recorded. The idle fraction is workload-specific — chat and coding pause a median 0.3 s between turns, RAG a median 5.3 s — so N RAG sessions occupy only ~0.53 N request slots while N chat sessions occupy ~0.9 N. Request-per-second figures are correspondingly far below what a single-turn benchmark would show at the same nominal concurrency.
- Single run per rung.
numRuns: 1, so the tables carry no run-to-run spread. ITL is a per-token statistic over thousands of intervals and is stable; TTFT varies most between repeats and should be read as an operating range, not a precise value. - Prefix caching is on and the corpus is multi-turn. The engine reported an 84–89 % prefix-cache hit rate, so only ~11 % of the ~22k input tokens per request are actually computed. A workload of independent one-shot prompts will show markedly higher TTFT and much lower session capacity at the same nominal input length.