DISCLAIMER: This hardware design is provided solely for academic purposes only. It has not undergone formal pre-silicon verification, validation, Emulation, or physical testing, and is not guaranteed for real-world silicon fabrication. The owner / creator accepts no responsibility and/or liability whatsoever for any consequences arising from its use. Use strictly at your own risk.
We are going to build a heterogeneous AI accelerator SoC, which consists of 8 NPU Compute Clusters, Distributed tensor/vector processing, a Centralized Scheduler and Command Processor + 1 x high-bandwidth QoS-aware NoC, 4 x HBM4 stacks, 4 x DDR memory, 8 x high-throughput DMA engines, 1 x PCIe/CXL host connectivity, 8 × 100G Ethernet ports. In addition, we will also describe multiple isolated CPU/firmware domains, and system-wide security, power-management, and RAS infrastructure. This can be used for following workload types described below.
Weights stay resident in HBM, spread across all 32 HBM ports by the AI datapath hash, so the 8 clusters together draw the full ≈7 TB/s (≈0.9 TB/s each through their four datapath ports). The scheduler batches requests from its 64 host queues and dispatches each decode step to all 8 clusters (tensor-parallel inside the chip); the C2C ring carries the per-layer all-reduce.
Sizing (one chip)
70B FP8: ≈100 tok/s for a single user; at 4K context and batch 64 ≈4,000 tok/s aggregate (≈62 tok/s per user, ≈30% of peak compute). Prompt processing ≈6,750 tok/s. 8B FP8: ≈875 tok/s single stream, ≈59,000 tok/s prompt processing. ≈300B FP4: ≈47 tok/s single stream.
Limiting resource
HBM bandwidth during token generation; compute only at very large batch (above ≈130 concurrent sequences for 70B FP8).
Subsystems exercised
NPU (SS1), AI datapath and C2C ring (SS2), scheduler (SS3), HBM (SS7), PCIe for request I/O (SS5).
Recommendations
Serve in FP8 or FP4 weights with FP8 KV cache; use continuous batching in the scheduler; keep tensor parallelism inside the chip and never across the Ethernet link.
Experts are distributed across the 8 clusters (expert parallelism inside the chip); token routing between clusters uses the C2C ring. Frequently-used experts live in HBM; rarely-used experts can live in DDR and be fetched on demand by the tile movers.
Sizing (one chip)
A ≈100B-total / 12B-active MoE in FP8 fits entirely in HBM: ≈580 tok/s single-stream bound. A ≈670B-total / 37B-active MoE in FP4 needs ≈335 GB, so it spans HBM + DDR: ≈380 tok/s bound if all active experts hit HBM, ≈77 tok/s with a 90% HBM hit rate because each DDR miss costs 40× more time.
Limiting resource
HBM capacity; the DDR fetch rate when experts spill out of HBM.
Subsystems exercised
SS1, SS2 (C2C ring for all-to-all), SS7 (HBM and DDR).
Recommendations
Profile expert popularity and pin the hot set in HBM; for the largest MoE models use two chips with experts split between them — the token all-to-all is tens of KB per token per layer, which the Ethernet link carries within the step time at moderate batch sizes (≈9 ms of transfer against ≈24 ms of HBM reads per step at batch 256).
Long Context Inference
Requests with very long prompts or conversations — whole codebases, books, long agent trajectories — where the KV cache, not the weights, dominates memory.
Aspect
Detail
How it runs
The KV cache sits in HBM next to the weights. A three-tier hierarchy extends it: HBM (active sessions), DDR (idle sessions and shared prompt prefixes), and host memory over CXL.mem.
Sizing (one chip)
70B FP8 with FP8 KV: ≈730K tokens of KV in HBM in total. A single 128K-token conversation generates at ≈77 tok/s; at 700K tokens ≈38 tok/s. An 8B model holds ≈2.7M tokens of KV. Restoring a 128K-token session from DDR takes ≈0.12 s versus ≈19 s to recompute it — prefix caching in DDR is about 150× faster.
Limiting resource
HBM capacity for KV; attention compute during prefill grows with the square of context (prefilling 1M tokens on an 8B model takes several minutes).
Use FP8 or FP4 KV cache, paged KV management in the runtime, and DDR as a prefix cache; keep decoding sequences’ KV in HBM because decoding from DDR-resident KV is ≈40× slower.
Disaggregated prefill / decode serving
Splitting each request so that compute-heavy prompt processing runs on one set of chips and bandwidth-heavy generation runs on another, each tuned for its phase.
Aspect
Detail
How it runs
Prefill chips run large batches near peak compute; when a prompt finishes, its KV cache is sent over RoCEv2 RDMA (SS6) directly into a decode chip’s HBM, without host involvement. The scheduler on the decode chip is notified through a doorbell write.
Sizing (one chip)
An 8K-token prompt on 70B FP8 produces 1.34 GB of KV, transferred in ≈14 ms over 97 GB/s Ethernet, while its prefill takes ≈1.2 s — the transfer is ≈1% overhead. One prefill chip generates only ≈1.1 GB/s of KV traffic, so a single Ethernet port pair is enough.
Limiting resource
None significant; the prefill:decode chip ratio is the tuning knob.
Subsystems exercised
SS6 (RDMA), SS3 (doorbells), SS7, SS1.
Recommendations
Pin RDMA traffic to traffic class TC3; use the collective engine’s scatter mode to fan one prompt’s KV out to several decode chips.
Speculative Decoding
A small draft model proposes several tokens and the large model verifies them in one pass, trading cheap compute for fewer bandwidth-bound steps.
Aspect
Detail
How it runs
The scheduler’s per-tenant cluster masks place the draft model (e.g. 1B) on one cluster and the target model on the other seven. The verification pass reads the target weights once for several tokens, raising arithmetic intensity.
Sizing (one chip)
With ≈3 accepted tokens per verification step, single-stream 70B FP8 generation rises from ≈100 tok/s to ≈250 tok/s.
Limiting resource
HBM bandwidth (shared between draft and target); acceptance rate.
Subsystems exercised
SS1, SS3 (cluster masks, barriers), SS7.
Recommendations
Most valuable for latency-sensitive, low-batch serving; at high batch the chip is already compute-efficient and speculation adds less.
Offline batch inference
Throughput-oriented jobs with no latency target: document summarization, data labeling, synthetic data generation, model evaluation, content moderation sweeps.
Aspect
Detail
How it runs
Very large batches make both prefill and decode compute-bound. DMA engines stage input data from host memory or DDR while clusters compute.
Sizing (one chip)
70B FP8: ≈6,750 tok/s, ≈580 million tokens per day; FP4 roughly doubles it. 8B FP8: ≈59,000 tok/s, ≈5 billion tokens per day.
Limiting resource
Tensor compute.
Subsystems exercised
SS1, SS3, SS8, SS7.
Recommendations
Use FP4 weights where quality allows; run at the TDP limit with power management favouring sustained frequency over turbo.
Agentic and tool-calling workloads
AI agents that alternate many short model calls with tool calls (search, code execution, APIs), reusing long shared prompts.
Aspect
Detail
How it runs
Each turn is a short prefill on top of a cached prefix plus a short generation. The hardware doorbell path gives sub-microsecond dispatch; shared prefixes (system prompts, tool schemas, conversation history) are kept in HBM or DDR.
Sizing (one chip)
Time to first token for 2K new tokens on 70B FP8 ≈0.3 s when the prefix is cached; recomputing a 128K-token prefix would add ≈19 s.
Limiting resource
Prefix-cache hit rate and capacity; per-turn latency.
Subsystems exercised
SS3 (doorbells, queues), SS7 (prefix cache), SS5.
Recommendations
Expose prefix caching as a first-class runtime feature; keep tool execution on the host or remote servers — the on-chip CPUs are firmware-only.
Fine-tuning (full, LoRA, QLoRA)
Adapting a pretrained model to a domain or task, from full-parameter updates to parameter-efficient adapters.
Aspect
Detail
How it runs
Weights, gradients, optimizer state and activations live in HBM; optimizer state can spill to DDR. Clusters run forward and backward passes; the vector units apply the optimizer update.
Sizing (one chip)
Full fine-tune of an 8B model in BF16 with Adam: ≈128 GB of state plus activations fits in HBM; ≈7,900 tok/s (1B tokens in ≈35 hours), ≈13,800 tok/s with FP8 training. 13B full fine-tune needs ≈208 GB of state, so the optimizer moves to DDR. LoRA on a 70B FP8 base fits on one chip at ≈2,400 tok/s; QLoRA with an FP4 base leaves even more room.
Limiting resource
HBM capacity for full fine-tuning; compute for LoRA.
Subsystems exercised
SS1, SS7 (HBM, DDR optimizer offload), SS8.
Recommendations
Use activation checkpointing; offload optimizer state to DDR above ≈10B parameters; scale data-parallel across chips for faster turnaround.
Reinforcement-learning post-training
RLHF and RL with verifiable rewards, where the model generates many rollouts, they are scored, and the model is updated — repeatedly.
Aspect
Detail
How it runs
Clusters are split between rollout generation (decode-bound) and policy training (compute-bound) on the same chip. Because both copies of the weights are in the same HBM, syncing new policy weights to the generators is an on-chip copy.
Sizing (one chip)
Weight sync for an 8B policy takes ≈5 ms on chip versus ≈0.16 s over Ethernet. Rollout and training throughput follow W1 and W8.
Limiting resource
Balance between generation and training phases.
Subsystems exercised
SS1, SS3 (cluster masks), SS7, SS6 for multi-chip jobs.
Recommendations
Re-balance the cluster split dynamically as rollout lengths change; reward models can run on the same chip as a third partition.
Knowledge Distillation
Training a smaller student model to imitate a larger teacher model’s outputs.
Aspect
Detail
How it runs
The teacher runs inference and the student trains, both on the same chip, so teacher logits never leave HBM.
Sizing (one chip)
70B FP8 teacher plus 8B BF16 student: ≈3,600 tok/s. Memory is tight (≈70 GB teacher + ≈128 GB student state); an FP4 teacher (≈35 GB) makes it fit comfortably.
Limiting resource
Compute; HBM capacity with an FP8 teacher.
Subsystems exercised
SS1, SS7.
Recommendations
Quantize the teacher to FP4 or precompute teacher outputs offline for very large teachers.
Multi-model fleets
Running many different, smaller models side by side — for example a chat model, an embedding model, a reranker and a speech model — on one chip.
Aspect
Detail
How it runs
Each model is pinned to a subset of clusters. In NUMA interleave mode each pair of clusters gets its own HBM stack (≈2 TB/s, 48 GB), giving each model predictable performance.
Sizing (one chip)
Eight 8B-class FP8 models, one per cluster, each with ≈236 TFLOPS FP8 and ≈24 GB of HBM; or four models with a full HBM stack each.
Limiting resource
HBM capacity and bandwidth per partition.
Subsystems exercised
SS1, SS2 (NUMA mode), SS3 (cluster masks).
Recommendations
Pair this with W11 so each model can belong to a different tenant.
Multi-tenant and Confidential Compute
Selling slices of the chip to different customers, including customers who require that neither the cloud operator nor other tenants can see their models or data.
Aspect
Detail
How it runs
SR-IOV gives up to 63 virtual functions; the scheduler supports 16 tenant contexts with weighted fair share and per-tenant cluster masks; NoC firewalls enforce hardware domain IDs. For confidential computing, PCIe/CXL IDE encrypts the host link, DDR uses per-tenant inline encryption keys, TDISP locks device assignment, and SPDM attestation proves the firmware state to the tenant.
Sizing (one chip)
Up to 8 hard partitions (one per cluster) or 16 soft tenants; encryption runs at line rate.
Limiting resource
Partition granularity (one cluster is the smallest hard slice); HBM is protected by isolation, not encryption.
Diffusion Transformer (Image and video generation)
Text-to-image, text-to-video and image editing with diffusion transformers (DiT) or similar denoising models.
Aspect
Detail
How it runs
Each denoising step is a full transformer pass over all latent tokens — compute-bound and well suited to FP8. Large HBM holds long video latent sequences and their activations.
Sizing (one chip)
A ≈12B image DiT at 1 megapixel (≈4K latent tokens), 28 steps with classifier-free guidance: ≈6 s per image; distilled few-step models under 1 s. A ≈14B video DiT generating 5 s of 720p (≈108K latent tokens), 50 steps: ≈11 minutes, dominated by attention.
Limiting resource
Tensor compute (attention dominates for video).
Subsystems exercised
SS1, SS7.
Recommendations
Split long video sequences across chips with sequence parallelism; the Ethernet traffic for this is modest relative to compute.
Computer Vision
Image classification, object detection, segmentation, pose estimation and 3D imaging (medical CT/MRI volumes, satellite imagery).
Aspect
Detail
How it runs
Convolutions and vision transformers map onto the tensor cores; large 3D volumes fit in HBM.
Sizing (one chip)
ResNet-50-class inference: ≈90,000 images per second at 40% utilization.
Limiting resource
Compute on chip; JPEG decoding must be done on the host because there is no hardware image decoder.
Subsystems exercised
SS1, SS5 (image ingest), SS8.
Recommendations
Ingest decoded tensors over PCIe; see W29 for adding hardware decoders.
Vision-language models, OCR and document AI
Models that read images and documents alongside text: visual question answering, chart and table understanding, invoice and form extraction.
Aspect
Detail
How it runs
A vision encoder converts each image into hundreds to thousands of tokens, then the language model processes them as a long prompt — a prefill-heavy workload.
Sizing (one chip)
A 1,000-page document (≈1.5M visual tokens) through an 8B VLM: ≈25 s.
Limiting resource
Compute (prefill).
Subsystems exercised
SS1, SS7, SS5.
Recommendations
Run the vision encoder and language model on separate cluster groups and pipeline pages through them.
Speech & Audio
Speech recognition, translation, diarization, text-to-speech and audio or music generation.
Aspect
Detail
How it runs
Encoder-decoder ASR models are small and compute-bound; TTS and audio-codec language models behave like small LLMs.
Sizing (one chip)
A ≈1.5B-parameter ASR model processes ≈380 thirty-second chunks per second — ≈11,000× real time, i.e. thousands of concurrent live streams.
Limiting resource
Compute; host-side audio I/O at very high stream counts.
Subsystems exercised
SS1, SS3 (many small queues).
Recommendations
Use many scheduler queues with small batches for streaming latency.
Embeddings, reranking and retrieval-augmented generation (RAG)
Converting documents and queries into vectors, re-scoring retrieved candidates, and generating answers grounded in retrieved text.
Aspect
Detail
How it runs
Embedding and reranker models are prefill-only workloads. The full RAG pipeline — embed the query, search (W18), rerank, generate (W1) — can run on one chip without leaving HBM.
Sizing (one chip)
A 7B embedding model processes ≈67,500 tok/s, embedding a 1-billion-token corpus in ≈4 hours.
Limiting resource
Compute
Subsystems exercised
SS1, SS7, SS3.
Recommendations
Co-locate retrieval and generation on the same chip to avoid host round-trips.
Vector Search
Finding the nearest neighbors of a query vector among millions to billions of stored vectors.
Aspect
Detail
How it runs
Exact search over a batch of queries is a matrix multiply against the stored vectors and runs on the tensor cores. Compressed indexes (IVF-PQ) hold far more vectors per byte.
Sizing (one chip)
≈180M vectors of 1024 dimensions in INT8 fit in HBM; batched exact search over all of them ≈2,500 queries/s; one full scan takes ≈27 ms. With 64-byte PQ codes, ≈2.8 billion vectors fit in HBM, plus more in DDR.
Limiting resource
HBM capacity; graph-based indexes (HNSW) run less efficiently because of random access.
Subsystems exercised
SS1, SS7, SS8.
Recommendations
Prefer brute-force or IVF-style indexes that map onto matrix multiplies; batch queries.
Recommendation and ranking
Ads click-through prediction, feed and search ranking, and newer transformer-based generative recommenders.
Aspect
Detail
How it runs
Huge embedding tables are spread across HBM (hot rows), DDR (warm rows) and host memory over CXL.mem (cold rows); lookups are gathered by the DMA and tile movers, and the dense interaction and MLP layers run on the tensor cores.
Sizing (one chip)
Up to ≈700 GB of tables on chip (HBM + DDR) plus CXL host memory; on the order of 20 billion 128-byte row lookups per second from HBM and ≈0.5 billion from DDR.
Limiting resource
Memory capacity and random-access rate; for multi-chip training of very large tables, the Ethernet all-to-all.
Subsystems exercised
SS7, SS8, SS5 (CXL.mem), SS6, SS1.
Recommendations
Cache hot embedding rows in HBM; use FP8 or INT8 embeddings; keep embedding sharding within as few chips as possible.
Time-series and financial ML
Forecasting, anomaly and fraud detection, risk-model inference, and market-data models that need low latency.
Aspect
Detail
How it runs
Small to medium models with strict latency. Requests can arrive directly over RoCE into device memory and trigger the scheduler without a host round-trip.
Sizing (one chip)
Thousands of concurrent small models or streams; latency dominated by network and model depth, with sub-microsecond dispatch.
Limiting resource
End-to-end latency; Monte-Carlo risk models that need FP64 are not supported (see W28).
Subsystems exercised
SS6 (RDMA ingest), SS3, SS1.
Recommendations
Keep models resident and use dedicated scheduler queues with the highest priority class.
Pretraining small and mid-size models (up to ≈13B)
Training language, vision or multimodal models from scratch where each chip holds a full model replica.
Aspect
Detail
How it runs
Data parallelism across chips: each chip trains on its own data shard and gradients are averaged with the Ethernet collective engine, overlapped with the backward pass.
Sizing (one chip)
8B in BF16 ≈7,900 tok/s per chip. Gradient all-reduce for 8B (16 GB) takes ≈0.33 s per step, easily hidden. 1 trillion tokens on 64 chips: ≈23 days in BF16, ≈13 days in FP8.
Limiting resource
Compute
Subsystems exercised
SS1, SS6 (collectives), SS7, SS8.
Recommendations
Use FP8 training with BF16 master weights; checkpoint to DDR and then to the host asynchronously.
Training models with hundreds of billions to trillions of parameters on thousands of chips.
Aspect
Detail
Why it is a Poor Fit
These jobs need tensor parallelism across chips and very large FSDP groups; at 800 Gb/s per chip the network, not compute, sets step time (see W22). Systems built for this use scale-up fabrics with many times more bandwidth per chip.
What would be needed
A scale-up interconnect port (e.g. UALink-class) and switch topology in a derivative chip.
Traditional FP64 HPC
Computational fluid dynamics, structural analysis, dense FP64 linear algebra, molecular dynamics with classical force fields.
Aspect
Detail
Why it is a Poor Fit
These jobs need tensor parallelism across chips and very large FSDP groups; at 800 Gb/s per chip the network, not compute, sets step time (see W22). Systems built for this use scale-up fabrics with many times more bandwidth per chip.
What would be needed
FP64 vector units, which would cost significant area for a market this product does not target.
End-to-end video analytics
Decoding many camera or video streams and running detection or understanding models on every frame.
Aspect
Detail
Why it is a Poor Fit
There is no hardware video or JPEG decoder, so frames must be decoded on the host. PCIe can carry ≈37,000 decoded 1080p frames per second, and the chip can infer on tens of thousands per second, but host CPUs typically decode far fewer — the host becomes the bottleneck.
What would be needed
4–8 hardware video/JPEG decode engines (H.264, H.265, AV1) attached to the NoC; this would move the workload to the strong-fit category.
General-purpose and branch-heavy code
Tree search, sorting-heavy pipelines, homomorphic encryption, genome alignment, and other irregular, control-flow-heavy algorithms.
Aspect
Detail
Why it is a Poor Fit
The NPU is optimized for dense tensor operations; the RISC-V CPU domains run firmware only and are not exposed to user code. Workloads with data-dependent branches, 64-bit modular arithmetic or dynamic-programming recurrences run at a small fraction of peak.
What would be needed
A user-programmable many-core CPU complex, which is outside this product’s scope.
Documentation
High Level Architecture
Target Workloads (Fit Summary)
Virtual Platform (Rust)
Design Environment
Verification Environment
Design
Verification
Emulation
Physical Design
Post-Si Validation
Virtual Platform
Click Here to go to github repository Follow README
Source Code
Click Here to go to github repository Follow README
Simulation
Click Here to go to github repository Follow README. License required
Emulation
Click Here to go to github repository Follow README. License required
