Compute Express Link 3.0 规范详解：协议规范和实现细节

需积分: 18 136 浏览量更新于2024-06-26 收藏 10.74MB PDF 举报

Compute Express Link（CXL）3.0 Specification Compute Express Link（CXL）是一种高速的点对点互连技术，旨在提供高带宽、低延迟的数据传输解决方案。CXL 3.0 Specification是最新的协议版本，提供了更高的带宽和更低的延迟。 CXL技术基于PCI Express（PCIe）协议，继承了PCIe的高带宽和低延迟特性，同时也引入了新的技术和功能来满足不断增长的数据传输需求。CXL 3.0 Specification相比于之前的版本，带来了许多新的特性和改进，例如： 1. 高速数据传输：CXL 3.0 Specification支持高达3200 MT/s的数据传输速率，远高于PCIe 4.0的速率。 2. 低延迟：CXL 3.0 Specification的延迟时间缩短了近一半，提供了更快的数据传输体验。 3. 增强的可扩展性：CXL 3.0 Specification引入了新的可扩展性机制，允许更容易地集成新的设备和技术。 4. 改进的可靠性：CXL 3.0 Specification引入了新的错误校正机制，提高了数据传输的可靠性。 CXL 3.0 Specification的主要应用场景包括： 1. 数据中心：CXL 3.0 Specification可以用于数据中心的高速互连，提供高带宽和低延迟的数据传输解决方案。 2. 人工智能：CXL 3.0 Specification可以用于人工智能和机器学习应用，提供高速的数据传输和低延迟的计算能力。 3. 高性能计算：CXL 3.0 Specification可以用于高性能计算应用，提供高速的数据传输和低延迟的计算能力。 CXL 3.0 Specification是一种高速、可靠的互连技术，旨在满足不断增长的数据传输需求。它可以广泛应用于数据中心、人工智能、高性能计算等领域，提供高速的数据传输和低延迟的计算能力。

Contents

August 1, 2022 Compute Express Link Specification

Revision 3.0, Version 1.0 16

14.11.3.4 Establish CXL.cachemem IDE (SHDA) Latency-Optimized 256B Flit

Mode.............................................................................. 811

14.11.3.5 Establish CXL.cachemem IDE (SHDA) 68B Flit Mode.............. 812

14.11.3.6 Locally Generate IV (SHDA)............................................... 813

14.11.3.7 Data Encryption – Decryption and Integrity Testing

with Containment Mode for MAC Generation and Checking..... 814

14.11.3.8 Data Encryption – Decryption and Integrity Testing

with Skid Mode for MAC Generation and Checking ................ 814

14.11.3.9 Key Refresh..................................................................... 815

14.11.3.10 Early MAC Termination...................................................... 815

14.11.3.11 Error Handling ................................................................. 816

14.11.4 Certificate Format/Certificate Chain ..................................................... 821

14.11.5 Security RAS .................................................................................... 822

14.11.5.1 CXL.io Poison Inject from Device ........................................ 822

14.11.5.2 CXL.cache Poison Inject from Device .................................. 823

14.11.5.3 CXL.cache CRC Inject from Device...................................... 824

14.11.5.4 CXL.mem Poison Injection................................................. 826

14.11.5.5 CXL.mem CRC Injection .................................................... 826

14.11.5.6 Flow Control Injection....................................................... 827

14.11.5.7 Unexpected Completion Injection ....................................... 828

14.11.5.8 Completion Timeout Injection ............................................ 830

14.11.5.9 Memory Error Injection and Logging ................................... 831

14.11.5.10 CXL.io Viral Inject from Device........................................... 832

14.11.5.11 CXL.cache Viral Inject from Device ..................................... 833

14.11.6 Security Protocol and Data Model ........................................................ 835

14.11.6.1 SPDM GET_VERSION ........................................................ 835

14.11.6.2 SPDM GET_CAPABILITIES ................................................. 836

14.11.6.3 SPDM NEGOTIATE_ALGORITHMS ....................................... 837

14.11.6.4 SPDM GET_DIGESTS ........................................................ 838

14.11.6.5 SPDM GET_CERTIFICATE................................................... 839

14.11.6.6 SPDM CHALLENGE............................................................ 840

14.11.6.7 SPDM GET_MEASUREMENTS Count..................................... 841

14.11.6.8 SPDM GET_MEASUREMENTS All ......................................... 842

14.11.6.9 SPDM GET_MEASUREMENTS Repeat with Signature .............. 843

14.11.6.10 SPDM CHALLENGE Sequences............................................ 844

14.11.6.11 SPDM ErrorCode Unsupported Request................................ 846

14.11.6.12 SPDM Major Version Invalid............................................... 846

14.11.6.13 SPDM ErrorCode UnexpectedRequest .................................. 847

14.12 Reliability, Availability, and Serviceability............................................................ 847

14.12.1 RAS Configuration............................................................................. 849

14.12.1.1 AER Support.................................................................... 849

14.12.1.2 CXL.io Poison Injection from Device to Host......................... 850

14.12.1.3 CXL.cache Poison Injection ................................................ 850

14.12.1.4 CXL.cache CRC Injection ................................................... 852

14.12.1.5 CXL.mem Link Poison Injection .......................................... 854

14.12.1.6 CXL.mem CRC Injection .................................................... 854

14.12.1.7 Flow Control Injection....................................................... 855

14.12.1.8 Unexpected Completion Injection ....................................... 856

14.12.1.9 Completion Timeout ......................................................... 856

14.12.1.10 CXL.mem Media Poison Injection........................................ 857

14.12.1.11 CXL.mem LSA Poison Injection........................................... 857

14.12.1.12 CXL.mem Device Health Injection....................................... 858

14.13 Memory Mapped Registers................................................................................ 858

14.13.1 CXL Capability Header ....................................................................... 858

14.13.2 CXL RAS Capability Header................................................................. 859

14.13.3 CXL Security Capability Header ........................................................... 859

14.13.4 CXL Link Capability Header................................................................. 860

14.13.5 CXL HDM Decoder Capability Header ................................................... 860

14.13.6 CXL Extended Security Capability Header ............................................. 861

Evaluation Copy

Contents

August 1, 2022 Compute Express Link Specification

Revision 3.0, Version 1.0 17

14.13.7 CXL IDE Capability Header ................................................................. 861

14.13.8 CXL HDM Decoder Capability Register .................................................. 862

14.13.9 CXL HDM Decoder Commit ................................................................. 862

14.13.10 CXL HDM Decoder Zero Size Commit ................................................... 863

14.13.11 CXL Snoop Filter Capability Header...................................................... 863

14.13.12 CXL Device Capabilities Array Register ................................................. 864

14.13.13 Device Status Registers Capabilities Header Register ............................. 865

14.13.14 Primary Mailbox Registers Capabilities Header Register .......................... 865

14.13.15 Secondary Mailbox Registers Capabilities Header Register ...................... 865

14.13.16 Memory Device Status Registers Capabilities Header Register ................. 866

14.13.17 CXL Timeout and Isolation Capability Header ........................................ 866

14.13.18 CXL.cachemem Extended Register Header............................................ 867

14.13.19 CXL BI Route Table Capability Header.................................................. 867

14.13.20 CXL BI Decoder Capability Header....................................................... 868

14.13.21 CXL Cache ID Route Table Header....................................................... 868

14.13.22 CXL Cache ID Decoder Capability Header ............................................. 869

14.13.23 CXL Extended HDM Decoder Capability Header...................................... 869

14.14 Memory Device Tests....................................................................................... 870

14.14.1 DVSEC CXL Range 1 Size Low Registers ............................................... 870

14.14.2 DVSEC CXL Range 2 Size Low Registers ............................................... 871

14.15 Sticky Register Tests ....................................................................................... 871

14.15.1 Sticky Register Test .......................................................................... 871

14.16 Device Capability and Test Configuration Control ................................................. 874

14.16.1 CXL Device Test Capability Advertisement ............................................ 874

14.16.2 Debug Capabilities in Device............................................................... 875

14.16.2.1 Error Logging .................................................................. 875

14.16.2.2 Event Monitors................................................................. 876

14.16.3 Compliance Mode DOE....................................................................... 877

14.16.3.1 Compliance Mode Capability .............................................. 877

14.16.3.2 Compliance Mode Status ................................................... 879

14.16.3.3 Compliance Mode Halt All .................................................. 879

14.16.3.4 Compliance Mode Multiple Write Streaming.......................... 880

14.16.3.5 Compliance Mode Producer-Consumer................................. 881

14.16.3.6 Test Algorithm 1b Multiple Write Streaming with Bogus Writes 881

14.16.3.7 Inject Link Poison............................................................. 882

14.16.3.8 Inject CRC ...................................................................... 883

14.16.3.9 Inject Flow Control ........................................................... 883

14.16.3.10 Toggle Cache Flush .......................................................... 884

14.16.3.11 Inject MAC Delay ............................................................. 884

14.16.3.12 Insert Unexpected MAC..................................................... 885

14.16.3.13 Inject Viral ...................................................................... 886

14.16.3.14 Inject ALMP in Any State................................................... 886

14.16.3.15 Ignore Received ALMP ...................................................... 887

14.16.3.16 Inject Bit Error in Flit ........................................................ 887

14.16.3.17 Inject Memory Device Poison ............................................. 888

A Taxonomy.............................................................................................................. 892

A.1 Accelerator Usage Taxonomy ............................................................................ 892

A.2 Bias Model Flow Example – From CPU ................................................................ 893

A.3 CPU Support for Bias Modes.............................................................................. 894

A.3.1 Remote Snoop Filter ............................................................................. 894

A.3.2 Directory in Accelerator-attached Memory ............................................... 894

A.4 Giant Cache Model........................................................................................... 895

B Unordered I/O to Support Peer-to-Peer Directly to HDM-DB.................................. 897

C Memory Protocol Tables ........................................................................................ 900

C.1 HDM-D and HDM-DB Requests .......................................................................... 901

Evaluation Copy

Figures

August 1, 2022 Compute Express Link Specification

Revision 3.0, Version 1.0 18

C.1.1 Forward Flows for HDM-D...................................................................... 905

C.1.2 BISnp for HDM-DB ............................................................................... 907

C.2 HDM-H Requests ............................................................................................. 909

C.3 HDM-D/HDM-DB RwD ...................................................................................... 910

C.4 HDM-H RwD ................................................................................................... 911

Figures

1-1 Conceptual Diagram of Device Attached to Processor via CXL................................ 49

1-2 Fan-out and Pooling Enabled by Switches........................................................... 50

1-3 Direct Peer-to-Peer Access to an HDM Memory by PCIe/CXL Devices without Going

through the Host ............................................................................................ 51

1-4 Shared Memory across Multiple Virtual Hierarchies .............................................. 51

1-5 CPU Flex Bus Port Example .............................................................................. 52

1-6 Flex Bus Usage Model Examples........................................................................ 53

1-7 Remote Far Memory Usage Model Example......................................................... 53

1-8 CXL Downstream Port Connections.................................................................... 54

1-9 Conceptual Diagram of Flex Bus Layering .......................................................... 55

2-1 CXL Device Types ........................................................................................... 58

2-2 Type 1 Device - Device with Cache.................................................................... 59

2-3 Type 2 Device - Device with Memory ................................................................. 60

2-4 Type 2 Device - Host Bias ................................................................................ 61

2-5 Type 2 Device - Device Bias ............................................................................. 62

2-6 Type 3 Device - Memory Expander .................................................................... 63

2-7 Head-to-LD Mapping in MH-SLDs ...................................................................... 68

2-8 Head-to-LD Mapping in MH-MLDs ...................................................................... 68

3-1 Flex Bus Layers - CXL.io Transaction Layer Highlighted ....................................... 72

3-2 CXL Power Management Messages Packet Format - Non-Flit Mode ......................... 74

3-3 CXL Power Management Messages Packet Format - Flit Mode................................ 74

3-4 Power Management Credits and Initialization ..................................................... 77

3-5 CXL EFN Messages Packet Format - Non-Flit Mode............................................... 79

3-6 CXL EFN Messages Packet Format - Flit Mode...................................................... 79

3-7 ATS 64-bit Request with CXL Indication - Non-Flit Mode ....................................... 80

3-8 Valid .io TLP Formats on PBR Links ................................................................... 83

3-9 CXL.cache Channels ....................................................................................... 84

3-10 CXL.cache Read Behavior................................................................................. 90

3-11 CXL.cache Read0 Behavior ............................................................................... 91

3-12 CXL.cache Device to Host Write Behavior .......................................................... 92

3-13 CXL.cache WrInv Transaction .......................................................................... 93

3-14 WOWrInv/F with FastGO/ExtCmp ..................................................................... 94

3-15 CXL.cache Read0-Write Semantics .................................................................... 95

3-16 CXL.cache Snoop Behavior ............................................................................. 103

3-17 CXL.mem Channels for Devices ...................................................................... 111

3-18 CXL.mem Channels for Hosts ......................................................................... 112

3-19 Flows for Back-Invalidation Snoops on CXL.mem Legend ................................... 138

3-20 Example BISnp with Blocking of M2S Req......................................................... 138

3-21 BISnp Early Conflict ..................................................................................... 139

3-22 BISnp Late Conflict ...................................................................................... 140

3-23 Block BISnp with Block Response ................................................................... 141

Evaluation Copy

Figures

August 1, 2022 Compute Express Link Specification

Revision 3.0, Version 1.0 19

3-24 Block BISnp with Cacheline Response ............................................................. 142

3-25 Flows for Type 1 Devices and Type 2 Devices Legend ........................................ 143

3-26 Example Cacheable Read from Host ................................................................ 143

3-27 Example Read for Ownership from Host ........................................................... 144

3-28 Example Non Cacheable Read from Host .......................................................... 145

3-29 Example Ownership Request from Host - No Data Required ................................ 145

3-30 Example Flush from Host ............................................................................... 146

3-31 Example Weakly Ordered Write from Host ....................................................... 147

3-32 Example Write from Host with Invalid Host Caches ............................................ 148

3-33 Example Write from Host with Valid Host Caches............................................... 149

3-34 Example Device Read to Device-attached Memory (HDM-D) ............................... 150

3-35 Example Device Read to Device-attached Memory (HDM-DB).............................. 151

3-36 Example Device Write to Device-Attached Memory in Host Bias (HDM-D).............. 152

3-37 Example Device Write to Device-attached Memory in Host Bias (HDM-DB)............ 153

3-38 Example Device Write to Device-attached Memory ........................................... 154

3-39 Example Host to Device Bias Flip (HDM-D) ...................................................... 155

3-40 Example MemSpecRd ................................................................................... 156

3-41 Read from Host to HDM-H.............................................................................. 156

3-42 Write from Host to All HDM Regions ................................................................ 157

4-1 Flex Bus Layers - CXL.io Link Layer Highlighted................................................. 159

4-2 Flex Bus Layers - CXL.cache + CXL.mem Link Layer Highlighted.......................... 161

4-3 CXL.cachemem Protocol Flit Overview.............................................................. 162

4-4 CXL.cachemem All Data Flit Overview .............................................................. 163

4-5 Example of a Protocol Flit from Device to Host .................................................. 164

4-6 H0 - H2D Req + H2D Rsp............................................................................... 170

4-7 H1 - H2D Data Header + H2D Rsp + H2D Rsp .................................................. 170

4-8 H2 - H2D Req + H2D Data Header .................................................................. 171

4-9 H3 - 4 H2D Data Header ................................................................................ 171

4-10 H4 - M2S RwD Header ................................................................................... 171

4-11 H5 - M2S Req............................................................................................... 172

4-12 H6 - MAC..................................................................................................... 172

4-13 G0 - H2D/M2S Data ...................................................................................... 172

4-14 G0 - M2S Byte Enable ................................................................................... 173

4-15 G1 - 4 H2D Rsp ............................................................................................ 173

4-16 G2 - H2D Req + H2D Data Header + H2D Rsp .................................................. 173

4-17 G3 - 4 H2D Data Header + H2D Rsp................................................................ 174

4-18 G4 - M2S Req + H2D Data Header .................................................................. 174

4-19 G5 - M2S RwD Header + H2D Rsp................................................................... 174

4-20 H0 - D2H Data Header + 2 D2H Rsp + S2M NDR............................................... 175

4-21 H1 - D2H Req + D2H Data Header .................................................................. 175

4-22 H2 - 4 D2H Data Header + D2H Rsp................................................................ 176

4-23 H3 - S2M DRS Header + S2M NDR .................................................................. 176

4-24 H4 - 2 S2M NDR ........................................................................................... 176

4-25 H5 - 2 S2M DRS Header................................................................................. 177

4-26 H6 - MAC..................................................................................................... 177

4-27 G0 - D2H/S2M Data ...................................................................................... 177

4-28 G0 - D2H Byte Enable ................................................................................... 178

4-29 G1 - D2H Req + 2 D2H Rsp............................................................................ 178

4-30 G2 - D2H Req + D2H Data Header + D2H Rsp .................................................. 178

4-31 G3 - 4 D2H Data Header................................................................................ 179

Evaluation Copy

Figures

August 1, 2022 Compute Express Link Specification

Revision 3.0, Version 1.0 20

4-32 G4 - S2M DRS Header + 2 S2M NDR ............................................................... 179

4-33 G5 - 2 S2M NDR ........................................................................................... 179

4-34 G6 - 3 S2M DRS Header ................................................................................ 180

4-35 LLCRD Flit Format (Only Slot 0 is Valid; Others are Reserved)............................. 185

4-36 RETRY Flit Format (Only Slot 0 is Valid; Others are Reserved)............................. 185

4-37 IDE Flit Format (Only Slot 0 is Valid; Others are Reserved)................................. 185

4-38 INIT Flit Format (Only Slot 0 is Valid; Others are Reserved)................................ 186

4-39 Retry Buffer and Related Pointers.................................................................... 191

4-40 CXL.cachemem Replay Diagram...................................................................... 196

4-41 Standard 256B Flit ....................................................................................... 200

4-42 Latency-Optimized (LOpt) 256B Flit ................................................................ 201

4-43 256B Packing: Slot and Subset Definition ......................................................... 204

4-44 256B Packing: G0/H0/HS0 HBR Messages ........................................................ 205

4-45 256B Packing: G0/H0 PBR Messages ............................................................... 206

4-46 256B Packing: G1/H1/HS1 HBR Messages ........................................................ 206

4-47 256B Packing: G1/H1 PBR Messages ............................................................... 207

4-48 256B Packing: G2/H2/HS2 HBR Messages ........................................................ 207

4-49 256B Packing: G2/H2 PBR Messages ............................................................... 208

4-50 256B Packing: G3/H3/HS3 HBR Messages ........................................................ 208

4-51 256B Packing: G3/H3 PBR Messages ............................................................... 209

4-52 256B Packing: G4/H4/HS4 HBR Messages ........................................................ 209

4-53 256B Packing: G4/H4 PBR Messages ............................................................... 210

4-54 256B Packing: G5/H5/HS5 HBR Messages ........................................................ 210

4-55 256B Packing: G5/H5 PBR Messages ............................................................... 211

4-56 256B Packing: G6/H6/HS6 HBR Messages ........................................................ 211

4-57 256B Packing: G6/H6 PBR Messages ............................................................... 212

4-58 256B Packing: G7/H7/HS7 HBR Messages ........................................................ 212

4-59 256B Packing: G7/H7 PBR Messages ............................................................... 213

4-60 256B Packing: G12/H12/HS12 HBR Messages................................................... 213

4-61 256B Packing: G12/H12 PBR Messages ............................................................ 214

4-62 256B Packing: G13/H13/HS13 HBR Messages................................................... 214

4-63 256B Packing: G13/H13 PBR Messages ............................................................ 215

4-64 256B Packing: G14/H14/HS14 HBR Messages................................................... 215

4-65 256B Packing: G14/H14 PBR Messages ............................................................ 216

4-66 256B Packing: G15/H15/HS15 HBR Messages................................................... 216

4-67 256B Packing: G15/H15 PBR Messages ............................................................ 217

4-68 Header Slot Decode Example ......................................................................... 218

4-69 256B Packing: H8/HS8 Link Layer Control Message Slot Format .......................... 224

4-70 Viral Error Message Injection Standard 256B Flit ............................................... 225

4-71 Viral Error Message Injection LOpt 256B Flit ..................................................... 225

5-1 Flex Bus Layers - CXL ARB/MUX Highlighted ..................................................... 228

5-2 Entry to Active Protocol Exchange ................................................................... 233

5-3 Example Status Exchange .............................................................................. 234

5-4 CXL Entry to Active Example Flow ................................................................... 236

5-5 CXL Entry to PM State Example....................................................................... 237

5-6 Successful PM Entry following PM Retry............................................................ 238

5-7 PM Abort before Downstream Port PM Acceptance ............................................. 238

5-8 PM Abort after Downstream Port PM Acceptance ............................................... 239

5-9 Example of a PMNAK Flow.............................................................................. 240

5-10 CXL Recovery Exit Example Flow..................................................................... 242

Evaluation Copy

剩余910页未读，继续阅读

彭本华

粉丝: 4
资源: 5

Compute Express Link 3.0 规范详解：协议规范和实现细节

CXL spec 3.0 Compute Express LInk 3.0

Compute EXpress Link（CXL）3.0规格说明书

CXL3.0协议规范文档

cxl 3.0 specification

CXL3.1 ， CXL Specification-rev3p1-ver1p0-2023August9

cxl2.0协议-CXL-2.0-Specification-r1.0-2020Oct26

CXL Fabric Manager API over MCTP Binding Specification

"开放的CXL1.1协议规范文档

CXL Type 3 设备组件命令接口通过 MCTP 绑定规范

CXL协议规范3.0最新版本分享

最新资源