How Do I Measure Performance?
www.nvidia.com
Best Practices For TensorRT Performance SWE-SWDOCTRT-001-BPRC _vTensorRT 7.0.0|3
These types of wall-clock times can be useful for measuring overall throughput and
latency of the application, and for placing inference times in context within a larger
system.
1.3.CUDA Events
One problem with timing on the host exclusively is that it requires host/device
synchronization. Optimized applications may have many inferences running in parallel
on the device with overlapping data movement. In addition, the synchronization itself
adds some amount of noise to timing measurements.
To help with these issues, CUDA provides an Event API. This API allows you to
place events into CUDA streams that will be time-stamped by the GPU as they are
encountered. Differences in timestamps can then tell you how long different operations
took.
The following example code snippet shows computing the time between two CUDA
events:
cudaEvent_t start, end;
cudaEventCreate(&start);
cudaEventCreate(&end);
cudaEventRecord(start, stream);
context->enqueue(batchSize, &buffers[0], stream, nullptr);
cudaEventRecord(end, stream);
cudaEventSynchronize(end);
float totalTime;
cudaEventElapsedTime(&totalTime, start, end);
TensorRT also includes an optional CUDA event in the method
IExecutionContext::enqueue that will be signaled once the input buffers are
free to be reused. This allows the application to immediately start refilling the input
buffer region for the next inference in parallel with finishing the current inference. For
example:
cudaEvent_t inputReady;
cudaEventCreate(&inputReady);
context->enqueue(batchSize, &buffers[0], stream, &inputReady);
cudaEventSynchronize(inputReady);
// At this point we can refill the input buffers, but output buffers may not be
done
1.4.Built-In TensorRT Profiling
To dig deeper into the performance of inference, it requires more fine-grained timing
measurements within the optimized network. The IExecutionContext interface
class provides a method called setProfiler that allows you to write a custom class
implementing the IProfiler interface.
剩余21页未读，继续阅读
君宝bob
- 粉丝: 197
- 资源: 20
我的内容管理
展开
最新资源
- 构建Cadence PSpice仿真模型库教程
- VMware 10.0安装指南：步骤详解与网络、文件共享解决方案
- 中国互联网20周年必读：影响行业的100本经典书籍
- SQL Server 2000 Analysis Services的经典MDX查询示例
- VC6.0 MFC操作Excel教程：亲测Win7下的应用与保存技巧
- 使用Python NetworkX处理网络图
- 科技驱动：计算机控制技术的革新与应用
- MF-1型机器人硬件与robobasic编程详解
- ADC性能指标解析：超越位数、SNR和谐波
- 通用示波器改造为逻辑分析仪：0-1字符显示与电路设计
- C++实现TCP控制台客户端
- SOA架构下ESB在卷烟厂的信息整合与决策支持
- 三维人脸识别：技术进展与应用解析
- 单张人脸图像的眼镜边框自动去除方法
- C语言绘制图形：余弦曲线与正弦函数示例
- Matlab 文件操作入门：fopen、fclose、fprintf、fscanf 等函数使用详解
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
