Ascend C Custom Operator Runtime Compilation and Offline Model Sinking Sample【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/geSample OverviewGraph construction entry:GE graph construction ATC offline compilationOperator programming language:Ascend CCompilation method:RTC operator runtime compilationModel sinking capability:Supports sinking to om offline modelCore pipeline:Ascend C kernel source code RTC operator runtime compilation - GE deliverables - AIR - ATC offline compilation - om offline model - ACL executionDifference from other samples: This sample completes Ascend C kernel runtime compilation inCompilecallback and serializes compilation products into om offline model, which can be deserialized and used in subsequent model execution phases.Multi-shape note: This sample mainly provides a multi-shape/multi-kernel processing framework that manages and sinks multiple kernel binaries by shape key.Compile-time input note: The sample reads input byTensorlayout inCompilephase, so it can simultaneously obtain metadata such asShape,DataType,Format; current key is still only generated by shape.This sample demonstrates a minimum runnable GE custom operator sinking pipeline: first construct graph to generateAIR, then generateom offline modelthroughATC offline compilation, finally load and execute byACLprogram. The AddCustom in the sample simultaneously inheritsCompilableOp,PortableOpandShapeInferOp, so it can not only complete Ascend C kernel runtime compilation through RTC inCompilecallback, but also serialize compilation results to final model, and complete Shape and DataType inference in the same implementation class.Applicable ScenariosWant to see Ascend C custom operator sample inGE graph construction ATC offline compilation om offline modelpipeline.Want to reference complete flow of custom operator RTC runtime compilation, model sinking and final model execution.Want to see minimum runnable pipeline ofAIR - ATC offline compilation - om offline model - ACL.Not suitable for understandingPyTorch/TorchAirorTensorFlow/Tritonscenarios.PrerequisitesCANNCANN environment has been correctly installed and configured, e.g., executedsource ${ASCEND_HOME_PATH}/set_env.sh.atccommand is available.Current environment hasACL,GE,Graphrelated headers and libraries.Follow installation guide to complete toolkit and ops package installation.Framework and PluginsThis sample does not depend on PyTorch or TensorFlow.Environment VariablesASCEND_HOME_PATHASCEND_CUSTOM_OPP_PATHwill be automatically appended withoutput/inrun.sh, and custom operator deliverables will be placed inoutput/op_graph/lib/linux/x86_64/oroutput/op_graph/lib/linux/aarch64/Additional DependenciescmakegQuick RunExecute inexamples/custom_op/compilable_add_customdirectory:Recommended Methodbash run.shrun.shwill automatically appendoutput/toASCEND_CUSTOM_OPP_PATHafter generatingoutput/op_graph/lib/os/arch/libcust_opapi.so. If successful,output/single_add.air,output/single_add.omwill be generated, and terminal will printModel executed successfully!andFirst element of output: 2.000000.Step-by-step MethodIf you want to manually observe artifacts at each stage, you can execute:cmake -S . -B build -DCMAKE_BUILD_TYPERelease cmake --build build -j$(nproc) cmake --install build export ASCEND_CUSTOM_OPP_PATH$(pwd)/output:$ASCEND_CUSTOM_OPP_PATH cd output ../build/compilable_add_graph_build cd .. atc \ --model$(pwd)/output/single_add.air \ --framework1 \ --output$(pwd)/output/single_add \ --soc_versionAscend910B1 cd build ./compilable_add_model_exec ../output/single_add.om cd ..Whereexport ASCEND_CUSTOM_OPP_PATH$(pwd)/output:$ASCEND_CUSTOM_OPP_PATHis used to add custom operator package root directory to environment variable, then GE/ATC will load deliverables according tooutput/op_graph/lib/os/arch/libcust_opapi.sorule.Directory Structure and Key Filescompilable_add_custom ├── CMakeLists.txt ├── README.md ├── run.sh ├── ge │ ├── add_custom_kernel.cpp // Ascend C kernel source code read during operator runtime compilation │ ├── add_custom.h // AddCustom unique proto definition, shared by deliverables and graph construction side │ ├── custom_op.cpp // CompilableOp/PortableOp/ShapeInferOp main process implementation │ └── utils │ ├── compile_utils.cpp // Compile-time input Tensor reading, platform info, kernel path and shape key helper logic │ └── kernel_binary_map_utils.cpp // Multi-bin serialization/deserialization logic ├── graph_build │ └── main.cc // Construct graph and export single_add.air └── model_exec └── main.cc // Load om offline model and executeKey files:ge/custom_op.cppCustom operator core main process, responsible for chaining operator runtime compilation, serialization, deserialization, Shape/DataType inference and execution; compile-time readsShape/DataType/Formatmetadata by input Tensor.ge/add_custom.hAddCustomunique proto definition; graph construction side uses the same name header file generated tooutput/op_graph/include/.ge/add_custom_kernel.cppAscend C kernel source code read during operator runtime compilation phase.ge/utils/compile_utils.cppEncapsulates compile-time input Tensor reading, platform info reading, kernel source path locating, and binary key generation logic based on shape, used to demonstrate binary management method in multi-shape scenarios.ge/utils/kernel_binary_map_utils.cppResponsible for serialization and deserialization of multiple kernel binaries, highlighting multi-key/multi-binary persistence and recovery.graph_build/main.ccBuild minimum Add graph and exportsingle_add.air.model_exec/main.ccLoad om offline model, prepare input/output and execute model.run.shChain configure, build, install, ATC offline compilation and model execution.Core Pipelinege/custom_op.cppfirst reads input TensorShape/DataType/Formatmetadata inCompilecallback, then readsge/add_custom_kernel.cpp, completes Add kernel operator runtime compilation viaaclrtc.graph_build/main.ccconstructs minimum Add graph and exportssingle_add.air.atcoffline compilessingle_add.airtosingle_add.om.model_exec/main.ccloadssingle_add.omand executes via ACL, outputs first result for verification.Note: Sample compile-time input is already organized by Tensor method, so shape, data type, format can be directly obtained; current still only uses input shape to generate key, and based on this caches, serializes and deserializes kernel binary, overall providing multi-shape/multi-kernel processing framework.Build Artifactsoutput/op_graph/lib/linux/x86_64/libcust_opapi.soCustom operator deliverables used by GE/ATC offline compilation in Linux x86_64 environment; aarch64 environment corresponds tooutput/op_graph/lib/linux/aarch64/libcust_opapi.so.output/op_graph/include/add_custom.hAddCustom proto header file that can be directly used by graph construction side, generated by copying fromge/add_custom.hduring build phase.output/op_graph/lib/linux/x86_64/add_custom_kernel.cppKernel source file output together with deliverables, read during operator runtime compilation phase; aarch64 environment corresponds tooutput/op_graph/lib/linux/aarch64/add_custom_kernel.cpp.output/single_add.airAIR file exported by graph construction program, as ATC offline compilation input.output/single_add.om.omoffline model file generated by ATC offline compilation.build/compilable_add_graph_buildgraph_build program.build/compilable_add_model_execom offline model execution program.Result VerificationWhen successful, you can observe:output/single_add.airhas been generated.output/single_add.omhas been generated.Terminal output containsModel executed successfully!.Terminal output containsFirst element of output: 2.000000.If failed, priority checks:WhetherASCEND_HOME_PATHhas been set and CANN environment has been correctlysourced.Whetheratcis available.Whetheroutput/op_graph/lib/os/arch/libcust_opapi.so,output/op_graph/lib/os/arch/add_custom_kernel.cppandoutput/op_graph/include/add_custom.hhave been generated.Whether currentsoc_versionmatches actual environment,run.shdefaults toAscend910B1.Notes / Limitationsrun.shcurrently fixessoc_versionAscend910B1, if you need to adapt to other products please adjust according to actual environment.build/compilable_add_model_execneeds to pass model path as positional parameter.run.shwill automatically appendASCEND_CUSTOM_OPP_PATHasoutput/, so thatATC offline compilationloadslibcust_opapi.soaccording toop_graph/lib/os/arch/directory specification.Current sample organizes multiple kernel binaries by shape key, mainly providing multi-shape/multi-kernel processing framework.Current sample depends on installed CANN toolkit headers and directly reads compile-time input viaOpCompileContext::GetInputTensor.AppendixReference DocumentationRTC operator runtime compilation reference: Ascend C Operator JIT CompilationATC offline compilation reference: ATC Tool Usage GuideOperator Runtime Compilation and Sinking MechanismCompilableOpis responsible for reading kernel source code inCompilecallback and callingaclrtcto complete operator runtime compilation.Compilephase input reading is already organized byTensor, so in addition to shape, data type and format can also be directly accessed.PortableOpis responsible for serializing compilation products to final model and deserializing for recovery in execution phase.ShapeInferOpis responsible for completing output shape and data type inference inInferShapeandInferDataTypecallbacks.After these parts cooperate, the om offline model generated by ATC offline compilation will contain device-side binaries needed for sample custom operator execution; if later extended to scenarios where key corresponds to different kernels, this sample already provides management and sinking method for multiple binaries.Key Implementation ResponsibilitiesCompile: Read source code and trigger operator runtime compilation.InferShape / InferDataType: Derive output description based on input shape and data type.Serialize / Deserialize: Write compilation products to model and recover in execution phase.Execute: Load device-side binary and initiate kernel call in model execution phase.【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 CANN/ge可编译加法算子样例
Ascend C Custom Operator Runtime Compilation and Offline Model Sinking Sample【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/geSample OverviewGraph construction entry:GE graph construction ATC offline compilationOperator programming language:Ascend CCompilation method:RTC operator runtime compilationModel sinking capability:Supports sinking to om offline modelCore pipeline:Ascend C kernel source code RTC operator runtime compilation - GE deliverables - AIR - ATC offline compilation - om offline model - ACL executionDifference from other samples: This sample completes Ascend C kernel runtime compilation inCompilecallback and serializes compilation products into om offline model, which can be deserialized and used in subsequent model execution phases.Multi-shape note: This sample mainly provides a multi-shape/multi-kernel processing framework that manages and sinks multiple kernel binaries by shape key.Compile-time input note: The sample reads input byTensorlayout inCompilephase, so it can simultaneously obtain metadata such asShape,DataType,Format; current key is still only generated by shape.This sample demonstrates a minimum runnable GE custom operator sinking pipeline: first construct graph to generateAIR, then generateom offline modelthroughATC offline compilation, finally load and execute byACLprogram. The AddCustom in the sample simultaneously inheritsCompilableOp,PortableOpandShapeInferOp, so it can not only complete Ascend C kernel runtime compilation through RTC inCompilecallback, but also serialize compilation results to final model, and complete Shape and DataType inference in the same implementation class.Applicable ScenariosWant to see Ascend C custom operator sample inGE graph construction ATC offline compilation om offline modelpipeline.Want to reference complete flow of custom operator RTC runtime compilation, model sinking and final model execution.Want to see minimum runnable pipeline ofAIR - ATC offline compilation - om offline model - ACL.Not suitable for understandingPyTorch/TorchAirorTensorFlow/Tritonscenarios.PrerequisitesCANNCANN environment has been correctly installed and configured, e.g., executedsource ${ASCEND_HOME_PATH}/set_env.sh.atccommand is available.Current environment hasACL,GE,Graphrelated headers and libraries.Follow installation guide to complete toolkit and ops package installation.Framework and PluginsThis sample does not depend on PyTorch or TensorFlow.Environment VariablesASCEND_HOME_PATHASCEND_CUSTOM_OPP_PATHwill be automatically appended withoutput/inrun.sh, and custom operator deliverables will be placed inoutput/op_graph/lib/linux/x86_64/oroutput/op_graph/lib/linux/aarch64/Additional DependenciescmakegQuick RunExecute inexamples/custom_op/compilable_add_customdirectory:Recommended Methodbash run.shrun.shwill automatically appendoutput/toASCEND_CUSTOM_OPP_PATHafter generatingoutput/op_graph/lib/os/arch/libcust_opapi.so. If successful,output/single_add.air,output/single_add.omwill be generated, and terminal will printModel executed successfully!andFirst element of output: 2.000000.Step-by-step MethodIf you want to manually observe artifacts at each stage, you can execute:cmake -S . -B build -DCMAKE_BUILD_TYPERelease cmake --build build -j$(nproc) cmake --install build export ASCEND_CUSTOM_OPP_PATH$(pwd)/output:$ASCEND_CUSTOM_OPP_PATH cd output ../build/compilable_add_graph_build cd .. atc \ --model$(pwd)/output/single_add.air \ --framework1 \ --output$(pwd)/output/single_add \ --soc_versionAscend910B1 cd build ./compilable_add_model_exec ../output/single_add.om cd ..Whereexport ASCEND_CUSTOM_OPP_PATH$(pwd)/output:$ASCEND_CUSTOM_OPP_PATHis used to add custom operator package root directory to environment variable, then GE/ATC will load deliverables according tooutput/op_graph/lib/os/arch/libcust_opapi.sorule.Directory Structure and Key Filescompilable_add_custom ├── CMakeLists.txt ├── README.md ├── run.sh ├── ge │ ├── add_custom_kernel.cpp // Ascend C kernel source code read during operator runtime compilation │ ├── add_custom.h // AddCustom unique proto definition, shared by deliverables and graph construction side │ ├── custom_op.cpp // CompilableOp/PortableOp/ShapeInferOp main process implementation │ └── utils │ ├── compile_utils.cpp // Compile-time input Tensor reading, platform info, kernel path and shape key helper logic │ └── kernel_binary_map_utils.cpp // Multi-bin serialization/deserialization logic ├── graph_build │ └── main.cc // Construct graph and export single_add.air └── model_exec └── main.cc // Load om offline model and executeKey files:ge/custom_op.cppCustom operator core main process, responsible for chaining operator runtime compilation, serialization, deserialization, Shape/DataType inference and execution; compile-time readsShape/DataType/Formatmetadata by input Tensor.ge/add_custom.hAddCustomunique proto definition; graph construction side uses the same name header file generated tooutput/op_graph/include/.ge/add_custom_kernel.cppAscend C kernel source code read during operator runtime compilation phase.ge/utils/compile_utils.cppEncapsulates compile-time input Tensor reading, platform info reading, kernel source path locating, and binary key generation logic based on shape, used to demonstrate binary management method in multi-shape scenarios.ge/utils/kernel_binary_map_utils.cppResponsible for serialization and deserialization of multiple kernel binaries, highlighting multi-key/multi-binary persistence and recovery.graph_build/main.ccBuild minimum Add graph and exportsingle_add.air.model_exec/main.ccLoad om offline model, prepare input/output and execute model.run.shChain configure, build, install, ATC offline compilation and model execution.Core Pipelinege/custom_op.cppfirst reads input TensorShape/DataType/Formatmetadata inCompilecallback, then readsge/add_custom_kernel.cpp, completes Add kernel operator runtime compilation viaaclrtc.graph_build/main.ccconstructs minimum Add graph and exportssingle_add.air.atcoffline compilessingle_add.airtosingle_add.om.model_exec/main.ccloadssingle_add.omand executes via ACL, outputs first result for verification.Note: Sample compile-time input is already organized by Tensor method, so shape, data type, format can be directly obtained; current still only uses input shape to generate key, and based on this caches, serializes and deserializes kernel binary, overall providing multi-shape/multi-kernel processing framework.Build Artifactsoutput/op_graph/lib/linux/x86_64/libcust_opapi.soCustom operator deliverables used by GE/ATC offline compilation in Linux x86_64 environment; aarch64 environment corresponds tooutput/op_graph/lib/linux/aarch64/libcust_opapi.so.output/op_graph/include/add_custom.hAddCustom proto header file that can be directly used by graph construction side, generated by copying fromge/add_custom.hduring build phase.output/op_graph/lib/linux/x86_64/add_custom_kernel.cppKernel source file output together with deliverables, read during operator runtime compilation phase; aarch64 environment corresponds tooutput/op_graph/lib/linux/aarch64/add_custom_kernel.cpp.output/single_add.airAIR file exported by graph construction program, as ATC offline compilation input.output/single_add.om.omoffline model file generated by ATC offline compilation.build/compilable_add_graph_buildgraph_build program.build/compilable_add_model_execom offline model execution program.Result VerificationWhen successful, you can observe:output/single_add.airhas been generated.output/single_add.omhas been generated.Terminal output containsModel executed successfully!.Terminal output containsFirst element of output: 2.000000.If failed, priority checks:WhetherASCEND_HOME_PATHhas been set and CANN environment has been correctlysourced.Whetheratcis available.Whetheroutput/op_graph/lib/os/arch/libcust_opapi.so,output/op_graph/lib/os/arch/add_custom_kernel.cppandoutput/op_graph/include/add_custom.hhave been generated.Whether currentsoc_versionmatches actual environment,run.shdefaults toAscend910B1.Notes / Limitationsrun.shcurrently fixessoc_versionAscend910B1, if you need to adapt to other products please adjust according to actual environment.build/compilable_add_model_execneeds to pass model path as positional parameter.run.shwill automatically appendASCEND_CUSTOM_OPP_PATHasoutput/, so thatATC offline compilationloadslibcust_opapi.soaccording toop_graph/lib/os/arch/directory specification.Current sample organizes multiple kernel binaries by shape key, mainly providing multi-shape/multi-kernel processing framework.Current sample depends on installed CANN toolkit headers and directly reads compile-time input viaOpCompileContext::GetInputTensor.AppendixReference DocumentationRTC operator runtime compilation reference: Ascend C Operator JIT CompilationATC offline compilation reference: ATC Tool Usage GuideOperator Runtime Compilation and Sinking MechanismCompilableOpis responsible for reading kernel source code inCompilecallback and callingaclrtcto complete operator runtime compilation.Compilephase input reading is already organized byTensor, so in addition to shape, data type and format can also be directly accessed.PortableOpis responsible for serializing compilation products to final model and deserializing for recovery in execution phase.ShapeInferOpis responsible for completing output shape and data type inference inInferShapeandInferDataTypecallbacks.After these parts cooperate, the om offline model generated by ATC offline compilation will contain device-side binaries needed for sample custom operator execution; if later extended to scenarios where key corresponds to different kernels, this sample already provides management and sinking method for multiple binaries.Key Implementation ResponsibilitiesCompile: Read source code and trigger operator runtime compilation.InferShape / InferDataType: Derive output description based on input shape and data type.Serialize / Deserialize: Write compilation products to model and recover in execution phase.Execute: Load device-side binary and initiate kernel call in model execution phase.【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考