API Summary

Summary of public functions and classes exposed in ONNX Runtime.

OrtValue

ONNX Runtime works with native Python data structures which are mapped into ONNX data formats : Numpy arrays (tensors), dictionaries (maps), and a list of Numpy arrays (sequences). The data backing these are on CPU.

ONNX Runtime supports a custom data structure that supports all ONNX data formats that allows users to place the data backing these on a device, for example, on a CUDA supported device. This allows for interesting IOBinding scenarios (discussed below). In addition, ONNX Runtime supports directly working with OrtValue (s) while inferencing a model if provided as part of the input feed.

Below is an example showing creation of an OrtValue from a Numpy array while placing its backing memory on a CUDA device:

#X is numpy array on cpu, create an OrtValue and place it on cuda device id = 0
ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
ortvalue.device_name()  # 'cuda'
ortvalue.shape()  # shape of the numpy array X
ortvalue.data_type()  # 'tensor(float)'
ortvalue.is_tensor()  # 'True'
np.array_equal(ortvalue.numpy(), X)  # 'True'

#ortvalue can be provided as part of the input feed to a model
ses = onnxruntime.InferenceSession('model.onnx')
res = sess.run(["Y"], {"X": ortvalue})

IOBinding

By default, ONNX Runtime always places input(s) and output(s) on CPU, which is not optimal if the input or output is consumed and produced on a device other than CPU because it introduces data copy between CPU and the device. ONNX Runtime provides a feature, IO Binding, which addresses this issue by enabling users to specify which device to place input(s) and output(s) on. Here are scenarios to use this feature.

(In the following code snippets, model.onnx is the model to execute, X is the input data to feed, and Y is the output data.)

Scenario 1:

A graph is executed on a device other than CPU, for instance CUDA. Users can use IOBinding to put input on CUDA as the follows.

#X is numpy array on cpu
session = onnxruntime.InferenceSession('model.onnx')
io_binding = session.io_binding()
# OnnxRuntime will copy the data over to the CUDA device if 'input' is consumed by nodes on the CUDA device
io_binding.bind_cpu_input('input', X)
io_binding.bind_output('output')
session.run_with_iobinding(io_binding)
Y = io_binding.copy_outputs_to_cpu()[0]

Scenario 2:

The input data is on a device, users directly use the input. The output data is on CPU.

#X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
session = onnxruntime.InferenceSession('model.onnx')
io_binding = session.io_binding()
io_binding.bind_input(name='input', device_type=X_ortvalue.device_name(), device_id=0, element_type=np.float32, shape=X_ortvalue.shape(), buffer_ptr=X_ortvalue.data_ptr())
io_binding.bind_output('output')
session.run_with_iobinding(io_binding)
Y = io_binding.copy_outputs_to_cpu()[0]

Scenario 3:

The input data and output data are both on a device, users directly use the input and also place output on the device.

#X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
Y_ortvalue = onnxruntime.OrtValue.ortvalue_from_shape_and_type([3, 2], np.float32, 'cuda', 0)  # Change the shape to the actual shape of the output being bound
session = onnxruntime.InferenceSession('model.onnx')
io_binding = session.io_binding()
io_binding.bind_input(name='input', device_type=X_ortvalue.device_name(), device_id=0, element_type=np.float32, shape=X_ortvalue.shape(), buffer_ptr=X_ortvalue.data_ptr())
io_binding.bind_output(name='output', device_type=Y_ortvalue.device_name(), device_id=0, element_type=np.float32, shape=Y_ortvalue.shape(), buffer_ptr=Y_ortvalue.data_ptr())
session.run_with_iobinding(io_binding)

Scenario 4:

Users can request ONNX Runtime to allocate an output on a device. This is particularly useful for dynamic shaped outputs. Users can use the get_outputs() API to get access to the OrtValue (s) corresponding to the allocated output(s). Users can thus consume the ONNX Runtime allocated memory for the output as an OrtValue.

#X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
session = onnxruntime.InferenceSession('model.onnx')
io_binding = session.io_binding()
io_binding.bind_input(name='input', device_type=X_ortvalue.device_name(), device_id=0, element_type=np.float32, shape=X_ortvalue.shape(), buffer_ptr=X_ortvalue.data_ptr())
#Request ONNX Runtime to bind and allocate memory on CUDA for 'output'
io_binding.bind_output('output', 'cuda')
session.run_with_iobinding(io_binding)
# The following call returns an OrtValue which has data allocated by ONNX Runtime on CUDA
ort_output = io_binding.get_outputs()[0]

Scenario 5:

Users can bind OrtValue (s) directly.

#X is numpy array on cpu
#X is numpy array on cpu
X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(X, 'cuda', 0)
Y_ortvalue = onnxruntime.OrtValue.ortvalue_from_shape_and_type([3, 2], np.float32, 'cuda', 0)  # Change the shape to the actual shape of the output being bound
session = onnxruntime.InferenceSession('model.onnx')
io_binding = session.io_binding()
io_binding.bind_ortvalue_input('input', X_ortvalue)
io_binding.bind_ortvalue_output('output', Y_ortvalue)
session.run_with_iobinding(io_binding)

Device

The package is compiled for a specific device, GPU or CPU. The CPU implementation includes optimizations such as MKL (Math Kernel Libary). The following function indicates the chosen option:

onnxruntime.get_device() → str

Return the device used to compute the prediction (CPU, MKL, …)

Examples and datasets

The package contains a few models stored in ONNX format used in the documentation. These don’t need to be downloaded as they are installed with the package.

onnxruntime.datasets.get_example(name)[source]

Retrieves the absolute file name of an example.

Load and run a model

ONNX Runtime reads a model saved in ONNX format. The main class InferenceSession wraps these functionalities in a single place.

class onnxruntime.ModelMetadata

Pre-defined and custom metadata about the model. It is usually used to identify the model used to run the prediction and facilitate the comparison.

property custom_metadata_map

additional metadata

property description

description of the model

property domain

ONNX domain

property graph_description

description of the graph hosted in the model

property graph_name

graph name

property producer_name

producer name

property version

version of the model

class onnxruntime.InferenceSession(path_or_bytes, sess_options=None, providers=None, provider_options=None)[source]

This is the main class used to run a model. The next release (ORT 1.10) will require explicitly setting the providers parameter if you want to use execution providers other than the default CPU provider (as opposed to the current behavior of providers getting set/registered by default based on the build flags) when instantiating InferenceSession.

class onnxruntime.NodeArg

Node argument definition, for both input and output, including arg name, arg type (contains both type and shape).

property name

node name

property shape

node shape (assuming the node holds a tensor)

property type

node type

class onnxruntime.RunOptions

Configuration information for a single Run.

property log_severity_level

Log severity level for a particular Run() invocation. 0:Verbose, 1:Info, 2:Warning. 3:Error, 4:Fatal. Default is 2.

property log_verbosity_level

VLOG level if DEBUG build and run_log_severity_level is 0. Applies to a particular Run() invocation. Default is 0.

property logid

To identify logs generated by a particular Run() invocation.

property only_execute_path_to_fetches

Only execute the nodes needed by fetch list

property terminate

Set to True to terminate any currently executing calls that are using this RunOptions instance. The individual calls will exit gracefully and return an error status.

class onnxruntime.SessionOptions

Configuration information for a session.

add_free_dimension_override_by_denotation(self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions, arg0: str, arg1: int) → None

Specify the dimension size for each denotation associated with an input’s free dimension.

add_free_dimension_override_by_name(self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions, arg0: str, arg1: int) → None

Specify values of named dimensions within model inputs.

add_initializer(self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions, arg0: str, arg1: object) → None
add_session_config_entry(self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions, arg0: str, arg1: str) → None

Set a single session configuration entry as a pair of strings.

property enable_cpu_mem_arena

Enables the memory arena on CPU. Arena may pre-allocate memory for future usage. Set this option to false if you don’t want it. Default is True.

property enable_mem_pattern

Enable the memory pattern optimization. Default is true.

property enable_profiling

Enable profiling for this session. Default is false.

property execution_mode

Sets the execution mode. Default is sequential.

property execution_order

Sets the execution order. Default is basic topological order.

get_session_config_entry(self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions, arg0: str) → str

Get a single session configuration value using the given configuration key.

property graph_optimization_level

Graph optimization level for this session.

property inter_op_num_threads

Sets the number of threads used to parallelize the execution of the graph (across nodes). Default is 0 to let onnxruntime choose.

property intra_op_num_threads

Sets the number of threads used to parallelize the execution within nodes. Default is 0 to let onnxruntime choose.

property log_severity_level

Log severity level. Applies to session load, initialization, etc. 0:Verbose, 1:Info, 2:Warning. 3:Error, 4:Fatal. Default is 2.

property log_verbosity_level

VLOG level if DEBUG build and session_log_severity_level is 0. Applies to session load, initialization, etc. Default is 0.

property logid

Logger id to use for session output.

property optimized_model_filepath

File path to serialize optimized model to. Optimized model is not serialized unless optimized_model_filepath is set. Serialized model format will default to ONNX unless:

  • add_session_config_entry is used to set ‘session.save_model_format’ to ‘ORT’, or

  • there is no ‘session.save_model_format’ config entry and optimized_model_filepath ends in ‘.ort’ (case insensitive)

property profile_file_prefix

The prefix of the profile file. The current time will be appended to the file name.

register_custom_ops_library(self: onnxruntime.capi.onnxruntime_pybind11_state.SessionOptions, arg0: str) → None

Specify the path to the shared library containing the custom op kernels required to run a model.

property use_deterministic_compute

Whether to use deterministic compute. Default is false.

Backend

In addition to the regular API which is optimized for performance and usability, ONNX Runtime also implements the ONNX backend API for verification of ONNX specification conformance. The following functions are supported:

onnxruntime.backend.is_compatible(model, device=None, **kwargs)

Return whether the model is compatible with the backend.

Parameters

  • model – unused

  • device – None to use the default device or a string (ex: ‘CPU’)

Returns

boolean

onnxruntime.backend.prepare(model, device=None, **kwargs)

Load the model and creates a onnxruntime.InferenceSession ready to be used as a backend.

Parameters

  • model – ModelProto (returned by onnx.load), string for a filename or bytes for a serialized model

  • device – requested device for the computation, None means the default one which depends on the compilation settings

  • kwargs – see onnxruntime.SessionOptions

Returns

onnxruntime.InferenceSession

onnxruntime.backend.run(model, inputs, device=None, **kwargs)

Compute the prediction.

Parameters
Returns

predictions

onnxruntime.backend.supports_device(device)

Check whether the backend is compiled with particular device support. In particular it’s used in the testing suite.