Class InferenceSession
Represents an Inference Session on an ONNX Model. This is a IDisposable class and it must be disposed of using either a explicit call to Dispose() method or a pattern of using() block. If this is a member of another class that class must also become IDisposable and it must dispose of InferfenceSession in its Dispose() method.
Inheritance
Namespace: Microsoft.ML.OnnxRuntime
Assembly: cs.temp.dll.dll
Syntax
public class InferenceSession : IDisposable
Constructors
InferenceSession(Byte[])
Constructs an InferenceSession from a model data in byte array
Declaration
public InferenceSession(byte[] model)
Parameters
Type | Name | Description |
---|---|---|
System.Byte[] | model |
InferenceSession(Byte[], PrePackedWeightsContainer)
Constructs an InferenceSession from a model data (in byte array) and it will use the provided pre-packed weights container to store and share pre-packed buffers of shared initializers across sessions if any.
Declaration
public InferenceSession(byte[] model, PrePackedWeightsContainer prepackedWeightsContainer)
Parameters
Type | Name | Description |
---|---|---|
System.Byte[] | model | Model as byte array |
PrePackedWeightsContainer | prepackedWeightsContainer | Instance of PrepackedWeightsContainer. Lifetime of 'prepackedWeightsContainer' must be managed by the user and it must outlive any sessions reliant on it |
InferenceSession(Byte[], SessionOptions)
Constructs an InferenceSession from a model data in byte array, with some additional session options
Declaration
public InferenceSession(byte[] model, SessionOptions options)
Parameters
Type | Name | Description |
---|---|---|
System.Byte[] | model | |
SessionOptions | options |
InferenceSession(Byte[], SessionOptions, PrePackedWeightsContainer)
Constructs an InferenceSession from a model data (in byte array) with some additional session options and it will use the provided pre-packed weights container to store and share pre-packed buffers of shared initializers across sessions if any.
Declaration
public InferenceSession(byte[] model, SessionOptions options, PrePackedWeightsContainer prepackedWeightsContainer)
Parameters
Type | Name | Description |
---|---|---|
System.Byte[] | model | Model as byte array |
SessionOptions | options | Session Options |
PrePackedWeightsContainer | prepackedWeightsContainer | Instance of PrepackedWeightsContainer. Lifetime of 'prepackedWeightsContainer' must be managed by the user and it must outlive any sessions reliant on it |
InferenceSession(String)
Constructs an InferenceSession from a model file
Declaration
public InferenceSession(string modelPath)
Parameters
Type | Name | Description |
---|---|---|
System.String | modelPath |
InferenceSession(String, PrePackedWeightsContainer)
Constructs an InferenceSession from a model file and it will use the provided pre-packed weights container to store and share pre-packed buffers of shared initializers across sessions if any.
Declaration
public InferenceSession(string modelPath, PrePackedWeightsContainer prepackedWeightsContainer)
Parameters
Type | Name | Description |
---|---|---|
System.String | modelPath | Model path |
PrePackedWeightsContainer | prepackedWeightsContainer | Instance of PrepackedWeightsContainer. Lifetime of 'prepackedWeightsContainer' must be managed by the user and it must outlive any sessions reliant on it |
InferenceSession(String, SessionOptions)
Constructs an InferenceSession from a model file, with some additional session options
Declaration
public InferenceSession(string modelPath, SessionOptions options)
Parameters
Type | Name | Description |
---|---|---|
System.String | modelPath | |
SessionOptions | options |
InferenceSession(String, SessionOptions, PrePackedWeightsContainer)
Constructs an InferenceSession from a model file, with some additional session options and it will use the provided pre-packed weights container to store and share pre-packed buffers of shared initializers across sessions if any.
Declaration
public InferenceSession(string modelPath, SessionOptions options, PrePackedWeightsContainer prepackedWeightsContainer)
Parameters
Type | Name | Description |
---|---|---|
System.String | modelPath | Model path |
SessionOptions | options | Session options |
PrePackedWeightsContainer | prepackedWeightsContainer | Instance of PrepackedWeightsContainer. Lifetime of 'prepackedWeightsContainer' must be managed by the user and it must outlive any sessions reliant on it |
Properties
InputMetadata
Meta data regarding the input nodes, keyed by input names
Declaration
public IReadOnlyDictionary<string, NodeMetadata> InputMetadata { get; }
Property Value
Type | Description |
---|---|
IReadOnlyDictionary<System.String, NodeMetadata> |
ModelMetadata
This property queries model metadata, constructs an instance of ModelMetadata and caches it
Declaration
public ModelMetadata ModelMetadata { get; }
Property Value
Type | Description |
---|---|
ModelMetadata | Instance of ModelMetdata |
OutputMetadata
Metadata regarding the output nodes, keyed by output names
Declaration
public IReadOnlyDictionary<string, NodeMetadata> OutputMetadata { get; }
Property Value
Type | Description |
---|---|
IReadOnlyDictionary<System.String, NodeMetadata> |
OverridableInitializerMetadata
Metadata regarding the overridable initializers, keyed by node names
Declaration
public IReadOnlyDictionary<string, NodeMetadata> OverridableInitializerMetadata { get; }
Property Value
Type | Description |
---|---|
IReadOnlyDictionary<System.String, NodeMetadata> |
ProfilingStartTimeNs
Return the nanoseconds of profiling's start time On some platforms, this timer may not be as precise as nanoseconds For instance, on Windows and MacOS, the precision will be ~100ns
Declaration
public ulong ProfilingStartTimeNs { get; }
Property Value
Type | Description |
---|---|
System.UInt64 |
Methods
CreateIoBinding()
Create OrtIoBinding instance to bind pre-allocated buffers to input/output
Declaration
public OrtIoBinding CreateIoBinding()
Returns
Type | Description |
---|---|
OrtIoBinding | A new instance of OrtIoBinding |
Dispose()
IDisposable implementation
Declaration
public void Dispose()
Dispose(Boolean)
IDisposable implementation
Declaration
protected virtual void Dispose(bool disposing)
Parameters
Type | Name | Description |
---|---|---|
System.Boolean | disposing | true if invoked from Dispose() method |
EndProfiling()
Ends profiling for the session.
Declaration
public string EndProfiling()
Returns
Type | Description |
---|---|
System.String | Returns the profile file name. |
Finalize()
Finalizer. to cleanup session in case it runs and the user forgets to Dispose() of the session
Declaration
protected void Finalize()
Run(IReadOnlyCollection<NamedOnnxValue>)
Runs the loaded model for the given inputs, and fetches all the outputs.
Declaration
public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<NamedOnnxValue> inputs)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<NamedOnnxValue> | inputs | specify a collection of NamedOnnxValue that indicates the input values. |
Returns
Type | Description |
---|---|
IDisposableReadOnlyCollection<DisposableNamedOnnxValue> | Output Tensors in a Collection of NamedOnnxValue. User must dispose the output. |
Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<NamedOnnxValue>)
Runs the loaded model for the given inputs and outputs.
Outputs need to be created with correct type and dimension to receive the fetched data.
Declaration
public void Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<NamedOnnxValue> outputs)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<NamedOnnxValue> | inputs | Specify a collection of NamedOnnxValue that indicates the input values. |
IReadOnlyCollection<NamedOnnxValue> | outputs | Specify a collection of NamedOnnxValue that indicates the output values. |
Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<NamedOnnxValue>, RunOptions)
Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.
Outputs need to be created with correct type and dimension to receive the fetched data.
Declaration
public void Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<NamedOnnxValue> outputs, RunOptions options)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<NamedOnnxValue> | inputs | Specify a collection of NamedOnnxValue that indicates the input values. |
IReadOnlyCollection<NamedOnnxValue> | outputs | Specify a collection of NamedOnnxValue that indicates the output values. |
RunOptions | options |
Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<String>)
Runs the loaded model for the given inputs, and fetches the outputs specified in outputNames
.
Declaration
public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<string> outputNames)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<NamedOnnxValue> | inputs | Specify a collection of NamedOnnxValue that indicates the input values. |
IReadOnlyCollection<System.String> | outputNames | Specify a collection of string that indicates the output names to fetch. |
Returns
Type | Description |
---|---|
IDisposableReadOnlyCollection<DisposableNamedOnnxValue> | Output Tensors in a Collection of NamedOnnxValue. User must dispose the output. |
Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>)
Runs the loaded model for the given inputs and outputs.
Outputs need to be created with correct type and dimension to receive the fetched data.
Declaration
public void Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<string> outputNames, IReadOnlyCollection<FixedBufferOnnxValue> outputValues)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<NamedOnnxValue> | inputs | Specify a collection of NamedOnnxValue that indicates the input values. |
IReadOnlyCollection<System.String> | outputNames | Specify a collection of string that indicates the output names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | outputValues | Specify a collection of FixedBufferOnnxValue that indicates the output values. |
Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, RunOptions)
Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.
Outputs need to be created with correct type and dimension to receive the fetched data.
Declaration
public void Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<string> outputNames, IReadOnlyCollection<FixedBufferOnnxValue> outputValues, RunOptions options)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<NamedOnnxValue> | inputs | Specify a collection of NamedOnnxValue that indicates the input values. |
IReadOnlyCollection<System.String> | outputNames | Specify a collection of string that indicates the output names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | outputValues | Specify a collection of FixedBufferOnnxValue that indicates the output values. |
RunOptions | options |
Run(IReadOnlyCollection<NamedOnnxValue>, IReadOnlyCollection<String>, RunOptions)
Runs the loaded model for the given inputs, and fetches the specified outputs in outputNames
. Uses the given RunOptions for this run.
Declaration
public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<NamedOnnxValue> inputs, IReadOnlyCollection<string> outputNames, RunOptions options)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<NamedOnnxValue> | inputs | Specify a collection of NamedOnnxValue that indicates the input values. |
IReadOnlyCollection<System.String> | outputNames | Specify a collection of string that indicates the output names to fetch. |
RunOptions | options |
Returns
Type | Description |
---|---|
IDisposableReadOnlyCollection<DisposableNamedOnnxValue> | Output Tensors in a Collection of NamedOnnxValue. User must dispose the output. |
Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>)
Runs the loaded model for the given inputs, and fetches all the outputs.
Declaration
public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<System.String> | inputNames | Specify a collection of string that indicates the input names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | inputValues | Specify a collection of FixedBufferOnnxValue that indicates the input values. |
Returns
Type | Description |
---|---|
IDisposableReadOnlyCollection<DisposableNamedOnnxValue> | Output Tensors in a Collection of NamedOnnxValue. User must dispose the output. |
Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<NamedOnnxValue>)
Runs the loaded model for the given inputs and outputs.
Outputs need to be created with correct type and dimension to receive the fetched data.
Declaration
public void Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<NamedOnnxValue> outputs)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<System.String> | inputNames | Specify a collection of string that indicates the input names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | inputValues | Specify a collection of FixedBufferOnnxValue that indicates the input values. |
IReadOnlyCollection<NamedOnnxValue> | outputs | Specify a collection of NamedOnnxValue that indicates the output values. |
Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<NamedOnnxValue>, RunOptions)
Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.
Outputs need to be created with correct type and dimension to receive the fetched data.
Declaration
public void Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<NamedOnnxValue> outputs, RunOptions options)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<System.String> | inputNames | Specify a collection of string that indicates the input names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | inputValues | Specify a collection of FixedBufferOnnxValue that indicates the input values. |
IReadOnlyCollection<NamedOnnxValue> | outputs | Specify a collection of NamedOnnxValue that indicates the output values. |
RunOptions | options |
Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<String>)
Runs the loaded model for the given inputs, and fetches the outputs specified in outputNames
.
Declaration
public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<string> outputNames)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<System.String> | inputNames | Specify a collection of string that indicates the input names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | inputValues | Specify a collection of FixedBufferOnnxValue that indicates the input values. |
IReadOnlyCollection<System.String> | outputNames | Specify a collection of string that indicates the output names to fetch. |
Returns
Type | Description |
---|---|
IDisposableReadOnlyCollection<DisposableNamedOnnxValue> | Output Tensors in a Collection of NamedOnnxValue. User must dispose the output. |
Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>)
Runs the loaded model for the given inputs and outputs.
Outputs need to be created with correct type and dimension to accept the fetched data.
Declaration
public void Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<string> outputNames, IReadOnlyCollection<FixedBufferOnnxValue> outputValues)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<System.String> | inputNames | Specify a collection of string that indicates the input names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | inputValues | Specify a collection of FixedBufferOnnxValue that indicates the input values. |
IReadOnlyCollection<System.String> | outputNames | Specify a collection of string that indicates the output names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | outputValues | Specify a collection of FixedBufferOnnxValue that indicates the output values. |
Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, RunOptions)
Runs the loaded model for the given inputs and outputs. Uses the given RunOptions for this run.
Outputs need to be created with correct type and dimension to accept the fetched data.
Declaration
public void Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<string> outputNames, IReadOnlyCollection<FixedBufferOnnxValue> outputValues, RunOptions options)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<System.String> | inputNames | Specify a collection of string that indicates the input names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | inputValues | Specify a collection of FixedBufferOnnxValue that indicates the input values. |
IReadOnlyCollection<System.String> | outputNames | Specify a collection of string that indicates the output names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | outputValues | Specify a collection of FixedBufferOnnxValue that indicates the output values. |
RunOptions | options |
Run(IReadOnlyCollection<String>, IReadOnlyCollection<FixedBufferOnnxValue>, IReadOnlyCollection<String>, RunOptions)
Runs the loaded model for the given inputs, and fetches the specified outputs in outputNames
. Uses the given RunOptions for this run.
Declaration
public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> Run(IReadOnlyCollection<string> inputNames, IReadOnlyCollection<FixedBufferOnnxValue> inputValues, IReadOnlyCollection<string> outputNames, RunOptions options)
Parameters
Type | Name | Description |
---|---|---|
IReadOnlyCollection<System.String> | inputNames | Specify a collection of string that indicates the input names. Should match |
IReadOnlyCollection<FixedBufferOnnxValue> | inputValues | Specify a collection of FixedBufferOnnxValue that indicates the input values. |
IReadOnlyCollection<System.String> | outputNames | Specify a collection of string that indicates the output names to fetch. |
RunOptions | options |
Returns
Type | Description |
---|---|
IDisposableReadOnlyCollection<DisposableNamedOnnxValue> | Output Tensors in a Collection of NamedOnnxValue. User must dispose the output. |
RunWithBinding(RunOptions, OrtIoBinding)
This method runs inference on the OrtIoBinding instance The method does not return anything. This is a lightweight version of RunWithBindingAndNames(). When you bind pre-allocated buffers to the output values you may not want to fetch the outputs since you already have access to them so you can spare the expense of fetching them and pairing with names. You can still fetch the outputs by calling OrtIOBinding.GetOutputValues()
Declaration
public void RunWithBinding(RunOptions runOptions, OrtIoBinding ioBinding)
Parameters
Type | Name | Description |
---|---|---|
RunOptions | runOptions | runOptions |
OrtIoBinding | ioBinding | ioBinding instance to use |
RunWithBindingAndNames(RunOptions, OrtIoBinding, String[])
This method return a collection of DisposableNamedOnnxValue as in other interfaces Query names from OrtIoBinding object and pair then with the array of OrtValues returned from OrtIoBinding.GetOutputValues()
Declaration
public IDisposableReadOnlyCollection<DisposableNamedOnnxValue> RunWithBindingAndNames(RunOptions runOptions, OrtIoBinding ioBinding, string[] names = null)
Parameters
Type | Name | Description |
---|---|---|
RunOptions | runOptions | RunOptions |
OrtIoBinding | ioBinding | OrtIoBinding instance with bindings |
System.String[] | names | optional parameter. If you already know the names of the outputs you can save a native call to retrieve output names. They will be paired with the returned OrtValues and combined into DisposbleNamedOnnxValues. Otherwise, the method will retrieve output names from the OrtIoBinding instance. It is an error if you supply a different number of names than the returned outputs |
Returns
Type | Description |
---|---|
IDisposableReadOnlyCollection<DisposableNamedOnnxValue> | A disposable collection of DisposableNamedOnnxValue that encapsulate output OrtValues |