Automatic Speech Recognition

4min

The below input parameters are for different attack types. To start working with the APIs, see Audio Speech Recognition﻿.
Audio Speech Recognition is an early access feature with limited functionality. It is not available as part of AIShield pypi package. For early access, kindly contact [email protected]
﻿
ASR Models Support:
﻿OpenAI-Whisper: All variants in PyTorch and Hugging Face formats are supported. Refer to the OpenAI-Whisper model card for the complete list of available variants. [Ref: OpenAI-Whisper-ModelCard]
﻿
Common Parameter
The below table parameters are common for Evasion Attack type.
Parameter
Data type
Description
Remark
model_Id
String
Model_id received during model registration. 
We need to provide this model ID in query parameter in URL.
You have to do model registration only once for a model to perform model analysis. This will help you track the no of api call made, and it's success metric.
Request Body (Json format)
﻿
﻿
﻿
model_framework 
String
Framework on which model is trained on. 
curretly supported framework are: onnx for Evasion and pytorch for data-poisoning .
﻿
To access all sample artifacts, please visit Artifacts﻿. 
For specific artifact details,  refer 
Vulnerability Report : Vulnerability Report﻿     
Sample Attacks : Sample Attacks﻿﻿
﻿
﻿
File upload format
 Data: The processed audio data, ready to be passed to the model for prediction, should be saved in a folder. 
﻿Download sample data﻿
Label: A CSV file should be created with two columns: "audio" and "label." The first column should contain the audio file name, and the second column should contain the label(text).  Check sample label file attached.
﻿Download sample label﻿
Model: The model should be saved in onnx format. This can be ignored when model is hosted as an API.
﻿Download sample model﻿
Note:
Format: All uploaded files must be in a zipped format.
Dataset: The provided files are sample audio data from the Librespeech dataset.
Audio File Properties: Each audio file should not exceed 30 seconds and must have a sampling rate of 16,000 Hz.
Sample Count: The minimum number of samples required must be between 300 and 500.
Data Sampling Strategy: Ensure that the uploaded samples are representative of the complete dataset, using techniques such as normal distributive sampling for even distribution.
 Parameter
Parameter
Data type
Description
Remark
Request Body (Json format)
﻿
﻿
﻿
model_api_details
String
Provide API details of hosted model as encrypted JSON string 
 
provide this only if use_model_api is "yes". 
use_model_api
String
Use model API to use your model endpoint instead of uploading the model as a zip file.
when this parameter is yes, you don't have to upload model as zip. You can pass api url along with other verification credential in json file. 
Convert pytorch model to ONNX format for ASR
Converting Pytorch model to onnx will create two models (Encoder and Decoder) in onnx format. 
To Load an OpenAI Model:
  load_model: "tiny.en", "base.en", "small.en", "medium.en", "large.en" 
To Load a Distil-Whisper Model:
load_model: model.bin or "distil-whisper/distil-large-v3"
Loading of Whisper model
Python
import whisper, torch, onnxruntime

# load tiny model trained on whisper dataset
device = torch.device('cpu')
model = whisper.load_model('tiny.en').to(device) #Hugging Face --> Model.bin
tokenizer = whisper.decoding.get_tokenizer(False,language='en',task='transcribe') ##initialoze whisper tokenizer
model.requires_grad_(False)
model.eval()
import whisper, torch, onnxruntime

# load tiny model trained on whisper dataset
device = torch.device('cpu')
model = whisper.load_model('tiny.en').to(device) #Hugging Face --> Model.bin
tokenizer = whisper.decoding.get_tokenizer(False,language='en',task='transcribe') ##initialoze whisper tokenizer
model.requires_grad_(False)
model.eval()
﻿
Convert to Onnx Format
 First load sample audio file to get the feature of audio signal:
python
import librosa

audio = librosa.load(path_of_audio_file,sr=16000)[0] #sample audio file to load audio features
audio = whisper.pad_or_trim(audio)
log_mel = whisper.log_mel_spectrogram(audio).unsqueeze(axis=0)
import librosa

audio = librosa.load(path_of_audio_file,sr=16000)[0] #sample audio file to load audio features
audio = whisper.pad_or_trim(audio)
log_mel = whisper.log_mel_spectrogram(audio).unsqueeze(axis=0)
﻿
Prepare the input for decoder model: 
python
input = log_mel.to(device)
audio_features = model.encoder(input)
dec_tokens = torch.tensor([tokenizer.sot_sequence_including_notimestamps],dtype=torch.long) ##decoder token

input = log_mel.to(device)
audio_features = model.encoder(input)
dec_tokens = torch.tensor([tokenizer.sot_sequence_including_notimestamps],dtype=torch.long) ##decoder token
﻿
Convert and save model (encoder and decoder) in onnx format:
python

#: ONNX export
##Encoder Export
torch.onnx.export(
    model.encoder, 
    (log_mel,), 
    "distil_encoder.onnx", 
    input_names=["input_mel"], 
    output_names=["out"],
    dynamic_axes={
        "input_mel": {0: "batch"},
        "out": {0: "batch"},
    },
)

#Decoder Export
torch.onnx.export(
    model.decoder, 
    (dec_tokens, audio_features), 
    "distil_decoder.onnx", 
    input_names=["tokens", "audio"], 
    output_names=["out"], 
    dynamic_axes={
        "tokens": {0: "batch", 1: "seq"},
        "audio": {0: "batch"},
        "out": {0: "batch", 1: "seq"},
    },
)

#: ONNX export
##Encoder Export
torch.onnx.export(
    model.encoder, 
    (log_mel,), 
    "distil_encoder.onnx", 
    input_names=["input_mel"], 
    output_names=["out"],
    dynamic_axes={
        "input_mel": {0: "batch"},
        "out": {0: "batch"},
    },
)

#Decoder Export
torch.onnx.export(
    model.decoder, 
    (dec_tokens, audio_features), 
    "distil_decoder.onnx", 
    input_names=["tokens", "audio"], 
    output_names=["out"], 
    dynamic_axes={
        "tokens": {0: "batch", 1: "seq"},
        "audio": {0: "batch"},
        "out": {0: "batch", 1: "seq"},
    },
)
﻿
File upload format
 Reference Data: The processed clean audio data, ready to be passed to the model for prediction, should be saved in a folder. 
﻿Download sample data﻿
Refernce Label: A CSV file corresponding to clean data should be created with two columns: "audio" and "label." The first column should contain the audio file name, and the second column should contain the label(text).  Check sample label file attached.
﻿Download sample label﻿
Model: The model should be saved in onnx format. This can be ignored when model is hosted as an API.
﻿Download sample model﻿
 Universal Data: Audio samples under test (poisoned, clean samples), should be saved in a folder . 
﻿Download sample universal data﻿
 Universal Label:  A CSV files corresponding to  universal data hould be created with two columns: "audio" and "label." The first column should contain the audio file name, and the second column should contain the label(text).  Check sample label file attached.
﻿Download sample universal label﻿
﻿
Note:
Format: All uploaded files must be in a zipped format.
Dataset: The provided files are sample audio data from the Librespeech dataset.
Audio File Properties: Each audio file should not exceed 30 seconds and must have a sampling rate of 16,000 Hz.
Sample Count: The minimum number of samples required must be between 500 to 700.
Data Sampling Strategy: Ensure that the uploaded samples are representative of the complete dataset, using techniques such as normal distributive sampling for even distribution.
Conversion of OpenAI-Whisper models to HuggingFace format
﻿
﻿
 

Parameter	Data type	Description	Remark
model_Id	String	Model_id received during model registration. We need to provide this model ID in query parameter in URL.	You have to do model registration only once for a model to perform model analysis. This will help you track the no of api call made, and it's success metric.
Request Body (Json format)
model_framework	String	Framework on which model is trained on.	curretly supported framework are: onnx for Evasion and pytorch for data-poisoning .

Parameter	Data type	Description	Remark
Request Body (Json format)
model_api_details	String	Provide API details of hosted model as encrypted JSON string	provide this only if use_model_api is "yes".
use_model_api	String	Use model API to use your model endpoint instead of uploading the model as a zip file.	when this parameter is yes, you don't have to upload model as zip. You can pass api url along with other verification credential in json file.

Updated 04 Feb 2025

Did this page help you?

Audio

Image

Automatic Speech Recognition

Common Parameter

Evasion

File upload format

Parameter

Convert pytorch model to ONNX format for ASR

Loading of Whisper model

Convert to Onnx Format

Data-Poisoning

File upload format

Conversion of OpenAI-Whisper models to HuggingFace format