Whisper Spoken LID (v1.0)

About this version

About this app (See raw metadata.json)

Spoken language identification over fixed-length windows of an audio track, using the language detection of OpenAI Whisper. Consecutive windows with the same most probable language become one TimeFrame.

Inputs

(Note: “*” as a property value means that the property is required but can be any value.)

One of the following is required: [

]

Configurable Parameters

(Note: Multivalued means the parameter can have one or more values.)

  • model: optional, defaults to tiny

    • Type: string
    • Multivalued: False
    • Choices: tiny, base, small, medium, large, turbo

    Whisper model size.

  • windowMs: optional, defaults to 30000

    • Type: integer
    • Multivalued: False

    Window length in milliseconds, at most 30000.

  • minLastWindowMs: optional, defaults to 0

    • Type: integer
    • Multivalued: False

    Shortest final window to keep, in milliseconds. 0 keeps any final window.

  • mergeWindows: optional, defaults to true

    • Type: boolean
    • Multivalued: False
    • Choices: false, true

    Join consecutive windows with the same most probable language into one TimeFrame. When false, each window becomes its own TimeFrame.

  • batchSize: optional, defaults to 1

    • Type: integer
    • Multivalued: False

    Number of windows passed to the model at once.

  • pretty: optional, defaults to false

    • Type: boolean
    • Multivalued: False
    • Choices: false, true

    The JSON body of the HTTP response will be re-formatted with 2-space indentation

  • runningTime: optional, defaults to true

    • Type: boolean
    • Multivalued: False
    • Choices: false, true

    The running time of the app will be recorded in the view metadata

  • hwFetch: optional, defaults to false

    • Type: boolean
    • Multivalued: False
    • Choices: false, true

    The hardware information (architecture, GPU and vRAM) will be recorded in the view metadata

  • tfSamplingMode: optional, defaults to representatives

    • Type: string
    • Multivalued: False
    • Choices: representatives, single, all

    Sampling mode for TimeFrame annotations. Has no effect when the app does not process TimeFrames. “representatives” uses all representative timepoints if present, otherwise skips the TimeFrame. “single” uses the middle representative if present, otherwise extracts an image from the midpoint of the start/end interval (midpoint is calculated by floor division of the sum of start and end). “all” uses all target timepoints if present, otherwise extracts all images from the time interval.

Outputs

(Note: “*” as a property value means that the property is required but can be any value.)

(Note: Not all output annotations are always generated.)

  • http://clams.ai/vocabulary/type/TimeFrame/v6
    • timeUnit = “milliseconds”
    • document = “*”
    • labelset = “*”
    • label = “*”
    • scores = “*”

    Each TimeFrame covers consecutive windows whose most probable language is label, a whisper language code. labelset lists all language codes of the whisper model in use. scores lists, in time order, the probability whisper gave label in each of those windows, one float in [0, 1] per window. Window i starts at start + i * windowMs; the last window ends at end and can be shorter. The probabilities are not calibrated.