Release Notes
(no notes provided by the developer)
Applies the SmolVLM2-2.2B-Instruct multimodal model to video frames selected by input TimeFrame annotations for prompt-driven captioning / scene description. Each invocation runs a single prompt against the TimeFrames selected by tfLabels; to apply different prompts to different label subsets (e.g. one prompt for slates, another for chyrons), run the app once per (prompt, tfLabels) combination. Per-TimeFrame captioning is composite: every frame sampled from a TF is fed to the model in a single prompt and yields one caption per TF. This app ships only the 2.2B-Instruct variant – the largest and most general-purpose model in the SmolVLM2 family. The smaller (256M and 500M) SmolVLM2 releases are post-trained specifically for video-QA tasks and we do not expect them to generalize well, given their size.
(Note: “*” as a property value means that the property is required but can be any value.)
http://clams.ai/vocabulary/type/VideoDocument/v2 (required) (of any properties)
http://clams.ai/vocabulary/type/TimeFrame/v6 (required)
Labeled TimeFrame annotations selecting which video segments to caption. Frame selection within each segment is controlled by the universal
tfSamplingModeparameter (see SDK docs). When present, therepresentativesproperty is consumed for representative-based sampling modes. Filter by label with thetfLabelsparameter.
(Note: Multivalued means the parameter can have one or more values.)
tfLabels: optional, defaults to []
Label(s) of input TimeFrame annotations to caption. By default (
[]), all TimeFrames are processed regardless of label. To restrict to specific labels, pass this parameter one or more times.
prompt: optional, defaults to You are looking at one or more frames sampled from a single segment of a news video. Describe what is shown, the purpose of this segment in the broader news video, and transcribe any visible text. Produce one consolidated caption across all provided frames.
User prompt(s) sent to the model. A single value runs as a one-shot generation. A multi-value list is interpreted as a multi-turn static prompt; see
promptModefor how turns are assembled.
systemPrompt: optional, defaults to ""
Optional system-role text prepended to the conversation. Empty by default.
promptMode: optional, defaults to turn-taking
user-only, turn-takingHow to interpret a multi-value
promptlist. Has no effect whenprompthas a single value. For semantics of each choice and worked examples, see https://clams.ai/clams-python/app-baseclasses.html#promptable-multiturn
maxNewTokens: optional, defaults to 200
Maximum number of new tokens generated per inference call. Forwarded to the backend’s
generate-equivalent. Larger values grow the KV cache linearly and increase GPU memory usage; reduce if VRAM is constrained.
temperature: optional, defaults to 0
Sampling temperature. The default
0.0selects deterministic / greedy decoding for maximum reproducibility; override for sampled generation.
topP: optional, defaults to 1
Nucleus-sampling cumulative probability cutoff. Only meaningful when
temperatureis greater than 0.
topK: optional, defaults to 50
Top-K sampling cutoff. Only meaningful when
temperatureis greater than 0.
parallelPrompts: optional, defaults to 1
Number of independent prompts the app runs in parallel (stacks into a single forward pass). The size of each prompt (how many images, how long the system/user text is, etc.) is NOT regulated by this parameter; that is each app’s responsibility. Prompt count and per-prompt content size combine multiplicatively for GPU memory, so the two can blow up together. Catastrophic example:
tfSamplingMode=allon a TimeFrame withouttargetsexpands that TF into one image per native-FPS frame (300 images for a 10-second TF at 30fps);parallelPrompts=4then runs 4 such prompts in one forward pass (~1200 images), guaranteed OOM. Keep at1on memory-tight setups; raise only when per-prompt content is small and bounded.
model: optional, defaults to HuggingFaceTB/SmolVLM2-2.2B-Instruct
HuggingFaceTB/SmolVLM2-2.2B-InstructHuggingFace model identifier to use for this request. Must be one of the model ids declared in this app’s
analyzer_versions; the SDK pins the corresponding commit hash at load time. When the app ships a single model (the typical case), this parameter defaults to that one model and can be omitted. Pass the full HF model id (e.g.org/repo-name); URL-encoding the/is optional.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to true
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
tfSamplingMode: optional, defaults to representatives
representatives, single, allSampling mode for TimeFrame annotations. Has no effect when the app does not process TimeFrames. “representatives” uses all representative timepoints if present, otherwise skips the TimeFrame. “single” uses the middle representative if present, otherwise extracts an image from the midpoint of the start/end interval (midpoint is calculated by floor division of the sum of start and end). “all” uses all target timepoints if present, otherwise extracts all images from the time interval.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Caption text generated by the SmolVLM2 model for each processed image. The
originsproperty points to theTimePointanchoring the image (an existing TimePoint reused when one already backs the image, or a TimePoint newly created in this view when the image was sampled from a TimeFrame interval without a backing TimePoint).
http://clams.ai/vocabulary/type/Alignment/v1 (of any properties)
Alignment between each parent TimeFrame and the TextDocument(s) derived from it.
Optional output. Newly-created TimePoint annotations for images that were sampled from a TimeFrame interval without an existing backing TimePoint (see
tfSamplingMode). When every sampled image came from an existing TimePoint, no TimePoints are created
Release Notes
New models with wider MLP classification head design
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnextv2_tiny
convnextv2_large, convnextv2_tinyModel name to use for classification, only applies when
useClassifier=true.
tpModelBatchSize: optional, defaults to 200
Number of images to process in a batch for classification. Smaller batch sizes will use less memory but may be slower. The default value of 200 is set to be the safely maximum size for “large” model running on desktop-grade GPU (12GB VRAM). Only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, collapse-close, collapse-close-reduce-difficulty, collapse-close-bin-lower-thirds, ignore-difficulties, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘IN:Chyron-person’, ‘CR:Credits’, ‘M:Main’, ‘Y:Chyron-other’, ‘KU:Chyron-other’, ‘GLOTW:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘P:-’, ‘-:-’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘IN:Chyron’, ‘CR:Credits’, ‘P:Neg’, ‘-:Neg’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘IN:Chyron-person’, ‘CR:Credits’, ‘Y:Other-text’, ‘KU:Other-text’, ‘GLOTW:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘M:Other-text’, ‘P:Neg’, ‘-:Neg’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘Y:Chyron’, ‘KU:Chyron’, ‘IN:Chyron’, ‘CR:Credits’, ‘GLOTW:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘M:Other-text’, ‘P:Neg’, ‘-:Neg’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’]
-binary-chyron-strict: [‘IN:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘KU:Chyron’, ‘IN:Chyron’]
-binary-credits: [‘CR:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to true
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
tfSamplingMode: optional, defaults to representatives
representatives, single, allSampling mode for TimeFrame annotations. Has no effect when the app does not process TimeFrames. “representatives” uses all representative timepoints if present, otherwise skips the TimeFrame. “single” uses the middle representative if present, otherwise extracts a frame from the midpoint of the start/end interval (midpoint is calculated by floor division of the sum of start and end). “all” uses all target timepoints if present, otherwise extracts all frames from the time interval.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Applies SmolVLM2-2.2B-Instruct multimodal model to video frames for image captioning.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/ImageDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v6 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
frameInterval: optional, defaults to 30
The interval at which to extract frames from the video if there are no timeframe annotations. Default is every 30 frames.
defaultPrompt: optional, defaults to Describe what is shown in this video frame. Analyze the purpose of this frame in the context of a news video. Transcribe any text present.
default prompt to use for timeframes not specified in the promptMap. If set to
-, timeframes not specified in the promptMap will be skipped.
promptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to new prompts. Must be formatted as “IN_LABEL:PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, any timeframe labels not mapped to a prompt will be used with the defaultprompt. In order to skip timeframes with a particular label, pass
-as the prompt value.in order to skip all timeframes not specified in the promptMap, set the defaultPromptparameter to-
defaultSystemPrompt: optional, defaults to ""
default system prompt to use for all timeframes. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior. The processor will format this properly using its chat template.
systemPromptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to system prompts. Must be formatted as “IN_LABEL:SYSTEM_PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior.
config: optional, defaults to config/default.yaml
Name of the config file to use.
num_beams: optional, defaults to 1
Number of beams for beam search during text generation. Default is 1. Higher values may improve quality but increase generation time.
batchSize: optional, defaults to 12
Number of images to process in each batch. Default is 12. Higher values may improve throughput but require more memory.
allRepresentatives: optional, defaults to false
false, trueDefault setting for processing all representative TimePoints in each TimeFrame. When true, all representatives are processed instead of just the first one. This can be overridden per-label in the config file using the all_representatives mapping (e.g., all_representatives: {slate: true, chyron: false}). Default is false (only the first representative is processed).
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
Release Notes
(no notes provided by the developer)
Applies SmolVLM2-2.2B-Instruct multimodal model to video frames for image captioning.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/ImageDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v6 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
frameInterval: optional, defaults to 30
The interval at which to extract frames from the video if there are no timeframe annotations. Default is every 30 frames.
defaultPrompt: optional, defaults to Describe what is shown in this video frame. Analyze the purpose of this frame in the context of a news video. Transcribe any text present.
default prompt to use for timeframes not specified in the promptMap. If set to
-, timeframes not specified in the promptMap will be skipped.
promptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to new prompts. Must be formatted as “IN_LABEL:PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, any timeframe labels not mapped to a prompt will be used with the defaultprompt. In order to skip timeframes with a particular label, pass
-as the prompt value.in order to skip all timeframes not specified in the promptMap, set the defaultPromptparameter to-
defaultSystemPrompt: optional, defaults to ""
default system prompt to use for all timeframes. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior. The processor will format this properly using its chat template.
systemPromptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to system prompts. Must be formatted as “IN_LABEL:SYSTEM_PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior.
config: optional, defaults to config/default.yaml
Name of the config file to use.
num_beams: optional, defaults to 1
Number of beams for beam search during text generation. Default is 1. Higher values may improve quality but increase generation time.
batchSize: optional, defaults to 12
Number of images to process in each batch. Default is 12. Higher values may improve throughput but require more memory.
allRepresentatives: optional, defaults to false
false, trueDefault setting for processing all representative TimePoints in each TimeFrame. When true, all representatives are processed instead of just the first one. This can be overridden per-label in the config file using the all_representatives mapping (e.g., all_representatives: {slate: true, chyron: false}). Default is false (only the first representative is processed).
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
Release Notes
(no notes provided by the developer)
CLAMS app wraps PySceneDetect and performs shot boundary detection on input videos
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
mode: optional, defaults to content
content, threshold, adaptivepick a scene detector algorithm, see http://scenedetect.com/projects/Manual/en/latest/cli/detectors.html
threshold: optional, defaults to 27
threshold value to use in the detection algorithm. Note that the meaning of this numerical value differs for different detector algorithms.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to true
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Applies SmolVLM2-2.2B-Instruct multimodal model to video frames for image captioning.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/ImageDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v6 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
frameInterval: optional, defaults to 30
The interval at which to extract frames from the video if there are no timeframe annotations. Default is every 30 frames.
defaultPrompt: optional, defaults to Describe what is shown in this video frame. Analyze the purpose of this frame in the context of a news video. Transcribe any text present.
default prompt to use for timeframes not specified in the promptMap. If set to
-, timeframes not specified in the promptMap will be skipped.
promptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to new prompts. Must be formatted as “IN_LABEL:PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, any timeframe labels not mapped to a prompt will be used with the defaultprompt. In order to skip timeframes with a particular label, pass
-as the prompt value.in order to skip all timeframes not specified in the promptMap, set the defaultPromptparameter to-
defaultSystemPrompt: optional, defaults to ""
default system prompt to use for all timeframes. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior. The processor will format this properly using its chat template.
systemPromptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to system prompts. Must be formatted as “IN_LABEL:SYSTEM_PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior.
config: optional, defaults to config/default.yaml
Name of the config file to use.
num_beams: optional, defaults to 1
Number of beams for beam search during text generation. Default is 1. Higher values may improve quality but increase generation time.
batchSize: optional, defaults to 12
Number of images to process in each batch. Default is 12. Higher values may improve throughput but require more memory.
allRepresentatives: optional, defaults to false
false, trueDefault setting for processing all representative TimePoints in each TimeFrame. When true, all representatives are processed instead of just the first one. This can be overridden per-label in the config file using the all_representatives mapping (e.g., all_representatives: {slate: true, chyron: false}). Default is false (only the first representative is processed).
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
Release Notes
(no notes provided by the developer)
Applies SmolVLM2-2.2B-Instruct multimodal model to video frames for image captioning.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/ImageDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v6 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
frameInterval: optional, defaults to 30
The interval at which to extract frames from the video if there are no timeframe annotations. Default is every 30 frames.
defaultPrompt: optional, defaults to Describe what is shown in this video frame. Analyze the purpose of this frame in the context of a news video. Transcribe any text present.
default prompt to use for timeframes not specified in the promptMap. If set to
-, timeframes not specified in the promptMap will be skipped.
promptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to new prompts. Must be formatted as “IN_LABEL:PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, any timeframe labels not mapped to a prompt will be used with the defaultprompt. In order to skip timeframes with a particular label, pass
-as the prompt value.in order to skip all timeframes not specified in the promptMap, set the defaultPromptparameter to-
defaultSystemPrompt: optional, defaults to ""
default system prompt to use for all timeframes. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior. The processor will format this properly using its chat template.
systemPromptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to system prompts. Must be formatted as “IN_LABEL:SYSTEM_PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior.
config: optional, defaults to config/default.yaml
Name of the config file to use.
num_beams: optional, defaults to 1
Number of beams for beam search during text generation. Default is 1. Higher values may improve quality but increase generation time.
batchSize: optional, defaults to 12
Number of images to process in each batch. Default is 12. Higher values may improve throughput but require more memory.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
Release Notes
(no notes provided by the developer)
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnextv2_tiny
convnextv2_large, convnextv2_tinyModel name to use for classification, only applies when
useClassifier=true.
tpModelBatchSize: optional, defaults to 200
Number of images to process in a batch for classification. Smaller batch sizes will use less memory but may be slower. The default value of 200 is set to be the safely maximum size for “large” model running on desktop-grade GPU (12GB VRAM). Only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, collapse-close, collapse-close-reduce-difficulty, collapse-close-bin-lower-thirds, ignore-difficulties, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘IN:Chyron-person’, ‘CR:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘KU:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’, ‘P:-’, ‘-:-’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘IN:Chyron’, ‘CR:Credits’, ‘P:Neg’, ‘-:Neg’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘IN:Chyron-person’, ‘CR:Credits’, ‘Y:Other-text’, ‘KU:Other-text’, ‘M:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘GLOTW:Other-text’, ‘P:Neg’, ‘-:Neg’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘Y:Chyron’, ‘KU:Chyron’, ‘IN:Chyron’, ‘CR:Credits’, ‘M:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘GLOTW:Other-text’, ‘P:Neg’, ‘-:Neg’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’]
-binary-chyron-strict: [‘IN:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘KU:Chyron’, ‘IN:Chyron’]
-binary-credits: [‘CR:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to true
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Applies SmolVLM2-2.2B-Instruct multimodal model to video frames for image captioning.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/ImageDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v6 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
frameInterval: optional, defaults to 30
The interval at which to extract frames from the video if there are no timeframe annotations. Default is every 30 frames.
defaultPrompt: optional, defaults to Describe what is shown in this video frame. Analyze the purpose of this frame in the context of a news video. Transcribe any text present.
default prompt to use for timeframes not specified in the promptMap. If set to
-, timeframes not specified in the promptMap will be skipped.
promptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to new prompts. Must be formatted as “IN_LABEL:PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, any timeframe labels not mapped to a prompt will be used with the defaultprompt. In order to skip timeframes with a particular label, pass
-as the prompt value.in order to skip all timeframes not specified in the promptMap, set the defaultPromptparameter to-
defaultSystemPrompt: optional, defaults to ""
default system prompt to use for all timeframes. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior. The processor will format this properly using its chat template.
systemPromptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to system prompts. Must be formatted as “IN_LABEL:SYSTEM_PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. System prompts are passed to the model using the messages format with role=”system”, providing context or instructions that guide the model’s behavior.
config: optional, defaults to config/default.yaml
Name of the config file to use.
num_beams: optional, defaults to 1
Number of beams for beam search during text generation. Default is 1. Higher values may improve quality but increase generation time.
batchSize: optional, defaults to 12
Number of images to process in each batch. Default is 12. Higher values may improve throughput but require more memory.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
Release Notes
What’s new
- updated to the latest SDK
- replaced aubio to avoid installation quirks
- removed
timeUnitandsampleSizeparams- renamed
lengthThresholdparam tominToneDuration
Detects spans of monotonic audio within an audio file
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
minToneDuration: optional, defaults to 2000
minimum length threshold (in ms)
stopAt: optional, defaults to 3600000
stop point for audio processing (in ms). Defaults to the length of the file
tolerance: optional, defaults to 1
threshold value for a “match” within audio processing
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
dependency version updates, fixes compat issue with new MMIF
inaSpeechSegmenter is a CNN-based audio segmentation toolkit. The original software can be found at https://github.com/ina-foss/inaSpeechSegmenter .
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
minTFDuration: optional, defaults to 0
minimum duration of a TimeFrame in milliseconds
silenceRatio: optional, defaults to 3
percentage ratio (0-100) of audio energy to to determine silence, ratio to mean every of the input audio.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
The INA semgmenter uses 5-way classification ([‘noEnergy’, ‘female’, ‘male’, ‘noise’, ‘music’]) and this wrapper remaps the labels to [‘silence’, ‘speech’, ‘noise’, ‘music’], by 1) renaming
noEnergytosilence2) collapsingfemaleandmaleintospeech(leaving additionalgenderproperty). Note that the time frame annotations do not exhaustively cover the input audio, but only the segments.
Release Notes
(no notes provided by the developer)
Chunk-level language ID over audio based on OpenAI Whisper
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
model: optional, defaults to tiny
tiny, base, small, medium, large, turboWhisper model size
chunk: optional, defaults to 30
chunk/window length in seconds
top: optional, defaults to 3
top-k language scores
batchSize: optional, defaults to 1
number of windows processed in a batch
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
experiment first release of the app
**Wraps the NVIDIA NeMo Forced Aligner tool to temporally align transcribed text with its audio source. **
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
http://mmif.clams.ai/vocabulary/TextDocument/v1 (required) (of any properties)
Text content transcribed from audio input with no existing annotations.
(Note: Multivalued means the parameter can have one or more values.)
model: optional, defaults to fc_hybrid
fc_hybrid, parakeet, conformer, fc_ctcNeMo ASR model to use. Choices: fc_hybrid, parakeet, conformer, fc_ctc. By default, the fc_hybrid model will be used.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/Token/v1 (of any properties)
Token from original text split on whitespace.
textproperty stores the string value of the token.startandendproperties indicate position of token in entire text.documentproperty identifies source text document.
TimeFrame annotation representing the source audio segment corresponding to a given transcribed token, with
startandendtimes given in milliseconds.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignment between
TokenandTimeFrameannotations.
Release Notes
This release includes a new parameter and a bugfix
tpModelBatchSizeparameter is added to control memory usage for TP annotation- fixed bug when user provided
tfLabelMapdoesn’t match TP’s labelSet
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnextv2_tiny
convnextv2_large, convnextv2_tinyModel name to use for classification, only applies when
useClassifier=true.
tpModelBatchSize: optional, defaults to 200
Number of images to process in a batch for classification. Smaller batch sizes will use less memory but may be slower. The default value of 200 is set to be the safely maximum size for “large” model running on desktop-grade GPU (12GB VRAM). Only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, collapse-close, collapse-close-reduce-difficulty, collapse-close-bin-lower-thirds, ignore-difficulties, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘IN:Chyron-person’, ‘CR:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘KU:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’, ‘P:-’, ‘-:-’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘IN:Chyron’, ‘CR:Credits’, ‘P:Neg’, ‘-:Neg’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘IN:Chyron-person’, ‘CR:Credits’, ‘Y:Other-text’, ‘KU:Other-text’, ‘M:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘GLOTW:Other-text’, ‘P:Neg’, ‘-:Neg’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘Y:Chyron’, ‘KU:Chyron’, ‘IN:Chyron’, ‘CR:Credits’, ‘M:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘GLOTW:Other-text’, ‘P:Neg’, ‘-:Neg’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’]
-binary-chyron-strict: [‘IN:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘KU:Chyron’, ‘IN:Chyron’]
-binary-credits: [‘CR:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
updated to the new whisper (python dependecy compatibilty)
Chunk-level language ID over audio based on OpenAI Whisper
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
model: optional, defaults to tiny
tiny, base, small, medium, large, turboWhisper model size
chunk: optional, defaults to 30
chunk/window length in seconds
top: optional, defaults to 3
top-k language scores
batchSize: optional, defaults to 1
number of windows processed in a batch
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
updated to whisper (2025 june ver)
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
model: optional, defaults to turbo
(from openai-whisper CLI) name of the Whisper model to use
language: optional, defaults to ""
(from openai-whisper CLI) language spoken in the audio, specify None to perform language detection
task: optional, defaults to transcribe
transcribe, translate(from openai-whisper CLI) whether to perform X->X speech recognition (‘transcribe’) or X->English translation (‘translate’)
initialPrompt: optional, defaults to ""
(from openai-whisper CLI) optional text to provide as a prompt for the first window.
conditionOnPreviousText: optional, defaults to True
(from openai-whisper CLI) if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop
noSpeechThreshold: optional, defaults to 0.6
(from openai-whisper CLI) if the probability of the < nospeech > token is higher than this value AND the decoding has failed due to logprob_threshold, consider the segment as silence
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
image processor comes with convnext models are now properly set to use the same loader as they are originally saved
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnextv2_tiny
convnextv2_large, convnextv2_tinyModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, collapse-close, collapse-close-reduce-difficulty, collapse-close-bin-lower-thirds, ignore-difficulties, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘IN:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘KU:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘IN:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘IN:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’, ‘GLOTW:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘KU:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘IN:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’, ‘GLOTW:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’, ‘IN:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘KU:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘IN:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’, ‘CR:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Major update includes …
modelSizeparameter is nowmodelmodelLangparameter is nowlanguage
(above changes to match openai’s CLI argument names)- fixed app crashing when word-level timestamping outputs hallucinations
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
model: optional, defaults to turbo
(from openai-whisper CLI) name of the Whisper model to use
language: optional, defaults to ""
(from openai-whisper CLI) language spoken in the audio, specify None to perform language detection
task: optional, defaults to transcribe
transcribe, translate(from openai-whisper CLI) whether to perform X->X speech recognition (‘transcribe’) or X->English translation (‘translate’)
initialPrompt: optional, defaults to ""
(from openai-whisper CLI) optional text to provide as a prompt for the first window.
conditionOnPreviousText: optional, defaults to True
(from openai-whisper CLI) if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop
noSpeechThreshold: optional, defaults to 0.6
(from openai-whisper CLI) if the probability of the < nospeech > token is higher than this value AND the decoding has failed due to logprob_threshold, consider the segment as silence
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
fixed import errors and outdated metadata
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnextv2_tiny
convnextv2_tiny, convnextv2_largeModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, collapse-close, collapse-close-reduce-difficulty, collapse-close-bin-lower-thirds, ignore-difficulties, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘IN:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘KU:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘IN:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘IN:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’, ‘GLOTW:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘KU:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘IN:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’, ‘GLOTW:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’, ‘IN:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘KU:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘IN:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’, ‘CR:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
experimental release of spoken LID app prototype based on whisper
Chunk-level language ID over audio based on OpenAI Whisper
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
model: optional, defaults to tiny
tiny, base, small, medium, large, turboWhisper model size
chunk: optional, defaults to 30
chunk/window length in seconds
top: optional, defaults to 3
top-k language scores
batchSize: optional, defaults to 1
number of windows processed in a batch
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnextv2_tiny
convnextv2_tiny, convnextv2_largeModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, collapse-close, collapse-close-reduce-difficulty, collapse-close-bin-lower-thirds, ignore-difficulties, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘IN:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘KU:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘IN:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘IN:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’, ‘GLOTW:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘KU:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘IN:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘CR:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’, ‘GLOTW:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’, ‘IN:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘KU:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘IN:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’, ‘CR:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
CLAMS app wraps the docTR, End-to-End OCR model. The model can detect text regions in the input image and recognize text in the regions (via parseq OCR model, only English is support at the moment). The text-localized regions are organized hierarchically by the model into “pages” > “blocks” > “lines” > “words”, and this CLAMS app translates them into TextDocument, Paragraphs, Sentence, and Token annotations to represent recognized text contents. See descriptions for I/O types below for details on how annotations are aligned to each other.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v6 (required)
The labeled TimeFrame annotation that represents the video segment to be processed. When
representativesproperty is present, the app will process videos still frames at the underlying time point annotations that are referred to by therepresentativesproperty. Otherwise, the app will process the middle frame of the video segment. Generic TimeFrames with nolabelproperty will not be processed.
(Note: Multivalued means the parameter can have one or more values.)
tfLabel: optional, defaults to []
The label of the TimeFrame annotation to be processed. By default (
[]), all TimeFrame annotations will be processed, regardless of theirlabelproperty values.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input images. Serialization isdone by concatenating
textvalues ofParagraphannotations with two newline characters.
Translation of the recognized docTR “words” in the input images.
textandwordproperties store the string values of the recognized text. The duplication is for keepingbackward compatibility and consistency withParagraphandSentenceannotations.
Translation of the recognized docTR “lines” in the input images.
textproperty stores the string value of space-joined words.
Translation of the recognized docTR “blocks” in the input images.
textproperty stores the string value of newline-joined sentences.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimePoint<->TextDocument, 2)TimePoint<->Token/Sentence/Paragraph, 3)BoundingBox<->Token/Sentence/Paragraph
Bounding boxes of the detected text regions in the input images. No corresponding box for the entire image (
TextDocument) region
Release Notes
SDK update
This tool applies Tesseract OCR to a video or image and generates text boxes and OCR results. Currenly only support English language.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v6 (required)
The Time frame annotation that represents the video segment to be processed. When
representativesproperty is present, the app will process videos still frames at the underlying time point annotations that are referred to by therepresentativesproperty. Otherwise, the app will process the middle frame of the video segment.
(Note: Multivalued means the parameter can have one or more values.)
tfLabel: optional, defaults to []
The label of the TimeFrame annotation to be processed. By default (
[]), all TimeFrame annotations will be processed, regardless of theirlabelproperty values.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input images. Serialization isdone by concatenating
textvalues ofParagraphannotations with two newline characters.
Translation of the recognized tesseract “words” in the input images.
tokenproperties store the string values of the recognized text. The duplication is for keepingbackward compatibility and consistency withParagraphandSentenceannotations.
Translation of the recognized tesseract “lines” in the input images.
sentenceproperty from LAPPS vocab stores the string value of space-joined words.
Translation of the recognized tesseract “blocks” in the input images.
paragraphproperty from LAPPS vocab stores the string value of newline-joined sentences.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimePoint<->TextDocument, 2)TimePoint<->Token/Sentence/Paragraph, 3)BoundingBox<->Token/Sentence/Paragraph
Bounding boxes of the detected text regions in the input images. No corresponding box for the entire image (
TextDocument) region
Release Notes
(no notes provided by the developer)
A CLAMS wrapper for NVIDIA NeMo Parakeet ASR models available on huggingface-hub with support for punctuation, capitalization, and word-level timestamping.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
contextSize: optional, defaults to 400
Local attention context size for the model. Can be any positive integer, or 0 to set global (full-context) attention. Larger context sizes may improve performance but require a lot more memory. For desktop CUDA device with 12GB VRAM, a context size of around 100 is recommended for full utilization of VRAM. Default is 400
modelSize: optional, defaults to 0.6b
110m, 0.6b, 1.1bParakeet model size to use. Choices: 110m, 0.6b, 1.1b
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v6 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
http://vocab.lappsgrid.org/Sentence (of any properties)
Release Notes
Updated to the latest clams SDK, annotation ID format changed to “long-form”.
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnext_small
convnext_lg, convnext_tiny, convnext_smallModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
CLAMS SDK and spacy version bump
Apply spaCy NLP to all text documents in a MMIF file.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (required) (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
pretokenized: optional, defaults to false
false, trueBoolean parameter to set the app to use existing tokenization, if available, for text documents for NLP processing. Useful to process ASR documents, for example.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://vocab.lappsgrid.org/Token (of any properties)
http://vocab.lappsgrid.org/Token#pos (of any properties)
http://vocab.lappsgrid.org/Token#lemma (of any properties)
http://vocab.lappsgrid.org/NounChunk (of any properties)
http://vocab.lappsgrid.org/Sentence (of any properties)
http://vocab.lappsgrid.org/NamedEntity (of any properties)
Release Notes
Initial release with naive line parsing and name normalization
Prototype to convert chyron text from docTR/Tesseract/LLaVA MMIF outputinto a name and list of attributes.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (required) (of any properties)
Text content transcribed from video input by docTR/Tesseract/LLAVA.
(Note: Multivalued means the parameter can have one or more values.)
note4mode: optional, defaults to false
false, trueBoolean to set the app to run in “note-4” mode and to take the second line (if available) from the input text to be the
name-normalizedvalue. The default is false, which means the app will try to generate normalization fromname-as-written(from the first line) value.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Reformatted chyron text.
documentproperty stores the ID of the original sourceVideoDocument.originproperty stores the ID of the original OCRTextDocumentannotation. Reformatted text is escaped JSON string with three fields:name-as-written,name-normalized, andattributes.
Release Notes
Updated to Tesseract5, and output MMIF format matches other TR apps
This tool applies Tesseract OCR to a video or image and generates text boxes and OCR results. Currenly only support English language.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v5 (required)
The Time frame annotation that represents the video segment to be processed. When
representativesproperty is present, the app will process videos still frames at the underlying time point annotations that are referred to by therepresentativesproperty. Otherwise, the app will process the middle frame of the video segment.
(Note: Multivalued means the parameter can have one or more values.)
tfLabel: optional, defaults to []
The label of the TimeFrame annotation to be processed. By default (
[]), all TimeFrame annotations will be processed, regardless of theirlabelproperty values.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input images. Serialization isdone by concatenating
textvalues ofParagraphannotations with two newline characters.
Translation of the recognized tesseract “words” in the input images.
tokenproperties store the string values of the recognized text. The duplication is for keepingbackward compatibility and consistency withParagraphandSentenceannotations.
Translation of the recognized tesseract “lines” in the input images.
sentenceproperty from LAPPS vocab stores the string value of space-joined words.
Translation of the recognized tesseract “blocks” in the input images.
paragraphproperty from LAPPS vocab stores the string value of newline-joined sentences.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimePoint<->TextDocument, 2)TimePoint<->Token/Sentence/Paragraph, 3)BoundingBox<->Token/Sentence/Paragraph
Bounding boxes of the detected text regions in the input images. No corresponding box for the entire image (
TextDocument) region
Release Notes
fixed critical bug that caused app crash
CLAMS app wraps the docTR, End-to-End OCR model. The model can detect text regions in the input image and recognize text in the regions (via parseq OCR model, only English is support at the moment). The text-localized regions are organized hierarchically by the model into “pages” > “blocks” > “lines” > “words”, and this CLAMS app translates them into TextDocument, Paragraphs, Sentence, and Token annotations to represent recognized text contents. See descriptions for I/O types below for details on how annotations are aligned to each other.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v5 (required)
The labeled TimeFrame annotation that represents the video segment to be processed. When
representativesproperty is present, the app will process videos still frames at the underlying time point annotations that are referred to by therepresentativesproperty. Otherwise, the app will process the middle frame of the video segment. Generic TimeFrames with nolabelproperty will not be processed.
(Note: Multivalued means the parameter can have one or more values.)
tfLabel: optional, defaults to []
The label of the TimeFrame annotation to be processed. By default (
[]), all TimeFrame annotations will be processed, regardless of theirlabelproperty values.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input images. Serialization isdone by concatenating
textvalues ofParagraphannotations with two newline characters.
Translation of the recognized docTR “words” in the input images.
textandwordproperties store the string values of the recognized text. The duplication is for keepingbackward compatibility and consistency withParagraphandSentenceannotations.
Translation of the recognized docTR “lines” in the input images.
textproperty stores the string value of space-joined words.
Translation of the recognized docTR “blocks” in the input images.
textproperty stores the string value of newline-joined sentences.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimePoint<->TextDocument, 2)TimePoint<->Token/Sentence/Paragraph, 3)BoundingBox<->Token/Sentence/Paragraph
Bounding boxes of the detected text regions in the input images. No corresponding box for the entire image (
TextDocument) region
Release Notes
Fixed .en suffix for whisper large and turbo models.* (closes #17)
Added validation for unsupported modelLang codes;
Updated wording in matadata.
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, True, base, b, small, s, medium, m, large, l, large-v2, l2, large-v3, l3, turbo, tuThe size of the model to use. When
modelLang=enis given, for non-largeand non -turbomodels, English-only models will be used instead of multilingual models for speed and accuracy. (Forlargeandturbomodels, English-only models are not available.) (also can be given as alias: tiny=t, base=b, small=s, medium=m, large=l, large-v2=l2, large-v3=l3, turbo=tu)
modelLang: optional, defaults to ""
Language of the model to use, accepts two- or three-letter ISO 639 language codes, however Whisper only supports a subset of languages. If the language is not supported, error will be raised.For the full list of supported languages, see https://github.com/openai/whisper/blob/20240930/whisper/tokenizer.py . In addition to the langauge code, two-letter region codes can be added to the language code, e.g. “en-US” for US English. Note that the region code is only for compatibility and recording purpose, and Whisper neither detects regional dialects, nor use the given one for transcription. When the langauge code is not given, Whisper will run in langauge detection mode, and will use first few seconds of the audio to detect the language.
task: optional, defaults to transcribe
transcribe, translate(from whisper CLI) whether to perform X->X speech recognition (‘transcribe’) or X->English translation (‘translate’)
initialPrompt: optional, defaults to ""
(from whisper CLI) optional text to provide as a prompt for the first window.
conditionOnPreviousText: optional, defaults to true
false, true(from whisper CLI) if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop
noSpeechThreshold: optional, defaults to 0.6
(from whisper CLI) if the probability of the < nospeech > token is higher than this value AND the decoding has failed due to logprob_threshold, consider the segment as silence
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
Made classification deterministic, meaning repeatedly running the app on the same video would result in the same
classificationsscores.
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnext_small
convnext_tiny, convnext_lg, convnext_smallModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Minor update to support latest SDK
- updated to the latext SDK, and hence added
cli.pyentry point- in various newly-generated annotation objects, all references to other annotation ID are now “long” form (
view_id:ann_id) even when referring annotations within the same view- now only processes
TimeFramewithlabelproperty (effectively ignores time frames from, for example, whisper-wrapper)- made model caching path inside the container consistent with other CLAMS apps (
/cache/doctr)
CLAMS app wraps the docTR, End-to-End OCR model. The model can detect text regions in the input image and recognize text in the regions (via parseq OCR model, only English is support at the moment). The text-localized regions are organized hierarchically by the model into “pages” > “blocks” > “lines” > “words”, and this CLAMS app translates them into TextDocument, Paragraphs, Sentence, and Token annotations to represent recognized text contents. See descriptions for I/O types below for details on how annotations are aligned to each other.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v5 (required)
The labeled TimeFrame annotation that represents the video segment to be processed. When
representativesproperty is present, the app will process videos still frames at the underlying time point annotations that are referred to by therepresentativesproperty. Otherwise, the app will process the middle frame of the video segment. Generic TimeFrames with nolabelproperty will not be processed.
(Note: Multivalued means the parameter can have one or more values.)
tfLabel: optional, defaults to []
The label of the TimeFrame annotation to be processed. By default (
[]), all TimeFrame annotations will be processed, regardless of theirlabelproperty values.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input images. Serialization isdone by concatenating
textvalues ofParagraphannotations with two newline characters.
Translation of the recognized docTR “words” in the input images.
textandwordproperties store the string values of the recognized text. The duplication is for keepingbackward compatibility and consistency withParagraphandSentenceannotations.
Translation of the recognized docTR “lines” in the input images.
textproperty stores the string value of space-joined words.
Translation of the recognized docTR “blocks” in the input images.
textproperty stores the string value of newline-joined sentences.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimePoint<->TextDocument, 2)TimePoint<->Token/Sentence/Paragraph, 3)BoundingBox<->Token/Sentence/Paragraph
Bounding boxes of the detected text regions in the input images. No corresponding box for the entire image (
TextDocument) region
Release Notes
Re-release of 7.4 with support for prebuilt image for arm64 architecture
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnext_small
convnext_small, convnext_tiny, convnext_lgModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Applies LLaVA v1.6 Mistral-7B to video frames for image captioning.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/ImageDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v5 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
frameInterval: optional, defaults to 30
The interval at which to extract frames from the video if there are no timeframe annotations. Default is every 30 frames.
defaultPrompt: optional, defaults to Describe what is shown in this video frame. Analyze the purpose of this frame in the context of a news video. Transcribe any text present.
default prompt to use for timeframes not specified in the promptMap. If set to
-, timeframes not specified in the promptMap will be skipped.
promptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to new prompts. Must be formatted as “IN_LABEL:PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, any timeframe labels not mapped to a prompt will be used with the defaultprompt. In order to skip timeframes with a particular label, pass
-as the prompt value.in order to skip all timeframes not specified in the promptMap, set the defaultPromptparameter to-
config: optional, defaults to config/default.yaml
Name of the config file to use.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
Release Notes
fixed rare rounding error with certain combo of sample_rate and video duration
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnext_small
convnext_small, convnext_tiny, convnext_lgModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
minor bugfix in the second (
TimeFrame) view metadata.
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnext_small
convnext_lg, convnext_tiny, convnext_smallModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Release to fix bugs…
- fixed unable to disable tfLabelMapPreset by nopreset value
- fixed some
tf-parameters weren’t recorded in the timepoint-only view
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnext_small
convnext_lg, convnext_tiny, convnext_smallModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits, nopreset(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Release with newly trained models:
- training data is expanded with new annotations: https://github.com/clamsproject/aapb-annotations/pull/98 and https://github.com/clamsproject/aapb-annotations/pull/104 .
- label
Uis added, total number of “raw” labels is now 18.- in additional to
convnext_lgandconvnext_tiny,convnext_small-based models are added. The default is nowconvnext_smallmodel.
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnext_small
convnext_lg, convnext_tiny, convnext_smallModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to []
(See also
tfLabelMapPreset, settfLabelMapPreset=nopresetto make sure that a preset does not overridetfLabelMapwhen using this) Mapping of a label in the input TimePoint annotations to a new label of the stitched TimeFrame annotations. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When two+ TP labels are mapped to a TF label, it essentially works as a “binning” operation. If no mapping is used, all the input labels are passed-through, meaning no change in both TP & TF labelsets. However, when at least one label is mapped, all the other “unset” labels are mapped to the negative label (-) and if-does not exist in the TF labelset, it is added automatically. Only applies whenuseStitcher=true.
tfLabelMapPreset: optional, defaults to relaxed
noprebin, nomap, strict, simpler, simple, relaxed, binary-bars, binary-slate, binary-chyron-strict, binary-chyron-relaxed, binary-credits(See also
tfLabelMap) Preset alias of a label mapping. If notnopreset, this parameter will override thetfLabelMapparameter. Available presets are:
-noprebin: []
-nomap: []
-strict: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Main’, ‘O:Opening’, ‘W:Opening’, ‘Y:Chyron-other’, ‘U:Chyron-other’, ‘K:Chyron-other’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-simpler: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’]
-simple: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘I:Chyron-person’, ‘N:Chyron-person’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘Y:Other-text’, ‘U:Other-text’, ‘K:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-relaxed: [‘B:Bars’, ‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’, ‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’, ‘C:Credits’, ‘R:Credits’, ‘M:Other-text’, ‘O:Other-text’, ‘W:Other-text’, ‘L:Other-text’, ‘G:Other-text’, ‘F:Other-text’, ‘E:Other-text’, ‘T:Other-text’]
-binary-bars: [‘B:Bars’]
-binary-slate: [‘S:Slate’, ‘S:H:Slate’, ‘S:C:Slate’, ‘S:D:Slate’, ‘S:B:Slate’, ‘S:G:Slate’]
-binary-chyron-strict: [‘I:Chyron-person’, ‘N:Chyron-person’]
-binary-chyron-relaxed: [‘Y:Chyron’, ‘U:Chyron’, ‘K:Chyron’, ‘I:Chyron’, ‘N:Chyron’]
-binary-credits: [‘C:Credits’, ‘R:Credits’]
Only applies whenuseStitcher=true.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
This version re-implements stitcher based on
simple-timepoints-stitcher
- app now can run stitch-only mode (
useClassifieranduseStitcher)
- simple-timepoints-stitcher app will retire
- prefixed all parameters with their corresponding modes (e.g.,
sampleRat e>tpSampleRate,minTPScore>tfMinTPScore- changes to parameters
minTFCount(frame count-based) becametfMinTFDuration(time-based)mapbecametfLabelMapto clarify what “map” the param setstfDynamicSceneLabelsis added to configure dynamic scene types that need multiple representative images/timepoints (defaults to [credit, ` credits`])- default values for stitcher parameters are changed based on recent experiments. Most significantly now
minTPScoredefaults to 0.5 andminTFScoredefaults to 0.9. See https://github.com/clamsproject/aapb-evaluations/issues/60 to read the full experimental reports.- changes to app behavior
- new stitcher implementation is not exactly the same as the old, and users should expect more “break-ups” in the middle of long time frames
- for dynamic scene types, the gap between representative time points is now twice the
tfMinTFDurationvalue- image classification is now done in batches (currently fixed to size 2000) to reduce memory usage. This will add some time overhead to image extraction process
Detects scenes with text, like slates, chyrons and credits. This app can run in three modes, depending on useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
useClassifier: optional, defaults to true
false, trueUse the image classifier model to generate TimePoint annotations.
tpModelName: optional, defaults to convnext_lg
convnext_tiny, convnext_lgModel name to use for classification, only applies when
useClassifier=true.
tpUsePosModel: optional, defaults to true
false, trueUse the model trained with positional features, only applies when
useClassifier=true.
tpStartAt: optional, defaults to 0
Number of milliseconds into the video to start processing, only applies when
useClassifier=true.
tpStopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing, only applies when
useClassifier=true.
tpSampleRate: optional, defaults to 1000
Milliseconds between sampled frames, only applies when
useClassifier=true.
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints.
tfMinTPScore: optional, defaults to 0.5
Minimum score for a TimePoint to be included in a TimeFrame. A lower value will include more TimePoints in the TimeFrame (increasing recall in exchange for precision). Only applies when
useStitcher=true.
tfMinTFScore: optional, defaults to 0.9
Minimum score for a TimeFrame. A lower value will include more TimeFrames in the output (increasing recall in exchange for precision). Only applies when
useStitcher=true
tfMinTFDuration: optional, defaults to 5000
Minimum duration of a TimeFrame in milliseconds, only applies when
useStitcher=true.
tfAllowOverlap: optional, defaults to false
false, trueAllow overlapping time frames, only applies when
useStitcher=true
tfDynamicSceneLabels: optional, defaults to ['credit', 'credits']
Labels that are considered dynamic scenes. For dynamic scenes, TimeFrame annotations contains multiple representative points to follow any changes in the scene. Only applies when
useStitcher=true
tfLabelMap: optional, defaults to ['B:bars', 'S:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credits', 'R:credits', 'W:other_opening', 'L:other_opening', 'O:other_opening', 'M:other_opening', 'E:other_text', 'K:other_text', 'G:other_text', 'T:other_text', 'F:other_text']
Mapping of a label in the input annotations to a new label. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. When no remap is used, all the input labels are passed as is, including any negative labels (
-). However, when at least one label is remapped, all the other “unset” labels are remapped to the negative label (-). Only applies whenuseStitcher=true
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
added support for arm64 container image
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, True, base, b, small, s, medium, m, large, l, large-v2, l2, large-v3, l3, turbo, tuThe size of the model to use. When
modelLang=enis given, for non-largemodels, English-only models will be used instead of multilingual models for speed and accuracy. (Forlargemodels, English-only models are not available.) (also can be given as alias: tiny=t, base=b, small=s, medium=m, large=l, large-v2=l2, large-v3=l3, turbo=tu)
modelLang: optional, defaults to ""
Language of the model to use, accepts two- or three-letter ISO 639 language codes, however Whisper only supports a subset of languages. If the language is not supported, error will be raised.For the full list of supported languages, see https://github.com/openai/whisper/blob/20240930/whisper/tokenizer.py . In addition to the langauge code, two-letter region codes can be added to the language code, e.g. “en-US” for US English. Note that the region code is only for compatibility and recording purpose, and Whisper neither detects regional dialects, nor use the given one for transcription. When the langauge code is not given, Whisper will run in langauge detection mode, and will use first few seconds of the audio to detect the language.
task: optional, defaults to transcribe
transcribe, translate(from whisper CLI) whether to perform X->X speech recognition (‘transcribe’) or X->English translation (‘translate’)
initialPrompt: optional, defaults to ""
(from whisper CLI) optional text to provide as a prompt for the first window.
conditionOnPreviousText: optional, defaults to true
false, true(from whisper CLI) if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop
noSpeechThreshold: optional, defaults to 0.6
(from whisper CLI) if the probability of the < nospeech > token is higher than this value AND the decoding has failed due to logprob_threshold, consider the segment as silence
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
- Now based on whisper 240930 version with support of
turbomodel- Beam search size is set to 5 for the decoder, following the
whispercommand’s default- (temporarily) Disabled multiprocessing web app via gunicorn and fell back to flask built-in to work around CUDA memory issue
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, True, base, b, small, s, medium, m, large, l, large-v2, l2, large-v3, l3, turbo, tuThe size of the model to use. When
modelLang=enis given, for non-largemodels, English-only models will be used instead of multilingual models for speed and accuracy. (Forlargemodels, English-only models are not available.) (also can be given as alias: tiny=t, base=b, small=s, medium=m, large=l, large-v2=l2, large-v3=l3, turbo=tu)
modelLang: optional, defaults to ""
Language of the model to use, accepts two- or three-letter ISO 639 language codes, however Whisper only supports a subset of languages. If the language is not supported, error will be raised.For the full list of supported languages, see https://github.com/openai/whisper/blob/20240930/whisper/tokenizer.py . In addition to the langauge code, two-letter region codes can be added to the language code, e.g. “en-US” for US English. Note that the region code is only for compatibility and recording purpose, and Whisper neither detects regional dialects, nor use the given one for transcription. When the langauge code is not given, Whisper will run in langauge detection mode, and will use first few seconds of the audio to detect the language.
task: optional, defaults to transcribe
transcribe, translate(from whisper CLI) whether to perform X->X speech recognition (‘transcribe’) or X->English translation (‘translate’)
initialPrompt: optional, defaults to ""
(from whisper CLI) optional text to provide as a prompt for the first window.
conditionOnPreviousText: optional, defaults to true
false, true(from whisper CLI) if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop
noSpeechThreshold: optional, defaults to 0.6
(from whisper CLI) if the probability of the < nospeech > token is higher than this value AND the decoding has failed due to logprob_threshold, consider the segment as silence
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
This version adds some delegation parameters to whisper.transcribe
task: delegate to--taskinitialPrompt: delegate to--initial-promptconditionOnPreviousText: delegate to--condition-on-previous-textnoSpeechThreshold: delegate to--no-speech-threshold
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, True, base, b, small, s, medium, m, large, l, large-v2, l2, large-v3, l3The size of the model to use. When
modelLang=enis given, for non-largemodels, English-only models will be used instead of multilingual models for speed and accuracy. (Forlargemodels, English-only models are not available.) (also can be given as alias: tiny=t, base=b, small=s, medium=m, large=l, large-v2=l2, large-v3=l3)
modelLang: optional, defaults to ""
Language of the model to use, accepts two- or three-letter ISO 639 language codes, however Whisper only supports a subset of languages. If the language is not supported, error will be raised.For the full list of supported languages, see https://github.com/openai/whisper/blob/20231117/whisper/tokenizer.py . In addition to the langauge code, two-letter region codes can be added to the language code, e.g. “en-US” for US English. Note that the region code is only for compatibility and recording purpose, and Whisper neither detects regional dialects, nor use the given one for transcription. When the langauge code is not given, Whisper will run in langauge detection mode, and will use first few seconds of the audio to detect the language.
task: optional, defaults to transcribe
transcribe, translate(from whisper CLI) whether to perform X->X speech recognition (‘transcribe’) or X->English translation (‘translate’)
initialPrompt: optional, defaults to ""
(from whisper CLI) optional text to provide as a prompt for the first window.
conditionOnPreviousText: optional, defaults to true
false, true(from whisper CLI) if True, provide the previous output of the model as a prompt for the next window; disabling may make the text inconsistent across windows, but the model becomes less prone to getting stuck in a failure loop
noSpeechThreshold: optional, defaults to 0.6
(from whisper CLI) if the probability of the < nospeech > token is higher than this value AND the decoding has failed due to logprob_threshold, consider the segment as silence
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
fixed inability to handle concurrent requests
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, True, base, b, small, s, medium, m, large, l, large-v2, l2, large-v3, l3The size of the model to use. When
modelLang=enis given, for non-largemodels, English-only models will be used instead of multilingual models for speed and accuracy. (Forlargemodels, English-only models are not available.) (also can be given as alias: tiny=t, base=b, small=s, medium=m, large=l, large-v2=l2, large-v3=l3)
modelLang: optional, defaults to ""
Language of the model to use, accepts two- or three-letter ISO 639 language codes, however Whisper only supports a subset of languages. If the language is not supported, error will be raised.For the full list of supported languages, see https://github.com/openai/whisper/blob/20231117/whisper/tokenizer.py . In addition to the langauge code, two-letter region codes can be added to the language code, e.g. “en-US” for US English. Note that the region code is only for compatibility and recording purpose, and Whisper neither detects regional dialects, nor use the given one for transcription. When the langauge code is not given, Whisper will run in langauge detection mode, and will use first few seconds of the audio to detect the language.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
reverting back to HF pipeline using chunking transcription
The wrapper of Distil-Whisper, avaliable models: distil-large-v3, distil-large-v2, distil-medium.en, distil-small.en. The default model is distil-small.en.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to distil-small.en
distil-large-v3, distil-large-v2, distil-medium.en, distil-small.en, small, s, medium, m, large-v2, l2, large-v3, l3The size of the model to use. There are four size of model to use distil-large-v3, distil-large-v2, distil-medium.en, distil-small.en. You can also enter the abbreviation of the model as parameter. ‘small’ and ‘s’ for distil-small.en; ‘medium’ and ‘m’ for distil-medium.en; ‘large-v2’ and ‘l2’ for distil-large-v2; ‘large-v3’ and ‘l3’ for distil-large-v3. The default model is distil-medium.en.)
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input audio/video.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimeFrame<->SENTENCE, 2)audio/video document<->TextDocument
http://vocab.lappsgrid.org/Sentence (of any properties)
The smallest recognized unit of distil-whisper. Normally a complete sentence.
Release Notes
Now can filter overlapping time frames
- via
allowOverlapparameter (default is true to keep the backward compatibility)- the filtering algorithm (based on https://github.com/clamsproject/app-swt-detection/blob/5092274d1ead64f21e0e3c2bbb7aa3ad4db75cd4/modeling/stitch.py#L88-L104)
- sort collected positive TP sequences based on their mean scores, regardless of the labels
- loop through the sorted seqs, if a seq in the loop does not include any “seen” TP, mark the seq as “final” and mark all the underlying TPs as “seen”, otherwise skip that one.
- return the “final” frames, re-sorted by the start TPs
Stitches a sequence of TimePoint annotations into a sequence of TimeFrame annotations, performing simple smoothing of short peaks of positive labels.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
TimePoint annotations to be stitched. Must be “exhaustive” in that it should cover an entire single time period in the input document, with a uniform sample rate.
(Note: Multivalued means the parameter can have one or more values.)
labelMap: optional, defaults to []
mapping of labels in the input annotations to new labels. Must be formatted as “IN_LABEL:OUT_LABEL” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any “negative” labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded as the negative label(“-“).
minTFDuration: optional, defaults to 1000
minimum duration of a TimeFrame in milliseconds
minTPScore: optional, defaults to 0.1
minimum score of a TimePoint to be considered as positive
minTFScore: optional, defaults to 0.5
minimum average score of TimePoints in a TimeFrame to be considered as positive
labelMapPreset: optional, defaults to null
null, swt-v4-4way, swt-v4-6waypreset of label mappings. If not
null, this parameter will override thelabelMapparameter. Available presets are:
-null:None
-swt-v4-4way:['B:bars', 'S:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credits', 'R:credits']
-swt-v4-6way:['B:bars', 'S:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credits', 'R:credits', 'E:other_text', 'K:other_text', 'G:other_text', 'T:other_text', 'F:other_text', 'W:other_opening', 'L:other_opening', 'O:other_opening', 'M:other_opening']
allowOverlap: optional, defaults to true
false, trueAllow overlapping time frames
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Stitched TimeFrame annotations. Each TimeFrame annotation represents a continuous segment of timepoints and its
labelproperty is determined by thelabelMapparameter (seeparameterssection). Therepresentativesis a singleton list of the TimePoint annotation that has the highest score in the TimeFrame.
Release Notes
SDK update to fix warning view bugs
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.5
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
modelName: optional, defaults to convnext_lg
convnext_tiny, convnext_lgmodel name to use for classification
usePosModel: optional, defaults to true
false, trueUse the model trained with positional features
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints
allowOverlap: optional, defaults to true
false, trueAllow overlapping time frames
map: optional, defaults to ['B:bars', 'S:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credits', 'R:credits', 'W:other_opening', 'L:other_opening', 'O:other_opening', 'M:other_opening', 'E:other_text', 'K:other_text', 'G:other_text', 'T:other_text', 'F:other_text']
Mapping of a label in the input annotations to a new label. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any negative labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded as a negative label.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
- updated to the latest SDK (adding some runtime recording params)
- added
usePosModelparam to toggle positional encoding- general re-implementation and bugfix for positional encoding
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.5
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
modelName: optional, defaults to convnext_lg
convnext_tiny, convnext_lgmodel name to use for classification
usePosModel: optional, defaults to true
false, trueUse the model trained with positional features
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints
allowOverlap: optional, defaults to true
false, trueAllow overlapping time frames
map: optional, defaults to ['B:bars', 'S:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credits', 'R:credits', 'W:other_opening', 'L:other_opening', 'O:other_opening', 'M:other_opening', 'E:other_text', 'K:other_text', 'G:other_text', 'T:other_text', 'F:other_text']
Mapping of a label in the input annotations to a new label. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any negative labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded as a negative label.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
SDK version update
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, True, base, b, small, s, medium, m, large, l, large-v2, l2, large-v3, l3The size of the model to use. When
modelLang=enis given, for non-largemodels, English-only models will be used instead of multilingual models for speed and accuracy. (Forlargemodels, English-only models are not available.) (also can be given as alias: tiny=t, base=b, small=s, medium=m, large=l, large-v2=l2, large-v3=l3)
modelLang: optional, defaults to ""
Language of the model to use, accepts two- or three-letter ISO 639 language codes, however Whisper only supports a subset of languages. If the language is not supported, error will be raised.For the full list of supported languages, see https://github.com/openai/whisper/blob/20231117/whisper/tokenizer.py . In addition to the langauge code, two-letter region codes can be added to the language code, e.g. “en-US” for US English. Note that the region code is only for compatibility and recording purpose, and Whisper neither detects regional dialects, nor use the given one for transcription. When the langauge code is not given, Whisper will run in langauge detection mode, and will use first few seconds of the audio to detect the language.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
Now based on SDK 1.3.0 with more universal parameters
The wrapper of Distil-Whisper, avaliable models: distil-large-v3, distil-large-v2, distil-medium.en, distil-small.en. The default model is distil-small.en.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to distil-small.en
distil-large-v3, distil-large-v2, distil-medium.en, distil-small.en, small, s, medium, m, large-v2, l2, large-v3, l3The size of the model to use. There are four size of model to use distil-large-v3, distil-large-v2, distil-medium.en, distil-small.en. You can also enter the abbreviation of the model as parameter. ‘small’ and ‘s’ for distil-small.en; ‘medium’ and ‘m’ for distil-medium.en; ‘large-v2’ and ‘l2’ for distil-large-v2; ‘large-v3’ and ‘l3’ for distil-large-v3. The default model is distil-medium.en.)
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
runningTime: optional, defaults to false
false, trueThe running time of the app will be recorded in the view metadata
hwFetch: optional, defaults to false
false, trueThe hardware information (architecture, GPU and vRAM) will be recorded in the view metadata
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input audio/video.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimeFrame<->SENTENCE, 2)audio/video document<->TextDocument
http://vocab.lappsgrid.org/Sentence (of any properties)
The smallest recognized unit of distil-whisper. Normally a complete sentence.
Release Notes
(no notes provided by the developer)
extract keywords of a text document according to TF-IDF values. IDF values and all features come from related pickle files in the current directory.App can either take a simple text document or take a MMIF file generated from the text slicer app.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
topN: optional, defaults to 10
top n keywords to extract from the current textfile.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Default property ‘text’ stores the extracted keywords (string). Added property ‘scores’ stores keywords’ TF-IDF values (float).
Release Notes
(no notes provided by the developer)
The wrapper of Distil-Whisper, avaliable models: distil-large-v3, distil-large-v2, distil-medium.en, distil-small.en. The default model is distil-small.en.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to distil-small.en
distil-large-v3, distil-large-v2, distil-medium.en, distil-small.en, small, s, medium, m, large-v2, l2, large-v3, l3The size of the model to use. There are four size of model to use distil-large-v3, distil-large-v2, distil-medium.en, distil-small.en. You can also enter the abbreviation of the model as parameter. ‘small’ and ‘s’ for distil-small.en; ‘medium’ and ‘m’ for distil-medium.en; ‘large-v2’ and ‘l2’ for distil-large-v2; ‘large-v3’ and ‘l3’ for distil-large-v3. The default model is distil-medium.en.)
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input audio/video.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimeFrame<->SENTENCE, 2)audio/video document<->TextDocument
http://vocab.lappsgrid.org/Sentence (of any properties)
The smallest recognized unit of distil-whisper. Normally a complete sentence.
Release Notes
(no notes provided by the developer)
Apply spaCy NLP to all text documents in a MMIF file.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (required) (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
pretokenized: optional, defaults to false
false, trueBoolean parameter to set the app to use existing tokenization, if available, for text documents for NLP processing. Useful to process ASR documents, for example.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://vocab.lappsgrid.org/Token (of any properties)
http://vocab.lappsgrid.org/Token#pos (of any properties)
http://vocab.lappsgrid.org/Token#lemma (of any properties)
http://vocab.lappsgrid.org/NounChunk (of any properties)
http://vocab.lappsgrid.org/Sentence (of any properties)
http://vocab.lappsgrid.org/NamedEntity (of any properties)
Release Notes
first version of text-slicer
Slice text snippets from a provided text document given time frames
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/TimeFrame/v5 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
containLabel: required
A list of labels that user expect TimeFrames contain.
Labels can be chosen from but not limited to:
[‘bars’, ‘tones’, ‘bars-and-tones’,’speech’,’noise’,
‘music’, ‘slate’, ‘chyron’, ‘lower-third’, ‘credits’]
Users are required to select at least one label. Otherwise, errors would be thrown instead
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Release Notes
Fixed subtype-label bug and …
- updating to clams-python 1.2.5,
- replaced cli.py with the one in the SDK to ensure consistency with other CLAMS apps
- added “full train” mode for trainer script to use all available data for training, without any validation at the train time
- refactor to declutter inter-dependency between app.py, Classifier, and Stitcher classes
- removed CLI in classifier.py, since app.py and cli.py can complete replace its functionality
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.5
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
modelName: optional, defaults to 20240626-205715.convnext_lg
20240626-205803.convnext_tiny, 20240626-205715.convnext_lgmodel name to use for classification
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints
allowOverlap: optional, defaults to true
false, trueAllow overlapping time frames
map: optional, defaults to ['B:bars', 'S:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credits', 'R:credits', 'W:other_opening', 'L:other_opening', 'O:other_opening', 'M:other_opening', 'E:other_text', 'K:other_text', 'G:other_text', 'T:other_text', 'F:other_text']
Mapping of a label in the input annotations to a new label. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any negative labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded as a negative label.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Minor release with updated SDK version and updated label mapping for compatibility with RFB and newer SWT.
Stitches a sequence of TimePoint annotations into a sequence of TimeFrame annotations, performing simple smoothing of short peaks of positive labels.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
TimePoint annotations to be stitched. Must be “exhaustive” in that it should cover an entire single time period in the input document, with a uniform sample rate.
(Note: Multivalued means the parameter can have one or more values.)
labelMap: optional, defaults to []
mapping of labels in the input annotations to new labels. Must be formatted as “IN_LABEL:OUT_LABEL” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any “negative” labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded as the negative label(“-“).
minTFDuration: optional, defaults to 1000
minimum duration of a TimeFrame in milliseconds
minTPScore: optional, defaults to 0.1
minimum score of a TimePoint to be considered as positive
minTFScore: optional, defaults to 0.5
minimum average score of TimePoints in a TimeFrame to be considered as positive
labelMapPreset: optional, defaults to null
null, swt-v4-4way, swt-v4-6waypreset of label mappings. If not
null, this parameter will override thelabelMapparameter. Available presets are:
-null:None
-swt-v4-4way:['B:bars', 'S:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credits', 'R:credits']
-swt-v4-6way:['B:bars', 'S:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credits', 'R:credits', 'E:other_text', 'K:other_text', 'G:other_text', 'T:other_text', 'F:other_text', 'W:other_opening', 'L:other_opening', 'O:other_opening', 'M:other_opening']
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Stitched TimeFrame annotations. Each TimeFrame annotation represents a continuous segment of timepoints and its
labelproperty is determined by thelabelMapparameter (seeparameterssection). Therepresentativesis a singleton list of the TimePoint annotation that has the highest score in the TimeFrame.
Release Notes
- Fixed minor typos in parameters
- Updated to clams-python 1.2.4, and added
cli.pyinterface
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, True, base, b, small, s, medium, m, large, l, large-v2, l2, large-v3, l3The size of the model to use. When
modelLang=enis given, for non-largemodels, English-only models will be used instead of multilingual models for speed and accuracy. (Forlargemodels, English-only models are not available.)
modelLang: optional, defaults to ""
Language of the model to use, accepts two- or three-letter ISO 639 language codes, however Whisper only supports a subset of languages. If the language is not supported, error will be raised.For the full list of supported languages, see https://github.com/openai/whisper/blob/20231117/whisper/tokenizer.py . In addition to the langauge code, two-letter region codes can be added to the language code, e.g. “en-US” for US English. Note that the region code is only for compatibility and recording purpose, and Whisper neither detects regional dialects, nor use the given one for transcription. When the langauge code is not given, Whisper will run in langauge detection mode, and will use first few seconds of the audio to detect the language.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
SDK v1.2.4 update
OpenCV-based text localization app that used EAST text detection model. Please visit the source code repository for full documentation.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/ImageDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, seconds, millisecondsUnit for time points in the output. Only works with VideoDocument input.
frameType: optional, defaults to ['']
slate, chyron, rolling-creditSegments of video to run on. Only works with VideoDocument input and TimeFrame input. Empty value means run on the every frame types.
sampleRate: optional, defaults to 30
Frequency to sample frames. Only works with VideoDocument input, and without TimeFrame input. (when
TimeFrameannotation is found, this parameter is ignored.)
stopAt: optional, defaults to 216000
Frame number to stop running. Only works with VideoDocument input. The default is roughly 2 hours of video at 30fps.
mergeBoxes: optional, defaults to false
false, trueif True, creates a single merged bounding box from all detected boxes.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Bumping Python SDK version, bug fixes and documentation updates
- Updated to clams-python 1.2.2
- Fixed token length (issue #30)
- Fixed problems with the pretokenized parameter (issue #32)
- Various documentation fixes.
Apply spaCy NLP to all text documents in a MMIF file.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (required) (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
pretokenized: optional, defaults to false
false, trueBoolean parameter to set the app to use existing tokenization, if available, for text documents for NLP processing. Useful to process ASR documents, for example.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://vocab.lappsgrid.org/Token (of any properties)
http://vocab.lappsgrid.org/Token#pos (of any properties)
http://vocab.lappsgrid.org/Token#lemma (of any properties)
http://vocab.lappsgrid.org/NounChunk (of any properties)
http://vocab.lappsgrid.org/Sentence (of any properties)
http://vocab.lappsgrid.org/NamedEntity (of any properties)
Release Notes
(no notes provided by the developer)
Applies llava to video frames.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v5 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
defaultPrompt: optional, defaults to What is shown in this video frame?
default prompt to use for timeframes not specified in the promptMap. If set to
-, timeframes not specified in the promptMap will be skipped.
promptMap: optional, defaults to []
mapping of labels of the input timeframe annotations to new prompts. Must be formatted as “IN_LABEL:PROMPT” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, any timeframe labels not mapped to a prompt will be used with the defaultprompt. In order to skip timeframes with a particular label, pass
-as the prompt value.in order to skip all timeframes not specified in the promptMap, set the defaultPromptparameter to-
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
Release Notes
(no notes provided by the developer)
Applies llava to video frames.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v5 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
Release Notes
This release adds a script to run a server-less app from the command line, adds and fixes parameters, and updates dependencies.
- Added CLI capabilities
- Added allowOverlap parameter
- Added map parameter for postbin mapping
- Updated to clams-python 1.2.2
- Updated base container to clams-python-opencv4-torch2:1.2.2
- Simplified model names
- Documentation updates
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 9223372036854775807
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.5
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
modelName: optional, defaults to 20240409-091401.convnext_lg
20240212-132306.convnext_lg, 20240409-093229.convnext_tiny, 20240409-091401.convnext_lg, 20240126-180026.convnext_lg, 20240212-131937.convnext_tinymodel name to use for classification
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints
allowOverlap: optional, defaults to true
false, trueAllow overlapping time frames
map: optional, defaults to ['B:bars', 'S:slate', 'S-H:slate', 'S-C:slate', 'S-D:slate', 'S-G:slate', 'W:other_opening', 'L:other_opening', 'O:other_opening', 'M:other_opening', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credit', 'R:credit', 'E:other_text', 'K:other_text', 'G:other_text', 'T:other_text', 'F:other_text']
Mapping of a label in the input annotations to a new label. Must be formatted as IN_LABEL:OUT_LABEL (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any negative labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded as a negative label.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Major update
TimeFrameproperties are updated to match new CLAMS vocab specificationnoEnergylabel from INA segmenter is stored assilencein MMIF for clarity- added parameter
silenceRatioto configure silence detection threshold- renamed
minDurationparameter tominTFDurationfor clarity- bugfixes in cli.py
inaSpeechSegmenter is a CNN-based audio segmentation toolkit. The original software can be found at https://github.com/ina-foss/inaSpeechSegmenter .
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
minTFDuration: optional, defaults to 0
minimum duration of a TimeFrame in milliseconds
silenceRatio: optional, defaults to 3
percentage ratio (0-100) of audio energy to to determine silence, ratio to mean every of the input audio.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
The INA semgmenter uses 5-way classification ([‘noEnergy’, ‘female’, ‘male’, ‘noise’, ‘music’]) and this wrapper remaps the labels to [‘silence’, ‘speech’, ‘noise’, ‘music’], by 1) renaming
noEnergytosilence2) collapsingfemaleandmaleintospeech(leaving additionalgenderproperty). Note that the time frame annotations do not exhaustively cover the input audio, but only the segments.
Release Notes
SDK update, added experimental cli.py
inaSpeechSegmenter is a CNN-based audio segmentation toolkit. The original software can be found at https://github.com/ina-foss/inaSpeechSegmenter .
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
minDuration: optional, defaults to 0
minimum duration of a TimeFrame in milliseconds
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Major update to add
- experimental CLI. CLI shares the same parameters specified in the app metadata. Try
python cli.py --help!- added presets for complex labelMap parameter. Find out about labelMapPreset parameter in the metadata.
Stitches a sequence of TimePoint annotations into a sequence of TimeFrame annotations, performing simple smoothing of short peaks of positive labels.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
TimePoint annotations to be stitched. Must be “exhaustive” in that it should cover an entire single time period in the input document, with a uniform sample rate.
(Note: Multivalued means the parameter can have one or more values.)
labelMap: optional, defaults to []
mapping of labels in the input annotations to new labels. Must be formatted as “IN_LABEL:OUT_LABEL” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any “negative” labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded as the negative label(“-“).
minTFDuration: optional, defaults to 1000
minimum duration of a TimeFrame in milliseconds
minTPScore: optional, defaults to 0.1
minimum score of a TimePoint to be considered as positive
minTFScore: optional, defaults to 0.5
minimum average score of TimePoints in a TimeFrame to be considered as positive
labelMapPreset: optional, defaults to null
null, swt-v4-4way, swt-v4-6waypreset of label mappings. If not
null, this parameter will override thelabelMapparameter. Available presets are:
-null:None
-swt-v4-4way:['B:bars', 'S:slate', 'S-H:slate', 'S-C:slate', 'S-D:slate', 'S-G:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credit', 'R:credit']
-swt-v4-6way:['B:bars', 'S:slate', 'S-H:slate', 'S-C:slate', 'S-D:slate', 'S-G:slate', 'I:chyron', 'N:chyron', 'Y:chyron', 'C:credit', 'R:credit', 'E:other_text', 'K:other_text', 'G:other_text', 'T:other_text', 'F:other_text', 'W:other_opening', 'L:other_opening', 'O:other_opening', 'M:other_opening']
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Stitched TimeFrame annotations. Each TimeFrame annotation represents a continuous segment of timepoints and its
labelproperty is determined by thelabelMapparameter (seeparameterssection). Therepresentativesis a singleton list of the TimePoint annotation that has the highest score in the TimeFrame.
Release Notes
Minor update with various fixes
- updated SDK version
- fixed container image dependency problem
- updated app metadata and made app description less verbose
CLAMS app wraps the docTR, End-to-End OCR model. The model can detect text regions in the input image and recognize text in the regions (via parseq OCR model, only English is support at the moment). The text-localized regions are organized hierarchically by the model into “pages” > “blocks” > “lines” > “words”, and this CLAMS app translates them into TextDocument, Paragraphs, Sentence, and Token annotations to represent recognized text contents. See descriptions for I/O types below for details on how annotations are aligned to each other.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v5 (required)
The Time frame annotation that represents the video segment to be processed. When
representativesproperty is present, the app will process videos still frames at the underlying time point annotations that are referred to by therepresentativesproperty. Otherwise, the app will process the middle frame of the video segment.
(Note: Multivalued means the parameter can have one or more values.)
tfLabel: optional, defaults to []
The label of the TimeFrame annotation to be processed. By default (
[]), all TimeFrame annotations will be processed, regardless of theirlabelproperty values.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input images. Serialization isdone by concatenating
textvalues ofParagraphannotations with two newline characters.
Translation of the recognized docTR “words” in the input images.
textandwordproperties store the string values of the recognized text. The duplication is for keepingbackward compatibility and consistency withParagraphandSentenceannotations.
Translation of the recognized docTR “lines” in the input images.
textproperty stores the string value of space-joined words.
Translation of the recognized docTR “blocks” in the input images.
textproperty stores the string value of newline-joined sentences.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimePoint<->TextDocument, 2)TimePoint<->Token/Sentence/Paragraph, 3)BoundingBox<->Token/Sentence/Paragraph
Bounding boxes of the detected text regions in the input images. No corresponding box for the entire image (
TextDocument) region
Release Notes
(no notes provided by the developer)
CLAMS app wraps the docTR, End-to-End OCR model, available at https://pypi.org/project/python-doctr . The model is capable of detecting text regions in the input image and recognizing text in the regions. The text-localized regions are organized hierarchically by the model into “pages” > “blocks” > “lines” > “words”, and this CLAMS app translates them into TextDocument, Paragraphs, Sentence, and Token annotations to represent recognized text contents, then aligns them to BoundingBox annotations that represent the detected geometries. This hierarchical structure is also represented in the TextDocument annotation output as two newlines (\n\n) between “paragraphs”, one newline (\n) between the “lines”, and one space (“ “) between the “words”. For the text recognition, the model is internally configured to use the “parseq” recognition model, and only works with English text at the moment.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v5 (required)
The Time frame annotation that represents the video segment to be processed. When
representativesproperty is present, the app will process videos still frames at the underlying time point annotations that are referred to by therepresentativesproperty. Otherwise, the app will process the middle frame of the video segment.
(Note: Multivalued means the parameter can have one or more values.)
tfLabel: optional, defaults to []
The label of the TimeFrame annotation to be processed. By default (
[]), all TimeFrame annotations will be processed, regardless of theirlabelproperty values.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Fully serialized text content of the recognized text in the input images. Serialization isdone by concatenating
textvalues ofParagraphannotations with two newline characters.
Translation of the recognized docTR “words” in the input images.
textandwordproperties store the string values of the recognized text. The duplication is for keepingbackward compatibility and consistency withParagraphandSentenceannotations.
Translation of the recognized docTR “lines” in the input images.
textproperty stores the string value of space-joined words.
Translation of the recognized docTR “blocks” in the input images.
textproperty stores the string value of newline-joined sentences.
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Alignments between 1)
TimePoint<->TextDocument, 2)TimePoint<->Token/Sentence/Paragraph, 3)BoundingBox<->Token/Sentence/Paragraph
Bounding boxes of the detected text regions in the input images. No corresponding box for the entire image (
TextDocument) region
Release Notes
fixed
labelproperty missing fromTimeFrame, updated to the new SDK version
Stitches a sequence of TimePoint annotations into a sequence of TimeFrame annotations, performing simple smoothing of short peaks of positive labels.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
TimePoint annotations to be stitched. Must be “exhaustive” in that it should cover an entire single time period in the input document, with a uniform sample rate.
(Note: Multivalued means the parameter can have one or more values.)
labelMap: optional, defaults to []
mapping of labels in the input annotations to new labels. Must be formatted as “IN_LABEL:OUT_LABEL” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any “negative” labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded asa negative label(“-“).
minTFDuration: optional, defaults to 1000
minimum duration of a TimeFrame in milliseconds
minTPScore: optional, defaults to 0.1
minimum score of a TimePoint to be considered as positive
minTFScore: optional, defaults to 0.5
minimum average score of TimePoints in a TimeFrame to be considered as positive
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Stitched TimeFrame annotations. Each TimeFrame annotation represents a continuous segment of timepoints and its
labelproperty is determined by thelabelMapparameter (seeparameterssection). Therepresentativesis a singleton list of the TimePoint annotation that has the highest score in the TimeFrame.
Release Notes
(no notes provided by the developer)
Stitches a sequence of TimePoint annotations into a sequence of TimeFrame annotations, performing simple smoothing of short peaks of positive labels.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
TimePoint annotations to be stitched. Must be “exhaustive” in that it should cover an entire single time period in the input document, with a uniform sample rate.
(Note: Multivalued means the parameter can have one or more values.)
labelMap: optional, defaults to []
mapping of labels in the input annotations to new labels. Must be formatted as “IN_LABEL:OUT_LABEL” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any “negative” labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded asa negative label(“-“).
minTFDuration: optional, defaults to 1000
minimum duration of a TimeFrame in milliseconds
minTPScore: optional, defaults to 0.1
minimum score of a TimePoint to be considered as positive
minTFScore: optional, defaults to 0.5
minimum average score of TimePoints in a TimeFrame to be considered as positive
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Stitched TimeFrame annotations. Each TimeFrame annotation represents a continuous segment of timepoints and its
labelproperty is determined by thelabelMapparameter (seeparameterssection). Therepresentativesis a singleton list of the TimePoint annotation that has the highest score in the TimeFrame.
Release Notes
This version is re-release of v4.3 from the right commit.
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 10000000
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.5
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
modelName: optional, defaults to 20240409-091401.convnext_lg.kfold_013
20240126-180026.convnext_lg.kfold_000, 20240409-091401.convnext_lg.kfold_013, 20240409-093229.convnext_tiny.kfold_012, 20240212-131937.convnext_tiny.kfold_000, 20240212-132306.convnext_lg.kfold_000model name to use for classification
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
This version brings many bug fixes and new models
- fixed missing
NEGlabel score in theclassificationproperty inTimePointannotations (#87), and app metadata is updated accordingly- fixed sampling rate disparity (#90)
- fixed sinusoidal positional features were not actually used (#47), and newly trained models with the fix are included
- miscellaneous code clean-up
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 10000000
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.5
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
modelName: optional, defaults to 20240126-180026.convnext_lg.kfold_000
20240126-180026.convnext_lg.kfold_000, 20240212-131937.convnext_tiny.kfold_000, 20240212-132306.convnext_lg.kfold_000model name to use for classification
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Stitches a sequence of TimePoint annotations into a sequence of TimeFrame annotations, performing simple smoothing of short peaks of positive labels.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
TimePoint annotations to be stitched. Must be “exhaustive” in that it should cover an entire single time period in the input document, with a uniform sample rate.
(Note: Multivalued means the parameter can have one or more values.)
labelMap: optional, defaults to []
mapping of labels in the input annotations to new labels. Must be formatted as “IN_LABEL:OUT_LABEL” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any “negative” labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded asa negative label(“-“).
minTFDuration: optional, defaults to 1000
minimum duration of a TimeFrame in milliseconds
minTPScore: optional, defaults to 0.1
minimum score of a TimePoint to be considered as positive
minTFScore: optional, defaults to 0.5
minimum average score of TimePoints in a TimeFrame to be considered as positive
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Stitched TimeFrame annotations. Each TimeFrame annotation represents a continuous segment of timepoints and its
labelproperty is determined by thelabelMapparameter (seeparameterssection). Therepresentativesis a singleton list of the TimePoint annotation that has the highest score in the TimeFrame.
Release Notes
(no notes provided by the developer)
Stitches a sequence of TimePoint annotations into a sequence of TimeFrame annotations, performing simple smoothing of short peaks of positive labels.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
TimePoint annotations to be stitched. Must be “exhaustive” in that it should cover an entire single time period in the input document, with a uniform sample rate.
(Note: Multivalued means the parameter can have one or more values.)
labelMap: optional, defaults to []
mapping of labels in the input annotations to new labels. Must be formatted as “IN_LABEL:OUT_LABEL” (with a colon). To pass multiple mappings, use this parameter multiple times. By default, all the input labels are passed as is, including any “negative” labels (with default value being no remapping at all). However, when at least one label is remapped, all the other “unset” labels are discarded asa negative label(“-“).
minTFDuration: optional, defaults to 1000
minimum duration of a TimeFrame in milliseconds
minTPScore: optional, defaults to 0.1
minimum score of a TimePoint to be considered as positive
minTFScore: optional, defaults to 0.5
minimum average score of TimePoints in a TimeFrame to be considered as positive
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Stitched TimeFrame annotations. Each TimeFrame annotation represents a continuous segment of timepoints and its
labelproperty is determined by thelabelMapparameter (seeparameterssection). Therepresentativesis a singleton list of the TimePoint annotation that has the highest score in the TimeFrame.
Release Notes
sdk version updates
CLAMS app wraps PySceneDetect and performs shot boundary detection on input videos
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
mode: optional, defaults to content
content, threshold, adaptivepick a scene detector algorithm, see http://scenedetect.com/projects/Manual/en/latest/cli/detectors.html
threshold: optional, defaults to 27.0
threshold value to use in the detection algorithm. Note that the meaning of this numerical value differs for different detector algorithms.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Includes bugfix and performance optimization
- fixed bug startAt/stopAt params were not honored
- faster video iteration for still image extraction
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 10000000
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.5
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
modelName: optional, defaults to 20240126-180026.convnext_lg.kfold_000
20240126-180026.convnext_lg.kfold_000, 20240212-132306.convnext_lg.kfold_000, 20240212-131937.convnext_tiny.kfold_000model name to use for classification
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
This version includes many bugfixes and clarification for the previous version.
- more informative, consistent, and flask-friendly debug-level logging for future development
- two additional pretrained models, including one based on convnext_tiny backbone for quicker annotation
TimePointannotations is re-worked
- classification-related props in TP anns are now all based on the “RAW” labels from classifier
- all images classification results are now recorded as TP annotations regardless of TF annotations
- added two runtime parameter
useStitcher- when"false", app will only generateTimePointannotations, not stitching them intoTimeFrameannotationsmodelName- to pick a model between pre-built classifiers (by default, app will use the best performing model from training experiments)- updated to the latest
mmif-pythonandclams-python, and thus no longer generates MMIFs with a non-existing version
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 10000000
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.5
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
modelName: optional, defaults to 20240126-180026.convnext_lg.kfold_000
20240126-180026.convnext_lg.kfold_000, 20240212-132306.convnext_lg.kfold_000, 20240212-131937.convnext_tiny.kfold_000model name to use for classification
useStitcher: optional, defaults to true
false, trueUse the stitcher after classifying the TimePoints
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Uses a new classification regime with a dictionary of labels and scores
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 10000000
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.5
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
v6 fixes a bug running en-only large models since there is no en-only large model
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, True, base, b, small, s, medium, m, large, l, large-v2, l2, large-v3, l3The size of the model to use. When
modelLand=enis given, for non-largemodels, English-only models will be used instead of multilingual models for speed and accuracy. (Forlargemodels, English-only models are not available.)
modelLang: optional, defaults to ""
Language of the model to use, accepts two- or three-letter ISO 639 language codes, however Whisper only supports a subset of languages. If the language is not supported, error will be raised.For the full list of supported languages, see https://github.com/openai/whisper/blob/20231117/whisper/tokenizer.py . In addition to the langauge code, two-letter region codes can be added to the language code, e.g. “en-US” for US English. Note that the region code is only for compatibility and recording purpose, and Whisper neither detects regional dialects, nor use the given one for transcription. When the langauge code is not given, Whisper will run in langauge detection mode, and will use first few seconds of the audio to detect the language.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
This version include addition of
modelLangparam and many other changes
- (BIG CHANGE) timeunit is now millisecond (integer)
modelLangparameter is added for selecting language to instruct Whisper to use, Whisper will run in language detection mode if this parameter is not given.- When
modelLangis set toen, load english only models instead of multilingual ones for speed and performance- skips empty segments from whisper when generating output MMIF
- updated to clams-python 1.1.1
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, base, small, medium, largeThe size of the model to use. Can be “tiny”, “base”, “small”, “medium”, or “large”.
modelLang: optional, defaults to ""
Language of the model to use, accepts two- or three-letter ISO 639 language codes, however Whisper only supports a subset of languages. If the language is not supported, error will be raised.For the full list of supported languages, see https://github.com/openai/whisper/blob/20231117/whisper/tokenizer.py . In addition to the langauge code, two-letter region codes can be added to the language code, e.g. “en-US” for US English. Note that the region code is only for compatibility and recording purpose, and Whisper neither detects regional dialects, nor use the given one for transcription. When the langauge code is not given, Whisper will run in langauge detection mode, and will use first few seconds of the audio to detect the language.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
New model, improved MMIF output and various fixes
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
startAt: optional, defaults to 0
Number of milliseconds into the video to start processing
stopAt: optional, defaults to 10000000
Number of milliseconds into the video to stop processing
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.25
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Using EasyOCR to extract text from timeframes
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v1 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/BoundingBox/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TimePoint/v1 (of any properties)
Release Notes
Fixed fatal error in 2.0 and added timeUnit
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.25
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Using EasyOCR to extract text from timeframes
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/TimeFrame/v1 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
sampleFrames: optional, defaults to 1
Number of frames to sample from timeframe
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
http://mmif.clams.ai/vocabulary/BoundingBox/v1 (of any properties)
http://mmif.clams.ai/vocabulary/TimePoint/v1 (of any properties)
Release Notes
- updated to the latest whisper models
- now ouputs
Sentenceannotations based on whisper’s segmentation
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, base, small, medium, largeThe size of the model to use. Can be “tiny”, “base”, “small”, “medium”, or “large”.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Release Notes
Version 2.1
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
sampleRate: optional, defaults to 1000
Milliseconds between sampled frames
minFrameScore: optional, defaults to 0.01
Minimum score for a still frame to be included in a TimeFrame
minTimeframeScore: optional, defaults to 0.25
Minimum score for a TimeFrame
minFrameCount: optional, defaults to 2
Minimum number of sampled frames required for a TimeFrame
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
Version 1.0 of the app
Detects scenes with text, like slates, chyrons and credits.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
model: optional, defaults to vgg16
the model to use, not implemented yet
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Apply named entity linking to all text documents in a MMIF file.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
confidence: optional, defaults to 0.5
disambiguation confidence score for linking
support: optional, defaults to 0
resource prominence, i.e. number of in-links in Wikipedia (lower bound)
types: required
limits recognition to certain types of named entities, e.g. DBpedia:Place
policy: optional, defaults to whitelist
whitelist, blacklist(whitelist) selects all entities of the same type; (blacklist) selects all entities not of the same type
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool detects slates.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, seconds, millisecondsUnit of time to use in output.
sampleRatio: optional, defaults to 30
Frequency to sample frames.
stopAt: optional, defaults to 9000
Frame number to stop processing
stopAfterOne: optional, defaults to true
false, trueWhen True, processing stops after first timeframe is found
minFrameCount: optional, defaults to 10
Minimum number of frames required for a timeframe to be included in the output
threshold: optional, defaults to 0.7
Threshold from 0-1, lower accepts more potential slates.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool uses a vision model to classify video segments. Currenly supports “chyron” frame type.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, millisecondsUnit for output timeframe
sampleRatio: optional, defaults to 30
Frequency to sample frames.
minFrameCount: optional, defaults to 60
Minimum number of frames required for a timeframe to be included in the output with a minimum value of 1
threshold: optional, defaults to 0.8
Threshold from 0-1, lower accepts more potential labels.
finetunedFrameType: optional, defaults to chyron
Name of fine-tuned model to use. All pre-installed models are named after the frame type they were fine-tuned for.
If an empty value is passed, the app will look for fewshots.csv file in the same directory as the app.py and create a new fine-tuned model based on the examples in that file.
At the moment, a model fine-tuned on “chyron” frame type is shipped as pre-installed.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool detects SMPTE color bars.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, seconds, millisecondsUnit for output typeframe.
sampleRatio: optional, defaults to 30
Frequency to sample frames.
stopAt: optional, defaults to 9000
Frame number to stop processing.
stopAfterOne: optional, defaults to true
false, trueWhen True, processing stops after first timeframe is found.
minFrameCount: optional, defaults to 10
minimum number of frames required for a timeframe to be included in the output.
threshold: optional, defaults to 0.7
Threshold from 0-1, lower accepts more potential slates.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
OpenCV-based text localization app that used EAST text detection model. Please visit the source code repository for full documentation.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/ImageDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, seconds, millisecondsUnit for time points in the output. Only works with VideoDocument input.
frameType: optional, defaults to ""
slate, chyron, rolling-creditSegments of video to run on. Only works with VideoDocument input and TimeFrame input. Empty value means run on the every frame types.
sampleRatio: optional, defaults to 30
Frequency to sample frames. Only works with VideoDocument input, and without TimeFrame input. (when
TimeFrameannotation is found, this parameter is ignored.)
stopAt: optional, defaults to 2 * 60 * 60 * 30
Frame number to stop running. Only works with VideoDocument input. The default is roughly 2 hours of video at 30fps.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool detects SMPTE color bars.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, seconds, millisecondsUnit for output typeframe.
sampleRatio: optional, defaults to 30
Frequency to sample frames.
stopAt: optional, defaults to 9000
Frame number to stop processing.
stopAfterOne: optional, defaults to true
false, trueWhen True, processing stops after first timeframe is found.
minFrameCount: optional, defaults to 10
minimum number of frames required for a timeframe to be included in the output.
threshold: optional, defaults to 0.7
Threshold from 0-1, lower accepts more potential slates.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool applies Parseq OCR to a video or image and generates text boxes and OCR results.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/BoundingBox/v1 (required)
(Note: Multivalued means the parameter can have one or more values.)
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Release Notes
(no notes provided by the developer)
This tool applies Tesseract OCR to a video or image and generates text boxes and OCR results.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
frameType: optional, defaults to ""
Use this to specify TimeFrame to use for filtering “text”-typed BoundingBox annotations. Can be “slate”, “chyron”, “speech”, etc.. If not set, the app won’t use TimeFrames for filtering.
threshold: optional, defaults to 0.9
Use this value between 0 and 1 to filter out low-confidence text boxes.
psm: optional, defaults to 0
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13Tesseract Page Segmentation Modes. See https://tesseract-ocr.github.io/tessdoc/ImproveQuality.html#page-segmentation-method
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Release Notes
(no notes provided by the developer)
Apply named entity linking to all text documents in a MMIF file.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
confidence: optional, defaults to 0.5
disambiguation confidence score for linking
support: optional, defaults to 0
resource prominence, i.e. number of in-links in Wikipedia (lower bound)
types: required
types filter
policy: optional, defaults to whitelist
whitelist, blacklist(whitelist) selects all entities of the same type; (blacklist) selects all entities not of the same type
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool detects chyrons, generates time segments.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, seconds, millisecondsunit for output timeframe
sampleRatio: optional, defaults to 5
Frequency to sample frames
minFrameCount: optional, defaults to 10
Minimum number of frames required for a timeframe to be included
threshold: optional, defaults to 0.5
Threshold from 0-1, lower accepts more potential chyrons
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This CLAMS app aligns transcript and audio track using Gentle. Gentle is a robust yet lenient forced aligner built on Kaldi.This app only works when Gentle is already installed locally.Unfortunately, Gentle is not distributed as a Python package distribution.To get Gentle installation instruction, see https://lowerquality.com/gentle/ Make sure install Gentle from the git commit specified in analyzer_version in this metadata.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
use_speech_segmentation: optional, defaults to true
false, trueWhen set true, use exising “speech”-typed
TimeFrameannotations and run aligner only on those frames, instead of entire audio files.
use_tokenization: optional, defaults to true
false, trueWhen set true,
Alignmentannotation output will honor existing latest tokenization (Tokenannotations). Due to a limitation of the way Kaldi reads in English tokens, existing tokens must not contain whitespaces.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://vocab.lappsgrid.org/Token (of any properties)
Release Notes
(no notes provided by the developer)
Detects spans of monotonic audio within an audio file
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to seconds
seconds, seconds, millisecondsthe unit for annotation output
lengthThreshold: optional, defaults to 2000
minimum length threshold (in ms)
sampleSize: optional, defaults to 512
length for each segment of samples to be compared
stopAt: optional, defaults to None
stop point for audio processing (in ms). Defaults to the length of the file
tolerance: optional, defaults to 1.0
threshold value for a “match” within audio processing
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
Apply spaCy NLP to all text documents in a MMIF file.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (required) (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
pretokenized: optional, defaults to false
false, trueBoolean parameter to set the app to use existing tokenization, if available, for text documents for NLP processing. Useful to process ASR documents, for example.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://vocab.lappsgrid.org/Token (of any properties)
http://vocab.lappsgrid.org/Token#pos (of any properties)
http://vocab.lappsgrid.org/Token#lemma (of any properties)
http://vocab.lappsgrid.org/NounChunk (of any properties)
http://vocab.lappsgrid.org/Sentence (of any properties)
http://vocab.lappsgrid.org/NamedEntity (of any properties)
Release Notes
(no notes provided by the developer)
Apply named entity linking to all text documents in a MMIF file.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
confidence: optional, defaults to 0.5
disambiguation confidence score for linking
support: optional, defaults to 0
resource prominence, i.e. number of in-links in Wikipedia (lower bound)
types: required
limits recognition to certain types of named entities, e.g. DBpedia:Place
policy: optional, defaults to whitelist
whitelist, blacklist(whitelist) selects all entities of the same type; (blacklist) selects all entities not of the same type
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
OpenCV-based text localization app that used EAST text detection model. Please visit the source code repository for full documentation.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/ImageDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, seconds, millisecondsUnit for time points in the output. Only works with VideoDocument input.
frameType: optional, defaults to ""
slate, chyron, rolling-creditSegments of video to run on. Only works with VideoDocument input and TimeFrame input. Empty value means run on the every frame types.
sampleRatio: optional, defaults to 30
Frequency to sample frames. Only works with VideoDocument input, and without TimeFrame input. (when
TimeFrameannotation is found, this parameter is ignored.)
stopAt: optional, defaults to 2 * 60 * 60 * 30
Frame number to stop running. Only works with VideoDocument input. The default is roughly 2 hours of video at 30fps.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
(no description provided by the developer)
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
mode: optional, defaults to content
content, threshold, adaptivepick a scene detector algorithm, see http://scenedetect.com/projects/Manual/en/latest/cli/detectors.html
threshold: optional, defaults to 27.0
threshold value to use in the detection algorithm. Note that the meaning of this numerical value differs for different detector algorithms.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
modelSize: optional, defaults to tiny
tiny, base, small, medium, largeThe size of the model to use. Can be “tiny”, “base”, “small”, “medium”, or “large”.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Release Notes
(no notes provided by the developer)
Brandeis Acoustic Classification & Segmentation (ACS) is a audio segmentation tool developed at Brandeis Lab for Linguistics and Computation. The original software can be found at https://github.com/brandeis-llc/acoustic-classification-segmentation .
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
inaSpeechSegmenter is a CNN-based audio segmentation toolkit. The original software can be found at https://github.com/ina-foss/inaSpeechSegmenter .
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool detects slates.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, seconds, millisecondsUnit of time to use in output.
sampleRatio: optional, defaults to 30
Frequency to sample frames.
stopAt: optional, defaults to 9000
Frame number to stop processing
stopAfterOne: optional, defaults to true
false, trueWhen True, processing stops after first timeframe is found
minFrameCount: optional, defaults to 10
Minimum number of frames required for a timeframe to be included in the output
threshold: optional, defaults to 0.7
Threshold from 0-1, lower accepts more potential slates.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
inaSpeechSegmenter is a CNN-based audio segmentation toolkit. The original software can be found at https://github.com/ina-foss/inaSpeechSegmenter .
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool detects slates.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, millisecondsUnit for output typeframe
sampleRatio: optional, defaults to 30
Frequency to sample frames.
stopAt: optional, defaults to 540000
Frame number to stop processing
stopAfterOne: optional, defaults to 1
false, trueWhen True, processing stops after first timeframe is found
minFrameCount: optional, defaults to 10
Minimum number of frames required for a timeframe to be included in the output
threshold: optional, defaults to 0
Threshold from 0-1, lower accepts more potential slates.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool detects slates.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, millisecondsUnit for output typeframe
sampleRatio: optional, defaults to 30
Frequency to sample frames.
stopAt: optional, defaults to 540000
Frame number to stop processing
stopAfterOne: optional, defaults to 1
false, trueWhen True, processing stops after first timeframe is found
minFrameCount: optional, defaults to 10
Minimum number of frames required for a timeframe to be included in the output
threshold: optional, defaults to 0
Threshold from 0-1, lower accepts more potential slates.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
(no description provided by the developer)
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
mode: optional, defaults to content
content, threshold, adaptivepick a scene detector algorithm, see http://scenedetect.com/projects/Manual/en/latest/cli/detectors.html
threshold: optional, defaults to 27
threshold value to use in the detection algorithm. Note that the meaning of this numerical value differs for different detector algorithms.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
This tool detects slates.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
timeUnit: optional, defaults to frames
frames, millisecondsUnit for output typeframe
sampleRatio: optional, defaults to 30
Frequency to sample frames.
stopAt: optional, defaults to 540000
Frame number to stop processing
stopAfterOne: optional, defaults to 1
false, trueWhen True, processing stops after first timeframe is found
minFrameCount: optional, defaults to 10
Minimum number of frames required for a timeframe to be included in the output
threshold: optional, defaults to 0
Threshold from 0-1, lower accepts more potential slates.
pretty: optional, defaults to false
false, trueThe JSON body of the HTTP response will be re-formatted with 2-space indentation
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
**A CLAMS wrapper for Kaldi-based ASR software originally developed by PopUpArchive and hipstas, and later updated by Kyeongmin Rim at Brandeis University. Wrapped software can be found at https://github.com/brandeis-llc/aapb-pua-kaldi-docker . **
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
use_speech_segmentation: optional, defaults to true
false, trueWhen true, the app looks for existing TimeFrame { “frameType”: “speech” } annotations, and runs ASR only on those frames, instead of entire audio files.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Release Notes
(no notes provided by the developer)
Brandeis Acoustic Classification & Segmentation (ACS) is a audio segmentation tool developed at Brandeis Lab for Linguistics and Computation. The original software can be found at https://github.com/brandeis-llc/acoustic-classification-segmentation .
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
Release Notes
(no notes provided by the developer)
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
One of the following is required: [
http://mmif.clams.ai/vocabulary/AudioDocument/v1 (required) (of any properties)
http://mmif.clams.ai/vocabulary/VideoDocument/v1 (required) (of any properties)
]
(Note: Multivalued means the parameter can have one or more values.)
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Release Notes
(no notes provided by the developer)
A CLAMS wrapper for Whisper-based ASR software originally developed by OpenAI.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Release Notes
(no notes provided by the developer)
**A CLAMS wrapper for Kaldi-based ASR software originally developed by PopUpArchive and hipstas, and later updated by Kyeongmin Rim at Brandeis University. Wrapped software can be found at https://github.com/brandeis-llc/aapb-pua-kaldi-docker . **
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Multivalued means the parameter can have one or more values.)
use_speech_segmentation: optional, defaults to true
false, trueWhen true, the app looks for existing TimeFrame { “frameType”: “speech” } annotations, and runs ASR only on those frames, instead of entire audio files.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (of any properties)
http://mmif.clams.ai/vocabulary/Alignment/v1 (of any properties)
Release Notes
(no notes provided by the developer)
Apply spaCy NLP to all text documents in a MMIF file.
(Note: “*” as a property value means that the property is required but can be any value.)
http://mmif.clams.ai/vocabulary/TextDocument/v1 (required) (of any properties)
http://vocab.lappsgrid.org/Token (of any properties)
(Note: Multivalued means the parameter can have one or more values.)
pretokenized: optional, defaults to false
false, trueBoolean parameter to set the app to use existing tokenization, if available, for text documents for NLP processing. Useful to process ASR documents, for example.
(Note: “*” as a property value means that the property is required but can be any value.)
(Note: Not all output annotations are always generated.)
http://vocab.lappsgrid.org/Token (of any properties)
http://vocab.lappsgrid.org/Token#pos (of any properties)
http://vocab.lappsgrid.org/Token#lemma (of any properties)
http://vocab.lappsgrid.org/NounChunk (of any properties)
http://vocab.lappsgrid.org/Sentence (of any properties)
http://vocab.lappsgrid.org/NamedEntity (of any properties)
@keighrim)@MrSqually)useClassifier, useStitcher parameters. When useClassifier=True, it runs in the “TimePoint mode” and generates TimePoint annotations. When useStitcher=True, it runs in the “TimeFrame mode” and generates TimeFrame annotations based on existing TimePoint annotations – if no TimePoint is found, it produces an error. By default, it runs in the ‘both’ mode and first generates TimePoint annotations and then TimeFrame annotations on them.
@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@keighrim)@marcverhagen)@keighrim)@keighrim)@keighrim)@keighrim)@marcverhagen)@marcverhagen)@marcverhagen)@marcverhagen)@marcverhagen)prompt against the TimeFrames selected by tfLabels; to apply different prompts to different label subsets (e.g. one prompt for slates, another for chyrons), run the app once per (prompt, tfLabels) combination. Per-TimeFrame captioning is composite: every frame sampled from a TF is fed to the model in a single prompt and yields one caption per TF. This app ships only the 2.2B-Instruct variant – the largest and most general-purpose model in the SmolVLM2 family. The smaller (256M and 500M) SmolVLM2 releases are post-trained specifically for video-QA tasks and we do not expect them to generalize well, given their size.
]]>TimePoint annotations into a sequence of TimeFrame annotations, performing simple smoothing of short peaks of positive labels.
]]>analyzer_version in this metadata.
]]>TextDocument, Paragraphs, Sentence, and Token annotations to represent recognized text contents. See descriptions for I/O types below for details on how annotations are aligned to each other.
]]>@wricketts)@keighrim)@wricketts)