Discuss: Event Assets

docs/event/assets.md

@@ -0,0 +1,101 @@
+---
+sidebar_position: 4
+---
+
+# Assets
+
+Assets are files associated with an event.
+Each asset has some asset-metadata that is stored in the DB, while the file itself is stored in the file system and is referenced by the database.
+
+Every asset has this metadata attached to it:
+- `id: ID`: unique identifier among all assets of all events. Assigned by Opencast and unchangable.
+- `flavor: NonBlankAsciiString` <sup>(1?)</sup>
+- `tags: string[]` <sup>(1?)</sup>
+- `properties: Map<Label, string>`: a `Label` to string map for custom properties. Values can be arbitrary strings.<sup>(7?)</sup>
+- `mimeType: NonBlankAsciiString`: a *lowercase* `NonBlankAsciiString` representing the MIME-type of the asset.
+- `size: uint64`: size of the file in bytes. This is always the actual file size and cannot be changed manually.
+- `checksum`: Checksum/hash of the file. Consists of `type` (e.g. `md5`, `sha256`) and the hex encoded value. In the API (and maybe in the database?) this should be serialized as `<type>:<value>`, e.g. `sha256:e3b0c44298f...`.
+- `source: bool`: whether this was directly uploaded by a user. `false` if it was processed or generated by Opencast.
+- `updated: Timestamp`: timestamp of when anything about this asset was last changed. <sup>(2?)</sup>
+- `internal: bool`: internal assets are not exposed like other assets in APIs. They are used to store source or intermediate artifacts, like the originally uploaded video. These cannot be queried or read by users with only `read` access. <sup>(3?)</sup>
+
+Additionally, in the API representation, assets have the following fields:
+- `uri: string`: a URI to the asset, i.e. where it can be downloaded.
+
+
+## Tracks
+
+A track is an attachment with time-component and "lives on a timeline", i.e. representing something that spans the video length.
+A cut operation on the video needs to modify all tracks.
+
+Each track has one or more streams.
+For example, an `mp4` track might have a video and audio stream, while a `vtt` track only has a single text track.
+
+- `tracks: Track[]`
+- `isLive: bool` <sup>(5?)</sup>
+- `isMaster: bool`: <sup>TODO: describe what exactly this means</sup>
+- `duration: Milliseconds`
+
+Non-`internal` tracks have to have the following properties:
+- The `duration` of the track has to match the `duration` of the event, i.e. all non-internal durations are the same.
+- There is only one video stream<sup>?</sup>
+
+
+### Streams
+
+A stream has the following properties:
+- `language: LangCode?`
+- `type: "video" | "audio" | "text"`: type of the track. Depending on the type, there are additional properties:
+  - `"video"`
+    - `resolution: [uint32, uint32]`
+    - `framerate: "dynamic" | float32`<sup>?</sup>
+    - `codec: "H264" | "H265" | "VP8" | "VP9" | "AV1" | ...`<sup>?</sup>
+  - `"audio"`
+    - `codec: "AAC" | "MP3" | "Opus" | ...`<sup>?</sup>
+  - `"text"`:
+    - `kind: string`: the kind of data this text track represents.
+      - `"subtitle"`: subtitles (does *not* contain speaker names, sounds, and the like)
+      - `"caption"`: closed captions (*does* contain speaker names, sounds, or the like)
+      - `"chapters"`: chapter markers with title per chapter
+      - `"in-video-text"`<sup>(4?)</sup>: representing text in the video (e.g. slide text)
+    - `origin: "manual" | "generated" | null`: describes how this track was created, `manual` meaning it was human crafted, while `generated` means it was automatically generated.
+    - `generator: string?`: non-empty name of the generating software (should only be set if `origin: "generated"`). Example: `"whisper"`.
+
+## Attachments
+
+Attachments are assets without "time-component", unlike tracks.
+They have the following additional metadata:
+- `language: LangCode?`
+
+There are some built-in attachments that have special meanings and requirements, each identified by the flavor:
+
+- `oc/thumbnail`: Thumbnail for the event, e.g. a preview image.
+  - `mimeType` must be `image/*`
+  - `properties` must include `w` and `h`, both holding numbers describing the width and height of the image.
+- `?/timeline-preview`: a sprite image, holding a raster of smaller images, all extracted from the video at regular intervals.
+  - `mimeType` must be `image/*`
+  - `properties` must include:
+    - `imageCountX`, `imageCountY`: how many smaller images are in each row/column
+    - `imageSizeX`, `imageSizeY`: size of each smaller image in pixels
+  - TODO: in the future, this might be better encoded as actual video file
+- `?/segment-preview`<sup>(6?)</sup>: image that is a preview for a segment of the video.
+  - `mimeType` must be `image/*`
+  - `properties` must include `startTime: Milliseconds`, denoting when the segment starts.
+
+TODO: generally make clear what properties are "user changable" and which are automatically set by OC, derived from files.
+
+---
+
+:::danger[Open questions]
+
+- (1?) Is it OK to require ASCII-only for tags and flavors?
+- (2?) Do we really need the `updated` field?
+- (3?) What permissions should be required to read internal assets? Is `write` access to the event enough? And/or should a special role be required?
+- (4?) Better name for this? `on-screen-text`? `text`? `video-text`?
+- (5?) Is `isLive` per track really the correct model? Should this be attached to the event instead? Like, how would a Tobira or LMS decide whether to display an event as live or not?
+- (6?) For timeline and segment previews, it is a bit unclear how to deal with dual stream videos. Right now, Opencast only generates these previews for one video (presentation) by default, I think? Is it useful to have previews for both? Then apps/the player need to support that.
+  - If we want to potentially show both, then the current `presenter/*` can stay.
+  - If we just want to have one preview per video, then it should be `oc/*`, as otherwise external apps have to arbitrarily chose one.
+- (7?) Do we want to allow values other than `string` in properties? Also compare `extraMetadata`
+
+:::