From patchwork Sun Sep 5 12:27:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Stone X-Patchwork-Id: 12476075 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28241C433EF for ; Sun, 5 Sep 2021 12:27:54 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D154C60C51 for ; Sun, 5 Sep 2021 12:27:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D154C60C51 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=collabora.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DE72089B9A; Sun, 5 Sep 2021 12:27:52 +0000 (UTC) Received: from bhuna.collabora.co.uk (bhuna.collabora.co.uk [IPv6:2a00:1098:0:82:1000:25:2eeb:e3e3]) by gabe.freedesktop.org (Postfix) with ESMTPS id 15EF289B9A for ; Sun, 5 Sep 2021 12:27:51 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) (Authenticated sender: daniels) with ESMTPSA id 783121F42F45 From: Daniel Stone To: dri-devel@lists.freedesktop.org Subject: [PATCH] doc: gpu: Add document describing buffer exchange Date: Sun, 5 Sep 2021 13:27:42 +0100 Message-Id: <20210905122742.86029-1-daniels@collabora.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Since there's a lot of confusion around this, document both the rules and the best practice around negotiating, allocating, importing, and using buffers when crossing context/process/device/subsystem boundaries. This ties up all of dmabuf, formats and modifiers, and their usage. Signed-off-by: Daniel Stone --- This is just a quick first draft, inspired by: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/3197#note_1048637 It's not complete or perfect, but I'm off to eat a roast then have a nice walk in the sun, so figured it'd be better to dash it off rather than let it rot on my hard drive. .../gpu/exchanging-pixel-buffers.rst | 285 ++++++++++++++++++ Documentation/gpu/index.rst | 1 + 2 files changed, 286 insertions(+) create mode 100644 Documentation/gpu/exchanging-pixel-buffers.rst diff --git a/Documentation/gpu/exchanging-pixel-buffers.rst b/Documentation/gpu/exchanging-pixel-buffers.rst new file mode 100644 index 000000000000..75c4de13d5c8 --- /dev/null +++ b/Documentation/gpu/exchanging-pixel-buffers.rst @@ -0,0 +1,285 @@ +.. Copyright 2021 Collabora Ltd. + +======================== +Exchanging pixel buffers +======================== + +As originally designed, the Linux graphics subsystem had extremely limited +support for sharing pixel-buffer allocations between processes, devices, and +subsystems. Modern systems require extensive integration between all three +classes; this document details how applications and kernel subsystems should +approach this sharing for two-dimensional image data. + +It is written with reference to the DRM subsystem for GPU and display devices, +V4L2 for media devices, and also to Vulkan, EGL and Wayland, for userspace +support, however any other subsystems should also follow this design and advice. + + +Formats and modifiers +===================== + +Each buffer must have an underlying format. This format describes the data which +can be stored and loaded for each pixel. Although each subsystem has its own +format descriptions (e.g. V4L2 and fbdev), the `DRM_FORMAT_*` tokens should be +reused wherever possible, as they are the standard descriptions used for +interchange. + +Each `DRM_FORMAT_*` token describes the per-pixel data available, in terms of +the translation between one or more pixels in memory, and the color data +contained within that memory. The number and type of color channels are +described: whether they are RGB or YUV, integer or floating-point, the size +of each channel and their locations within the pixel memory, and the +relationship between color planes. + +For example, `DRM_FORMAT_ARGB8888` describes a format in which each pixel has a +single 32-bit value in memory. Alpha, red, green, and blue, color channels are +available at 8-byte precision per channel, ordered respectively from most to +least significant bits in little-endian storage. As a more complex example, +`DRM_FORMAT_NV12` describes a format in which luma and chroma YUV samples are +stored in separate memory planes, where the chroma plane is stored at half the +resolution in both dimensions (i.e. one U/V chroma sample is stored for each 2x2 +pixel grouping). + +Format modifiers describe a translation mechanism between these per-pixel memory +samples, and the actual memory storage for the buffer. The most straightforward +modifier is `DRM_FORMAT_MOD_LINEAR`, describing a scheme in which each pixel has +contiguous storage beginning at (0,0); each pixel's location in memory will be +`base + (y * stride) + (x * bpp)`. This is considered the baseline interchange +format, and most convenient for CPU access. + +Modern hardware employs much more sophisticated access mechanisms, typically +making use of tiled access and possibly also compression. For example, the +`DRM_FORMAT_MOD_VIVANTE_TILED` modifier describes memory storage where pixels +are stored in 4x4 blocks arranged in row-major ordering, i.e. the first tile in +memory stores pixels (0,0) to (3,3) inclusive, and the second tile in memory +stores pixels (4,0) to (7,3) inclusive. + +Some modifiers may modify the number of memory buffers required to store the +data; for example, the `I915_FORMAT_MOD_Y_TILED_CCS` modifier adds a second +memory buffer to RGB formats in which it stores data about the status of every +tile, notably including whether the tile is fully populated with pixel data, or +can be expanded from a single solid color. + +These extended layouts are highly vendor-specific, and even specific to +particular generations or configurations of devices per-vendor. For this reason, +support of modifiers must be explicitly enumerated and negotiated by all users +in order to ensure a compatible and optimal pipeline, as discussed below. + + +Dimensions and size +=================== + +Each pixel buffer must be accompanied by logical pixel dimensions. This refers +to the number of unique samples which can be extracted from, or stored to, the +underlying memory storage. For example, even though a 1920x1080 +`DRM_FORMAT_NV12` buffer has a luma plane containing 1920x1080 samples for the Y +component, and 960x540 samples for the U and V components, the overall buffer is +still described as having dimensions of 1920x1080. + +The in-memory storage of a buffer is not guaranteed to begin immediately at the +base address of the underlying memory, nor is it guaranteed that the memory +storage is tightly clipped to either dimension. + +Each plane must therefore be described with an `offset` in bytes, which will be +added to the base address of the memory storage before performing any per-pixel +calculations. This may be used to combine multiple planes into a single pixel +buffer; for example, `DRM_FORMAT_NV12` may be stored in a single memory buffer +where the luma plane's storage begins immediately at the start of the buffer +with an offset of 0, and the chroma plane's storage begins after the offset of +the luma plane as expressed through its offset. + +Each plane must also have a `stride` in bytes, expressing the offset in memory +between two contiguous scanlines. For example, a `DRM_FORMAT_MOD_LINEAR` buffer +with dimensions of 1000x1000 may have been allocated as if it were 1024x1000, in +order to allow for aligned access patterns. In this case, the buffer will still +be described with a width of 1000, however the stride will be `1024 * bpp`, +indicating that there are 24 pixels at the positive extreme of the x axis whose +values are not significant. + +Buffers may also be padded further in the y dimension, simply by allocating a +larger area than would ordinarily be required. For example, many media decoders +are not able to natively output buffers of height 1080, but instead require an +effective height of 1088 pixels. In this case, the buffer continues to be +described as having a height of 1080, with the memory allocation for each buffer +being increased to account for the extra padding. + + +Enumeration +=========== + +Every user of pixel buffers must be able to enumerate a set of supported formats +and modifiers, described together. Within KMS, this is achieved with the +`IN_FORMATS` property on each DRM plane, listing the supported DRM formats, and +the modifiers supported for each format. In userspace, this is supported through +the `EGL_EXT_image_dma_buf_import_modifiers` extension entrypoints for EGL, the +`VK_EXT_image_drm_format_modifier` extension for Vulkan, and the +`zwp_linux_dmabuf_v1` extension for Wayland. + +Each of these interfaces allows users to query a set of supported +format+modifier combinations. + + +Negotiation +=========== + +It is the responsibility of userspace to negotiate an acceptable format+modifier +combination for its usage. This is performed through a simple intersection of +lists. For example, if a user wants to use Vulkan to render an image to be +displayed on a KMS plane, it must: + - query KMS for the `IN_FORMATS` property for the given plane + - query Vulkan for the supported formats for its physical device + - intersect these formats to determine the most appropriate one + - for this format, intersect the lists of supported modifiers for both KMS and + Vulkan, to obtain a final list of acceptable modifiers for that format + +This intersection must be performed for all usages. For example, if the user +also wishes to encode the image to a video stream, it must query the media API +it intends to use for encoding for the set of modifiers it supports, and +additionally intersect against this list. + +If the intersection of all lists is an empty list, it is not possible to share +buffers in this way, and an alternate strategy must be considered (e.g. using +CPU access routines to copy data between the different uses, with the +corresponding performance cost). + +The resulting modifier list is unsorted; the order is not significant. + + +Allocation +========== + +Once userspace has determined an appropriate format, and corresponding list of +acceptable modifiers, it must allocate the buffer. As there is no universal +buffer-allocation interface available at either kernel or userspace level, the +client makes an arbitrary choice of allocation interface such as Vulkan, GBM, or +a media API. + +Each allocation request must take, at a minimum: the pixel format, a list of +acceptable modifiers, and the buffer's width and height. Each API may extend +this set of properties in different ways, such as allowing allocation in more +than two dimensions, intended usage patterns, etc. + +The component which allocates the buffer will make an arbitrary choice of what +it considers the 'best' modifier within the acceptable list for the requested +allocation, any padding required, and further properties of the underlying +memory buffers such as whether they are stored in system or device-specific +memory, whether or not they are physically contiguous, and their cache mode. +These properties of the memory buffer are not visible to userspace, however the +`dma-heaps` API is an effort to address this. + +After allocation, the client must query the allocator to determine the actual +modifier selected for the buffer, as well as the per-plane offset and stride. +Allocators are not permitted to vary the format in use, to select a modifier not +provided within the acceptable list, nor to vary the pixel dimensions other than +the padding expressed through offset, stride, and size. + + +Import +====== + +To use a buffer within a different context, device, or subsystem, the user +passes these parameters (format, modifier, width, height, and per-plane offset +and stride) to an importing API. + +Each memory plane is referred to by a buffer handle, which may be unique or +duplicated within a buffer. For example, a `DRM_FORMAT_NV12` buffer may have the +luma and chroma buffers combined into a single memory buffer by use of the +per-plane offset parameters, or they may be completely separate allocations in +memory. For this reason, each import and allocation API must provide a separate +handle for each plane. + +Each kernel subsystem has its own types and interfaces for buffer management. +DRM uses GEM buffer objects (BOs), V4L2 has its own references, etc. These types +are not portable between contexts, processes, devices, or subsystems. + +To address this, `dma-buf` handles are used as the universal interchange for +buffers. Subsystem-specific operations are used to export native buffer handles +to a `dma-buf` file descriptor, and to import those file descriptors into a +native buffer handle. dma-buf file descriptors can be transferred between +contexts, processes, devices, and subsystems. + +For example, a Wayland media player may use V4L2 to decode a video frame into +a `DRM_FORMAT_NV12` buffer. This will result in two memory planes (luma and +chroma) being dequeued by the user from V4L2. These planes are then exported to +one dma-buf file descriptor per plane, these descriptors are then sent along +with the metadata (format, modifier, width, height, per-plane offset and stride) +to the Wayland server. The Wayland server will then import these file +descriptors as an EGLImage for use through EGL/OpenGL (ES), a VkImage for use +through Vulkan, or a `drm_fb` for use through KMS; each of these import +operations will take the same metadata and convert the dma-buf file descriptors +into their native buffer handles. + + +Implicit modifiers +================== + +The concept of modifiers post-dates all of the subsystems mentioned above. As +such, it has been retrofitted into all of these APIs, and in order to ensure +backwards compatibility, support is needed for drivers and userspace which do +not (yet) support modifiers. + +As an example, GBM is used to allocate buffers to be shared between EGL for +rendering and KMS for display. It has two entrypoints for allocating buffers: +`gbm_bo_create` which only takes the format, width, height, and a usage token, +and `gbm_bo_create_with_modifiers` which extends this with a list of modifiers. + +In the latter case, the allocation is as discussed above, being provided with a +list of acceptable modifiers that the implementation can choose from (or fail if +it is not possible to allocate within those constraints). In the former case +where modifiers are not provided, the GBM implementation must make its own +choice as to what is likely to be the 'best' layout. Such a choice is entirely +implementation-specific: some will internally use tiled layouts which are not +CPU-accessible if the implementation decides that is a good idea through +whatever heuristic. It is the implementation's responsibility to ensure that +this choice is appropriate. + +To support this case where the layout is not known because there is no awareness +of modifiers, a special `DRM_FORMAT_MOD_INVALID` token has been defined. This +pseudo-modifier declares that the layout is not known, and that the driver +should use its own logic to determine what the underlying layout may be. + +There are four cases where this token may be used: + - during enumeration, an interface may return `DRM_FORMAT_MOD_INVALID`, either + as the sole member of a modifier list to declare that explicit modifiers are + not supported, or as part of a larger list to declare that implicit modifiers + may be used + - during allocation, a user may supply `DRM_FORMAT_MOD_INVALID`, either as the + sole member of a modifier list (equivalent to not supplying a modifier list + at all) to declare that explicit modifiers are not supported and must not be + used, or as part of a larger list to declare that an allocation using implicit + modifiers is acceptable + - in a post-allocation query, an implementation may return + `DRM_FORMAT_MOD_INVALID` as the modifier of the allocated buffer to declare + that the underlying layout is implementation-defined and that an explicit + modifier description is not available; per the above rules, this may only be + returned when the user has included `DRM_FORMAT_MOD_INVALID` as part of the + list of acceptable modifiers, or not provided a list + - when importing a buffer, the user may supply `DRM_FORMAT_MOD_INVALID` as the + buffer modifier (or not supply a modifier) to indicate that the modifier is + unknown for whatever reason; this is only acceptable when the buffer has + not been allocated with an explicit modifier + +It follows from this that a buffer chain must be either fully implicit or fully +explicit. For example, if a user wishes to allocate a buffer for use between +GPU, display, and media, but the media API does not support modifiers, then the +user **must not** allocate the buffer with explicit modifiers and attempt to +import the buffer into the media API with no modifier, but either perform the +allocation using implicit modifiers, or allocate the buffer for media use +separately and copy between the two buffers. + +As one exception to the above, allocations may be 'upgraded' from implicit +to explicit modifiers. For example, if the buffer is allocated with +`gbm_bo_create` (taking no modifiers), the user may then query the modifier with +`gbm_bo_get_modifier` and then use this modifier as an explicit modifier token +if a valid modifier is returned. + +When allocating buffers for exchange between different users and modifiers are +not available, implementations are strongly encouraged to use +`DRM_FORMAT_MOD_LINEAR` for their allocation, as this is the universal baseline +for exchange. + +Any new users - userspace programs and protocols, kernel subsystems, etc - +wishing to exchange buffers must offer interoperability through dma-buf file +descriptors for memory planes, DRM format tokens to describe the format, DRM +format modifiers to describe the layout in memory, at least width and height for +dimensions, and at least offset and stride for each memory plane. diff --git a/Documentation/gpu/index.rst b/Documentation/gpu/index.rst index b9c1214d8f23..cb12f2654ed7 100644 --- a/Documentation/gpu/index.rst +++ b/Documentation/gpu/index.rst @@ -10,6 +10,7 @@ Linux GPU Driver Developer's Guide drm-kms drm-kms-helpers drm-uapi + exchanging-pixel-buffers driver-uapi drm-client drivers From patchwork Thu Aug 3 15:47:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Stone X-Patchwork-Id: 13340276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EBCD8EB64DD for ; Thu, 3 Aug 2023 15:49:44 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 03F3310E625; Thu, 3 Aug 2023 15:49:39 +0000 (UTC) Received: from madras.collabora.co.uk (madras.collabora.co.uk [IPv6:2a00:1098:0:82:1000:25:2eeb:e5ab]) by gabe.freedesktop.org (Postfix) with ESMTPS id ACE9710E624 for ; Thu, 3 Aug 2023 15:49:36 +0000 (UTC) Received: from strictly.printclub (zone.collabora.co.uk [167.235.23.81]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: daniels) by madras.collabora.co.uk (Postfix) with ESMTPSA id 2AC2866071A0 for ; Thu, 3 Aug 2023 16:49:35 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1691077775; bh=hEF9AbkpLFJjarqL0WcBarx2nrfXTiUUrHOt6Fb0FgI=; h=From:To:Subject:Date:In-Reply-To:References:From; b=DNL3uYUtOk0GOUXiNnfFBPoO5xfseyTdGUR2AtCJa4QQ/Hv0Ss59T4ElG6U/SIQxU hhr5XMWYCla08OPNh46oqBeb4KnX3YercN8Mexgt1sQ6AjLwkSGvOrt4e0b3yomsFw +AmcBwu9ypqjAtJt9LRbE0GTq2aWMdsRjZE0IUePUBQ/3y+UYY8mvOAmabC3gAMTOb vj/9OQHqkQpgQc0GBU1r8QHRoMMtFFQv3efHhfwJilDLmz75bgpxaEkhM14FkUtJ+r F0Mh1D6rTTZGow9vj8ZHY9D8F7jM2aWMsFB+pynNtJRnAV5PeY0Er8r6WKaNLvuxbw 97WPFzFMOYWBg== From: Daniel Stone To: dri-devel@lists.freedesktop.org Subject: [PATCH v2 2/2] doc: uapi: Add document describing dma-buf semantics Date: Thu, 3 Aug 2023 16:47:29 +0100 Message-ID: <20230803154908.105124-4-daniels@collabora.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20210905122742.86029-1-daniels@collabora.com> References: <20210905122742.86029-1-daniels@collabora.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Since there's a lot of confusion around this, document both the rules and the best practice around negotiating, allocating, importing, and using buffers when crossing context/process/device/subsystem boundaries. This ties up all of dma-buf, formats and modifiers, and their usage. Signed-off-by: Daniel Stone Reviewed-by: Sui Jingfeng --- Documentation/driver-api/dma-buf.rst | 8 + Documentation/gpu/drm-uapi.rst | 7 + .../userspace-api/dma-buf-alloc-exchange.rst | 384 ++++++++++++++++++ Documentation/userspace-api/index.rst | 1 + 4 files changed, 400 insertions(+) create mode 100644 Documentation/userspace-api/dma-buf-alloc-exchange.rst v2: - Moved to general uAPI section, cross-referenced from dma-buf/DRM - Added Pekka's suggested glossary with some small changes - Cleanups and clarifications from Simon and James diff --git a/Documentation/driver-api/dma-buf.rst b/Documentation/driver-api/dma-buf.rst index 862dbc2759d0..0c153d79ccc4 100644 --- a/Documentation/driver-api/dma-buf.rst +++ b/Documentation/driver-api/dma-buf.rst @@ -22,6 +22,14 @@ interact with the three main primitives offered by dma-buf: allowing implicit (kernel-ordered) synchronization of work to preserve the illusion of coherent access + +Userspace API principles and use +-------------------------------- + +For more details on how to design your subsystem's API for dma-buf use, please +see Documentation/userspace-api/dma-buf-alloc-exchange.rst. + + Shared DMA Buffers ------------------ diff --git a/Documentation/gpu/drm-uapi.rst b/Documentation/gpu/drm-uapi.rst index 65fb3036a580..eef5fd19bc92 100644 --- a/Documentation/gpu/drm-uapi.rst +++ b/Documentation/gpu/drm-uapi.rst @@ -486,3 +486,10 @@ and the CRTC index is its position in this array. .. kernel-doc:: include/uapi/drm/drm_mode.h :internal: + + +dma-buf interoperability +======================== + +Please see Documentation/userspace-api/dma-buf-alloc-exchange.rst for +information on how dma-buf is integrated and exposed within DRM. diff --git a/Documentation/userspace-api/dma-buf-alloc-exchange.rst b/Documentation/userspace-api/dma-buf-alloc-exchange.rst new file mode 100644 index 000000000000..090453d2ad78 --- /dev/null +++ b/Documentation/userspace-api/dma-buf-alloc-exchange.rst @@ -0,0 +1,384 @@ +.. Copyright 2021-2023 Collabora Ltd. + +======================== +Exchanging pixel buffers +======================== + +As originally designed, the Linux graphics subsystem had extremely limited +support for sharing pixel-buffer allocations between processes, devices, and +subsystems. Modern systems require extensive integration between all three +classes; this document details how applications and kernel subsystems should +approach this sharing for two-dimensional image data. + +It is written with reference to the DRM subsystem for GPU and display devices, +V4L2 for media devices, and also to Vulkan, EGL and Wayland, for userspace +support, however any other subsystems should also follow this design and advice. + + +Glossary of terms +================= + +.. glossary:: + + image: + Conceptually a two-dimensional array of pixels. The pixels may be stored + in one or more memory buffers. Has width and height in pixels, pixel + format and modifier (implicit or explicit). + + row: + A span along a single y-axis value, e.g. from co-ordinates (0,100) to + (200,100). + + scanline: + Synonym for row. + + column: + A span along a single x-axis value, e.g. from co-ordinates (100,0) to + (100,100). + + memory buffer: + A piece of memory for storing (parts of) pixel data. Has stride and size + in bytes and at least one handle in some API. May contain one or more + planes. + + plane: + A two-dimensional array of some or all of an image's color and alpha + channel values. + + pixel: + A picture element. Has a single color value which is defined by one or + more color channels values, e.g. R, G and B, or Y, Cb and Cr. May also + have an alpha value as an additional channel. + + pixel data: + Bytes or bits that represent some or all of the color/alpha channel values + of a pixel or an image. The data for one pixel may be spread over several + planes or memory buffers depending on format and modifier. + + color value: + A tuple of numbers, representing a color. Each element in the tuple is a + color channel value. + + color channel: + One of the dimensions in a color model. For example, RGB model has + channels R, G, and B. Alpha channel is sometimes counted as a color + channel as well. + + pixel format: + A description of how pixel data represents the pixel's color and alpha + values. + + modifier: + A description of how pixel data is laid out in memory buffers. + + alpha: + A value that denotes the color coverage in a pixel. Sometimes used for + translucency instead. + + stride: + A value that denotes the relationship between pixel-location co-ordinates + and byte-offset values. Typically used as the byte offset between two + pixels at the start of vertically-consecutive tiling blocks. For linear + layouts, the byte offset between two vertically-adjacent pixels. + + pitch: + Synonym for stride. + + +Formats and modifiers +===================== + +Each buffer must have an underlying format. This format describes the color +values provided for each pixel. Although each subsystem has its own format +descriptions (e.g. V4L2 and fbdev), the ``DRM_FORMAT_*`` tokens should be reused +wherever possible, as they are the standard descriptions used for interchange. +These tokens are described in the ``drm_fourcc.h`` file, which is a part of +DRM's uAPI. + +Each ``DRM_FORMAT_*`` token describes the translation between a pixel +co-ordinate in an image, and the color values for that pixel contained within +its memory buffers. The number and type of color channels are described: +whether they are RGB or YUV, integer or floating-point, the size of each channel +and their locations within the pixel memory, and the relationship between color +planes. + +For example, ``DRM_FORMAT_ARGB8888`` describes a format in which each pixel has +a single 32-bit value in memory. Alpha, red, green, and blue, color channels are +available at 8-bit precision per channel, ordered respectively from most to +least significant bits in little-endian storage. ``DRM_FORMAT_*`` is not +affected by either CPU or device endianness; the byte pattern in memory is +always as described in the format definition, which is usually little-endian. + +As a more complex example, ``DRM_FORMAT_NV12`` describes a format in which luma +and chroma YUV samples are stored in separate planes, where the chroma plane is +stored at half the resolution in both dimensions (i.e. one U/V chroma +sample is stored for each 2x2 pixel grouping). + +Format modifiers describe a translation mechanism between these per-pixel memory +samples, and the actual memory storage for the buffer. The most straightforward +modifier is ``DRM_FORMAT_MOD_LINEAR``, describing a scheme in which each plane +is laid out row-sequentially, from the top-left to the bottom-right corner. +This is considered the baseline interchange format, and most convenient for CPU +access. + +Modern hardware employs much more sophisticated access mechanisms, typically +making use of tiled access and possibly also compression. For example, the +``DRM_FORMAT_MOD_VIVANTE_TILED`` modifier describes memory storage where pixels +are stored in 4x4 blocks arranged in row-major ordering, i.e. the first tile in +a plane stores pixels (0,0) to (3,3) inclusive, and the second tile in a plane +stores pixels (4,0) to (7,3) inclusive. + +Some modifiers may modify the number of planes required for an image; for +example, the ``I915_FORMAT_MOD_Y_TILED_CCS`` modifier adds a second plane to RGB +formats in which it stores data about the status of every tile, notably +including whether the tile is fully populated with pixel data, or can be +expanded from a single solid color. + +These extended layouts are highly vendor-specific, and even specific to +particular generations or configurations of devices per-vendor. For this reason, +support of modifiers must be explicitly enumerated and negotiated by all users +in order to ensure a compatible and optimal pipeline, as discussed below. + + +Dimensions and size +=================== + +Each pixel buffer must be accompanied by logical pixel dimensions. This refers +to the number of unique samples which can be extracted from, or stored to, the +underlying memory storage. For example, even though a 1920x1080 +``DRM_FORMAT_NV12`` buffer has a luma plane containing 1920x1080 samples for the Y +component, and 960x540 samples for the U and V components, the overall buffer is +still described as having dimensions of 1920x1080. + +The in-memory storage of a buffer is not guaranteed to begin immediately at the +base address of the underlying memory, nor is it guaranteed that the memory +storage is tightly clipped to either dimension. + +Each plane must therefore be described with an ``offset`` in bytes, which will be +added to the base address of the memory storage before performing any per-pixel +calculations. This may be used to combine multiple planes into a single memory +buffer; for example, ``DRM_FORMAT_NV12`` may be stored in a single memory buffer +where the luma plane's storage begins immediately at the start of the buffer +with an offset of 0, and the chroma plane's storage follows within the same buffer +beginning from the byte offset for that plane. + +Each plane must also have a ``stride`` in bytes, expressing the offset in memory +between two contiguous row. For example, a ``DRM_FORMAT_MOD_LINEAR`` buffer +with dimensions of 1000x1000 may have been allocated as if it were 1024x1000, in +order to allow for aligned access patterns. In this case, the buffer will still +be described with a width of 1000, however the stride will be ``1024 * bpp``, +indicating that there are 24 pixels at the positive extreme of the x axis whose +values are not significant. + +Buffers may also be padded further in the y dimension, simply by allocating a +larger area than would ordinarily be required. For example, many media decoders +are not able to natively output buffers of height 1080, but instead require an +effective height of 1088 pixels. In this case, the buffer continues to be +described as having a height of 1080, with the memory allocation for each buffer +being increased to account for the extra padding. + + +Enumeration +=========== + +Every user of pixel buffers must be able to enumerate a set of supported formats +and modifiers, described together. Within KMS, this is achieved with the +``IN_FORMATS`` property on each DRM plane, listing the supported DRM formats, and +the modifiers supported for each format. In userspace, this is supported through +the `EGL_EXT_image_dma_buf_import_modifiers`_ extension entrypoints for EGL, the +`VK_EXT_image_drm_format_modifier`_ extension for Vulkan, and the +`zwp_linux_dmabuf_v1`_ extension for Wayland. + +Each of these interfaces allows users to query a set of supported +format+modifier combinations. + + +Negotiation +=========== + +It is the responsibility of userspace to negotiate an acceptable format+modifier +combination for its usage. This is performed through a simple intersection of +lists. For example, if a user wants to use Vulkan to render an image to be +displayed on a KMS plane, it must: + + - query KMS for the ``IN_FORMATS`` property for the given plane + - query Vulkan for the supported formats for its physical device, making sure + to pass the ``VkImageUsageFlagBits`` and ``VkImageCreateFlagBits`` + corresponding to the intended rendering use + - intersect these formats to determine the most appropriate one + - for this format, intersect the lists of supported modifiers for both KMS and + Vulkan, to obtain a final list of acceptable modifiers for that format + +This intersection must be performed for all usages. For example, if the user +also wishes to encode the image to a video stream, it must query the media API +it intends to use for encoding for the set of modifiers it supports, and +additionally intersect against this list. + +If the intersection of all lists is an empty list, it is not possible to share +buffers in this way, and an alternate strategy must be considered (e.g. using +CPU access routines to copy data between the different uses, with the +corresponding performance cost). + +The resulting modifier list is unsorted; the order is not significant. + + +Allocation +========== + +Once userspace has determined an appropriate format, and corresponding list of +acceptable modifiers, it must allocate the buffer. As there is no universal +buffer-allocation interface available at either kernel or userspace level, the +client makes an arbitrary choice of allocation interface such as Vulkan, GBM, or +a media API. + +Each allocation request must take, at a minimum: the pixel format, a list of +acceptable modifiers, and the buffer's width and height. Each API may extend +this set of properties in different ways, such as allowing allocation in more +than two dimensions, intended usage patterns, etc. + +The component which allocates the buffer will make an arbitrary choice of what +it considers the 'best' modifier within the acceptable list for the requested +allocation, any padding required, and further properties of the underlying +memory buffers such as whether they are stored in system or device-specific +memory, whether or not they are physically contiguous, and their cache mode. +These properties of the memory buffer are not visible to userspace, however the +``dma-heaps`` API is an effort to address this. + +After allocation, the client must query the allocator to determine the actual +modifier selected for the buffer, as well as the per-plane offset and stride. +Allocators are not permitted to vary the format in use, to select a modifier not +provided within the acceptable list, nor to vary the pixel dimensions other than +the padding expressed through offset, stride, and size. + +Communicating additional constraints, such as alignment of stride or offset, +placement within a particular memory area, etc, is out of scope of dma-buf, +and is not solved by format and modifier tokens. + + +Import +====== + +To use a buffer within a different context, device, or subsystem, the user +passes these parameters (format, modifier, width, height, and per-plane offset +and stride) to an importing API. + +Each memory buffer is referred to by a buffer handle, which may be unique or +duplicated within an image. For example, a ``DRM_FORMAT_NV12`` buffer may have +the luma and chroma buffers combined into a single memory buffer by use of the +per-plane offset parameters, or they may be completely separate allocations in +memory. For this reason, each import and allocation API must provide a separate +handle for each plane. + +Each kernel subsystem has its own types and interfaces for buffer management. +DRM uses GEM buffer objects (BOs), V4L2 has its own references, etc. These types +are not portable between contexts, processes, devices, or subsystems. + +To address this, ``dma-buf`` handles are used as the universal interchange for +buffers. Subsystem-specific operations are used to export native buffer handles +to a ``dma-buf`` file descriptor, and to import those file descriptors into a +native buffer handle. dma-buf file descriptors can be transferred between +contexts, processes, devices, and subsystems. + +For example, a Wayland media player may use V4L2 to decode a video frame into a +``DRM_FORMAT_NV12`` buffer. This will result in two memory planes (luma and +chroma) being dequeued by the user from V4L2. These planes are then exported to +one dma-buf file descriptor per plane, these descriptors are then sent along +with the metadata (format, modifier, width, height, per-plane offset and stride) +to the Wayland server. The Wayland server will then import these file +descriptors as an EGLImage for use through EGL/OpenGL (ES), a VkImage for use +through Vulkan, or a KMS framebuffer object; each of these import operations +will take the same metadata and convert the dma-buf file descriptors into their +native buffer handles. + +Having a non-empty intersection of supported modifiers does not guarantee that +import will succeed into all consumers; they may have constraints beyond those +impliied by modifiers which must be satisfied. + + +Implicit modifiers +================== + +The concept of modifiers post-dates all of the subsystems mentioned above. As +such, it has been retrofitted into all of these APIs, and in order to ensure +backwards compatibility, support is needed for drivers and userspace which do +not (yet) support modifiers. + +As an example, GBM is used to allocate buffers to be shared between EGL for +rendering and KMS for display. It has two entrypoints for allocating buffers: +``gbm_bo_create`` which only takes the format, width, height, and a usage token, +and ``gbm_bo_create_with_modifiers`` which extends this with a list of modifiers. + +In the latter case, the allocation is as discussed above, being provided with a +list of acceptable modifiers that the implementation can choose from (or fail if +it is not possible to allocate within those constraints). In the former case +where modifiers are not provided, the GBM implementation must make its own +choice as to what is likely to be the 'best' layout. Such a choice is entirely +implementation-specific: some will internally use tiled layouts which are not +CPU-accessible if the implementation decides that is a good idea through +whatever heuristic. It is the implementation's responsibility to ensure that +this choice is appropriate. + +To support this case where the layout is not known because there is no awareness +of modifiers, a special ``DRM_FORMAT_MOD_INVALID`` token has been defined. This +pseudo-modifier declares that the layout is not known, and that the driver +should use its own logic to determine what the underlying layout may be. + +.. note:: + + ``DRM_FORMAT_MOD_INVALID`` is a non-zero value. The modifier value zero is + ``DRM_FORMAT_MOD_LINEAR``, which is an explicit guarantee that the image + has the linear layout. Care and attention should be taken to ensure that + zero as a default uninitialized value signals no modifier. + +There are four cases where this token may be used: + - during enumeration, an interface may return ``DRM_FORMAT_MOD_INVALID``, either + as the sole member of a modifier list to declare that explicit modifiers are + not supported, or as part of a larger list to declare that implicit modifiers + may be used + - during allocation, a user may supply ``DRM_FORMAT_MOD_INVALID``, either as the + sole member of a modifier list (equivalent to not supplying a modifier list + at all) to declare that explicit modifiers are not supported and must not be + used, or as part of a larger list to declare that an allocation using implicit + modifiers is acceptable + - in a post-allocation query, an implementation may return + ``DRM_FORMAT_MOD_INVALID`` as the modifier of the allocated buffer to declare + that the underlying layout is implementation-defined and that an explicit + modifier description is not available; per the above rules, this may only be + returned when the user has included ``DRM_FORMAT_MOD_INVALID`` as part of the + list of acceptable modifiers, or not provided a list + - when importing a buffer, the user may supply ``DRM_FORMAT_MOD_INVALID`` as the + buffer modifier (or not supply a modifier) to indicate that the modifier is + unknown for whatever reason; this is only acceptable when the buffer has + not been allocated with an explicit modifier + +It follows from this that for any single buffer, the complete chain of operations +formed by the producer and all the consumers must be either fully implicit or fully +explicit. For example, if a user wishes to allocate a buffer for use between +GPU, display, and media, but the media API does not support modifiers, then the +user **must not** allocate the buffer with explicit modifiers and attempt to +import the buffer into the media API with no modifier, but either perform the +allocation using implicit modifiers, or allocate the buffer for media use +separately and copy between the two buffers. + +As one exception to the above, allocations may be 'upgraded' from implicit +to explicit modifiers. For example, if the buffer is allocated with +``gbm_bo_create`` (taking no modifiers), the user may then query the modifier with +``gbm_bo_get_modifier`` and then use this modifier as an explicit modifier token +if a valid modifier is returned. + +When allocating buffers for exchange between different users and modifiers are +not available, implementations are strongly encouraged to use +``DRM_FORMAT_MOD_LINEAR`` for their allocation, as this is the universal baseline +for exchange. However, it is not guaranteed that this will result in the correct +interpretation of buffer content, as implicit modifier operation may still be +subject to driver-specific heuristics. + +Any new users - userspace programs and protocols, kernel subsystems, etc - +wishing to exchange buffers must offer interoperability through dma-buf file +descriptors for memory planes, DRM format tokens to describe the format, DRM +format modifiers to describe the layout in memory, at least width and height for +dimensions, and at least offset and stride for each memory plane. + +.. _zwp_linux_dmabuf_v1: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml +.. _VK_EXT_image_drm_format_modifier: https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VK_EXT_image_drm_format_modifier.html +.. _EGL_EXT_image_dma_buf_import_modifiers: https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_image_dma_buf_import_modifiers.txt \ No newline at end of file diff --git a/Documentation/userspace-api/index.rst b/Documentation/userspace-api/index.rst index 72a65db0c498..031df47a7c19 100644 --- a/Documentation/userspace-api/index.rst +++ b/Documentation/userspace-api/index.rst @@ -22,6 +22,7 @@ place where this information is gathered. unshare spec_ctrl accelerators/ocxl + dma-buf-alloc-exchange ebpf/index ELF ioctl/index