[v2] drm/doc: add rfc section for small BAR uapi

Commit Message

Matthew Auld April 20, 2022, 5:13 p.m. UTC

Add an entry for the new uapi needed for small BAR on DG2+.

v2:
  - Some spelling fixes and other small tweaks. (Akeem & Thomas)
  - Rework error capture interactions, including no longer needing
    NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
  - Add probed_cpu_visible_size. (Lionel)

Signed-off-by: Matthew Auld <[e-mail]>
Cc: Thomas Hellström <[e-mail]>
Cc: Lionel Landwerlin <[e-mail]>
Cc: Jon Bloomfield <[e-mail]>
Cc: Daniel Vetter <[e-mail]>
Cc: Jordan Justen <[e-mail]>
Cc: Kenneth Graunke <[e-mail]>
Cc: Akeem G Abodunrin <[e-mail]>
Cc: [e-mail]
---
 Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
 Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
 Documentation/gpu/rfc/index.rst          |   4 +
 3 files changed, 252 insertions(+)
 create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
 create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst

Comments

Lionel Landwerlin April 27, 2022, 6:35 a.m. UTC | #1

Hi Matt,


The proposal looks good to me.

Looking forward to try it on drm-tip.


-Lionel

On 20/04/2022 20:13, Matthew Auld wrote:
> Add an entry for the new uapi needed for small BAR on DG2+.
>
> v2:
>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>    - Rework error capture interactions, including no longer needing
>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>    - Add probed_cpu_visible_size. (Lionel)
>
> Signed-off-by: Matthew Auld <[e-mail]>
> Cc: Thomas Hellström <[e-mail]>
> Cc: Lionel Landwerlin <[e-mail]>
> Cc: Jon Bloomfield <[e-mail]>
> Cc: Daniel Vetter <[e-mail]>
> Cc: Jordan Justen <[e-mail]>
> Cc: Kenneth Graunke <[e-mail]>
> Cc: Akeem G Abodunrin <[e-mail]>
> Cc: [e-mail]
> ---
>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>   Documentation/gpu/rfc/index.rst          |   4 +
>   3 files changed, 252 insertions(+)
>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>
> diff --git a/Documentation/gpu/rfc/i915_small_bar.h b/Documentation/gpu/rfc/i915_small_bar.h
> new file mode 100644
> index 000000000000..7bfd0cf44d35
> --- /dev/null
> +++ b/Documentation/gpu/rfc/i915_small_bar.h
> @@ -0,0 +1,190 @@
> +/**
> + * struct __drm_i915_memory_region_info - Describes one region as known to the
> + * driver.
> + *
> + * Note this is using both struct drm_i915_query_item and struct drm_i915_query.
> + * For this new query we are adding the new query id DRM_I915_QUERY_MEMORY_REGIONS
> + * at &drm_i915_query_item.query_id.
> + */
> +struct __drm_i915_memory_region_info {
> +	/** @region: The class:instance pair encoding */
> +	struct drm_i915_gem_memory_class_instance region;
> +
> +	/** @rsvd0: MBZ */
> +	__u32 rsvd0;
> +
> +	/** @probed_size: Memory probed by the driver (-1 = unknown) */
> +	__u64 probed_size;
> +
> +	/** @unallocated_size: Estimate of memory remaining (-1 = unknown) */
> +	__u64 unallocated_size;
> +
> +	union {
> +		/** @rsvd1: MBZ */
> +		__u64 rsvd1[8];
> +		struct {
> +			/**
> +			 * @probed_cpu_visible_size: Memory probed by the driver
> +			 * that is CPU accessible. (-1 = unknown).
> +			 *
> +			 * This will be always be <= @probed_size, and the
> +			 * remainder(if there is any) will not be CPU
> +			 * accessible.
> +			 */
> +			__u64 probed_cpu_visible_size;
> +		};
> +	};
> +};
> +
> +/**
> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, with added
> + * extension support using struct i915_user_extension.
> + *
> + * Note that new buffer flags should be added here, at least for the stuff that
> + * is immutable. Previously we would have two ioctls, one to create the object
> + * with gem_create, and another to apply various parameters, however this
> + * creates some ambiguity for the params which are considered immutable. Also in
> + * general we're phasing out the various SET/GET ioctls.
> + */
> +struct __drm_i915_gem_create_ext {
> +	/**
> +	 * @size: Requested size for the object.
> +	 *
> +	 * The (page-aligned) allocated size for the object will be returned.
> +	 *
> +	 * Note that for some devices we have might have further minimum
> +	 * page-size restrictions(larger than 4K), like for device local-memory.
> +	 * However in general the final size here should always reflect any
> +	 * rounding up, if for example using the I915_GEM_CREATE_EXT_MEMORY_REGIONS
> +	 * extension to place the object in device local-memory.
> +	 */
> +	__u64 size;
> +	/**
> +	 * @handle: Returned handle for the object.
> +	 *
> +	 * Object handles are nonzero.
> +	 */
> +	__u32 handle;
> +	/**
> +	 * @flags: Optional flags.
> +	 *
> +	 * Supported values:
> +	 *
> +	 * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the kernel that
> +	 * the object will need to be accessed via the CPU.
> +	 *
> +	 * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
> +	 * only strictly required on platforms where only some of the device
> +	 * memory is directly visible or mappable through the CPU, like on DG2+.
> +	 *
> +	 * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
> +	 * ensure we can always spill the allocation to system memory, if we
> +	 * can't place the object in the mappable part of
> +	 * I915_MEMORY_CLASS_DEVICE.
> +	 *
> +	 * Note that since the kernel only supports flat-CCS on objects that can
> +	 * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
> +	 * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
> +	 * flat-CCS.
> +	 *
> +	 * Without this hint, the kernel will assume that non-mappable
> +	 * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note that the
> +	 * kernel can still migrate the object to the mappable part, as a last
> +	 * resort, if userspace ever CPU faults this object, but this might be
> +	 * expensive, and so ideally should be avoided.
> +	 */
> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
> +	__u32 flags;
> +	/**
> +	 * @extensions: The chain of extensions to apply to this object.
> +	 *
> +	 * This will be useful in the future when we need to support several
> +	 * different extensions, and we need to apply more than one when
> +	 * creating the object. See struct i915_user_extension.
> +	 *
> +	 * If we don't supply any extensions then we get the same old gem_create
> +	 * behaviour.
> +	 *
> +	 * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
> +	 * struct drm_i915_gem_create_ext_memory_regions.
> +	 *
> +	 * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
> +	 * struct drm_i915_gem_create_ext_protected_content.
> +	 */
> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
> +	__u64 extensions;
> +};
> +
> +#define DRM_I915_QUERY_VMA_INFO	5
> +
> +/**
> + * struct __drm_i915_query_vma_info
> + *
> + * Given a vm and GTT address, lookup the corresponding vma, returning its set
> + * of attributes.
> + *
> + * .. code-block:: C
> + *
> + *	struct drm_i915_query_vma_info info = {};
> + *	struct drm_i915_query_item item = {
> + *		.data_ptr = (uintptr_t)&info,
> + *		.query_id = DRM_I915_QUERY_VMA_INFO,
> + *	};
> + *	struct drm_i915_query query = {
> + *		.num_items = 1,
> + *		.items_ptr = (uintptr_t)&item,
> + *	};
> + *	int err;
> + *
> + *	// Unlike some other types of queries, there is no need to first query
> + *	// the size of the data_ptr blob here, since we already know ahead of
> + *	// time how big this needs to be.
> + *	item.length = sizeof(info);
> + *
> + *	// Next we fill in the vm_id and ppGTT address of the vma we wish
> + *	// to query, before then firing off the query.
> + *	info.vm_id = vm_id;
> + *	info.offset = gtt_address;
> + *	err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
> + *	if (err || item.length < 0) ...
> + *
> + *	// If all went well we can now inspect the returned attributes.
> + *	if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
> + */
> +struct __drm_i915_query_vma_info {
> +	/**
> +	 * @vm_id: The given vm id that contains the vma. The id is the value
> +	 * returned by the DRM_I915_GEM_VM_CREATE. See struct
> +	 * drm_i915_gem_vm_control.vm_id.
> +	 */
> +	__u32 vm_id;
> +	/** @pad: MBZ. */
> +	__u32 pad;
> +	/**
> +	 * @offset: The corresponding ppGTT address of the vma which the kernel
> +	 * will use to perform the lookup.
> +	 */
> +	__u64 offset;
> +	/**
> +	 * @attributes: The returned attributes for the given vma.
> +	 *
> +	 * Possible values:
> +	 *
> +	 * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages backing the
> +	 * vma are currently CPU accessible. If this is not set then the vma is
> +	 * currently backed by I915_MEMORY_CLASS_DEVICE memory, which the CPU
> +	 * cannot directly access(this is only possible on discrete devices with
> +	 * a small BAR). Attempting to MMAP and fault such an object will
> +	 * require the kernel first synchronising any GPU work tied to the
> +	 * object, before then migrating the pages, either to the CPU accessible
> +	 * part of I915_MEMORY_CLASS_DEVICE, or I915_MEMORY_CLASS_SYSTEM, if the
> +	 * placements permit it. See I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
> +	 *
> +	 * Note that this is inherently racy.
> +	 */
> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
> +	__u64 attributes;
> +	/** @rsvd: MBZ */
> +	__u32 rsvd[4];
> +};
> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst b/Documentation/gpu/rfc/i915_small_bar.rst
> new file mode 100644
> index 000000000000..be3d9bcdd86d
> --- /dev/null
> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
> @@ -0,0 +1,58 @@
> +==========================
> +I915 Small BAR RFC Section
> +==========================
> +Starting from DG2 we will have resizable BAR support for device local-memory(i.e
> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size might still be
> +smaller than the total probed_size. In such cases, only some subset of
> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the first 256M),
> +while the remainder is only accessible via the GPU.
> +
> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
> +----------------------------------------------
> +New gem_create_ext flag to tell the kernel that a BO will require CPU access.
> +This becomes important when placing an object in I915_MEMORY_CLASS_DEVICE, where
> +underneath the device has a small BAR, meaning only some portion of it is CPU
> +accessible. Without this flag the kernel will assume that CPU access is not
> +required, and prioritize using the non-CPU visible portion of
> +I915_MEMORY_CLASS_DEVICE.
> +
> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> +   :functions: __drm_i915_gem_create_ext
> +
> +probed_cpu_visible_size attribute
> +---------------------------------
> +New struct__drm_i915_memory_region attribute which returns the total size of the
> +CPU accessible portion, for the particular region. This should only be
> +applicable for I915_MEMORY_CLASS_DEVICE.
> +
> +Vulkan will need this as part of creating a separate VkMemoryHeap with the
> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU visible portion,
> +where the total size of the heap needs to be known.
> +
> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> +   :functions: __drm_i915_memory_region_info
> +
> +DRM_I915_QUERY_VMA_INFO query
> +-----------------------------
> +Query the attributes of some vma. Given a vm and GTT offset, find the
> +respective vma, and return its set of attributes. For now we only support
> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
> +currently placed in memory that is accessible by the CPU. This should always be
> +set on devices where the CPU probed_cpu_visible_size of I915_MEMORY_CLASS_DEVICE
> +matches the probed_size. If this is not set then CPU faulting the object will
> +likely first require migrating the pages.
> +
> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> +   :functions: __drm_i915_query_vma_info
> +
> +Error Capture restrictions
> +--------------------------
> +With error capture we have two new restrictions:
> +
> +    1) Error capture is best effort on small BAR systems; if the pages are not
> +    CPU accessible, at the time of capture, then the kernel is free to skip
> +    trying to capture them.
> +
> +    2) On discrete we now reject error capture on recoverable contexts. In the
> +    future the kernel may want to blit during error capture, when for example
> +    something is not currently CPU accessible.
> diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
> index 91e93a705230..5a3bd3924ba6 100644
> --- a/Documentation/gpu/rfc/index.rst
> +++ b/Documentation/gpu/rfc/index.rst
> @@ -23,3 +23,7 @@ host such documentation:
>   .. toctree::
>   
>       i915_scheduler.rst
> +
> +.. toctree::
> +
> +    i915_small_bar.rst

Lionel Landwerlin April 27, 2022, 6:48 a.m. UTC | #2

One question though, how do we detect that this flag 
(I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS) is accepted on a given kernel?
I assume older kernels are going to reject object creation if we use 
this flag?

I didn't plan to use __drm_i915_query_vma_info, but isn't it 
inconsistent to select the placement on the GEM object and then query 
whether it's mappable by address?
You made a comment stating this is racy, wouldn't querying on the GEM 
object prevent this?

Thanks,

-Lionel

On 27/04/2022 09:35, Lionel Landwerlin wrote:
> Hi Matt,
>
>
> The proposal looks good to me.
>
> Looking forward to try it on drm-tip.
>
>
> -Lionel
>
> On 20/04/2022 20:13, Matthew Auld wrote:
>> Add an entry for the new uapi needed for small BAR on DG2+.
>>
>> v2:
>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>    - Rework error capture interactions, including no longer needing
>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>    - Add probed_cpu_visible_size. (Lionel)
>>
>> Signed-off-by: Matthew Auld <[e-mail]>
>> Cc: Thomas Hellström <[e-mail]>
>> Cc: Lionel Landwerlin <[e-mail]>
>> Cc: Jon Bloomfield <[e-mail]>
>> Cc: Daniel Vetter <[e-mail]>
>> Cc: Jordan Justen <[e-mail]>
>> Cc: Kenneth Graunke <[e-mail]>
>> Cc: Akeem G Abodunrin <[e-mail]>
>> Cc: [e-mail]
>> ---
>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>   Documentation/gpu/rfc/index.rst          |   4 +
>>   3 files changed, 252 insertions(+)
>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>
>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>> b/Documentation/gpu/rfc/i915_small_bar.h
>> new file mode 100644
>> index 000000000000..7bfd0cf44d35
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>> @@ -0,0 +1,190 @@
>> +/**
>> + * struct __drm_i915_memory_region_info - Describes one region as 
>> known to the
>> + * driver.
>> + *
>> + * Note this is using both struct drm_i915_query_item and struct 
>> drm_i915_query.
>> + * For this new query we are adding the new query id 
>> DRM_I915_QUERY_MEMORY_REGIONS
>> + * at &drm_i915_query_item.query_id.
>> + */
>> +struct __drm_i915_memory_region_info {
>> +    /** @region: The class:instance pair encoding */
>> +    struct drm_i915_gem_memory_class_instance region;
>> +
>> +    /** @rsvd0: MBZ */
>> +    __u32 rsvd0;
>> +
>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>> +    __u64 probed_size;
>> +
>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>> unknown) */
>> +    __u64 unallocated_size;
>> +
>> +    union {
>> +        /** @rsvd1: MBZ */
>> +        __u64 rsvd1[8];
>> +        struct {
>> +            /**
>> +             * @probed_cpu_visible_size: Memory probed by the driver
>> +             * that is CPU accessible. (-1 = unknown).
>> +             *
>> +             * This will be always be <= @probed_size, and the
>> +             * remainder(if there is any) will not be CPU
>> +             * accessible.
>> +             */
>> +            __u64 probed_cpu_visible_size;
>> +        };
>> +    };
>> +};
>> +
>> +/**
>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, 
>> with added
>> + * extension support using struct i915_user_extension.
>> + *
>> + * Note that new buffer flags should be added here, at least for the 
>> stuff that
>> + * is immutable. Previously we would have two ioctls, one to create 
>> the object
>> + * with gem_create, and another to apply various parameters, however 
>> this
>> + * creates some ambiguity for the params which are considered 
>> immutable. Also in
>> + * general we're phasing out the various SET/GET ioctls.
>> + */
>> +struct __drm_i915_gem_create_ext {
>> +    /**
>> +     * @size: Requested size for the object.
>> +     *
>> +     * The (page-aligned) allocated size for the object will be 
>> returned.
>> +     *
>> +     * Note that for some devices we have might have further minimum
>> +     * page-size restrictions(larger than 4K), like for device 
>> local-memory.
>> +     * However in general the final size here should always reflect any
>> +     * rounding up, if for example using the 
>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>> +     * extension to place the object in device local-memory.
>> +     */
>> +    __u64 size;
>> +    /**
>> +     * @handle: Returned handle for the object.
>> +     *
>> +     * Object handles are nonzero.
>> +     */
>> +    __u32 handle;
>> +    /**
>> +     * @flags: Optional flags.
>> +     *
>> +     * Supported values:
>> +     *
>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>> kernel that
>> +     * the object will need to be accessed via the CPU.
>> +     *
>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
>> +     * only strictly required on platforms where only some of the 
>> device
>> +     * memory is directly visible or mappable through the CPU, like 
>> on DG2+.
>> +     *
>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>> +     * ensure we can always spill the allocation to system memory, 
>> if we
>> +     * can't place the object in the mappable part of
>> +     * I915_MEMORY_CLASS_DEVICE.
>> +     *
>> +     * Note that since the kernel only supports flat-CCS on objects 
>> that can
>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>> +     * flat-CCS.
>> +     *
>> +     * Without this hint, the kernel will assume that non-mappable
>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>> that the
>> +     * kernel can still migrate the object to the mappable part, as 
>> a last
>> +     * resort, if userspace ever CPU faults this object, but this 
>> might be
>> +     * expensive, and so ideally should be avoided.
>> +     */
>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>> +    __u32 flags;
>> +    /**
>> +     * @extensions: The chain of extensions to apply to this object.
>> +     *
>> +     * This will be useful in the future when we need to support 
>> several
>> +     * different extensions, and we need to apply more than one when
>> +     * creating the object. See struct i915_user_extension.
>> +     *
>> +     * If we don't supply any extensions then we get the same old 
>> gem_create
>> +     * behaviour.
>> +     *
>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>> +     * struct drm_i915_gem_create_ext_memory_regions.
>> +     *
>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>> +     * struct drm_i915_gem_create_ext_protected_content.
>> +     */
>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>> +    __u64 extensions;
>> +};
>> +
>> +#define DRM_I915_QUERY_VMA_INFO    5
>> +
>> +/**
>> + * struct __drm_i915_query_vma_info
>> + *
>> + * Given a vm and GTT address, lookup the corresponding vma, 
>> returning its set
>> + * of attributes.
>> + *
>> + * .. code-block:: C
>> + *
>> + *    struct drm_i915_query_vma_info info = {};
>> + *    struct drm_i915_query_item item = {
>> + *        .data_ptr = (uintptr_t)&info,
>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>> + *    };
>> + *    struct drm_i915_query query = {
>> + *        .num_items = 1,
>> + *        .items_ptr = (uintptr_t)&item,
>> + *    };
>> + *    int err;
>> + *
>> + *    // Unlike some other types of queries, there is no need to 
>> first query
>> + *    // the size of the data_ptr blob here, since we already know 
>> ahead of
>> + *    // time how big this needs to be.
>> + *    item.length = sizeof(info);
>> + *
>> + *    // Next we fill in the vm_id and ppGTT address of the vma we wish
>> + *    // to query, before then firing off the query.
>> + *    info.vm_id = vm_id;
>> + *    info.offset = gtt_address;
>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>> + *    if (err || item.length < 0) ...
>> + *
>> + *    // If all went well we can now inspect the returned attributes.
>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>> + */
>> +struct __drm_i915_query_vma_info {
>> +    /**
>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>> value
>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>> +     * drm_i915_gem_vm_control.vm_id.
>> +     */
>> +    __u32 vm_id;
>> +    /** @pad: MBZ. */
>> +    __u32 pad;
>> +    /**
>> +     * @offset: The corresponding ppGTT address of the vma which the 
>> kernel
>> +     * will use to perform the lookup.
>> +     */
>> +    __u64 offset;
>> +    /**
>> +     * @attributes: The returned attributes for the given vma.
>> +     *
>> +     * Possible values:
>> +     *
>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>> backing the
>> +     * vma are currently CPU accessible. If this is not set then the 
>> vma is
>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>> the CPU
>> +     * cannot directly access(this is only possible on discrete 
>> devices with
>> +     * a small BAR). Attempting to MMAP and fault such an object will
>> +     * require the kernel first synchronising any GPU work tied to the
>> +     * object, before then migrating the pages, either to the CPU 
>> accessible
>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>> I915_MEMORY_CLASS_SYSTEM, if the
>> +     * placements permit it. See 
>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>> +     *
>> +     * Note that this is inherently racy.
>> +     */
>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>> +    __u64 attributes;
>> +    /** @rsvd: MBZ */
>> +    __u32 rsvd[4];
>> +};
>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>> b/Documentation/gpu/rfc/i915_small_bar.rst
>> new file mode 100644
>> index 000000000000..be3d9bcdd86d
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>> @@ -0,0 +1,58 @@
>> +==========================
>> +I915 Small BAR RFC Section
>> +==========================
>> +Starting from DG2 we will have resizable BAR support for device 
>> local-memory(i.e
>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>> might still be
>> +smaller than the total probed_size. In such cases, only some subset of
>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>> first 256M),
>> +while the remainder is only accessible via the GPU.
>> +
>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>> +----------------------------------------------
>> +New gem_create_ext flag to tell the kernel that a BO will require 
>> CPU access.
>> +This becomes important when placing an object in 
>> I915_MEMORY_CLASS_DEVICE, where
>> +underneath the device has a small BAR, meaning only some portion of 
>> it is CPU
>> +accessible. Without this flag the kernel will assume that CPU access 
>> is not
>> +required, and prioritize using the non-CPU visible portion of
>> +I915_MEMORY_CLASS_DEVICE.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_gem_create_ext
>> +
>> +probed_cpu_visible_size attribute
>> +---------------------------------
>> +New struct__drm_i915_memory_region attribute which returns the total 
>> size of the
>> +CPU accessible portion, for the particular region. This should only be
>> +applicable for I915_MEMORY_CLASS_DEVICE.
>> +
>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>> with the
>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>> visible portion,
>> +where the total size of the heap needs to be known.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_memory_region_info
>> +
>> +DRM_I915_QUERY_VMA_INFO query
>> +-----------------------------
>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>> +respective vma, and return its set of attributes. For now we only 
>> support
>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>> +currently placed in memory that is accessible by the CPU. This 
>> should always be
>> +set on devices where the CPU probed_cpu_visible_size of 
>> I915_MEMORY_CLASS_DEVICE
>> +matches the probed_size. If this is not set then CPU faulting the 
>> object will
>> +likely first require migrating the pages.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_query_vma_info
>> +
>> +Error Capture restrictions
>> +--------------------------
>> +With error capture we have two new restrictions:
>> +
>> +    1) Error capture is best effort on small BAR systems; if the 
>> pages are not
>> +    CPU accessible, at the time of capture, then the kernel is free 
>> to skip
>> +    trying to capture them.
>> +
>> +    2) On discrete we now reject error capture on recoverable 
>> contexts. In the
>> +    future the kernel may want to blit during error capture, when 
>> for example
>> +    something is not currently CPU accessible.
>> diff --git a/Documentation/gpu/rfc/index.rst 
>> b/Documentation/gpu/rfc/index.rst
>> index 91e93a705230..5a3bd3924ba6 100644
>> --- a/Documentation/gpu/rfc/index.rst
>> +++ b/Documentation/gpu/rfc/index.rst
>> @@ -23,3 +23,7 @@ host such documentation:
>>   .. toctree::
>>         i915_scheduler.rst
>> +
>> +.. toctree::
>> +
>> +    i915_small_bar.rst
>
>

Christian König April 27, 2022, 6:55 a.m. UTC | #3

Well usually we increment the drm minor version when adding some new 
flags on amdgpu.

Additional to that just one comment from our experience with that: You 
don't just need one flag, but two. The first one is a hint which says 
"CPU access needed" and the second is a promise which says "CPU access 
never needed".

The background is that on a whole bunch of buffers you can 100% certain 
say that you will never ever need CPU access.

Then at least we have a whole bunch of buffers where we might need CPU 
access, but can't tell for sure.

And last we have stuff like transfer buffers you can be 100% sure that 
you need CPU access.

Separating it like this helped a lot with performance on small BAR systems.

Regards,
Christian.

Am 27.04.22 um 08:48 schrieb Lionel Landwerlin:
> One question though, how do we detect that this flag 
> (I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS) is accepted on a given 
> kernel?
> I assume older kernels are going to reject object creation if we use 
> this flag?
>
> I didn't plan to use __drm_i915_query_vma_info, but isn't it 
> inconsistent to select the placement on the GEM object and then query 
> whether it's mappable by address?
> You made a comment stating this is racy, wouldn't querying on the GEM 
> object prevent this?
>
> Thanks,
>
> -Lionel
>
> On 27/04/2022 09:35, Lionel Landwerlin wrote:
>> Hi Matt,
>>
>>
>> The proposal looks good to me.
>>
>> Looking forward to try it on drm-tip.
>>
>>
>> -Lionel
>>
>> On 20/04/2022 20:13, Matthew Auld wrote:
>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>
>>> v2:
>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>    - Rework error capture interactions, including no longer needing
>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>    - Add probed_cpu_visible_size. (Lionel)
>>>
>>> Signed-off-by: Matthew Auld <[e-mail]>
>>> Cc: Thomas Hellström <[e-mail]>
>>> Cc: Lionel Landwerlin <[e-mail]>
>>> Cc: Jon Bloomfield <[e-mail]>
>>> Cc: Daniel Vetter <[e-mail]>
>>> Cc: Jordan Justen <[e-mail]>
>>> Cc: Kenneth Graunke <[e-mail]>
>>> Cc: Akeem G Abodunrin <[e-mail]>
>>> Cc: [e-mail]
>>> ---
>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>> +++++++++++++++++++++++
>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>   3 files changed, 252 insertions(+)
>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>> new file mode 100644
>>> index 000000000000..7bfd0cf44d35
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>> @@ -0,0 +1,190 @@
>>> +/**
>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>> known to the
>>> + * driver.
>>> + *
>>> + * Note this is using both struct drm_i915_query_item and struct 
>>> drm_i915_query.
>>> + * For this new query we are adding the new query id 
>>> DRM_I915_QUERY_MEMORY_REGIONS
>>> + * at &drm_i915_query_item.query_id.
>>> + */
>>> +struct __drm_i915_memory_region_info {
>>> +    /** @region: The class:instance pair encoding */
>>> +    struct drm_i915_gem_memory_class_instance region;
>>> +
>>> +    /** @rsvd0: MBZ */
>>> +    __u32 rsvd0;
>>> +
>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>> +    __u64 probed_size;
>>> +
>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>> unknown) */
>>> +    __u64 unallocated_size;
>>> +
>>> +    union {
>>> +        /** @rsvd1: MBZ */
>>> +        __u64 rsvd1[8];
>>> +        struct {
>>> +            /**
>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>> +             * that is CPU accessible. (-1 = unknown).
>>> +             *
>>> +             * This will be always be <= @probed_size, and the
>>> +             * remainder(if there is any) will not be CPU
>>> +             * accessible.
>>> +             */
>>> +            __u64 probed_cpu_visible_size;
>>> +        };
>>> +    };
>>> +};
>>> +
>>> +/**
>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>> behaviour, with added
>>> + * extension support using struct i915_user_extension.
>>> + *
>>> + * Note that new buffer flags should be added here, at least for 
>>> the stuff that
>>> + * is immutable. Previously we would have two ioctls, one to create 
>>> the object
>>> + * with gem_create, and another to apply various parameters, 
>>> however this
>>> + * creates some ambiguity for the params which are considered 
>>> immutable. Also in
>>> + * general we're phasing out the various SET/GET ioctls.
>>> + */
>>> +struct __drm_i915_gem_create_ext {
>>> +    /**
>>> +     * @size: Requested size for the object.
>>> +     *
>>> +     * The (page-aligned) allocated size for the object will be 
>>> returned.
>>> +     *
>>> +     * Note that for some devices we have might have further minimum
>>> +     * page-size restrictions(larger than 4K), like for device 
>>> local-memory.
>>> +     * However in general the final size here should always reflect 
>>> any
>>> +     * rounding up, if for example using the 
>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>> +     * extension to place the object in device local-memory.
>>> +     */
>>> +    __u64 size;
>>> +    /**
>>> +     * @handle: Returned handle for the object.
>>> +     *
>>> +     * Object handles are nonzero.
>>> +     */
>>> +    __u32 handle;
>>> +    /**
>>> +     * @flags: Optional flags.
>>> +     *
>>> +     * Supported values:
>>> +     *
>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>> kernel that
>>> +     * the object will need to be accessed via the CPU.
>>> +     *
>>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, 
>>> and
>>> +     * only strictly required on platforms where only some of the 
>>> device
>>> +     * memory is directly visible or mappable through the CPU, like 
>>> on DG2+.
>>> +     *
>>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>>> +     * ensure we can always spill the allocation to system memory, 
>>> if we
>>> +     * can't place the object in the mappable part of
>>> +     * I915_MEMORY_CLASS_DEVICE.
>>> +     *
>>> +     * Note that since the kernel only supports flat-CCS on objects 
>>> that can
>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>> don't
>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>>> +     * flat-CCS.
>>> +     *
>>> +     * Without this hint, the kernel will assume that non-mappable
>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>> that the
>>> +     * kernel can still migrate the object to the mappable part, as 
>>> a last
>>> +     * resort, if userspace ever CPU faults this object, but this 
>>> might be
>>> +     * expensive, and so ideally should be avoided.
>>> +     */
>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>> +    __u32 flags;
>>> +    /**
>>> +     * @extensions: The chain of extensions to apply to this object.
>>> +     *
>>> +     * This will be useful in the future when we need to support 
>>> several
>>> +     * different extensions, and we need to apply more than one when
>>> +     * creating the object. See struct i915_user_extension.
>>> +     *
>>> +     * If we don't supply any extensions then we get the same old 
>>> gem_create
>>> +     * behaviour.
>>> +     *
>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>> +     *
>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>> +     */
>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>> +    __u64 extensions;
>>> +};
>>> +
>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>> +
>>> +/**
>>> + * struct __drm_i915_query_vma_info
>>> + *
>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>> returning its set
>>> + * of attributes.
>>> + *
>>> + * .. code-block:: C
>>> + *
>>> + *    struct drm_i915_query_vma_info info = {};
>>> + *    struct drm_i915_query_item item = {
>>> + *        .data_ptr = (uintptr_t)&info,
>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>> + *    };
>>> + *    struct drm_i915_query query = {
>>> + *        .num_items = 1,
>>> + *        .items_ptr = (uintptr_t)&item,
>>> + *    };
>>> + *    int err;
>>> + *
>>> + *    // Unlike some other types of queries, there is no need to 
>>> first query
>>> + *    // the size of the data_ptr blob here, since we already know 
>>> ahead of
>>> + *    // time how big this needs to be.
>>> + *    item.length = sizeof(info);
>>> + *
>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we 
>>> wish
>>> + *    // to query, before then firing off the query.
>>> + *    info.vm_id = vm_id;
>>> + *    info.offset = gtt_address;
>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>> + *    if (err || item.length < 0) ...
>>> + *
>>> + *    // If all went well we can now inspect the returned attributes.
>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>> + */
>>> +struct __drm_i915_query_vma_info {
>>> +    /**
>>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>>> value
>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>> +     * drm_i915_gem_vm_control.vm_id.
>>> +     */
>>> +    __u32 vm_id;
>>> +    /** @pad: MBZ. */
>>> +    __u32 pad;
>>> +    /**
>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>> the kernel
>>> +     * will use to perform the lookup.
>>> +     */
>>> +    __u64 offset;
>>> +    /**
>>> +     * @attributes: The returned attributes for the given vma.
>>> +     *
>>> +     * Possible values:
>>> +     *
>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>> backing the
>>> +     * vma are currently CPU accessible. If this is not set then 
>>> the vma is
>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>>> the CPU
>>> +     * cannot directly access(this is only possible on discrete 
>>> devices with
>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>> +     * require the kernel first synchronising any GPU work tied to the
>>> +     * object, before then migrating the pages, either to the CPU 
>>> accessible
>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>> I915_MEMORY_CLASS_SYSTEM, if the
>>> +     * placements permit it. See 
>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>> +     *
>>> +     * Note that this is inherently racy.
>>> +     */
>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>> +    __u64 attributes;
>>> +    /** @rsvd: MBZ */
>>> +    __u32 rsvd[4];
>>> +};
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>> new file mode 100644
>>> index 000000000000..be3d9bcdd86d
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>> @@ -0,0 +1,58 @@
>>> +==========================
>>> +I915 Small BAR RFC Section
>>> +==========================
>>> +Starting from DG2 we will have resizable BAR support for device 
>>> local-memory(i.e
>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>> might still be
>>> +smaller than the total probed_size. In such cases, only some subset of
>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>> first 256M),
>>> +while the remainder is only accessible via the GPU.
>>> +
>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>> +----------------------------------------------
>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>> CPU access.
>>> +This becomes important when placing an object in 
>>> I915_MEMORY_CLASS_DEVICE, where
>>> +underneath the device has a small BAR, meaning only some portion of 
>>> it is CPU
>>> +accessible. Without this flag the kernel will assume that CPU 
>>> access is not
>>> +required, and prioritize using the non-CPU visible portion of
>>> +I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_gem_create_ext
>>> +
>>> +probed_cpu_visible_size attribute
>>> +---------------------------------
>>> +New struct__drm_i915_memory_region attribute which returns the 
>>> total size of the
>>> +CPU accessible portion, for the particular region. This should only be
>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>> with the
>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>> visible portion,
>>> +where the total size of the heap needs to be known.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_memory_region_info
>>> +
>>> +DRM_I915_QUERY_VMA_INFO query
>>> +-----------------------------
>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>> +respective vma, and return its set of attributes. For now we only 
>>> support
>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>>> +currently placed in memory that is accessible by the CPU. This 
>>> should always be
>>> +set on devices where the CPU probed_cpu_visible_size of 
>>> I915_MEMORY_CLASS_DEVICE
>>> +matches the probed_size. If this is not set then CPU faulting the 
>>> object will
>>> +likely first require migrating the pages.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_query_vma_info
>>> +
>>> +Error Capture restrictions
>>> +--------------------------
>>> +With error capture we have two new restrictions:
>>> +
>>> +    1) Error capture is best effort on small BAR systems; if the 
>>> pages are not
>>> +    CPU accessible, at the time of capture, then the kernel is free 
>>> to skip
>>> +    trying to capture them.
>>> +
>>> +    2) On discrete we now reject error capture on recoverable 
>>> contexts. In the
>>> +    future the kernel may want to blit during error capture, when 
>>> for example
>>> +    something is not currently CPU accessible.
>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>> b/Documentation/gpu/rfc/index.rst
>>> index 91e93a705230..5a3bd3924ba6 100644
>>> --- a/Documentation/gpu/rfc/index.rst
>>> +++ b/Documentation/gpu/rfc/index.rst
>>> @@ -23,3 +23,7 @@ host such documentation:
>>>   .. toctree::
>>>         i915_scheduler.rst
>>> +
>>> +.. toctree::
>>> +
>>> +    i915_small_bar.rst
>>
>>
>

Tvrtko Ursulin April 27, 2022, 8:36 a.m. UTC | #4

On 20/04/2022 18:13, Matthew Auld wrote:
> Add an entry for the new uapi needed for small BAR on DG2+.
> 
> v2:
>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>    - Rework error capture interactions, including no longer needing
>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>    - Add probed_cpu_visible_size. (Lionel)
> 
> Signed-off-by: Matthew Auld <[e-mail]>
> Cc: Thomas Hellström <[e-mail]>
> Cc: Lionel Landwerlin <[e-mail]>
> Cc: Jon Bloomfield <[e-mail]>
> Cc: Daniel Vetter <[e-mail]>
> Cc: Jordan Justen <[e-mail]>
> Cc: Kenneth Graunke <[e-mail]>
> Cc: Akeem G Abodunrin <[e-mail]>
> Cc: [e-mail]
> ---
>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>   Documentation/gpu/rfc/index.rst          |   4 +
>   3 files changed, 252 insertions(+)
>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
> 
> diff --git a/Documentation/gpu/rfc/i915_small_bar.h b/Documentation/gpu/rfc/i915_small_bar.h
> new file mode 100644
> index 000000000000..7bfd0cf44d35
> --- /dev/null
> +++ b/Documentation/gpu/rfc/i915_small_bar.h
> @@ -0,0 +1,190 @@
> +/**
> + * struct __drm_i915_memory_region_info - Describes one region as known to the
> + * driver.
> + *
> + * Note this is using both struct drm_i915_query_item and struct drm_i915_query.
> + * For this new query we are adding the new query id DRM_I915_QUERY_MEMORY_REGIONS
> + * at &drm_i915_query_item.query_id.
> + */
> +struct __drm_i915_memory_region_info {
> +	/** @region: The class:instance pair encoding */
> +	struct drm_i915_gem_memory_class_instance region;
> +
> +	/** @rsvd0: MBZ */
> +	__u32 rsvd0;
> +
> +	/** @probed_size: Memory probed by the driver (-1 = unknown) */
> +	__u64 probed_size;
> +
> +	/** @unallocated_size: Estimate of memory remaining (-1 = unknown) */
> +	__u64 unallocated_size;
> +
> +	union {
> +		/** @rsvd1: MBZ */
> +		__u64 rsvd1[8];
> +		struct {
> +			/**
> +			 * @probed_cpu_visible_size: Memory probed by the driver
> +			 * that is CPU accessible. (-1 = unknown).
> +			 *
> +			 * This will be always be <= @probed_size, and the
> +			 * remainder(if there is any) will not be CPU
> +			 * accessible.
> +			 */
> +			__u64 probed_cpu_visible_size;

Would unallocated_cpu_visible_size be useful, to follow the total 
unallocated_size?

Btw, have we ever considered whether unallocated_size should require 
CAP_SYS_ADMIN/PERFMON or something?

> +		};
> +	};
> +};
> +
> +/**
> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, with added
> + * extension support using struct i915_user_extension.
> + *
> + * Note that new buffer flags should be added here, at least for the stuff that
> + * is immutable. Previously we would have two ioctls, one to create the object
> + * with gem_create, and another to apply various parameters, however this
> + * creates some ambiguity for the params which are considered immutable. Also in
> + * general we're phasing out the various SET/GET ioctls.
> + */
> +struct __drm_i915_gem_create_ext {
> +	/**
> +	 * @size: Requested size for the object.
> +	 *
> +	 * The (page-aligned) allocated size for the object will be returned.
> +	 *
> +	 * Note that for some devices we have might have further minimum
> +	 * page-size restrictions(larger than 4K), like for device local-memory.
> +	 * However in general the final size here should always reflect any
> +	 * rounding up, if for example using the I915_GEM_CREATE_EXT_MEMORY_REGIONS
> +	 * extension to place the object in device local-memory.
> +	 */
> +	__u64 size;
> +	/**
> +	 * @handle: Returned handle for the object.
> +	 *
> +	 * Object handles are nonzero.
> +	 */
> +	__u32 handle;
> +	/**
> +	 * @flags: Optional flags.
> +	 *
> +	 * Supported values:
> +	 *
> +	 * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the kernel that
> +	 * the object will need to be accessed via the CPU.
> +	 *
> +	 * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
> +	 * only strictly required on platforms where only some of the device
> +	 * memory is directly visible or mappable through the CPU, like on DG2+.
> +	 *
> +	 * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
> +	 * ensure we can always spill the allocation to system memory, if we
> +	 * can't place the object in the mappable part of
> +	 * I915_MEMORY_CLASS_DEVICE.
> +	 *
> +	 * Note that since the kernel only supports flat-CCS on objects that can
> +	 * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
> +	 * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
> +	 * flat-CCS.
> +	 *
> +	 * Without this hint, the kernel will assume that non-mappable
> +	 * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note that the
> +	 * kernel can still migrate the object to the mappable part, as a last
> +	 * resort, if userspace ever CPU faults this object, but this might be
> +	 * expensive, and so ideally should be avoided.
> +	 */

So "needs_cpu_access" flag could almost be viewed as a sub-region 
placement priority? What I mean is this:

1)
placements=device,system flags=

This results in placement priorities: device, device_cpu_mappable, system.

2)
placements=device,system flags=needs_cpu_access

This results in placement priorities: device_cpu_mappable, device, system.

Is this correct?

The benefit of the flag is that i915 can place the object to the right 
place from the start instead of on the first CPU access? Is that worth 
it or is there more to it?

> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
> +	__u32 flags;
> +	/**
> +	 * @extensions: The chain of extensions to apply to this object.
> +	 *
> +	 * This will be useful in the future when we need to support several
> +	 * different extensions, and we need to apply more than one when
> +	 * creating the object. See struct i915_user_extension.
> +	 *
> +	 * If we don't supply any extensions then we get the same old gem_create
> +	 * behaviour.
> +	 *
> +	 * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
> +	 * struct drm_i915_gem_create_ext_memory_regions.
> +	 *
> +	 * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
> +	 * struct drm_i915_gem_create_ext_protected_content.
> +	 */
> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
> +	__u64 extensions;
> +};
> +
> +#define DRM_I915_QUERY_VMA_INFO	5
> +
> +/**
> + * struct __drm_i915_query_vma_info
> + *
> + * Given a vm and GTT address, lookup the corresponding vma, returning its set
> + * of attributes.
> + *
> + * .. code-block:: C
> + *
> + *	struct drm_i915_query_vma_info info = {};
> + *	struct drm_i915_query_item item = {
> + *		.data_ptr = (uintptr_t)&info,
> + *		.query_id = DRM_I915_QUERY_VMA_INFO,
> + *	};
> + *	struct drm_i915_query query = {
> + *		.num_items = 1,
> + *		.items_ptr = (uintptr_t)&item,
> + *	};
> + *	int err;
> + *
> + *	// Unlike some other types of queries, there is no need to first query
> + *	// the size of the data_ptr blob here, since we already know ahead of
> + *	// time how big this needs to be.
> + *	item.length = sizeof(info);
> + *
> + *	// Next we fill in the vm_id and ppGTT address of the vma we wish
> + *	// to query, before then firing off the query.
> + *	info.vm_id = vm_id;
> + *	info.offset = gtt_address;
> + *	err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
> + *	if (err || item.length < 0) ...
> + *
> + *	// If all went well we can now inspect the returned attributes.
> + *	if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
> + */
> +struct __drm_i915_query_vma_info {
> +	/**
> +	 * @vm_id: The given vm id that contains the vma. The id is the value
> +	 * returned by the DRM_I915_GEM_VM_CREATE. See struct
> +	 * drm_i915_gem_vm_control.vm_id.
> +	 */
> +	__u32 vm_id;
> +	/** @pad: MBZ. */
> +	__u32 pad;
> +	/**
> +	 * @offset: The corresponding ppGTT address of the vma which the kernel
> +	 * will use to perform the lookup.
> +	 */
> +	__u64 offset;
> +	/**
> +	 * @attributes: The returned attributes for the given vma.
> +	 *
> +	 * Possible values:
> +	 *
> +	 * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages backing the
> +	 * vma are currently CPU accessible. If this is not set then the vma is
> +	 * currently backed by I915_MEMORY_CLASS_DEVICE memory, which the CPU
> +	 * cannot directly access(this is only possible on discrete devices with
> +	 * a small BAR). Attempting to MMAP and fault such an object will
> +	 * require the kernel first synchronising any GPU work tied to the
> +	 * object, before then migrating the pages, either to the CPU accessible
> +	 * part of I915_MEMORY_CLASS_DEVICE, or I915_MEMORY_CLASS_SYSTEM, if the
> +	 * placements permit it. See I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
> +	 *
> +	 * Note that this is inherently racy.
> +	 */
> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
> +	__u64 attributes;
> +	/** @rsvd: MBZ */
> +	__u32 rsvd[4];
> +};
> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst b/Documentation/gpu/rfc/i915_small_bar.rst
> new file mode 100644
> index 000000000000..be3d9bcdd86d
> --- /dev/null
> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
> @@ -0,0 +1,58 @@
> +==========================
> +I915 Small BAR RFC Section
> +==========================
> +Starting from DG2 we will have resizable BAR support for device local-memory(i.e
> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size might still be
> +smaller than the total probed_size. In such cases, only some subset of
> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the first 256M),
> +while the remainder is only accessible via the GPU.
> +
> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
> +----------------------------------------------
> +New gem_create_ext flag to tell the kernel that a BO will require CPU access.
> +This becomes important when placing an object in I915_MEMORY_CLASS_DEVICE, where
> +underneath the device has a small BAR, meaning only some portion of it is CPU
> +accessible. Without this flag the kernel will assume that CPU access is not
> +required, and prioritize using the non-CPU visible portion of
> +I915_MEMORY_CLASS_DEVICE.
> +
> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> +   :functions: __drm_i915_gem_create_ext
> +
> +probed_cpu_visible_size attribute
> +---------------------------------
> +New struct__drm_i915_memory_region attribute which returns the total size of the
> +CPU accessible portion, for the particular region. This should only be
> +applicable for I915_MEMORY_CLASS_DEVICE.
> +
> +Vulkan will need this as part of creating a separate VkMemoryHeap with the
> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU visible portion,
> +where the total size of the heap needs to be known.
> +
> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> +   :functions: __drm_i915_memory_region_info
> +
> +DRM_I915_QUERY_VMA_INFO query
> +-----------------------------
> +Query the attributes of some vma. Given a vm and GTT offset, find the
> +respective vma, and return its set of attributes. For now we only support
> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
> +currently placed in memory that is accessible by the CPU. This should always be
> +set on devices where the CPU probed_cpu_visible_size of I915_MEMORY_CLASS_DEVICE
> +matches the probed_size. If this is not set then CPU faulting the object will
> +likely first require migrating the pages.

I think there should be justification for the new query documented as 
well. (Why on top of what.)

Without it personally I can't immediately understand why the disconnect 
between the object based and VMA based API. Userspace has to do some 
intervening operations like either execbuf, or vm bind in the future, to 
make this query usable after object creation. So question is why 
wouldn't it know already which placements it allowed and so would i915 
auto-migrate or not for this particular object. No? Or in other words 
why this wouldn't be an object based query since the question it is 
answering is about the object backing store and not the VMA.

Regards,

Tvrtko

> +
> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> +   :functions: __drm_i915_query_vma_info
> +
> +Error Capture restrictions
> +--------------------------
> +With error capture we have two new restrictions:
> +
> +    1) Error capture is best effort on small BAR systems; if the pages are not
> +    CPU accessible, at the time of capture, then the kernel is free to skip
> +    trying to capture them.
> +
> +    2) On discrete we now reject error capture on recoverable contexts. In the
> +    future the kernel may want to blit during error capture, when for example
> +    something is not currently CPU accessible.
> diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
> index 91e93a705230..5a3bd3924ba6 100644
> --- a/Documentation/gpu/rfc/index.rst
> +++ b/Documentation/gpu/rfc/index.rst
> @@ -23,3 +23,7 @@ host such documentation:
>   .. toctree::
>   
>       i915_scheduler.rst
> +
> +.. toctree::
> +
> +    i915_small_bar.rst

Matthew Auld April 27, 2022, 3:02 p.m. UTC | #5

On 27/04/2022 07:55, Christian König wrote:
> Well usually we increment the drm minor version when adding some new 
> flags on amdgpu.
> 
> Additional to that just one comment from our experience with that: You 
> don't just need one flag, but two. The first one is a hint which says 
> "CPU access needed" and the second is a promise which says "CPU access 
> never needed".
> 
> The background is that on a whole bunch of buffers you can 100% certain 
> say that you will never ever need CPU access.
> 
> Then at least we have a whole bunch of buffers where we might need CPU 
> access, but can't tell for sure.
> 
> And last we have stuff like transfer buffers you can be 100% sure that 
> you need CPU access.
> 
> Separating it like this helped a lot with performance on small BAR systems.

Thanks for the comments. For the "CPU access never needed" flag, what 
extra stuff does that do on the kernel side vs not specifying any 
flag/hint? I assume it still prioritizes using the non-CPU visible 
portion first? What else does it do?

> 
> Regards,
> Christian.
> 
> Am 27.04.22 um 08:48 schrieb Lionel Landwerlin:
>> One question though, how do we detect that this flag 
>> (I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS) is accepted on a given 
>> kernel?
>> I assume older kernels are going to reject object creation if we use 
>> this flag?
>>
>> I didn't plan to use __drm_i915_query_vma_info, but isn't it 
>> inconsistent to select the placement on the GEM object and then query 
>> whether it's mappable by address?
>> You made a comment stating this is racy, wouldn't querying on the GEM 
>> object prevent this?
>>
>> Thanks,
>>
>> -Lionel
>>
>> On 27/04/2022 09:35, Lionel Landwerlin wrote:
>>> Hi Matt,
>>>
>>>
>>> The proposal looks good to me.
>>>
>>> Looking forward to try it on drm-tip.
>>>
>>>
>>> -Lionel
>>>
>>> On 20/04/2022 20:13, Matthew Auld wrote:
>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>
>>>> v2:
>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>    - Rework error capture interactions, including no longer needing
>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>
>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>> Cc: Thomas Hellström <[e-mail]>
>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>> Cc: Jon Bloomfield <[e-mail]>
>>>> Cc: Daniel Vetter <[e-mail]>
>>>> Cc: Jordan Justen <[e-mail]>
>>>> Cc: Kenneth Graunke <[e-mail]>
>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>> Cc: [e-mail]
>>>> ---
>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>> +++++++++++++++++++++++
>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>   3 files changed, 252 insertions(+)
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>> new file mode 100644
>>>> index 000000000000..7bfd0cf44d35
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>> @@ -0,0 +1,190 @@
>>>> +/**
>>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>>> known to the
>>>> + * driver.
>>>> + *
>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>> drm_i915_query.
>>>> + * For this new query we are adding the new query id 
>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>> + * at &drm_i915_query_item.query_id.
>>>> + */
>>>> +struct __drm_i915_memory_region_info {
>>>> +    /** @region: The class:instance pair encoding */
>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>> +
>>>> +    /** @rsvd0: MBZ */
>>>> +    __u32 rsvd0;
>>>> +
>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>> +    __u64 probed_size;
>>>> +
>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>> unknown) */
>>>> +    __u64 unallocated_size;
>>>> +
>>>> +    union {
>>>> +        /** @rsvd1: MBZ */
>>>> +        __u64 rsvd1[8];
>>>> +        struct {
>>>> +            /**
>>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>>> +             * that is CPU accessible. (-1 = unknown).
>>>> +             *
>>>> +             * This will be always be <= @probed_size, and the
>>>> +             * remainder(if there is any) will not be CPU
>>>> +             * accessible.
>>>> +             */
>>>> +            __u64 probed_cpu_visible_size;
>>>> +        };
>>>> +    };
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>> behaviour, with added
>>>> + * extension support using struct i915_user_extension.
>>>> + *
>>>> + * Note that new buffer flags should be added here, at least for 
>>>> the stuff that
>>>> + * is immutable. Previously we would have two ioctls, one to create 
>>>> the object
>>>> + * with gem_create, and another to apply various parameters, 
>>>> however this
>>>> + * creates some ambiguity for the params which are considered 
>>>> immutable. Also in
>>>> + * general we're phasing out the various SET/GET ioctls.
>>>> + */
>>>> +struct __drm_i915_gem_create_ext {
>>>> +    /**
>>>> +     * @size: Requested size for the object.
>>>> +     *
>>>> +     * The (page-aligned) allocated size for the object will be 
>>>> returned.
>>>> +     *
>>>> +     * Note that for some devices we have might have further minimum
>>>> +     * page-size restrictions(larger than 4K), like for device 
>>>> local-memory.
>>>> +     * However in general the final size here should always reflect 
>>>> any
>>>> +     * rounding up, if for example using the 
>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>> +     * extension to place the object in device local-memory.
>>>> +     */
>>>> +    __u64 size;
>>>> +    /**
>>>> +     * @handle: Returned handle for the object.
>>>> +     *
>>>> +     * Object handles are nonzero.
>>>> +     */
>>>> +    __u32 handle;
>>>> +    /**
>>>> +     * @flags: Optional flags.
>>>> +     *
>>>> +     * Supported values:
>>>> +     *
>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>> kernel that
>>>> +     * the object will need to be accessed via the CPU.
>>>> +     *
>>>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, 
>>>> and
>>>> +     * only strictly required on platforms where only some of the 
>>>> device
>>>> +     * memory is directly visible or mappable through the CPU, like 
>>>> on DG2+.
>>>> +     *
>>>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>>>> +     * ensure we can always spill the allocation to system memory, 
>>>> if we
>>>> +     * can't place the object in the mappable part of
>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>> +     *
>>>> +     * Note that since the kernel only supports flat-CCS on objects 
>>>> that can
>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>>> don't
>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>>>> +     * flat-CCS.
>>>> +     *
>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>>> that the
>>>> +     * kernel can still migrate the object to the mappable part, as 
>>>> a last
>>>> +     * resort, if userspace ever CPU faults this object, but this 
>>>> might be
>>>> +     * expensive, and so ideally should be avoided.
>>>> +     */
>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>> +    __u32 flags;
>>>> +    /**
>>>> +     * @extensions: The chain of extensions to apply to this object.
>>>> +     *
>>>> +     * This will be useful in the future when we need to support 
>>>> several
>>>> +     * different extensions, and we need to apply more than one when
>>>> +     * creating the object. See struct i915_user_extension.
>>>> +     *
>>>> +     * If we don't supply any extensions then we get the same old 
>>>> gem_create
>>>> +     * behaviour.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>> +     */
>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>> +    __u64 extensions;
>>>> +};
>>>> +
>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_query_vma_info
>>>> + *
>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>> returning its set
>>>> + * of attributes.
>>>> + *
>>>> + * .. code-block:: C
>>>> + *
>>>> + *    struct drm_i915_query_vma_info info = {};
>>>> + *    struct drm_i915_query_item item = {
>>>> + *        .data_ptr = (uintptr_t)&info,
>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>> + *    };
>>>> + *    struct drm_i915_query query = {
>>>> + *        .num_items = 1,
>>>> + *        .items_ptr = (uintptr_t)&item,
>>>> + *    };
>>>> + *    int err;
>>>> + *
>>>> + *    // Unlike some other types of queries, there is no need to 
>>>> first query
>>>> + *    // the size of the data_ptr blob here, since we already know 
>>>> ahead of
>>>> + *    // time how big this needs to be.
>>>> + *    item.length = sizeof(info);
>>>> + *
>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we 
>>>> wish
>>>> + *    // to query, before then firing off the query.
>>>> + *    info.vm_id = vm_id;
>>>> + *    info.offset = gtt_address;
>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>> + *    if (err || item.length < 0) ...
>>>> + *
>>>> + *    // If all went well we can now inspect the returned attributes.
>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>> + */
>>>> +struct __drm_i915_query_vma_info {
>>>> +    /**
>>>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>>>> value
>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>> +     */
>>>> +    __u32 vm_id;
>>>> +    /** @pad: MBZ. */
>>>> +    __u32 pad;
>>>> +    /**
>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>> the kernel
>>>> +     * will use to perform the lookup.
>>>> +     */
>>>> +    __u64 offset;
>>>> +    /**
>>>> +     * @attributes: The returned attributes for the given vma.
>>>> +     *
>>>> +     * Possible values:
>>>> +     *
>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>>> backing the
>>>> +     * vma are currently CPU accessible. If this is not set then 
>>>> the vma is
>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>>>> the CPU
>>>> +     * cannot directly access(this is only possible on discrete 
>>>> devices with
>>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>>> +     * require the kernel first synchronising any GPU work tied to the
>>>> +     * object, before then migrating the pages, either to the CPU 
>>>> accessible
>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>> +     * placements permit it. See 
>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>> +     *
>>>> +     * Note that this is inherently racy.
>>>> +     */
>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>> +    __u64 attributes;
>>>> +    /** @rsvd: MBZ */
>>>> +    __u32 rsvd[4];
>>>> +};
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> new file mode 100644
>>>> index 000000000000..be3d9bcdd86d
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> @@ -0,0 +1,58 @@
>>>> +==========================
>>>> +I915 Small BAR RFC Section
>>>> +==========================
>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>> local-memory(i.e
>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>> might still be
>>>> +smaller than the total probed_size. In such cases, only some subset of
>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>> first 256M),
>>>> +while the remainder is only accessible via the GPU.
>>>> +
>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>> +----------------------------------------------
>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>> CPU access.
>>>> +This becomes important when placing an object in 
>>>> I915_MEMORY_CLASS_DEVICE, where
>>>> +underneath the device has a small BAR, meaning only some portion of 
>>>> it is CPU
>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>> access is not
>>>> +required, and prioritize using the non-CPU visible portion of
>>>> +I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_gem_create_ext
>>>> +
>>>> +probed_cpu_visible_size attribute
>>>> +---------------------------------
>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>> total size of the
>>>> +CPU accessible portion, for the particular region. This should only be
>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>>> with the
>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>> visible portion,
>>>> +where the total size of the heap needs to be known.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_memory_region_info
>>>> +
>>>> +DRM_I915_QUERY_VMA_INFO query
>>>> +-----------------------------
>>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>>> +respective vma, and return its set of attributes. For now we only 
>>>> support
>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>>>> +currently placed in memory that is accessible by the CPU. This 
>>>> should always be
>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>> I915_MEMORY_CLASS_DEVICE
>>>> +matches the probed_size. If this is not set then CPU faulting the 
>>>> object will
>>>> +likely first require migrating the pages.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_query_vma_info
>>>> +
>>>> +Error Capture restrictions
>>>> +--------------------------
>>>> +With error capture we have two new restrictions:
>>>> +
>>>> +    1) Error capture is best effort on small BAR systems; if the 
>>>> pages are not
>>>> +    CPU accessible, at the time of capture, then the kernel is free 
>>>> to skip
>>>> +    trying to capture them.
>>>> +
>>>> +    2) On discrete we now reject error capture on recoverable 
>>>> contexts. In the
>>>> +    future the kernel may want to blit during error capture, when 
>>>> for example
>>>> +    something is not currently CPU accessible.
>>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>>> b/Documentation/gpu/rfc/index.rst
>>>> index 91e93a705230..5a3bd3924ba6 100644
>>>> --- a/Documentation/gpu/rfc/index.rst
>>>> +++ b/Documentation/gpu/rfc/index.rst
>>>> @@ -23,3 +23,7 @@ host such documentation:
>>>>   .. toctree::
>>>>         i915_scheduler.rst
>>>> +
>>>> +.. toctree::
>>>> +
>>>> +    i915_small_bar.rst
>>>
>>>
>>
>

Christian König April 27, 2022, 3:04 p.m. UTC | #6

Am 27.04.22 um 17:02 schrieb Matthew Auld:
> On 27/04/2022 07:55, Christian König wrote:
>> Well usually we increment the drm minor version when adding some new 
>> flags on amdgpu.
>>
>> Additional to that just one comment from our experience with that: 
>> You don't just need one flag, but two. The first one is a hint which 
>> says "CPU access needed" and the second is a promise which says "CPU 
>> access never needed".
>>
>> The background is that on a whole bunch of buffers you can 100% 
>> certain say that you will never ever need CPU access.
>>
>> Then at least we have a whole bunch of buffers where we might need 
>> CPU access, but can't tell for sure.
>>
>> And last we have stuff like transfer buffers you can be 100% sure 
>> that you need CPU access.
>>
>> Separating it like this helped a lot with performance on small BAR 
>> systems.
>
> Thanks for the comments. For the "CPU access never needed" flag, what 
> extra stuff does that do on the kernel side vs not specifying any 
> flag/hint? I assume it still prioritizes using the non-CPU visible 
> portion first? What else does it do?

It's used as a hint when you need to pin BOs for scanout for example.

In general we try to allocate BOs which are marked "CPU access needed" 
in the CPU visible window if possible, but fallback to any memory if 
that won't fit.

Christian.

>
>>
>> Regards,
>> Christian.
>>
>> Am 27.04.22 um 08:48 schrieb Lionel Landwerlin:
>>> One question though, how do we detect that this flag 
>>> (I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS) is accepted on a given 
>>> kernel?
>>> I assume older kernels are going to reject object creation if we use 
>>> this flag?
>>>
>>> I didn't plan to use __drm_i915_query_vma_info, but isn't it 
>>> inconsistent to select the placement on the GEM object and then 
>>> query whether it's mappable by address?
>>> You made a comment stating this is racy, wouldn't querying on the 
>>> GEM object prevent this?
>>>
>>> Thanks,
>>>
>>> -Lionel
>>>
>>> On 27/04/2022 09:35, Lionel Landwerlin wrote:
>>>> Hi Matt,
>>>>
>>>>
>>>> The proposal looks good to me.
>>>>
>>>> Looking forward to try it on drm-tip.
>>>>
>>>>
>>>> -Lionel
>>>>
>>>> On 20/04/2022 20:13, Matthew Auld wrote:
>>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>>
>>>>> v2:
>>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>>    - Rework error capture interactions, including no longer needing
>>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>>
>>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>>> Cc: Thomas Hellström <[e-mail]>
>>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>>> Cc: Jon Bloomfield <[e-mail]>
>>>>> Cc: Daniel Vetter <[e-mail]>
>>>>> Cc: Jordan Justen <[e-mail]>
>>>>> Cc: Kenneth Graunke <[e-mail]>
>>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>>> Cc: [e-mail]
>>>>> ---
>>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>>> +++++++++++++++++++++++
>>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>>   3 files changed, 252 insertions(+)
>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>>
>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>>> new file mode 100644
>>>>> index 000000000000..7bfd0cf44d35
>>>>> --- /dev/null
>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>>> @@ -0,0 +1,190 @@
>>>>> +/**
>>>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>>>> known to the
>>>>> + * driver.
>>>>> + *
>>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>>> drm_i915_query.
>>>>> + * For this new query we are adding the new query id 
>>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>>> + * at &drm_i915_query_item.query_id.
>>>>> + */
>>>>> +struct __drm_i915_memory_region_info {
>>>>> +    /** @region: The class:instance pair encoding */
>>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>>> +
>>>>> +    /** @rsvd0: MBZ */
>>>>> +    __u32 rsvd0;
>>>>> +
>>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>>> +    __u64 probed_size;
>>>>> +
>>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>>> unknown) */
>>>>> +    __u64 unallocated_size;
>>>>> +
>>>>> +    union {
>>>>> +        /** @rsvd1: MBZ */
>>>>> +        __u64 rsvd1[8];
>>>>> +        struct {
>>>>> +            /**
>>>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>>>> +             * that is CPU accessible. (-1 = unknown).
>>>>> +             *
>>>>> +             * This will be always be <= @probed_size, and the
>>>>> +             * remainder(if there is any) will not be CPU
>>>>> +             * accessible.
>>>>> +             */
>>>>> +            __u64 probed_cpu_visible_size;
>>>>> +        };
>>>>> +    };
>>>>> +};
>>>>> +
>>>>> +/**
>>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>>> behaviour, with added
>>>>> + * extension support using struct i915_user_extension.
>>>>> + *
>>>>> + * Note that new buffer flags should be added here, at least for 
>>>>> the stuff that
>>>>> + * is immutable. Previously we would have two ioctls, one to 
>>>>> create the object
>>>>> + * with gem_create, and another to apply various parameters, 
>>>>> however this
>>>>> + * creates some ambiguity for the params which are considered 
>>>>> immutable. Also in
>>>>> + * general we're phasing out the various SET/GET ioctls.
>>>>> + */
>>>>> +struct __drm_i915_gem_create_ext {
>>>>> +    /**
>>>>> +     * @size: Requested size for the object.
>>>>> +     *
>>>>> +     * The (page-aligned) allocated size for the object will be 
>>>>> returned.
>>>>> +     *
>>>>> +     * Note that for some devices we have might have further minimum
>>>>> +     * page-size restrictions(larger than 4K), like for device 
>>>>> local-memory.
>>>>> +     * However in general the final size here should always 
>>>>> reflect any
>>>>> +     * rounding up, if for example using the 
>>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>>> +     * extension to place the object in device local-memory.
>>>>> +     */
>>>>> +    __u64 size;
>>>>> +    /**
>>>>> +     * @handle: Returned handle for the object.
>>>>> +     *
>>>>> +     * Object handles are nonzero.
>>>>> +     */
>>>>> +    __u32 handle;
>>>>> +    /**
>>>>> +     * @flags: Optional flags.
>>>>> +     *
>>>>> +     * Supported values:
>>>>> +     *
>>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>>> kernel that
>>>>> +     * the object will need to be accessed via the CPU.
>>>>> +     *
>>>>> +     * Only valid when placing objects in 
>>>>> I915_MEMORY_CLASS_DEVICE, and
>>>>> +     * only strictly required on platforms where only some of the 
>>>>> device
>>>>> +     * memory is directly visible or mappable through the CPU, 
>>>>> like on DG2+.
>>>>> +     *
>>>>> +     * One of the placements MUST also be 
>>>>> I915_MEMORY_CLASS_SYSTEM, to
>>>>> +     * ensure we can always spill the allocation to system 
>>>>> memory, if we
>>>>> +     * can't place the object in the mappable part of
>>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>>> +     *
>>>>> +     * Note that since the kernel only supports flat-CCS on 
>>>>> objects that can
>>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>>>> don't
>>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together 
>>>>> with
>>>>> +     * flat-CCS.
>>>>> +     *
>>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. 
>>>>> Note that the
>>>>> +     * kernel can still migrate the object to the mappable part, 
>>>>> as a last
>>>>> +     * resort, if userspace ever CPU faults this object, but this 
>>>>> might be
>>>>> +     * expensive, and so ideally should be avoided.
>>>>> +     */
>>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>>> +    __u32 flags;
>>>>> +    /**
>>>>> +     * @extensions: The chain of extensions to apply to this object.
>>>>> +     *
>>>>> +     * This will be useful in the future when we need to support 
>>>>> several
>>>>> +     * different extensions, and we need to apply more than one when
>>>>> +     * creating the object. See struct i915_user_extension.
>>>>> +     *
>>>>> +     * If we don't supply any extensions then we get the same old 
>>>>> gem_create
>>>>> +     * behaviour.
>>>>> +     *
>>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>>> +     *
>>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>>> +     */
>>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>>> +    __u64 extensions;
>>>>> +};
>>>>> +
>>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>>> +
>>>>> +/**
>>>>> + * struct __drm_i915_query_vma_info
>>>>> + *
>>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>>> returning its set
>>>>> + * of attributes.
>>>>> + *
>>>>> + * .. code-block:: C
>>>>> + *
>>>>> + *    struct drm_i915_query_vma_info info = {};
>>>>> + *    struct drm_i915_query_item item = {
>>>>> + *        .data_ptr = (uintptr_t)&info,
>>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>>> + *    };
>>>>> + *    struct drm_i915_query query = {
>>>>> + *        .num_items = 1,
>>>>> + *        .items_ptr = (uintptr_t)&item,
>>>>> + *    };
>>>>> + *    int err;
>>>>> + *
>>>>> + *    // Unlike some other types of queries, there is no need to 
>>>>> first query
>>>>> + *    // the size of the data_ptr blob here, since we already 
>>>>> know ahead of
>>>>> + *    // time how big this needs to be.
>>>>> + *    item.length = sizeof(info);
>>>>> + *
>>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma 
>>>>> we wish
>>>>> + *    // to query, before then firing off the query.
>>>>> + *    info.vm_id = vm_id;
>>>>> + *    info.offset = gtt_address;
>>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>>> + *    if (err || item.length < 0) ...
>>>>> + *
>>>>> + *    // If all went well we can now inspect the returned 
>>>>> attributes.
>>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>>> + */
>>>>> +struct __drm_i915_query_vma_info {
>>>>> +    /**
>>>>> +     * @vm_id: The given vm id that contains the vma. The id is 
>>>>> the value
>>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>>> +     */
>>>>> +    __u32 vm_id;
>>>>> +    /** @pad: MBZ. */
>>>>> +    __u32 pad;
>>>>> +    /**
>>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>>> the kernel
>>>>> +     * will use to perform the lookup.
>>>>> +     */
>>>>> +    __u64 offset;
>>>>> +    /**
>>>>> +     * @attributes: The returned attributes for the given vma.
>>>>> +     *
>>>>> +     * Possible values:
>>>>> +     *
>>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>>>> backing the
>>>>> +     * vma are currently CPU accessible. If this is not set then 
>>>>> the vma is
>>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>>>>> the CPU
>>>>> +     * cannot directly access(this is only possible on discrete 
>>>>> devices with
>>>>> +     * a small BAR). Attempting to MMAP and fault such an object 
>>>>> will
>>>>> +     * require the kernel first synchronising any GPU work tied 
>>>>> to the
>>>>> +     * object, before then migrating the pages, either to the CPU 
>>>>> accessible
>>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>>> +     * placements permit it. See 
>>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>>> +     *
>>>>> +     * Note that this is inherently racy.
>>>>> +     */
>>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>>> +    __u64 attributes;
>>>>> +    /** @rsvd: MBZ */
>>>>> +    __u32 rsvd[4];
>>>>> +};
>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>> new file mode 100644
>>>>> index 000000000000..be3d9bcdd86d
>>>>> --- /dev/null
>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>> @@ -0,0 +1,58 @@
>>>>> +==========================
>>>>> +I915 Small BAR RFC Section
>>>>> +==========================
>>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>>> local-memory(i.e
>>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>>> might still be
>>>>> +smaller than the total probed_size. In such cases, only some 
>>>>> subset of
>>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>>> first 256M),
>>>>> +while the remainder is only accessible via the GPU.
>>>>> +
>>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>>> +----------------------------------------------
>>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>>> CPU access.
>>>>> +This becomes important when placing an object in 
>>>>> I915_MEMORY_CLASS_DEVICE, where
>>>>> +underneath the device has a small BAR, meaning only some portion 
>>>>> of it is CPU
>>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>>> access is not
>>>>> +required, and prioritize using the non-CPU visible portion of
>>>>> +I915_MEMORY_CLASS_DEVICE.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_gem_create_ext
>>>>> +
>>>>> +probed_cpu_visible_size attribute
>>>>> +---------------------------------
>>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>>> total size of the
>>>>> +CPU accessible portion, for the particular region. This should 
>>>>> only be
>>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>>> +
>>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>>>> with the
>>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>>> visible portion,
>>>>> +where the total size of the heap needs to be known.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_memory_region_info
>>>>> +
>>>>> +DRM_I915_QUERY_VMA_INFO query
>>>>> +-----------------------------
>>>>> +Query the attributes of some vma. Given a vm and GTT offset, find 
>>>>> the
>>>>> +respective vma, and return its set of attributes. For now we only 
>>>>> support
>>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the 
>>>>> object/vma is
>>>>> +currently placed in memory that is accessible by the CPU. This 
>>>>> should always be
>>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>>> I915_MEMORY_CLASS_DEVICE
>>>>> +matches the probed_size. If this is not set then CPU faulting the 
>>>>> object will
>>>>> +likely first require migrating the pages.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_query_vma_info
>>>>> +
>>>>> +Error Capture restrictions
>>>>> +--------------------------
>>>>> +With error capture we have two new restrictions:
>>>>> +
>>>>> +    1) Error capture is best effort on small BAR systems; if the 
>>>>> pages are not
>>>>> +    CPU accessible, at the time of capture, then the kernel is 
>>>>> free to skip
>>>>> +    trying to capture them.
>>>>> +
>>>>> +    2) On discrete we now reject error capture on recoverable 
>>>>> contexts. In the
>>>>> +    future the kernel may want to blit during error capture, when 
>>>>> for example
>>>>> +    something is not currently CPU accessible.
>>>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>>>> b/Documentation/gpu/rfc/index.rst
>>>>> index 91e93a705230..5a3bd3924ba6 100644
>>>>> --- a/Documentation/gpu/rfc/index.rst
>>>>> +++ b/Documentation/gpu/rfc/index.rst
>>>>> @@ -23,3 +23,7 @@ host such documentation:
>>>>>   .. toctree::
>>>>>         i915_scheduler.rst
>>>>> +
>>>>> +.. toctree::
>>>>> +
>>>>> +    i915_small_bar.rst
>>>>
>>>>
>>>
>>

Matthew Auld April 27, 2022, 3:18 p.m. UTC | #7

On 27/04/2022 07:48, Lionel Landwerlin wrote:
> One question though, how do we detect that this flag 
> (I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS) is accepted on a given kernel?
> I assume older kernels are going to reject object creation if we use 
> this flag?

 From some offline discussion with Lionel, the plan here is to just do a 
dummy gem_create_ext to check if the kernel throws an error with the new 
flag or not.

> 
> I didn't plan to use __drm_i915_query_vma_info, but isn't it 
> inconsistent to select the placement on the GEM object and then query 
> whether it's mappable by address?
> You made a comment stating this is racy, wouldn't querying on the GEM 
> object prevent this?

Since mesa at this time doesn't currently have a use for this one, then 
I guess we should maybe just drop this part of the uapi, in this version 
at least, if no objections.

> 
> Thanks,
> 
> -Lionel
> 
> On 27/04/2022 09:35, Lionel Landwerlin wrote:
>> Hi Matt,
>>
>>
>> The proposal looks good to me.
>>
>> Looking forward to try it on drm-tip.
>>
>>
>> -Lionel
>>
>> On 20/04/2022 20:13, Matthew Auld wrote:
>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>
>>> v2:
>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>    - Rework error capture interactions, including no longer needing
>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>    - Add probed_cpu_visible_size. (Lionel)
>>>
>>> Signed-off-by: Matthew Auld <[e-mail]>
>>> Cc: Thomas Hellström <[e-mail]>
>>> Cc: Lionel Landwerlin <[e-mail]>
>>> Cc: Jon Bloomfield <[e-mail]>
>>> Cc: Daniel Vetter <[e-mail]>
>>> Cc: Jordan Justen <[e-mail]>
>>> Cc: Kenneth Graunke <[e-mail]>
>>> Cc: Akeem G Abodunrin <[e-mail]>
>>> Cc: [e-mail]
>>> ---
>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>   3 files changed, 252 insertions(+)
>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>> new file mode 100644
>>> index 000000000000..7bfd0cf44d35
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>> @@ -0,0 +1,190 @@
>>> +/**
>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>> known to the
>>> + * driver.
>>> + *
>>> + * Note this is using both struct drm_i915_query_item and struct 
>>> drm_i915_query.
>>> + * For this new query we are adding the new query id 
>>> DRM_I915_QUERY_MEMORY_REGIONS
>>> + * at &drm_i915_query_item.query_id.
>>> + */
>>> +struct __drm_i915_memory_region_info {
>>> +    /** @region: The class:instance pair encoding */
>>> +    struct drm_i915_gem_memory_class_instance region;
>>> +
>>> +    /** @rsvd0: MBZ */
>>> +    __u32 rsvd0;
>>> +
>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>> +    __u64 probed_size;
>>> +
>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>> unknown) */
>>> +    __u64 unallocated_size;
>>> +
>>> +    union {
>>> +        /** @rsvd1: MBZ */
>>> +        __u64 rsvd1[8];
>>> +        struct {
>>> +            /**
>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>> +             * that is CPU accessible. (-1 = unknown).
>>> +             *
>>> +             * This will be always be <= @probed_size, and the
>>> +             * remainder(if there is any) will not be CPU
>>> +             * accessible.
>>> +             */
>>> +            __u64 probed_cpu_visible_size;
>>> +        };
>>> +    };
>>> +};
>>> +
>>> +/**
>>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, 
>>> with added
>>> + * extension support using struct i915_user_extension.
>>> + *
>>> + * Note that new buffer flags should be added here, at least for the 
>>> stuff that
>>> + * is immutable. Previously we would have two ioctls, one to create 
>>> the object
>>> + * with gem_create, and another to apply various parameters, however 
>>> this
>>> + * creates some ambiguity for the params which are considered 
>>> immutable. Also in
>>> + * general we're phasing out the various SET/GET ioctls.
>>> + */
>>> +struct __drm_i915_gem_create_ext {
>>> +    /**
>>> +     * @size: Requested size for the object.
>>> +     *
>>> +     * The (page-aligned) allocated size for the object will be 
>>> returned.
>>> +     *
>>> +     * Note that for some devices we have might have further minimum
>>> +     * page-size restrictions(larger than 4K), like for device 
>>> local-memory.
>>> +     * However in general the final size here should always reflect any
>>> +     * rounding up, if for example using the 
>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>> +     * extension to place the object in device local-memory.
>>> +     */
>>> +    __u64 size;
>>> +    /**
>>> +     * @handle: Returned handle for the object.
>>> +     *
>>> +     * Object handles are nonzero.
>>> +     */
>>> +    __u32 handle;
>>> +    /**
>>> +     * @flags: Optional flags.
>>> +     *
>>> +     * Supported values:
>>> +     *
>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>> kernel that
>>> +     * the object will need to be accessed via the CPU.
>>> +     *
>>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
>>> +     * only strictly required on platforms where only some of the 
>>> device
>>> +     * memory is directly visible or mappable through the CPU, like 
>>> on DG2+.
>>> +     *
>>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>>> +     * ensure we can always spill the allocation to system memory, 
>>> if we
>>> +     * can't place the object in the mappable part of
>>> +     * I915_MEMORY_CLASS_DEVICE.
>>> +     *
>>> +     * Note that since the kernel only supports flat-CCS on objects 
>>> that can
>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>>> +     * flat-CCS.
>>> +     *
>>> +     * Without this hint, the kernel will assume that non-mappable
>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>> that the
>>> +     * kernel can still migrate the object to the mappable part, as 
>>> a last
>>> +     * resort, if userspace ever CPU faults this object, but this 
>>> might be
>>> +     * expensive, and so ideally should be avoided.
>>> +     */
>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>> +    __u32 flags;
>>> +    /**
>>> +     * @extensions: The chain of extensions to apply to this object.
>>> +     *
>>> +     * This will be useful in the future when we need to support 
>>> several
>>> +     * different extensions, and we need to apply more than one when
>>> +     * creating the object. See struct i915_user_extension.
>>> +     *
>>> +     * If we don't supply any extensions then we get the same old 
>>> gem_create
>>> +     * behaviour.
>>> +     *
>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>> +     *
>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>> +     */
>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>> +    __u64 extensions;
>>> +};
>>> +
>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>> +
>>> +/**
>>> + * struct __drm_i915_query_vma_info
>>> + *
>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>> returning its set
>>> + * of attributes.
>>> + *
>>> + * .. code-block:: C
>>> + *
>>> + *    struct drm_i915_query_vma_info info = {};
>>> + *    struct drm_i915_query_item item = {
>>> + *        .data_ptr = (uintptr_t)&info,
>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>> + *    };
>>> + *    struct drm_i915_query query = {
>>> + *        .num_items = 1,
>>> + *        .items_ptr = (uintptr_t)&item,
>>> + *    };
>>> + *    int err;
>>> + *
>>> + *    // Unlike some other types of queries, there is no need to 
>>> first query
>>> + *    // the size of the data_ptr blob here, since we already know 
>>> ahead of
>>> + *    // time how big this needs to be.
>>> + *    item.length = sizeof(info);
>>> + *
>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we wish
>>> + *    // to query, before then firing off the query.
>>> + *    info.vm_id = vm_id;
>>> + *    info.offset = gtt_address;
>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>> + *    if (err || item.length < 0) ...
>>> + *
>>> + *    // If all went well we can now inspect the returned attributes.
>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>> + */
>>> +struct __drm_i915_query_vma_info {
>>> +    /**
>>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>>> value
>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>> +     * drm_i915_gem_vm_control.vm_id.
>>> +     */
>>> +    __u32 vm_id;
>>> +    /** @pad: MBZ. */
>>> +    __u32 pad;
>>> +    /**
>>> +     * @offset: The corresponding ppGTT address of the vma which the 
>>> kernel
>>> +     * will use to perform the lookup.
>>> +     */
>>> +    __u64 offset;
>>> +    /**
>>> +     * @attributes: The returned attributes for the given vma.
>>> +     *
>>> +     * Possible values:
>>> +     *
>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>> backing the
>>> +     * vma are currently CPU accessible. If this is not set then the 
>>> vma is
>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>>> the CPU
>>> +     * cannot directly access(this is only possible on discrete 
>>> devices with
>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>> +     * require the kernel first synchronising any GPU work tied to the
>>> +     * object, before then migrating the pages, either to the CPU 
>>> accessible
>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>> I915_MEMORY_CLASS_SYSTEM, if the
>>> +     * placements permit it. See 
>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>> +     *
>>> +     * Note that this is inherently racy.
>>> +     */
>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>> +    __u64 attributes;
>>> +    /** @rsvd: MBZ */
>>> +    __u32 rsvd[4];
>>> +};
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>> new file mode 100644
>>> index 000000000000..be3d9bcdd86d
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>> @@ -0,0 +1,58 @@
>>> +==========================
>>> +I915 Small BAR RFC Section
>>> +==========================
>>> +Starting from DG2 we will have resizable BAR support for device 
>>> local-memory(i.e
>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>> might still be
>>> +smaller than the total probed_size. In such cases, only some subset of
>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>> first 256M),
>>> +while the remainder is only accessible via the GPU.
>>> +
>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>> +----------------------------------------------
>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>> CPU access.
>>> +This becomes important when placing an object in 
>>> I915_MEMORY_CLASS_DEVICE, where
>>> +underneath the device has a small BAR, meaning only some portion of 
>>> it is CPU
>>> +accessible. Without this flag the kernel will assume that CPU access 
>>> is not
>>> +required, and prioritize using the non-CPU visible portion of
>>> +I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_gem_create_ext
>>> +
>>> +probed_cpu_visible_size attribute
>>> +---------------------------------
>>> +New struct__drm_i915_memory_region attribute which returns the total 
>>> size of the
>>> +CPU accessible portion, for the particular region. This should only be
>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>> with the
>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>> visible portion,
>>> +where the total size of the heap needs to be known.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_memory_region_info
>>> +
>>> +DRM_I915_QUERY_VMA_INFO query
>>> +-----------------------------
>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>> +respective vma, and return its set of attributes. For now we only 
>>> support
>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>>> +currently placed in memory that is accessible by the CPU. This 
>>> should always be
>>> +set on devices where the CPU probed_cpu_visible_size of 
>>> I915_MEMORY_CLASS_DEVICE
>>> +matches the probed_size. If this is not set then CPU faulting the 
>>> object will
>>> +likely first require migrating the pages.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_query_vma_info
>>> +
>>> +Error Capture restrictions
>>> +--------------------------
>>> +With error capture we have two new restrictions:
>>> +
>>> +    1) Error capture is best effort on small BAR systems; if the 
>>> pages are not
>>> +    CPU accessible, at the time of capture, then the kernel is free 
>>> to skip
>>> +    trying to capture them.
>>> +
>>> +    2) On discrete we now reject error capture on recoverable 
>>> contexts. In the
>>> +    future the kernel may want to blit during error capture, when 
>>> for example
>>> +    something is not currently CPU accessible.
>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>> b/Documentation/gpu/rfc/index.rst
>>> index 91e93a705230..5a3bd3924ba6 100644
>>> --- a/Documentation/gpu/rfc/index.rst
>>> +++ b/Documentation/gpu/rfc/index.rst
>>> @@ -23,3 +23,7 @@ host such documentation:
>>>   .. toctree::
>>>         i915_scheduler.rst
>>> +
>>> +.. toctree::
>>> +
>>> +    i915_small_bar.rst
>>
>>
>

Lionel Landwerlin April 27, 2022, 3:37 p.m. UTC | #8

On 27/04/2022 18:18, Matthew Auld wrote:
> On 27/04/2022 07:48, Lionel Landwerlin wrote:
>> One question though, how do we detect that this flag 
>> (I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS) is accepted on a given 
>> kernel?
>> I assume older kernels are going to reject object creation if we use 
>> this flag?
>
> From some offline discussion with Lionel, the plan here is to just do 
> a dummy gem_create_ext to check if the kernel throws an error with the 
> new flag or not.
>
>>
>> I didn't plan to use __drm_i915_query_vma_info, but isn't it 
>> inconsistent to select the placement on the GEM object and then query 
>> whether it's mappable by address?
>> You made a comment stating this is racy, wouldn't querying on the GEM 
>> object prevent this?
>
> Since mesa at this time doesn't currently have a use for this one, 
> then I guess we should maybe just drop this part of the uapi, in this 
> version at least, if no objections.


Just repeating what we discussed (maybe I missed some other discussion 
and that's why I was confused) :


The way I was planning to use this is to have 3 heaps in Vulkan :

     - heap0: local only, no cpu visible

     - heap1: system, cpu visible

     - heap2: local & cpu visible


With heap2 having the reported probed_cpu_visible_size size.

It is an error for the application to map from heap0 [1].


With that said, it means if we created a GEM BO without 
I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS, we'll never mmap it.

So why the query?

I guess it would be useful when we import a buffer from another 
application. But in that case, why not have the query on the BO?


-Lionel


[1] : 
https://www.khronos.org/registry/vulkan/specs/1.3-extensions/man/html/vkMapMemory.html 
(VUID-vkMapMemory-memory-00682)

>
>>
>> Thanks,
>>
>> -Lionel
>>
>> On 27/04/2022 09:35, Lionel Landwerlin wrote:
>>> Hi Matt,
>>>
>>>
>>> The proposal looks good to me.
>>>
>>> Looking forward to try it on drm-tip.
>>>
>>>
>>> -Lionel
>>>
>>> On 20/04/2022 20:13, Matthew Auld wrote:
>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>
>>>> v2:
>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>    - Rework error capture interactions, including no longer needing
>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>
>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>> Cc: Thomas Hellström <[e-mail]>
>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>> Cc: Jon Bloomfield <[e-mail]>
>>>> Cc: Daniel Vetter <[e-mail]>
>>>> Cc: Jordan Justen <[e-mail]>
>>>> Cc: Kenneth Graunke <[e-mail]>
>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>> Cc: [e-mail]
>>>> ---
>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>> +++++++++++++++++++++++
>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>   3 files changed, 252 insertions(+)
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>> new file mode 100644
>>>> index 000000000000..7bfd0cf44d35
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>> @@ -0,0 +1,190 @@
>>>> +/**
>>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>>> known to the
>>>> + * driver.
>>>> + *
>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>> drm_i915_query.
>>>> + * For this new query we are adding the new query id 
>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>> + * at &drm_i915_query_item.query_id.
>>>> + */
>>>> +struct __drm_i915_memory_region_info {
>>>> +    /** @region: The class:instance pair encoding */
>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>> +
>>>> +    /** @rsvd0: MBZ */
>>>> +    __u32 rsvd0;
>>>> +
>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>> +    __u64 probed_size;
>>>> +
>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>> unknown) */
>>>> +    __u64 unallocated_size;
>>>> +
>>>> +    union {
>>>> +        /** @rsvd1: MBZ */
>>>> +        __u64 rsvd1[8];
>>>> +        struct {
>>>> +            /**
>>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>>> +             * that is CPU accessible. (-1 = unknown).
>>>> +             *
>>>> +             * This will be always be <= @probed_size, and the
>>>> +             * remainder(if there is any) will not be CPU
>>>> +             * accessible.
>>>> +             */
>>>> +            __u64 probed_cpu_visible_size;
>>>> +        };
>>>> +    };
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>> behaviour, with added
>>>> + * extension support using struct i915_user_extension.
>>>> + *
>>>> + * Note that new buffer flags should be added here, at least for 
>>>> the stuff that
>>>> + * is immutable. Previously we would have two ioctls, one to 
>>>> create the object
>>>> + * with gem_create, and another to apply various parameters, 
>>>> however this
>>>> + * creates some ambiguity for the params which are considered 
>>>> immutable. Also in
>>>> + * general we're phasing out the various SET/GET ioctls.
>>>> + */
>>>> +struct __drm_i915_gem_create_ext {
>>>> +    /**
>>>> +     * @size: Requested size for the object.
>>>> +     *
>>>> +     * The (page-aligned) allocated size for the object will be 
>>>> returned.
>>>> +     *
>>>> +     * Note that for some devices we have might have further minimum
>>>> +     * page-size restrictions(larger than 4K), like for device 
>>>> local-memory.
>>>> +     * However in general the final size here should always 
>>>> reflect any
>>>> +     * rounding up, if for example using the 
>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>> +     * extension to place the object in device local-memory.
>>>> +     */
>>>> +    __u64 size;
>>>> +    /**
>>>> +     * @handle: Returned handle for the object.
>>>> +     *
>>>> +     * Object handles are nonzero.
>>>> +     */
>>>> +    __u32 handle;
>>>> +    /**
>>>> +     * @flags: Optional flags.
>>>> +     *
>>>> +     * Supported values:
>>>> +     *
>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>> kernel that
>>>> +     * the object will need to be accessed via the CPU.
>>>> +     *
>>>> +     * Only valid when placing objects in 
>>>> I915_MEMORY_CLASS_DEVICE, and
>>>> +     * only strictly required on platforms where only some of the 
>>>> device
>>>> +     * memory is directly visible or mappable through the CPU, 
>>>> like on DG2+.
>>>> +     *
>>>> +     * One of the placements MUST also be 
>>>> I915_MEMORY_CLASS_SYSTEM, to
>>>> +     * ensure we can always spill the allocation to system memory, 
>>>> if we
>>>> +     * can't place the object in the mappable part of
>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>> +     *
>>>> +     * Note that since the kernel only supports flat-CCS on 
>>>> objects that can
>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>>> don't
>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together 
>>>> with
>>>> +     * flat-CCS.
>>>> +     *
>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>>> that the
>>>> +     * kernel can still migrate the object to the mappable part, 
>>>> as a last
>>>> +     * resort, if userspace ever CPU faults this object, but this 
>>>> might be
>>>> +     * expensive, and so ideally should be avoided.
>>>> +     */
>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>> +    __u32 flags;
>>>> +    /**
>>>> +     * @extensions: The chain of extensions to apply to this object.
>>>> +     *
>>>> +     * This will be useful in the future when we need to support 
>>>> several
>>>> +     * different extensions, and we need to apply more than one when
>>>> +     * creating the object. See struct i915_user_extension.
>>>> +     *
>>>> +     * If we don't supply any extensions then we get the same old 
>>>> gem_create
>>>> +     * behaviour.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>> +     */
>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>> +    __u64 extensions;
>>>> +};
>>>> +
>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_query_vma_info
>>>> + *
>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>> returning its set
>>>> + * of attributes.
>>>> + *
>>>> + * .. code-block:: C
>>>> + *
>>>> + *    struct drm_i915_query_vma_info info = {};
>>>> + *    struct drm_i915_query_item item = {
>>>> + *        .data_ptr = (uintptr_t)&info,
>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>> + *    };
>>>> + *    struct drm_i915_query query = {
>>>> + *        .num_items = 1,
>>>> + *        .items_ptr = (uintptr_t)&item,
>>>> + *    };
>>>> + *    int err;
>>>> + *
>>>> + *    // Unlike some other types of queries, there is no need to 
>>>> first query
>>>> + *    // the size of the data_ptr blob here, since we already know 
>>>> ahead of
>>>> + *    // time how big this needs to be.
>>>> + *    item.length = sizeof(info);
>>>> + *
>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we 
>>>> wish
>>>> + *    // to query, before then firing off the query.
>>>> + *    info.vm_id = vm_id;
>>>> + *    info.offset = gtt_address;
>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>> + *    if (err || item.length < 0) ...
>>>> + *
>>>> + *    // If all went well we can now inspect the returned attributes.
>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>> + */
>>>> +struct __drm_i915_query_vma_info {
>>>> +    /**
>>>> +     * @vm_id: The given vm id that contains the vma. The id is 
>>>> the value
>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>> +     */
>>>> +    __u32 vm_id;
>>>> +    /** @pad: MBZ. */
>>>> +    __u32 pad;
>>>> +    /**
>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>> the kernel
>>>> +     * will use to perform the lookup.
>>>> +     */
>>>> +    __u64 offset;
>>>> +    /**
>>>> +     * @attributes: The returned attributes for the given vma.
>>>> +     *
>>>> +     * Possible values:
>>>> +     *
>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>>> backing the
>>>> +     * vma are currently CPU accessible. If this is not set then 
>>>> the vma is
>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>>>> the CPU
>>>> +     * cannot directly access(this is only possible on discrete 
>>>> devices with
>>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>>> +     * require the kernel first synchronising any GPU work tied to 
>>>> the
>>>> +     * object, before then migrating the pages, either to the CPU 
>>>> accessible
>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>> +     * placements permit it. See 
>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>> +     *
>>>> +     * Note that this is inherently racy.
>>>> +     */
>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>> +    __u64 attributes;
>>>> +    /** @rsvd: MBZ */
>>>> +    __u32 rsvd[4];
>>>> +};
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> new file mode 100644
>>>> index 000000000000..be3d9bcdd86d
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> @@ -0,0 +1,58 @@
>>>> +==========================
>>>> +I915 Small BAR RFC Section
>>>> +==========================
>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>> local-memory(i.e
>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>> might still be
>>>> +smaller than the total probed_size. In such cases, only some 
>>>> subset of
>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>> first 256M),
>>>> +while the remainder is only accessible via the GPU.
>>>> +
>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>> +----------------------------------------------
>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>> CPU access.
>>>> +This becomes important when placing an object in 
>>>> I915_MEMORY_CLASS_DEVICE, where
>>>> +underneath the device has a small BAR, meaning only some portion 
>>>> of it is CPU
>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>> access is not
>>>> +required, and prioritize using the non-CPU visible portion of
>>>> +I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_gem_create_ext
>>>> +
>>>> +probed_cpu_visible_size attribute
>>>> +---------------------------------
>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>> total size of the
>>>> +CPU accessible portion, for the particular region. This should 
>>>> only be
>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>>> with the
>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>> visible portion,
>>>> +where the total size of the heap needs to be known.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_memory_region_info
>>>> +
>>>> +DRM_I915_QUERY_VMA_INFO query
>>>> +-----------------------------
>>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>>> +respective vma, and return its set of attributes. For now we only 
>>>> support
>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the 
>>>> object/vma is
>>>> +currently placed in memory that is accessible by the CPU. This 
>>>> should always be
>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>> I915_MEMORY_CLASS_DEVICE
>>>> +matches the probed_size. If this is not set then CPU faulting the 
>>>> object will
>>>> +likely first require migrating the pages.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_query_vma_info
>>>> +
>>>> +Error Capture restrictions
>>>> +--------------------------
>>>> +With error capture we have two new restrictions:
>>>> +
>>>> +    1) Error capture is best effort on small BAR systems; if the 
>>>> pages are not
>>>> +    CPU accessible, at the time of capture, then the kernel is 
>>>> free to skip
>>>> +    trying to capture them.
>>>> +
>>>> +    2) On discrete we now reject error capture on recoverable 
>>>> contexts. In the
>>>> +    future the kernel may want to blit during error capture, when 
>>>> for example
>>>> +    something is not currently CPU accessible.
>>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>>> b/Documentation/gpu/rfc/index.rst
>>>> index 91e93a705230..5a3bd3924ba6 100644
>>>> --- a/Documentation/gpu/rfc/index.rst
>>>> +++ b/Documentation/gpu/rfc/index.rst
>>>> @@ -23,3 +23,7 @@ host such documentation:
>>>>   .. toctree::
>>>>         i915_scheduler.rst
>>>> +
>>>> +.. toctree::
>>>> +
>>>> +    i915_small_bar.rst
>>>
>>>
>>

Daniel Vetter April 27, 2022, 3:38 p.m. UTC | #9

On Wed, Apr 27, 2022 at 08:55:07AM +0200, Christian König wrote:
> Well usually we increment the drm minor version when adding some new flags
> on amdgpu.
> 
> Additional to that just one comment from our experience with that: You don't
> just need one flag, but two. The first one is a hint which says "CPU access
> needed" and the second is a promise which says "CPU access never needed".
> 
> The background is that on a whole bunch of buffers you can 100% certain say
> that you will never ever need CPU access.
> 
> Then at least we have a whole bunch of buffers where we might need CPU
> access, but can't tell for sure.
> 
> And last we have stuff like transfer buffers you can be 100% sure that you
> need CPU access.
> 
> Separating it like this helped a lot with performance on small BAR systems.

So my assumption was that for transfer buffers you'd fill them with the
cpu first anyway, so no need for the extra flag.

I guess this if for transfer buffers for gpu -> cpu transfers, where it
would result in costly bo move and stalls and it's better to make sure
it's cpu accessible from the start? At least on current gpu we have where
there's no coherent interconnect, those buffers have to be in system
memory or your cpu access will be a disaster, so again they're naturally
cpu accessible.

What's the use-case for the "cpu access required" flag where "cpu access
before gpu access" isn't a good enough hint already to get the same perf
benefits?

Also for scanout my idea at least is that we just fail mmap when you
haven't set the flag and the scanout is pinned to unmappable, for two
reasons:
- 4k buffers are big, if we force them all into mappable things are
  non-pretty.
- You need mesa anyway to access tiled buffers, and mesa knows how to use
  a transfer buffer. That should work even when you do desktop switching
  and fastboot and stuff like that with the getfb2 ioctl should all work
  (and without getfb2 it's doomed to garbage anyway).

So only dumb kms buffers (which are linear) would ever get the
NEEDS_CPU_ACCESS flag, and only those we'd ever pin into cpu accessible
range for scanout. Is there a hole in that plan?

Cheers, Daniel

> 
> Regards,
> Christian.
> 
> Am 27.04.22 um 08:48 schrieb Lionel Landwerlin:
> > One question though, how do we detect that this flag
> > (I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS) is accepted on a given
> > kernel?
> > I assume older kernels are going to reject object creation if we use
> > this flag?
> > 
> > I didn't plan to use __drm_i915_query_vma_info, but isn't it
> > inconsistent to select the placement on the GEM object and then query
> > whether it's mappable by address?
> > You made a comment stating this is racy, wouldn't querying on the GEM
> > object prevent this?
> > 
> > Thanks,
> > 
> > -Lionel
> > 
> > On 27/04/2022 09:35, Lionel Landwerlin wrote:
> > > Hi Matt,
> > > 
> > > 
> > > The proposal looks good to me.
> > > 
> > > Looking forward to try it on drm-tip.
> > > 
> > > 
> > > -Lionel
> > > 
> > > On 20/04/2022 20:13, Matthew Auld wrote:
> > > > Add an entry for the new uapi needed for small BAR on DG2+.
> > > > 
> > > > v2:
> > > >    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
> > > >    - Rework error capture interactions, including no longer needing
> > > >      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
> > > >    - Add probed_cpu_visible_size. (Lionel)
> > > > 
> > > > Signed-off-by: Matthew Auld <[e-mail]>
> > > > Cc: Thomas Hellström <[e-mail]>
> > > > Cc: Lionel Landwerlin <[e-mail]>
> > > > Cc: Jon Bloomfield <[e-mail]>
> > > > Cc: Daniel Vetter <[e-mail]>
> > > > Cc: Jordan Justen <[e-mail]>
> > > > Cc: Kenneth Graunke <[e-mail]>
> > > > Cc: Akeem G Abodunrin <[e-mail]>
> > > > Cc: [e-mail]
> > > > ---
> > > >   Documentation/gpu/rfc/i915_small_bar.h   | 190
> > > > +++++++++++++++++++++++
> > > >   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
> > > >   Documentation/gpu/rfc/index.rst          |   4 +
> > > >   3 files changed, 252 insertions(+)
> > > >   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
> > > >   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
> > > > 
> > > > diff --git a/Documentation/gpu/rfc/i915_small_bar.h
> > > > b/Documentation/gpu/rfc/i915_small_bar.h
> > > > new file mode 100644
> > > > index 000000000000..7bfd0cf44d35
> > > > --- /dev/null
> > > > +++ b/Documentation/gpu/rfc/i915_small_bar.h
> > > > @@ -0,0 +1,190 @@
> > > > +/**
> > > > + * struct __drm_i915_memory_region_info - Describes one region
> > > > as known to the
> > > > + * driver.
> > > > + *
> > > > + * Note this is using both struct drm_i915_query_item and
> > > > struct drm_i915_query.
> > > > + * For this new query we are adding the new query id
> > > > DRM_I915_QUERY_MEMORY_REGIONS
> > > > + * at &drm_i915_query_item.query_id.
> > > > + */
> > > > +struct __drm_i915_memory_region_info {
> > > > +    /** @region: The class:instance pair encoding */
> > > > +    struct drm_i915_gem_memory_class_instance region;
> > > > +
> > > > +    /** @rsvd0: MBZ */
> > > > +    __u32 rsvd0;
> > > > +
> > > > +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
> > > > +    __u64 probed_size;
> > > > +
> > > > +    /** @unallocated_size: Estimate of memory remaining (-1 =
> > > > unknown) */
> > > > +    __u64 unallocated_size;
> > > > +
> > > > +    union {
> > > > +        /** @rsvd1: MBZ */
> > > > +        __u64 rsvd1[8];
> > > > +        struct {
> > > > +            /**
> > > > +             * @probed_cpu_visible_size: Memory probed by the driver
> > > > +             * that is CPU accessible. (-1 = unknown).
> > > > +             *
> > > > +             * This will be always be <= @probed_size, and the
> > > > +             * remainder(if there is any) will not be CPU
> > > > +             * accessible.
> > > > +             */
> > > > +            __u64 probed_cpu_visible_size;
> > > > +        };
> > > > +    };
> > > > +};
> > > > +
> > > > +/**
> > > > + * struct __drm_i915_gem_create_ext - Existing gem_create
> > > > behaviour, with added
> > > > + * extension support using struct i915_user_extension.
> > > > + *
> > > > + * Note that new buffer flags should be added here, at least
> > > > for the stuff that
> > > > + * is immutable. Previously we would have two ioctls, one to
> > > > create the object
> > > > + * with gem_create, and another to apply various parameters,
> > > > however this
> > > > + * creates some ambiguity for the params which are considered
> > > > immutable. Also in
> > > > + * general we're phasing out the various SET/GET ioctls.
> > > > + */
> > > > +struct __drm_i915_gem_create_ext {
> > > > +    /**
> > > > +     * @size: Requested size for the object.
> > > > +     *
> > > > +     * The (page-aligned) allocated size for the object will be
> > > > returned.
> > > > +     *
> > > > +     * Note that for some devices we have might have further minimum
> > > > +     * page-size restrictions(larger than 4K), like for device
> > > > local-memory.
> > > > +     * However in general the final size here should always
> > > > reflect any
> > > > +     * rounding up, if for example using the
> > > > I915_GEM_CREATE_EXT_MEMORY_REGIONS
> > > > +     * extension to place the object in device local-memory.
> > > > +     */
> > > > +    __u64 size;
> > > > +    /**
> > > > +     * @handle: Returned handle for the object.
> > > > +     *
> > > > +     * Object handles are nonzero.
> > > > +     */
> > > > +    __u32 handle;
> > > > +    /**
> > > > +     * @flags: Optional flags.
> > > > +     *
> > > > +     * Supported values:
> > > > +     *
> > > > +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to
> > > > the kernel that
> > > > +     * the object will need to be accessed via the CPU.
> > > > +     *
> > > > +     * Only valid when placing objects in
> > > > I915_MEMORY_CLASS_DEVICE, and
> > > > +     * only strictly required on platforms where only some of
> > > > the device
> > > > +     * memory is directly visible or mappable through the CPU,
> > > > like on DG2+.
> > > > +     *
> > > > +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
> > > > +     * ensure we can always spill the allocation to system
> > > > memory, if we
> > > > +     * can't place the object in the mappable part of
> > > > +     * I915_MEMORY_CLASS_DEVICE.
> > > > +     *
> > > > +     * Note that since the kernel only supports flat-CCS on
> > > > objects that can
> > > > +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we
> > > > therefore don't
> > > > +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
> > > > +     * flat-CCS.
> > > > +     *
> > > > +     * Without this hint, the kernel will assume that non-mappable
> > > > +     * I915_MEMORY_CLASS_DEVICE is preferred for this object.
> > > > Note that the
> > > > +     * kernel can still migrate the object to the mappable
> > > > part, as a last
> > > > +     * resort, if userspace ever CPU faults this object, but
> > > > this might be
> > > > +     * expensive, and so ideally should be avoided.
> > > > +     */
> > > > +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
> > > > +    __u32 flags;
> > > > +    /**
> > > > +     * @extensions: The chain of extensions to apply to this object.
> > > > +     *
> > > > +     * This will be useful in the future when we need to
> > > > support several
> > > > +     * different extensions, and we need to apply more than one when
> > > > +     * creating the object. See struct i915_user_extension.
> > > > +     *
> > > > +     * If we don't supply any extensions then we get the same
> > > > old gem_create
> > > > +     * behaviour.
> > > > +     *
> > > > +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
> > > > +     * struct drm_i915_gem_create_ext_memory_regions.
> > > > +     *
> > > > +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
> > > > +     * struct drm_i915_gem_create_ext_protected_content.
> > > > +     */
> > > > +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
> > > > +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
> > > > +    __u64 extensions;
> > > > +};
> > > > +
> > > > +#define DRM_I915_QUERY_VMA_INFO    5
> > > > +
> > > > +/**
> > > > + * struct __drm_i915_query_vma_info
> > > > + *
> > > > + * Given a vm and GTT address, lookup the corresponding vma,
> > > > returning its set
> > > > + * of attributes.
> > > > + *
> > > > + * .. code-block:: C
> > > > + *
> > > > + *    struct drm_i915_query_vma_info info = {};
> > > > + *    struct drm_i915_query_item item = {
> > > > + *        .data_ptr = (uintptr_t)&info,
> > > > + *        .query_id = DRM_I915_QUERY_VMA_INFO,
> > > > + *    };
> > > > + *    struct drm_i915_query query = {
> > > > + *        .num_items = 1,
> > > > + *        .items_ptr = (uintptr_t)&item,
> > > > + *    };
> > > > + *    int err;
> > > > + *
> > > > + *    // Unlike some other types of queries, there is no need
> > > > to first query
> > > > + *    // the size of the data_ptr blob here, since we already
> > > > know ahead of
> > > > + *    // time how big this needs to be.
> > > > + *    item.length = sizeof(info);
> > > > + *
> > > > + *    // Next we fill in the vm_id and ppGTT address of the vma
> > > > we wish
> > > > + *    // to query, before then firing off the query.
> > > > + *    info.vm_id = vm_id;
> > > > + *    info.offset = gtt_address;
> > > > + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
> > > > + *    if (err || item.length < 0) ...
> > > > + *
> > > > + *    // If all went well we can now inspect the returned attributes.
> > > > + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
> > > > + */
> > > > +struct __drm_i915_query_vma_info {
> > > > +    /**
> > > > +     * @vm_id: The given vm id that contains the vma. The id is
> > > > the value
> > > > +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
> > > > +     * drm_i915_gem_vm_control.vm_id.
> > > > +     */
> > > > +    __u32 vm_id;
> > > > +    /** @pad: MBZ. */
> > > > +    __u32 pad;
> > > > +    /**
> > > > +     * @offset: The corresponding ppGTT address of the vma
> > > > which the kernel
> > > > +     * will use to perform the lookup.
> > > > +     */
> > > > +    __u64 offset;
> > > > +    /**
> > > > +     * @attributes: The returned attributes for the given vma.
> > > > +     *
> > > > +     * Possible values:
> > > > +     *
> > > > +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages
> > > > backing the
> > > > +     * vma are currently CPU accessible. If this is not set
> > > > then the vma is
> > > > +     * currently backed by I915_MEMORY_CLASS_DEVICE memory,
> > > > which the CPU
> > > > +     * cannot directly access(this is only possible on discrete
> > > > devices with
> > > > +     * a small BAR). Attempting to MMAP and fault such an object will
> > > > +     * require the kernel first synchronising any GPU work tied to the
> > > > +     * object, before then migrating the pages, either to the
> > > > CPU accessible
> > > > +     * part of I915_MEMORY_CLASS_DEVICE, or
> > > > I915_MEMORY_CLASS_SYSTEM, if the
> > > > +     * placements permit it. See
> > > > I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
> > > > +     *
> > > > +     * Note that this is inherently racy.
> > > > +     */
> > > > +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
> > > > +    __u64 attributes;
> > > > +    /** @rsvd: MBZ */
> > > > +    __u32 rsvd[4];
> > > > +};
> > > > diff --git a/Documentation/gpu/rfc/i915_small_bar.rst
> > > > b/Documentation/gpu/rfc/i915_small_bar.rst
> > > > new file mode 100644
> > > > index 000000000000..be3d9bcdd86d
> > > > --- /dev/null
> > > > +++ b/Documentation/gpu/rfc/i915_small_bar.rst
> > > > @@ -0,0 +1,58 @@
> > > > +==========================
> > > > +I915 Small BAR RFC Section
> > > > +==========================
> > > > +Starting from DG2 we will have resizable BAR support for device
> > > > local-memory(i.e
> > > > +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size
> > > > might still be
> > > > +smaller than the total probed_size. In such cases, only some subset of
> > > > +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the
> > > > first 256M),
> > > > +while the remainder is only accessible via the GPU.
> > > > +
> > > > +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
> > > > +----------------------------------------------
> > > > +New gem_create_ext flag to tell the kernel that a BO will
> > > > require CPU access.
> > > > +This becomes important when placing an object in
> > > > I915_MEMORY_CLASS_DEVICE, where
> > > > +underneath the device has a small BAR, meaning only some
> > > > portion of it is CPU
> > > > +accessible. Without this flag the kernel will assume that CPU
> > > > access is not
> > > > +required, and prioritize using the non-CPU visible portion of
> > > > +I915_MEMORY_CLASS_DEVICE.
> > > > +
> > > > +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> > > > +   :functions: __drm_i915_gem_create_ext
> > > > +
> > > > +probed_cpu_visible_size attribute
> > > > +---------------------------------
> > > > +New struct__drm_i915_memory_region attribute which returns the
> > > > total size of the
> > > > +CPU accessible portion, for the particular region. This should only be
> > > > +applicable for I915_MEMORY_CLASS_DEVICE.
> > > > +
> > > > +Vulkan will need this as part of creating a separate
> > > > VkMemoryHeap with the
> > > > +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU
> > > > visible portion,
> > > > +where the total size of the heap needs to be known.
> > > > +
> > > > +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> > > > +   :functions: __drm_i915_memory_region_info
> > > > +
> > > > +DRM_I915_QUERY_VMA_INFO query
> > > > +-----------------------------
> > > > +Query the attributes of some vma. Given a vm and GTT offset, find the
> > > > +respective vma, and return its set of attributes. For now we
> > > > only support
> > > > +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
> > > > +currently placed in memory that is accessible by the CPU. This
> > > > should always be
> > > > +set on devices where the CPU probed_cpu_visible_size of
> > > > I915_MEMORY_CLASS_DEVICE
> > > > +matches the probed_size. If this is not set then CPU faulting
> > > > the object will
> > > > +likely first require migrating the pages.
> > > > +
> > > > +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> > > > +   :functions: __drm_i915_query_vma_info
> > > > +
> > > > +Error Capture restrictions
> > > > +--------------------------
> > > > +With error capture we have two new restrictions:
> > > > +
> > > > +    1) Error capture is best effort on small BAR systems; if
> > > > the pages are not
> > > > +    CPU accessible, at the time of capture, then the kernel is
> > > > free to skip
> > > > +    trying to capture them.
> > > > +
> > > > +    2) On discrete we now reject error capture on recoverable
> > > > contexts. In the
> > > > +    future the kernel may want to blit during error capture,
> > > > when for example
> > > > +    something is not currently CPU accessible.
> > > > diff --git a/Documentation/gpu/rfc/index.rst
> > > > b/Documentation/gpu/rfc/index.rst
> > > > index 91e93a705230..5a3bd3924ba6 100644
> > > > --- a/Documentation/gpu/rfc/index.rst
> > > > +++ b/Documentation/gpu/rfc/index.rst
> > > > @@ -23,3 +23,7 @@ host such documentation:
> > > >   .. toctree::
> > > >         i915_scheduler.rst
> > > > +
> > > > +.. toctree::
> > > > +
> > > > +    i915_small_bar.rst
> > > 
> > > 
> > 
>

Matthew Auld April 27, 2022, 5:36 p.m. UTC | #10

On 27/04/2022 09:36, Tvrtko Ursulin wrote:
> 
> On 20/04/2022 18:13, Matthew Auld wrote:
>> Add an entry for the new uapi needed for small BAR on DG2+.
>>
>> v2:
>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>    - Rework error capture interactions, including no longer needing
>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>    - Add probed_cpu_visible_size. (Lionel)
>>
>> Signed-off-by: Matthew Auld <[e-mail]>
>> Cc: Thomas Hellström <[e-mail]>
>> Cc: Lionel Landwerlin <[e-mail]>
>> Cc: Jon Bloomfield <[e-mail]>
>> Cc: Daniel Vetter <[e-mail]>
>> Cc: Jordan Justen <[e-mail]>
>> Cc: Kenneth Graunke <[e-mail]>
>> Cc: Akeem G Abodunrin <[e-mail]>
>> Cc: [e-mail]
>> ---
>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>   Documentation/gpu/rfc/index.rst          |   4 +
>>   3 files changed, 252 insertions(+)
>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>
>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>> b/Documentation/gpu/rfc/i915_small_bar.h
>> new file mode 100644
>> index 000000000000..7bfd0cf44d35
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>> @@ -0,0 +1,190 @@
>> +/**
>> + * struct __drm_i915_memory_region_info - Describes one region as 
>> known to the
>> + * driver.
>> + *
>> + * Note this is using both struct drm_i915_query_item and struct 
>> drm_i915_query.
>> + * For this new query we are adding the new query id 
>> DRM_I915_QUERY_MEMORY_REGIONS
>> + * at &drm_i915_query_item.query_id.
>> + */
>> +struct __drm_i915_memory_region_info {
>> +    /** @region: The class:instance pair encoding */
>> +    struct drm_i915_gem_memory_class_instance region;
>> +
>> +    /** @rsvd0: MBZ */
>> +    __u32 rsvd0;
>> +
>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>> +    __u64 probed_size;
>> +
>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>> unknown) */
>> +    __u64 unallocated_size;
>> +
>> +    union {
>> +        /** @rsvd1: MBZ */
>> +        __u64 rsvd1[8];
>> +        struct {
>> +            /**
>> +             * @probed_cpu_visible_size: Memory probed by the driver
>> +             * that is CPU accessible. (-1 = unknown).
>> +             *
>> +             * This will be always be <= @probed_size, and the
>> +             * remainder(if there is any) will not be CPU
>> +             * accessible.
>> +             */
>> +            __u64 probed_cpu_visible_size;
> 
> Would unallocated_cpu_visible_size be useful, to follow the total 
> unallocated_size?

Make sense. But I don't think unallocated_size has actually been 
properly wired up yet. It still just gives the same value as 
probed_size. IIRC for unallocated_size we still need a real 
user/usecase/umd, before wiring that up for real with the existing avail 
tracking. Once we have that we can also add unallocated_cpu_visible_size.

> 
> Btw, have we ever considered whether unallocated_size should require 
> CAP_SYS_ADMIN/PERFMON or something?

Note sure. But just in case we do add it for real at some point, why the 
added restriction?

> 
>> +        };
>> +    };
>> +};
>> +
>> +/**
>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, 
>> with added
>> + * extension support using struct i915_user_extension.
>> + *
>> + * Note that new buffer flags should be added here, at least for the 
>> stuff that
>> + * is immutable. Previously we would have two ioctls, one to create 
>> the object
>> + * with gem_create, and another to apply various parameters, however 
>> this
>> + * creates some ambiguity for the params which are considered 
>> immutable. Also in
>> + * general we're phasing out the various SET/GET ioctls.
>> + */
>> +struct __drm_i915_gem_create_ext {
>> +    /**
>> +     * @size: Requested size for the object.
>> +     *
>> +     * The (page-aligned) allocated size for the object will be 
>> returned.
>> +     *
>> +     * Note that for some devices we have might have further minimum
>> +     * page-size restrictions(larger than 4K), likefor device 
>> local-memory.
>> +     * However in general the final size here should always reflect any
>> +     * rounding up, if for example using the 
>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>> +     * extension to place the object in device local-memory.
>> +     */
>> +    __u64 size;
>> +    /**
>> +     * @handle: Returned handle for the object.
>> +     *
>> +     * Object handles are nonzero.
>> +     */
>> +    __u32 handle;
>> +    /**
>> +     * @flags: Optional flags.
>> +     *
>> +     * Supported values:
>> +     *
>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>> kernel that
>> +     * the object will need to be accessed via the CPU.
>> +     *
>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
>> +     * only strictly required on platforms where only some of the device
>> +     * memory is directly visible or mappable through the CPU, like 
>> on DG2+.
>> +     *
>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>> +     * ensure we can always spill the allocation tosystem memory, if we
>> +     * can't place the object in the mappable part of
>> +     * I915_MEMORY_CLASS_DEVICE.
>> +     *
>> +     * Note that since the kernel only supports flat-CCS on objects 
>> that can
>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>> +     * flat-CCS.
>> +     *
>> +     * Without this hint, the kernel will assume that non-mappable
>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>> that the
>> +     * kernel can still migrate the object to the mappable part, as a 
>> last
>> +     * resort, if userspace ever CPU faults this object, but this 
>> might be
>> +     * expensive, and so ideally should be avoided.
>> +     */
> 
> So "needs_cpu_access" flag could almost be viewed as a sub-region 
> placement priority? What I mean is this:
> 
> 1)
> placements=device,system flags=
> 
> This results in placement priorities: device, device_cpu_mappable, system.

Yup.

> 
> 2)
> placements=device,system flags=needs_cpu_access
> 
> This results in placement priorities: device_cpu_mappable, device, system.

Here it would only be: device_cpu_mappable, system. We would completely 
ignore "device" in this case.

> 
> Is this correct?
> 
> The benefit of the flag is that i915 can place the object to the right 
> place from the start instead of on the first CPU access? Is that worth 
> it or is there more to it?

Yeah, the object will only be placed somewhere that is also CPU 
mappable, with the flag set.

> 
>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>> +    __u32 flags;
>> +    /**
>> +     * @extensions: The chain of extensions to apply to this object.
>> +     *
>> +     * This will be useful in the future when we need to support several
>> +     * different extensions, and we need to apply more than one when
>> +     * creating the object. See struct i915_user_extension.
>> +     *
>> +     * If we don't supply any extensions then we get the same old 
>> gem_create
>> +     * behaviour.
>> +     *
>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usagesee
>> +     * struct drm_i915_gem_create_ext_memory_regions.
>> +     *
>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>> +     * struct drm_i915_gem_create_ext_protected_content.
>> +     */
>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>> +    __u64 extensions;
>> +};
>> +
>> +#define DRM_I915_QUERY_VMA_INFO    5
>> +
>> +/**
>> + * struct __drm_i915_query_vma_info
>> + *
>> + * Given a vm and GTT address, lookup the corresponding vma, 
>> returning its set
>> + * of attributes.
>> + *
>> + * .. code-block:: C
>> + *
>> + *    struct drm_i915_query_vma_info info = {};
>> + *    struct drm_i915_query_item item = {
>> + *        .data_ptr = (uintptr_t)&info,
>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>> + *    };
>> + *    struct drm_i915_query query = {
>> + *        .num_items = 1,
>> + *        .items_ptr = (uintptr_t)&item,
>> + *    };
>> + *    int err;
>> + *
>> + *    // Unlike some other types of queries, there is noneed to 
>> first query
>> + *    // the size of the data_ptr blob here, since we already know 
>> ahead of
>> + *    // time how big this needs to be.
>> + *    item.length = sizeof(info);
>> + *
>> + *    // Next we fill in the vm_id and ppGTT address of the vma we wish
>> + *    // to query, before then firing off the query.
>> + *    info.vm_id = vm_id;
>> + *    info.offset = gtt_address;
>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>> + *    if (err || item.length < 0) ...
>> + *
>> + *    // If all went well we can now inspect the returned attributes.
>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>> + */
>> +struct __drm_i915_query_vma_info {
>> +    /**
>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>> value
>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>> +     * drm_i915_gem_vm_control.vm_id.
>> +     */
>> +    __u32 vm_id;
>> +    /** @pad: MBZ. */
>> +    __u32 pad;
>> +    /**
>> +     * @offset: The corresponding ppGTT address of the vma which the 
>> kernel
>> +     * will use to perform the lookup.
>> +     */
>> +    __u64 offset;
>> +    /**
>> +     * @attributes: The returned attributes for thegiven vma.
>> +     *
>> +     * Possible values:
>> +     *
>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set ifthe pages backing 
>> the
>> +     * vma are currently CPU accessible. If this isnot set then the 
>> vma is
>> +     * currently backed by I915_MEMORY_CLASS_DEVICEmemory, which the 
>> CPU
>> +     * cannot directly access(this is only possibleon discrete 
>> devices with
>> +     * a small BAR). Attempting to MMAP and fault such an object will
>> +     * require the kernel first synchronising any GPU work tied to the
>> +     * object, before then migrating the pages, either to the CPU 
>> accessible
>> +     * part of I915_MEMORY_CLASS_DEVICE, or I915_MEMORY_CLASS_SYSTEM, 
>> if the
>> +     * placements permit it. See 
>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>> +     *
>> +     * Note that this is inherently racy.
>> +     */
>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>> +    __u64 attributes;
>> +    /** @rsvd: MBZ */
>> +    __u32 rsvd[4];
>> +};
>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>> b/Documentation/gpu/rfc/i915_small_bar.rst
>> new file mode 100644
>> index 000000000000..be3d9bcdd86d
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>> @@ -0,0 +1,58 @@
>> +==========================
>> +I915 Small BAR RFC Section
>> +==========================
>> +Starting from DG2 we will have resizable BAR support for device 
>> local-memory(i.e
>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size might 
>> still be
>> +smaller than the total probed_size. In such cases, only some subset of
>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the first 
>> 256M),
>> +while the remainder is only accessible via the GPU.
>> +
>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>> +----------------------------------------------
>> +New gem_create_ext flag to tell the kernel that a BO will require CPU 
>> access.
>> +This becomes important when placing an object in 
>> I915_MEMORY_CLASS_DEVICE, where
>> +underneath the device has a small BAR, meaning only some portion of 
>> it is CPU
>> +accessible. Without this flag the kernel will assume that CPU access 
>> is not
>> +required, and prioritize using the non-CPU visible portion of
>> +I915_MEMORY_CLASS_DEVICE.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_gem_create_ext
>> +
>> +probed_cpu_visible_size attribute
>> +---------------------------------
>> +New struct__drm_i915_memory_region attribute which returns the total 
>> size of the
>> +CPU accessible portion, for the particular region. This should only be
>> +applicable for I915_MEMORY_CLASS_DEVICE.
>> +
>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>> with the
>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU visible 
>> portion,
>> +where the total size of the heap needs to be known.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_memory_region_info
>> +
>> +DRM_I915_QUERY_VMA_INFO query
>> +-----------------------------
>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>> +respective vma, and return its set of attributes. For now we only 
>> support
>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>> +currently placed in memory that is accessible by the CPU. This should 
>> always be
>> +set on devices where the CPU probed_cpu_visible_size of 
>> I915_MEMORY_CLASS_DEVICE
>> +matches the probed_size. If this is not set then CPU faulting the 
>> object will
>> +likely first require migrating the pages.
> 
> I think there should be justification for the new query documented as 
> well. (Why on top of what.)

Yeah, I'm wondering now if we can just drop this part of the uapi, for 
now at least, and focus on landing the new flag stuff first.

> 
> Without it personally I can't immediately understand why the disconnect 
> between the object based and VMA based API. Userspace has to do some 
> intervening operations like either execbuf, or vm bind in the future, to 
> make this query usable after object creation. So question is why 
> wouldn't it know already which placements it allowed and so would i915 
> auto-migrate or not for this particular object. No? Or in other words 
> why this wouldn't be an object based query since the question it is 
> answering is about the object backing store and not the VMA.

Yeah, just using the object handle or so I guess would also work. Thanks 
for the comments.

> 
> Regards,
> 
> Tvrtko
> 
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_query_vma_info
>> +
>> +Error Capture restrictions
>> +--------------------------
>> +With error capture we have two new restrictions:
>> +
>> +    1) Error capture is best effort on small BAR systems; if the 
>> pages are not
>> +    CPU accessible, at the time of capture, then the kernel is free 
>> to skip
>> +    trying to capture them.
>> +
>> +    2) On discrete we now reject error capture on recoverable 
>> contexts. In the
>> +    future the kernel may want to blit during error capture, when for 
>> example
>> +    something is not currently CPU accessible.
>> diff --git a/Documentation/gpu/rfc/index.rst 
>> b/Documentation/gpu/rfc/index.rst
>> index 91e93a705230..5a3bd3924ba6 100644
>> --- a/Documentation/gpu/rfc/index.rst
>> +++ b/Documentation/gpu/rfc/index.rst
>> @@ -23,3 +23,7 @@ host such documentation:
>>   .. toctree::
>>       i915_scheduler.rst
>> +
>> +.. toctree::
>> +
>> +    i915_small_bar.rst

Tvrtko Ursulin April 28, 2022, 8:55 a.m. UTC | #11

On 27/04/2022 18:36, Matthew Auld wrote:
> On 27/04/2022 09:36, Tvrtko Ursulin wrote:
>>
>> On 20/04/2022 18:13, Matthew Auld wrote:
>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>
>>> v2:
>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>    - Rework error capture interactions, including no longer needing
>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>    - Add probed_cpu_visible_size. (Lionel)
>>>
>>> Signed-off-by: Matthew Auld <[e-mail]>
>>> Cc: Thomas Hellström <[e-mail]>
>>> Cc: Lionel Landwerlin <[e-mail]>
>>> Cc: Jon Bloomfield <[e-mail]>
>>> Cc: Daniel Vetter <[e-mail]>
>>> Cc: Jordan Justen <[e-mail]>
>>> Cc: Kenneth Graunke <[e-mail]>
>>> Cc: Akeem G Abodunrin <[e-mail]>
>>> Cc: [e-mail]
>>> ---
>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>   3 files changed, 252 insertions(+)
>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>> new file mode 100644
>>> index 000000000000..7bfd0cf44d35
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>> @@ -0,0 +1,190 @@
>>> +/**
>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>> known to the
>>> + * driver.
>>> + *
>>> + * Note this is using both struct drm_i915_query_item and struct 
>>> drm_i915_query.
>>> + * For this new query we are adding the new query id 
>>> DRM_I915_QUERY_MEMORY_REGIONS
>>> + * at &drm_i915_query_item.query_id.
>>> + */
>>> +struct __drm_i915_memory_region_info {
>>> +    /** @region: The class:instance pair encoding */
>>> +    struct drm_i915_gem_memory_class_instance region;
>>> +
>>> +    /** @rsvd0: MBZ */
>>> +    __u32 rsvd0;
>>> +
>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>> +    __u64 probed_size;
>>> +
>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>> unknown) */
>>> +    __u64 unallocated_size;
>>> +
>>> +    union {
>>> +        /** @rsvd1: MBZ */
>>> +        __u64 rsvd1[8];
>>> +        struct {
>>> +            /**
>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>> +             * that is CPU accessible. (-1 = unknown).
>>> +             *
>>> +             * This will be always be <= @probed_size, and the
>>> +             * remainder(if there is any) will not be CPU
>>> +             * accessible.
>>> +             */
>>> +            __u64 probed_cpu_visible_size;
>>
>> Would unallocated_cpu_visible_size be useful, to follow the total 
>> unallocated_size?
> 
> Make sense. But I don't think unallocated_size has actually been 
> properly wired up yet. It still just gives the same value as 
> probed_size. IIRC for unallocated_size we still need a real 
> user/usecase/umd, before wiring that up for real with the existing avail 
> tracking. Once we have that we can also add unallocated_cpu_visible_size.

So this does nothing at the moment:

  info.unallocated_size = mr->avail;

Right, it is set to "mem->avail = mem->total;" at region init time and I 
indeed can't find it ever getting modified. Okay.

>> Btw, have we ever considered whether unallocated_size should require 
>> CAP_SYS_ADMIN/PERFMON or something?
> 
> Note sure. But just in case we do add it for real at some point, why the 
> added restriction?

To avoid a side channel, albeit perhaps a very weak one. For engine 
utilization we require CAP_SYS_PERFMON, but that is implied by the perf 
core API. It's open for discussion. I guess it may make sense to limit 
it also because it is questionable the field(s) are even useful.

> 
>>
>>> +        };
>>> +    };
>>> +};
>>> +
>>> +/**
>>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, 
>>> with added
>>> + * extension support using struct i915_user_extension.
>>> + *
>>> + * Note that new buffer flags should be added here, at least for the 
>>> stuff that
>>> + * is immutable. Previously we would have two ioctls, one to create 
>>> the object
>>> + * with gem_create, and another to apply various parameters, however 
>>> this
>>> + * creates some ambiguity for the params which are considered 
>>> immutable. Also in
>>> + * general we're phasing out the various SET/GET ioctls.
>>> + */
>>> +struct __drm_i915_gem_create_ext {
>>> +    /**
>>> +     * @size: Requested size for the object.
>>> +     *
>>> +     * The (page-aligned) allocated size for the object will be 
>>> returned.
>>> +     *
>>> +     * Note that for some devices we have might have further minimum
>>> +     * page-size restrictions(larger than 4K), likefor device 
>>> local-memory.
>>> +     * However in general the final size here should always reflect any
>>> +     * rounding up, if for example using the 
>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>> +     * extension to place the object in device local-memory.
>>> +     */
>>> +    __u64 size;
>>> +    /**
>>> +     * @handle: Returned handle for the object.
>>> +     *
>>> +     * Object handles are nonzero.
>>> +     */
>>> +    __u32 handle;
>>> +    /**
>>> +     * @flags: Optional flags.
>>> +     *
>>> +     * Supported values:
>>> +     *
>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>> kernel that
>>> +     * the object will need to be accessed via the CPU.
>>> +     *
>>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
>>> +     * only strictly required on platforms where only some of the 
>>> device
>>> +     * memory is directly visible or mappable through the CPU, like 
>>> on DG2+.
>>> +     *
>>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>>> +     * ensure we can always spill the allocation tosystem memory, if we
>>> +     * can't place the object in the mappable part of
>>> +     * I915_MEMORY_CLASS_DEVICE.
>>> +     *
>>> +     * Note that since the kernel only supports flat-CCS on objects 
>>> that can
>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>>> +     * flat-CCS.
>>> +     *
>>> +     * Without this hint, the kernel will assume that non-mappable
>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>> that the
>>> +     * kernel can still migrate the object to the mappable part, as 
>>> a last
>>> +     * resort, if userspace ever CPU faults this object, but this 
>>> might be
>>> +     * expensive, and so ideally should be avoided.
>>> +     */
>>
>> So "needs_cpu_access" flag could almost be viewed as a sub-region 
>> placement priority? What I mean is this:
>>
>> 1)
>> placements=device,system flags=
>>
>> This results in placement priorities: device, device_cpu_mappable, 
>> system.
> 
> Yup.
> 
>>
>> 2)
>> placements=device,system flags=needs_cpu_access
>>
>> This results in placement priorities: device_cpu_mappable, device, 
>> system.
> 
> Here it would only be: device_cpu_mappable, system. We would completely 
> ignore "device" in this case.
> 
>>
>> Is this correct?
>>
>> The benefit of the flag is that i915 can place the object to the right 
>> place from the start instead of on the first CPU access? Is that worth 
>> it or is there more to it?
> 
> Yeah, the object will only be placed somewhere that is also CPU 
> mappable, with the flag set.

Hm, wouldn't it be more efficient to be able to migrate it over to 
non-mappable in cases when mappable is over-subscribed?

>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>> +    __u32 flags;
>>> +    /**
>>> +     * @extensions: The chain of extensions to apply to this object.
>>> +     *
>>> +     * This will be useful in the future when we need to support 
>>> several
>>> +     * different extensions, and we need to apply more than one when
>>> +     * creating the object. See struct i915_user_extension.
>>> +     *
>>> +     * If we don't supply any extensions then we get the same old 
>>> gem_create
>>> +     * behaviour.
>>> +     *
>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usagesee
>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>> +     *
>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>> +     */
>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>> +    __u64 extensions;
>>> +};
>>> +
>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>> +
>>> +/**
>>> + * struct __drm_i915_query_vma_info
>>> + *
>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>> returning its set
>>> + * of attributes.
>>> + *
>>> + * .. code-block:: C
>>> + *
>>> + *    struct drm_i915_query_vma_info info = {};
>>> + *    struct drm_i915_query_item item = {
>>> + *        .data_ptr = (uintptr_t)&info,
>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>> + *    };
>>> + *    struct drm_i915_query query = {
>>> + *        .num_items = 1,
>>> + *        .items_ptr = (uintptr_t)&item,
>>> + *    };
>>> + *    int err;
>>> + *
>>> + *    // Unlike some other types of queries, there is noneed to 
>>> first query
>>> + *    // the size of the data_ptr blob here, since we already know 
>>> ahead of
>>> + *    // time how big this needs to be.
>>> + *    item.length = sizeof(info);
>>> + *
>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we wish
>>> + *    // to query, before then firing off the query.
>>> + *    info.vm_id = vm_id;
>>> + *    info.offset = gtt_address;
>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>> + *    if (err || item.length < 0) ...
>>> + *
>>> + *    // If all went well we can now inspect the returned attributes.
>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>> + */
>>> +struct __drm_i915_query_vma_info {
>>> +    /**
>>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>>> value
>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>> +     * drm_i915_gem_vm_control.vm_id.
>>> +     */
>>> +    __u32 vm_id;
>>> +    /** @pad: MBZ. */
>>> +    __u32 pad;
>>> +    /**
>>> +     * @offset: The corresponding ppGTT address of the vma which the 
>>> kernel
>>> +     * will use to perform the lookup.
>>> +     */
>>> +    __u64 offset;
>>> +    /**
>>> +     * @attributes: The returned attributes for thegiven vma.
>>> +     *
>>> +     * Possible values:
>>> +     *
>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set ifthe pages backing 
>>> the
>>> +     * vma are currently CPU accessible. If this isnot set then the 
>>> vma is
>>> +     * currently backed by I915_MEMORY_CLASS_DEVICEmemory, which the 
>>> CPU
>>> +     * cannot directly access(this is only possibleon discrete 
>>> devices with
>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>> +     * require the kernel first synchronising any GPU work tied to the
>>> +     * object, before then migrating the pages, either to the CPU 
>>> accessible
>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>> I915_MEMORY_CLASS_SYSTEM, if the
>>> +     * placements permit it. See 
>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>> +     *
>>> +     * Note that this is inherently racy.
>>> +     */
>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>> +    __u64 attributes;
>>> +    /** @rsvd: MBZ */
>>> +    __u32 rsvd[4];
>>> +};
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>> new file mode 100644
>>> index 000000000000..be3d9bcdd86d
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>> @@ -0,0 +1,58 @@
>>> +==========================
>>> +I915 Small BAR RFC Section
>>> +==========================
>>> +Starting from DG2 we will have resizable BAR support for device 
>>> local-memory(i.e
>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>> might still be
>>> +smaller than the total probed_size. In such cases, only some subset of
>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>> first 256M),
>>> +while the remainder is only accessible via the GPU.
>>> +
>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>> +----------------------------------------------
>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>> CPU access.
>>> +This becomes important when placing an object in 
>>> I915_MEMORY_CLASS_DEVICE, where
>>> +underneath the device has a small BAR, meaning only some portion of 
>>> it is CPU
>>> +accessible. Without this flag the kernel will assume that CPU access 
>>> is not
>>> +required, and prioritize using the non-CPU visible portion of
>>> +I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_gem_create_ext
>>> +
>>> +probed_cpu_visible_size attribute
>>> +---------------------------------
>>> +New struct__drm_i915_memory_region attribute which returns the total 
>>> size of the
>>> +CPU accessible portion, for the particular region. This should only be
>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>> with the
>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>> visible portion,
>>> +where the total size of the heap needs to be known.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_memory_region_info
>>> +
>>> +DRM_I915_QUERY_VMA_INFO query
>>> +-----------------------------
>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>> +respective vma, and return its set of attributes. For now we only 
>>> support
>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>>> +currently placed in memory that is accessible by the CPU. This 
>>> should always be
>>> +set on devices where the CPU probed_cpu_visible_size of 
>>> I915_MEMORY_CLASS_DEVICE
>>> +matches the probed_size. If this is not set then CPU faulting the 
>>> object will
>>> +likely first require migrating the pages.
>>
>> I think there should be justification for the new query documented as 
>> well. (Why on top of what.)
> 
> Yeah, I'm wondering now if we can just drop this part of the uapi, for 
> now at least, and focus on landing the new flag stuff first.
> 
>>
>> Without it personally I can't immediately understand why the 
>> disconnect between the object based and VMA based API. Userspace has 
>> to do some intervening operations like either execbuf, or vm bind in 
>> the future, to make this query usable after object creation. So 
>> question is why wouldn't it know already which placements it allowed 
>> and so would i915 auto-migrate or not for this particular object. No? 
>> Or in other words why this wouldn't be an object based query since the 
>> question it is answering is about the object backing store and not the 
>> VMA.
> 
> Yeah, just using the object handle or so I guess would also work. Thanks 
> for the comments.

I saw other folks have said the same so omitting for now sounds good to 
me indeed.

Regards,

Tvrtko

Matthew Auld April 28, 2022, 10:25 a.m. UTC | #12

On 28/04/2022 09:55, Tvrtko Ursulin wrote:
> 
> On 27/04/2022 18:36, Matthew Auld wrote:
>> On 27/04/2022 09:36, Tvrtko Ursulin wrote:
>>>
>>> On 20/04/2022 18:13, Matthew Auld wrote:
>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>
>>>> v2:
>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>    - Rework error capture interactions, including no longer needing
>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>
>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>> Cc: Thomas Hellström <[e-mail]>
>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>> Cc: Jon Bloomfield <[e-mail]>
>>>> Cc: Daniel Vetter <[e-mail]>
>>>> Cc: Jordan Justen <[e-mail]>
>>>> Cc: Kenneth Graunke <[e-mail]>
>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>> Cc: [e-mail]
>>>> ---
>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>> +++++++++++++++++++++++
>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>   3 files changed, 252 insertions(+)
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>> new file mode 100644
>>>> index 000000000000..7bfd0cf44d35
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>> @@ -0,0 +1,190 @@
>>>> +/**
>>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>>> known to the
>>>> + * driver.
>>>> + *
>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>> drm_i915_query.
>>>> + * For this new query we are adding the new query id 
>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>> + * at &drm_i915_query_item.query_id.
>>>> + */
>>>> +struct __drm_i915_memory_region_info {
>>>> +    /** @region: The class:instance pair encoding */
>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>> +
>>>> +    /** @rsvd0: MBZ */
>>>> +    __u32 rsvd0;
>>>> +
>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>> +    __u64 probed_size;
>>>> +
>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>> unknown) */
>>>> +    __u64 unallocated_size;
>>>> +
>>>> +    union {
>>>> +        /** @rsvd1: MBZ */
>>>> +        __u64 rsvd1[8];
>>>> +        struct {
>>>> +            /**
>>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>>> +             * that is CPU accessible. (-1 = unknown).
>>>> +             *
>>>> +             * This will be always be <= @probed_size, and the
>>>> +             * remainder(if there is any) will not be CPU
>>>> +             * accessible.
>>>> +             */
>>>> +            __u64 probed_cpu_visible_size;
>>>
>>> Would unallocated_cpu_visible_size be useful, to follow the total 
>>> unallocated_size?
>>
>> Make sense. But I don't think unallocated_size has actually been 
>> properly wired up yet. It still just gives the same value as 
>> probed_size. IIRC for unallocated_size we still need a real 
>> user/usecase/umd, before wiring that up for real with the existing 
>> avail tracking. Once we have that we can also add 
>> unallocated_cpu_visible_size.
> 
> So this does nothing at the moment:
> 
>   info.unallocated_size = mr->avail;
> 
> Right, it is set to "mem->avail = mem->total;" at region init time and I 
> indeed can't find it ever getting modified. Okay.
> 
>>> Btw, have we ever considered whether unallocated_size should require 
>>> CAP_SYS_ADMIN/PERFMON or something?
>>
>> Note sure. But just in case we do add it for real at some point, why 
>> the added restriction?
> 
> To avoid a side channel, albeit perhaps a very weak one. For engine 
> utilization we require CAP_SYS_PERFMON, but that is implied by the perf 
> core API. It's open for discussion. I guess it may make sense to limit 
> it also because it is questionable the field(s) are even useful.
> 
>>
>>>
>>>> +        };
>>>> +    };
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>> behaviour, with added
>>>> + * extension support using struct i915_user_extension.
>>>> + *
>>>> + * Note that new buffer flags should be added here, at least for 
>>>> the stuff that
>>>> + * is immutable. Previously we would have two ioctls, one to create 
>>>> the object
>>>> + * with gem_create, and another to apply various parameters, 
>>>> however this
>>>> + * creates some ambiguity for the params which are considered 
>>>> immutable. Also in
>>>> + * general we're phasing out the various SET/GET ioctls.
>>>> + */
>>>> +struct __drm_i915_gem_create_ext {
>>>> +    /**
>>>> +     * @size: Requested size for the object.
>>>> +     *
>>>> +     * The (page-aligned) allocated size for the object will be 
>>>> returned.
>>>> +     *
>>>> +     * Note that for some devices we have might have further minimum
>>>> +     * page-size restrictions(larger than 4K), likefor device 
>>>> local-memory.
>>>> +     * However in general the final size here should always reflect 
>>>> any
>>>> +     * rounding up, if for example using the 
>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>> +     * extension to place the object in device local-memory.
>>>> +     */
>>>> +    __u64 size;
>>>> +    /**
>>>> +     * @handle: Returned handle for the object.
>>>> +     *
>>>> +     * Object handles are nonzero.
>>>> +     */
>>>> +    __u32 handle;
>>>> +    /**
>>>> +     * @flags: Optional flags.
>>>> +     *
>>>> +     * Supported values:
>>>> +     *
>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>> kernel that
>>>> +     * the object will need to be accessed via the CPU.
>>>> +     *
>>>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, 
>>>> and
>>>> +     * only strictly required on platforms where only some of the 
>>>> device
>>>> +     * memory is directly visible or mappable through the CPU, like 
>>>> on DG2+.
>>>> +     *
>>>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>>>> +     * ensure we can always spill the allocation tosystem memory, 
>>>> if we
>>>> +     * can't place the object in the mappable part of
>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>> +     *
>>>> +     * Note that since the kernel only supports flat-CCS on objects 
>>>> that can
>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>>> don't
>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>>>> +     * flat-CCS.
>>>> +     *
>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>>> that the
>>>> +     * kernel can still migrate the object to the mappable part, as 
>>>> a last
>>>> +     * resort, if userspace ever CPU faults this object, but this 
>>>> might be
>>>> +     * expensive, and so ideally should be avoided.
>>>> +     */
>>>
>>> So "needs_cpu_access" flag could almost be viewed as a sub-region 
>>> placement priority? What I mean is this:
>>>
>>> 1)
>>> placements=device,system flags=
>>>
>>> This results in placement priorities: device, device_cpu_mappable, 
>>> system.
>>
>> Yup.
>>
>>>
>>> 2)
>>> placements=device,system flags=needs_cpu_access
>>>
>>> This results in placement priorities: device_cpu_mappable, device, 
>>> system.
>>
>> Here it would only be: device_cpu_mappable, system. We would 
>> completely ignore "device" in this case.
>>
>>>
>>> Is this correct?
>>>
>>> The benefit of the flag is that i915 can place the object to the 
>>> right place from the start instead of on the first CPU access? Is 
>>> that worth it or is there more to it?
>>
>> Yeah, the object will only be placed somewhere that is also CPU 
>> mappable, with the flag set.
> 
> Hm, wouldn't it be more efficient to be able to migrate it over to 
> non-mappable in cases when mappable is over-subscribed?

Not sure. As an alternative strategy, I guess that might be interesting, 
and if userspace wants something like that we can always add a new flag 
I guess? It's a toss up whether just using system memory is better/worse 
than incurring an extra move at fault time?

> 
>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>> +    __u32 flags;
>>>> +    /**
>>>> +     * @extensions: The chain of extensions to apply to this object.
>>>> +     *
>>>> +     * This will be useful in the future when we need to support 
>>>> several
>>>> +     * different extensions, and we need to apply more than one when
>>>> +     * creating the object. See struct i915_user_extension.
>>>> +     *
>>>> +     * If we don't supply any extensions then we get the same old 
>>>> gem_create
>>>> +     * behaviour.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usagesee
>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>> +     */
>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>> +    __u64 extensions;
>>>> +};
>>>> +
>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_query_vma_info
>>>> + *
>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>> returning its set
>>>> + * of attributes.
>>>> + *
>>>> + * .. code-block:: C
>>>> + *
>>>> + *    struct drm_i915_query_vma_info info = {};
>>>> + *    struct drm_i915_query_item item = {
>>>> + *        .data_ptr = (uintptr_t)&info,
>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>> + *    };
>>>> + *    struct drm_i915_query query = {
>>>> + *        .num_items = 1,
>>>> + *        .items_ptr = (uintptr_t)&item,
>>>> + *    };
>>>> + *    int err;
>>>> + *
>>>> + *    // Unlike some other types of queries, there is noneed to 
>>>> first query
>>>> + *    // the size of the data_ptr blob here, since we already know 
>>>> ahead of
>>>> + *    // time how big this needs to be.
>>>> + *    item.length = sizeof(info);
>>>> + *
>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we 
>>>> wish
>>>> + *    // to query, before then firing off the query.
>>>> + *    info.vm_id = vm_id;
>>>> + *    info.offset = gtt_address;
>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>> + *    if (err || item.length < 0) ...
>>>> + *
>>>> + *    // If all went well we can now inspect the returned attributes.
>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>> + */
>>>> +struct __drm_i915_query_vma_info {
>>>> +    /**
>>>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>>>> value
>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>> +     */
>>>> +    __u32 vm_id;
>>>> +    /** @pad: MBZ. */
>>>> +    __u32 pad;
>>>> +    /**
>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>> the kernel
>>>> +     * will use to perform the lookup.
>>>> +     */
>>>> +    __u64 offset;
>>>> +    /**
>>>> +     * @attributes: The returned attributes for thegiven vma.
>>>> +     *
>>>> +     * Possible values:
>>>> +     *
>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set ifthe pages 
>>>> backing the
>>>> +     * vma are currently CPU accessible. If this isnot set then the 
>>>> vma is
>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICEmemory, which 
>>>> the CPU
>>>> +     * cannot directly access(this is only possibleon discrete 
>>>> devices with
>>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>>> +     * require the kernel first synchronising any GPU work tied to the
>>>> +     * object, before then migrating the pages, either to the CPU 
>>>> accessible
>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>> +     * placements permit it. See 
>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>> +     *
>>>> +     * Note that this is inherently racy.
>>>> +     */
>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>> +    __u64 attributes;
>>>> +    /** @rsvd: MBZ */
>>>> +    __u32 rsvd[4];
>>>> +};
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> new file mode 100644
>>>> index 000000000000..be3d9bcdd86d
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> @@ -0,0 +1,58 @@
>>>> +==========================
>>>> +I915 Small BAR RFC Section
>>>> +==========================
>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>> local-memory(i.e
>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>> might still be
>>>> +smaller than the total probed_size. In such cases, only some subset of
>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>> first 256M),
>>>> +while the remainder is only accessible via the GPU.
>>>> +
>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>> +----------------------------------------------
>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>> CPU access.
>>>> +This becomes important when placing an object in 
>>>> I915_MEMORY_CLASS_DEVICE, where
>>>> +underneath the device has a small BAR, meaning only some portion of 
>>>> it is CPU
>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>> access is not
>>>> +required, and prioritize using the non-CPU visible portion of
>>>> +I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_gem_create_ext
>>>> +
>>>> +probed_cpu_visible_size attribute
>>>> +---------------------------------
>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>> total size of the
>>>> +CPU accessible portion, for the particular region. This should only be
>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>>> with the
>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>> visible portion,
>>>> +where the total size of the heap needs to be known.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_memory_region_info
>>>> +
>>>> +DRM_I915_QUERY_VMA_INFO query
>>>> +-----------------------------
>>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>>> +respective vma, and return its set of attributes. For now we only 
>>>> support
>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>>>> +currently placed in memory that is accessible by the CPU. This 
>>>> should always be
>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>> I915_MEMORY_CLASS_DEVICE
>>>> +matches the probed_size. If this is not set then CPU faulting the 
>>>> object will
>>>> +likely first require migrating the pages.
>>>
>>> I think there should be justification for the new query documented as 
>>> well. (Why on top of what.)
>>
>> Yeah, I'm wondering now if we can just drop this part of the uapi, for 
>> now at least, and focus on landing the new flag stuff first.
>>
>>>
>>> Without it personally I can't immediately understand why the 
>>> disconnect between the object based and VMA based API. Userspace has 
>>> to do some intervening operations like either execbuf, or vm bind in 
>>> the future, to make this query usable after object creation. So 
>>> question is why wouldn't it know already which placements it allowed 
>>> and so would i915 auto-migrate or not for this particular object. No? 
>>> Or in other words why this wouldn't be an object based query since 
>>> the question it is answering is about the object backing store and 
>>> not the VMA.
>>
>> Yeah, just using the object handle or so I guess would also work. 
>> Thanks for the comments.
> 
> I saw other folks have said the same so omitting for now sounds good to 
> me indeed.
> 
> Regards,
> 
> Tvrtko

Tvrtko Ursulin April 28, 2022, 11:11 a.m. UTC | #13

On 28/04/2022 11:25, Matthew Auld wrote:
> On 28/04/2022 09:55, Tvrtko Ursulin wrote:
>>
>> On 27/04/2022 18:36, Matthew Auld wrote:
>>> On 27/04/2022 09:36, Tvrtko Ursulin wrote:
>>>>
>>>> On 20/04/2022 18:13, Matthew Auld wrote:
>>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>>
>>>>> v2:
>>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>>    - Rework error capture interactions, including no longer needing
>>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>>
>>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>>> Cc: Thomas Hellström <[e-mail]>
>>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>>> Cc: Jon Bloomfield <[e-mail]>
>>>>> Cc: Daniel Vetter <[e-mail]>
>>>>> Cc: Jordan Justen <[e-mail]>
>>>>> Cc: Kenneth Graunke <[e-mail]>
>>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>>> Cc: [e-mail]
>>>>> ---
>>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>>> +++++++++++++++++++++++
>>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>>   3 files changed, 252 insertions(+)
>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>>
>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>>> new file mode 100644
>>>>> index 000000000000..7bfd0cf44d35
>>>>> --- /dev/null
>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>>> @@ -0,0 +1,190 @@
>>>>> +/**
>>>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>>>> known to the
>>>>> + * driver.
>>>>> + *
>>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>>> drm_i915_query.
>>>>> + * For this new query we are adding the new query id 
>>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>>> + * at &drm_i915_query_item.query_id.
>>>>> + */
>>>>> +struct __drm_i915_memory_region_info {
>>>>> +    /** @region: The class:instance pair encoding */
>>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>>> +
>>>>> +    /** @rsvd0: MBZ */
>>>>> +    __u32 rsvd0;
>>>>> +
>>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>>> +    __u64 probed_size;
>>>>> +
>>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>>> unknown) */
>>>>> +    __u64 unallocated_size;
>>>>> +
>>>>> +    union {
>>>>> +        /** @rsvd1: MBZ */
>>>>> +        __u64 rsvd1[8];
>>>>> +        struct {
>>>>> +            /**
>>>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>>>> +             * that is CPU accessible. (-1 = unknown).
>>>>> +             *
>>>>> +             * This will be always be <= @probed_size, and the
>>>>> +             * remainder(if there is any) will not be CPU
>>>>> +             * accessible.
>>>>> +             */
>>>>> +            __u64 probed_cpu_visible_size;
>>>>
>>>> Would unallocated_cpu_visible_size be useful, to follow the total 
>>>> unallocated_size?
>>>
>>> Make sense. But I don't think unallocated_size has actually been 
>>> properly wired up yet. It still just gives the same value as 
>>> probed_size. IIRC for unallocated_size we still need a real 
>>> user/usecase/umd, before wiring that up for real with the existing 
>>> avail tracking. Once we have that we can also add 
>>> unallocated_cpu_visible_size.
>>
>> So this does nothing at the moment:
>>
>>   info.unallocated_size = mr->avail;
>>
>> Right, it is set to "mem->avail = mem->total;" at region init time and 
>> I indeed can't find it ever getting modified. Okay.
>>
>>>> Btw, have we ever considered whether unallocated_size should require 
>>>> CAP_SYS_ADMIN/PERFMON or something?
>>>
>>> Note sure. But just in case we do add it for real at some point, why 
>>> the added restriction?
>>
>> To avoid a side channel, albeit perhaps a very weak one. For engine 
>> utilization we require CAP_SYS_PERFMON, but that is implied by the 
>> perf core API. It's open for discussion. I guess it may make sense to 
>> limit it also because it is questionable the field(s) are even useful.
>>
>>>
>>>>
>>>>> +        };
>>>>> +    };
>>>>> +};
>>>>> +
>>>>> +/**
>>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>>> behaviour, with added
>>>>> + * extension support using struct i915_user_extension.
>>>>> + *
>>>>> + * Note that new buffer flags should be added here, at least for 
>>>>> the stuff that
>>>>> + * is immutable. Previously we would have two ioctls, one to 
>>>>> create the object
>>>>> + * with gem_create, and another to apply various parameters, 
>>>>> however this
>>>>> + * creates some ambiguity for the params which are considered 
>>>>> immutable. Also in
>>>>> + * general we're phasing out the various SET/GET ioctls.
>>>>> + */
>>>>> +struct __drm_i915_gem_create_ext {
>>>>> +    /**
>>>>> +     * @size: Requested size for the object.
>>>>> +     *
>>>>> +     * The (page-aligned) allocated size for the object will be 
>>>>> returned.
>>>>> +     *
>>>>> +     * Note that for some devices we have might have further minimum
>>>>> +     * page-size restrictions(larger than 4K), likefor device 
>>>>> local-memory.
>>>>> +     * However in general the final size here should always 
>>>>> reflect any
>>>>> +     * rounding up, if for example using the 
>>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>>> +     * extension to place the object in device local-memory.
>>>>> +     */
>>>>> +    __u64 size;
>>>>> +    /**
>>>>> +     * @handle: Returned handle for the object.
>>>>> +     *
>>>>> +     * Object handles are nonzero.
>>>>> +     */
>>>>> +    __u32 handle;
>>>>> +    /**
>>>>> +     * @flags: Optional flags.
>>>>> +     *
>>>>> +     * Supported values:
>>>>> +     *
>>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>>> kernel that
>>>>> +     * the object will need to be accessed via the CPU.
>>>>> +     *
>>>>> +     * Only valid when placing objects in 
>>>>> I915_MEMORY_CLASS_DEVICE, and
>>>>> +     * only strictly required on platforms where only some of the 
>>>>> device
>>>>> +     * memory is directly visible or mappable through the CPU, 
>>>>> like on DG2+.
>>>>> +     *
>>>>> +     * One of the placements MUST also be 
>>>>> I915_MEMORY_CLASS_SYSTEM, to
>>>>> +     * ensure we can always spill the allocation tosystem memory, 
>>>>> if we
>>>>> +     * can't place the object in the mappable part of
>>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>>> +     *
>>>>> +     * Note that since the kernel only supports flat-CCS on 
>>>>> objects that can
>>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>>>> don't
>>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together 
>>>>> with
>>>>> +     * flat-CCS.
>>>>> +     *
>>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>>>> that the
>>>>> +     * kernel can still migrate the object to the mappable part, 
>>>>> as a last
>>>>> +     * resort, if userspace ever CPU faults this object, but this 
>>>>> might be
>>>>> +     * expensive, and so ideally should be avoided.
>>>>> +     */
>>>>
>>>> So "needs_cpu_access" flag could almost be viewed as a sub-region 
>>>> placement priority? What I mean is this:
>>>>
>>>> 1)
>>>> placements=device,system flags=
>>>>
>>>> This results in placement priorities: device, device_cpu_mappable, 
>>>> system.
>>>
>>> Yup.
>>>
>>>>
>>>> 2)
>>>> placements=device,system flags=needs_cpu_access
>>>>
>>>> This results in placement priorities: device_cpu_mappable, device, 
>>>> system.
>>>
>>> Here it would only be: device_cpu_mappable, system. We would 
>>> completely ignore "device" in this case.
>>>
>>>>
>>>> Is this correct?
>>>>
>>>> The benefit of the flag is that i915 can place the object to the 
>>>> right place from the start instead of on the first CPU access? Is 
>>>> that worth it or is there more to it?
>>>
>>> Yeah, the object will only be placed somewhere that is also CPU 
>>> mappable, with the flag set.
>>
>> Hm, wouldn't it be more efficient to be able to migrate it over to 
>> non-mappable in cases when mappable is over-subscribed?
> 
> Not sure. As an alternative strategy, I guess that might be interesting, 
> and if userspace wants something like that we can always add a new flag 
> I guess? It's a toss up whether just using system memory is better/worse 
> than incurring an extra move at fault time?

For me it doesn't make sense to allow bos wo/ cpu mappable flag to use 
the mappable section (albeit as 2nd priority) and not allowing the cpu 
mappable ones to temporarily go anywhere in lmem.

 From i915 side it needs to know the allowed regions (for this argument 
I speak of mappable / non-mappable as separate regions, even if uapi 
does not expose them as separate memory regions). i915 does not know in 
advance the exact usage pattern.

In case of multiple clients, one might touch a buffer from the CPU once 
and then render with GPU many times. Another client might touch from CPU 
a lot more. With mappable space contention this would cause the buffer 
from the first client to constantly get migrated between smem and lmem, 
while in reality it could have been migrated to non mappable lmem and 
used by the GPU without any problems.

As some sort of diagram:

	Client A BO	Client B BO
	--------------	--------------
	CPU access		
			CPU access (Client A BO "evicted" to smem)
	GPU access	GPU access
	GPU access	CPU access
	GPU access	GPU access

If we assume there is only space for one BO in mappable, allowing 
non-mappable placement allows Client A to be unaffected by Client B 
activity. While with the current proposal it needlessly takes a hit on 
every, or every other GPU access.

So I think, if there isn't a fundamental reason to disallow it which I 
am missing, not limiting the implied placement when cpu access flag is 
given is beneficial to flexibility of migration decisions i915 can make.

Regards,

Tvrtko

> 
>>
>>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>>> +    __u32 flags;
>>>>> +    /**
>>>>> +     * @extensions: The chain of extensions to apply to this object.
>>>>> +     *
>>>>> +     * This will be useful in the future when we need to support 
>>>>> several
>>>>> +     * different extensions, and we need to apply more than one when
>>>>> +     * creating the object. See struct i915_user_extension.
>>>>> +     *
>>>>> +     * If we don't supply any extensions then we get the same old 
>>>>> gem_create
>>>>> +     * behaviour.
>>>>> +     *
>>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usagesee
>>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>>> +     *
>>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>>> +     */
>>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>>> +    __u64 extensions;
>>>>> +};
>>>>> +
>>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>>> +
>>>>> +/**
>>>>> + * struct __drm_i915_query_vma_info
>>>>> + *
>>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>>> returning its set
>>>>> + * of attributes.
>>>>> + *
>>>>> + * .. code-block:: C
>>>>> + *
>>>>> + *    struct drm_i915_query_vma_info info = {};
>>>>> + *    struct drm_i915_query_item item = {
>>>>> + *        .data_ptr = (uintptr_t)&info,
>>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>>> + *    };
>>>>> + *    struct drm_i915_query query = {
>>>>> + *        .num_items = 1,
>>>>> + *        .items_ptr = (uintptr_t)&item,
>>>>> + *    };
>>>>> + *    int err;
>>>>> + *
>>>>> + *    // Unlike some other types of queries, there is noneed to 
>>>>> first query
>>>>> + *    // the size of the data_ptr blob here, since we already know 
>>>>> ahead of
>>>>> + *    // time how big this needs to be.
>>>>> + *    item.length = sizeof(info);
>>>>> + *
>>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we 
>>>>> wish
>>>>> + *    // to query, before then firing off the query.
>>>>> + *    info.vm_id = vm_id;
>>>>> + *    info.offset = gtt_address;
>>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>>> + *    if (err || item.length < 0) ...
>>>>> + *
>>>>> + *    // If all went well we can now inspect the returned attributes.
>>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>>> + */
>>>>> +struct __drm_i915_query_vma_info {
>>>>> +    /**
>>>>> +     * @vm_id: The given vm id that contains the vma. The id is 
>>>>> the value
>>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>>> +     */
>>>>> +    __u32 vm_id;
>>>>> +    /** @pad: MBZ. */
>>>>> +    __u32 pad;
>>>>> +    /**
>>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>>> the kernel
>>>>> +     * will use to perform the lookup.
>>>>> +     */
>>>>> +    __u64 offset;
>>>>> +    /**
>>>>> +     * @attributes: The returned attributes for thegiven vma.
>>>>> +     *
>>>>> +     * Possible values:
>>>>> +     *
>>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set ifthe pages 
>>>>> backing the
>>>>> +     * vma are currently CPU accessible. If this isnot set then 
>>>>> the vma is
>>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICEmemory, which 
>>>>> the CPU
>>>>> +     * cannot directly access(this is only possibleon discrete 
>>>>> devices with
>>>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>>>> +     * require the kernel first synchronising any GPU work tied to 
>>>>> the
>>>>> +     * object, before then migrating the pages, either to the CPU 
>>>>> accessible
>>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>>> +     * placements permit it. See 
>>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>>> +     *
>>>>> +     * Note that this is inherently racy.
>>>>> +     */
>>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>>> +    __u64 attributes;
>>>>> +    /** @rsvd: MBZ */
>>>>> +    __u32 rsvd[4];
>>>>> +};
>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>> new file mode 100644
>>>>> index 000000000000..be3d9bcdd86d
>>>>> --- /dev/null
>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>> @@ -0,0 +1,58 @@
>>>>> +==========================
>>>>> +I915 Small BAR RFC Section
>>>>> +==========================
>>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>>> local-memory(i.e
>>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>>> might still be
>>>>> +smaller than the total probed_size. In such cases, only some 
>>>>> subset of
>>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>>> first 256M),
>>>>> +while the remainder is only accessible via the GPU.
>>>>> +
>>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>>> +----------------------------------------------
>>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>>> CPU access.
>>>>> +This becomes important when placing an object in 
>>>>> I915_MEMORY_CLASS_DEVICE, where
>>>>> +underneath the device has a small BAR, meaning only some portion 
>>>>> of it is CPU
>>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>>> access is not
>>>>> +required, and prioritize using the non-CPU visible portion of
>>>>> +I915_MEMORY_CLASS_DEVICE.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_gem_create_ext
>>>>> +
>>>>> +probed_cpu_visible_size attribute
>>>>> +---------------------------------
>>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>>> total size of the
>>>>> +CPU accessible portion, for the particular region. This should 
>>>>> only be
>>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>>> +
>>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>>>> with the
>>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>>> visible portion,
>>>>> +where the total size of the heap needs to be known.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_memory_region_info
>>>>> +
>>>>> +DRM_I915_QUERY_VMA_INFO query
>>>>> +-----------------------------
>>>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>>>> +respective vma, and return its set of attributes. For now we only 
>>>>> support
>>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the 
>>>>> object/vma is
>>>>> +currently placed in memory that is accessible by the CPU. This 
>>>>> should always be
>>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>>> I915_MEMORY_CLASS_DEVICE
>>>>> +matches the probed_size. If this is not set then CPU faulting the 
>>>>> object will
>>>>> +likely first require migrating the pages.
>>>>
>>>> I think there should be justification for the new query documented 
>>>> as well. (Why on top of what.)
>>>
>>> Yeah, I'm wondering now if we can just drop this part of the uapi, 
>>> for now at least, and focus on landing the new flag stuff first.
>>>
>>>>
>>>> Without it personally I can't immediately understand why the 
>>>> disconnect between the object based and VMA based API. Userspace has 
>>>> to do some intervening operations like either execbuf, or vm bind in 
>>>> the future, to make this query usable after object creation. So 
>>>> question is why wouldn't it know already which placements it allowed 
>>>> and so would i915 auto-migrate or not for this particular object. 
>>>> No? Or in other words why this wouldn't be an object based query 
>>>> since the question it is answering is about the object backing store 
>>>> and not the VMA.
>>>
>>> Yeah, just using the object handle or so I guess would also work. 
>>> Thanks for the comments.
>>
>> I saw other folks have said the same so omitting for now sounds good 
>> to me indeed.
>>
>> Regards,
>>
>> Tvrtko

Lionel Landwerlin May 2, 2022, 7:54 a.m. UTC | #14

On 20/04/2022 20:13, Matthew Auld wrote:
> Add an entry for the new uapi needed for small BAR on DG2+.
>
> v2:
>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>    - Rework error capture interactions, including no longer needing
>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>    - Add probed_cpu_visible_size. (Lionel)
>
> Signed-off-by: Matthew Auld <[e-mail]>
> Cc: Thomas Hellström <[e-mail]>
> Cc: Lionel Landwerlin <[e-mail]>
> Cc: Jon Bloomfield <[e-mail]>
> Cc: Daniel Vetter <[e-mail]>
> Cc: Jordan Justen <[e-mail]>
> Cc: Kenneth Graunke <[e-mail]>
> Cc: Akeem G Abodunrin <[e-mail]>
> Cc: [e-mail]
> ---
>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>   Documentation/gpu/rfc/index.rst          |   4 +
>   3 files changed, 252 insertions(+)
>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>
> diff --git a/Documentation/gpu/rfc/i915_small_bar.h b/Documentation/gpu/rfc/i915_small_bar.h
> new file mode 100644
> index 000000000000..7bfd0cf44d35
> --- /dev/null
> +++ b/Documentation/gpu/rfc/i915_small_bar.h
> @@ -0,0 +1,190 @@
> +/**
> + * struct __drm_i915_memory_region_info - Describes one region as known to the
> + * driver.
> + *
> + * Note this is using both struct drm_i915_query_item and struct drm_i915_query.
> + * For this new query we are adding the new query id DRM_I915_QUERY_MEMORY_REGIONS
> + * at &drm_i915_query_item.query_id.
> + */
> +struct __drm_i915_memory_region_info {
> +	/** @region: The class:instance pair encoding */
> +	struct drm_i915_gem_memory_class_instance region;
> +
> +	/** @rsvd0: MBZ */
> +	__u32 rsvd0;
> +
> +	/** @probed_size: Memory probed by the driver (-1 = unknown) */
> +	__u64 probed_size;
> +
> +	/** @unallocated_size: Estimate of memory remaining (-1 = unknown) */
> +	__u64 unallocated_size;
> +
> +	union {
> +		/** @rsvd1: MBZ */
> +		__u64 rsvd1[8];
> +		struct {
> +			/**
> +			 * @probed_cpu_visible_size: Memory probed by the driver
> +			 * that is CPU accessible. (-1 = unknown).
> +			 *
> +			 * This will be always be <= @probed_size, and the
> +			 * remainder(if there is any) will not be CPU
> +			 * accessible.
> +			 */
> +			__u64 probed_cpu_visible_size;
> +		};


Trying to implement userspace support in Vulkan for this, I have an 
additional question about the value of probed_cpu_visible_size.

When is it set to -1?

I'm guessing before there is support for this value it'll be 0 (MBZ).

After after it should either be the entire lmem or something smaller.


-Lionel


> +	};
> +};
> +
> +/**
> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, with added
> + * extension support using struct i915_user_extension.
> + *
> + * Note that new buffer flags should be added here, at least for the stuff that
> + * is immutable. Previously we would have two ioctls, one to create the object
> + * with gem_create, and another to apply various parameters, however this
> + * creates some ambiguity for the params which are considered immutable. Also in
> + * general we're phasing out the various SET/GET ioctls.
> + */
> +struct __drm_i915_gem_create_ext {
> +	/**
> +	 * @size: Requested size for the object.
> +	 *
> +	 * The (page-aligned) allocated size for the object will be returned.
> +	 *
> +	 * Note that for some devices we have might have further minimum
> +	 * page-size restrictions(larger than 4K), like for device local-memory.
> +	 * However in general the final size here should always reflect any
> +	 * rounding up, if for example using the I915_GEM_CREATE_EXT_MEMORY_REGIONS
> +	 * extension to place the object in device local-memory.
> +	 */
> +	__u64 size;
> +	/**
> +	 * @handle: Returned handle for the object.
> +	 *
> +	 * Object handles are nonzero.
> +	 */
> +	__u32 handle;
> +	/**
> +	 * @flags: Optional flags.
> +	 *
> +	 * Supported values:
> +	 *
> +	 * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the kernel that
> +	 * the object will need to be accessed via the CPU.
> +	 *
> +	 * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
> +	 * only strictly required on platforms where only some of the device
> +	 * memory is directly visible or mappable through the CPU, like on DG2+.
> +	 *
> +	 * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
> +	 * ensure we can always spill the allocation to system memory, if we
> +	 * can't place the object in the mappable part of
> +	 * I915_MEMORY_CLASS_DEVICE.
> +	 *
> +	 * Note that since the kernel only supports flat-CCS on objects that can
> +	 * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
> +	 * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
> +	 * flat-CCS.
> +	 *
> +	 * Without this hint, the kernel will assume that non-mappable
> +	 * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note that the
> +	 * kernel can still migrate the object to the mappable part, as a last
> +	 * resort, if userspace ever CPU faults this object, but this might be
> +	 * expensive, and so ideally should be avoided.
> +	 */
> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
> +	__u32 flags;
> +	/**
> +	 * @extensions: The chain of extensions to apply to this object.
> +	 *
> +	 * This will be useful in the future when we need to support several
> +	 * different extensions, and we need to apply more than one when
> +	 * creating the object. See struct i915_user_extension.
> +	 *
> +	 * If we don't supply any extensions then we get the same old gem_create
> +	 * behaviour.
> +	 *
> +	 * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
> +	 * struct drm_i915_gem_create_ext_memory_regions.
> +	 *
> +	 * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
> +	 * struct drm_i915_gem_create_ext_protected_content.
> +	 */
> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
> +	__u64 extensions;
> +};
> +
> +#define DRM_I915_QUERY_VMA_INFO	5
> +
> +/**
> + * struct __drm_i915_query_vma_info
> + *
> + * Given a vm and GTT address, lookup the corresponding vma, returning its set
> + * of attributes.
> + *
> + * .. code-block:: C
> + *
> + *	struct drm_i915_query_vma_info info = {};
> + *	struct drm_i915_query_item item = {
> + *		.data_ptr = (uintptr_t)&info,
> + *		.query_id = DRM_I915_QUERY_VMA_INFO,
> + *	};
> + *	struct drm_i915_query query = {
> + *		.num_items = 1,
> + *		.items_ptr = (uintptr_t)&item,
> + *	};
> + *	int err;
> + *
> + *	// Unlike some other types of queries, there is no need to first query
> + *	// the size of the data_ptr blob here, since we already know ahead of
> + *	// time how big this needs to be.
> + *	item.length = sizeof(info);
> + *
> + *	// Next we fill in the vm_id and ppGTT address of the vma we wish
> + *	// to query, before then firing off the query.
> + *	info.vm_id = vm_id;
> + *	info.offset = gtt_address;
> + *	err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
> + *	if (err || item.length < 0) ...
> + *
> + *	// If all went well we can now inspect the returned attributes.
> + *	if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
> + */
> +struct __drm_i915_query_vma_info {
> +	/**
> +	 * @vm_id: The given vm id that contains the vma. The id is the value
> +	 * returned by the DRM_I915_GEM_VM_CREATE. See struct
> +	 * drm_i915_gem_vm_control.vm_id.
> +	 */
> +	__u32 vm_id;
> +	/** @pad: MBZ. */
> +	__u32 pad;
> +	/**
> +	 * @offset: The corresponding ppGTT address of the vma which the kernel
> +	 * will use to perform the lookup.
> +	 */
> +	__u64 offset;
> +	/**
> +	 * @attributes: The returned attributes for the given vma.
> +	 *
> +	 * Possible values:
> +	 *
> +	 * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages backing the
> +	 * vma are currently CPU accessible. If this is not set then the vma is
> +	 * currently backed by I915_MEMORY_CLASS_DEVICE memory, which the CPU
> +	 * cannot directly access(this is only possible on discrete devices with
> +	 * a small BAR). Attempting to MMAP and fault such an object will
> +	 * require the kernel first synchronising any GPU work tied to the
> +	 * object, before then migrating the pages, either to the CPU accessible
> +	 * part of I915_MEMORY_CLASS_DEVICE, or I915_MEMORY_CLASS_SYSTEM, if the
> +	 * placements permit it. See I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
> +	 *
> +	 * Note that this is inherently racy.
> +	 */
> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
> +	__u64 attributes;
> +	/** @rsvd: MBZ */
> +	__u32 rsvd[4];
> +};
> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst b/Documentation/gpu/rfc/i915_small_bar.rst
> new file mode 100644
> index 000000000000..be3d9bcdd86d
> --- /dev/null
> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
> @@ -0,0 +1,58 @@
> +==========================
> +I915 Small BAR RFC Section
> +==========================
> +Starting from DG2 we will have resizable BAR support for device local-memory(i.e
> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size might still be
> +smaller than the total probed_size. In such cases, only some subset of
> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the first 256M),
> +while the remainder is only accessible via the GPU.
> +
> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
> +----------------------------------------------
> +New gem_create_ext flag to tell the kernel that a BO will require CPU access.
> +This becomes important when placing an object in I915_MEMORY_CLASS_DEVICE, where
> +underneath the device has a small BAR, meaning only some portion of it is CPU
> +accessible. Without this flag the kernel will assume that CPU access is not
> +required, and prioritize using the non-CPU visible portion of
> +I915_MEMORY_CLASS_DEVICE.
> +
> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> +   :functions: __drm_i915_gem_create_ext
> +
> +probed_cpu_visible_size attribute
> +---------------------------------
> +New struct__drm_i915_memory_region attribute which returns the total size of the
> +CPU accessible portion, for the particular region. This should only be
> +applicable for I915_MEMORY_CLASS_DEVICE.
> +
> +Vulkan will need this as part of creating a separate VkMemoryHeap with the
> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU visible portion,
> +where the total size of the heap needs to be known.
> +
> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> +   :functions: __drm_i915_memory_region_info
> +
> +DRM_I915_QUERY_VMA_INFO query
> +-----------------------------
> +Query the attributes of some vma. Given a vm and GTT offset, find the
> +respective vma, and return its set of attributes. For now we only support
> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
> +currently placed in memory that is accessible by the CPU. This should always be
> +set on devices where the CPU probed_cpu_visible_size of I915_MEMORY_CLASS_DEVICE
> +matches the probed_size. If this is not set then CPU faulting the object will
> +likely first require migrating the pages.
> +
> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> +   :functions: __drm_i915_query_vma_info
> +
> +Error Capture restrictions
> +--------------------------
> +With error capture we have two new restrictions:
> +
> +    1) Error capture is best effort on small BAR systems; if the pages are not
> +    CPU accessible, at the time of capture, then the kernel is free to skip
> +    trying to capture them.
> +
> +    2) On discrete we now reject error capture on recoverable contexts. In the
> +    future the kernel may want to blit during error capture, when for example
> +    something is not currently CPU accessible.
> diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
> index 91e93a705230..5a3bd3924ba6 100644
> --- a/Documentation/gpu/rfc/index.rst
> +++ b/Documentation/gpu/rfc/index.rst
> @@ -23,3 +23,7 @@ host such documentation:
>   .. toctree::
>   
>       i915_scheduler.rst
> +
> +.. toctree::
> +
> +    i915_small_bar.rst

Lionel Landwerlin May 2, 2022, 8:53 a.m. UTC | #15

On 02/05/2022 10:54, Lionel Landwerlin wrote:
> On 20/04/2022 20:13, Matthew Auld wrote:
>> Add an entry for the new uapi needed for small BAR on DG2+.
>>
>> v2:
>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>    - Rework error capture interactions, including no longer needing
>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>    - Add probed_cpu_visible_size. (Lionel)
>>
>> Signed-off-by: Matthew Auld <[e-mail]>
>> Cc: Thomas Hellström <[e-mail]>
>> Cc: Lionel Landwerlin <[e-mail]>
>> Cc: Jon Bloomfield <[e-mail]>
>> Cc: Daniel Vetter <[e-mail]>
>> Cc: Jordan Justen <[e-mail]>
>> Cc: Kenneth Graunke <[e-mail]>
>> Cc: Akeem G Abodunrin <[e-mail]>
>> Cc: [e-mail]
>> ---
>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>   Documentation/gpu/rfc/index.rst          |   4 +
>>   3 files changed, 252 insertions(+)
>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>
>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>> b/Documentation/gpu/rfc/i915_small_bar.h
>> new file mode 100644
>> index 000000000000..7bfd0cf44d35
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>> @@ -0,0 +1,190 @@
>> +/**
>> + * struct __drm_i915_memory_region_info - Describes one region as 
>> known to the
>> + * driver.
>> + *
>> + * Note this is using both struct drm_i915_query_item and struct 
>> drm_i915_query.
>> + * For this new query we are adding the new query id 
>> DRM_I915_QUERY_MEMORY_REGIONS
>> + * at &drm_i915_query_item.query_id.
>> + */
>> +struct __drm_i915_memory_region_info {
>> +    /** @region: The class:instance pair encoding */
>> +    struct drm_i915_gem_memory_class_instance region;
>> +
>> +    /** @rsvd0: MBZ */
>> +    __u32 rsvd0;
>> +
>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>> +    __u64 probed_size;
>> +
>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>> unknown) */
>> +    __u64 unallocated_size;
>> +
>> +    union {
>> +        /** @rsvd1: MBZ */
>> +        __u64 rsvd1[8];
>> +        struct {
>> +            /**
>> +             * @probed_cpu_visible_size: Memory probed by the driver
>> +             * that is CPU accessible. (-1 = unknown).
>> +             *
>> +             * This will be always be <= @probed_size, and the
>> +             * remainder(if there is any) will not be CPU
>> +             * accessible.
>> +             */
>> +            __u64 probed_cpu_visible_size;
>> +        };
>
>
> Trying to implement userspace support in Vulkan for this, I have an 
> additional question about the value of probed_cpu_visible_size.
>
> When is it set to -1?
>
> I'm guessing before there is support for this value it'll be 0 (MBZ).
>
> After after it should either be the entire lmem or something smaller.
>
>
> -Lionel


Other pain point of this new uAPI, previously we could query the 
unallocated size for each heap.

Now lmem is effectively divided into 2 heaps, but unallocated_size is 
tracking allocation from both parts of lmem.

Is adding new I915_MEMORY_CLASS_DEVICE_NON_MAPPABLE out of question?


-Lionel


>
>
>> +    };
>> +};
>> +
>> +/**
>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, 
>> with added
>> + * extension support using struct i915_user_extension.
>> + *
>> + * Note that new buffer flags should be added here, at least for the 
>> stuff that
>> + * is immutable. Previously we would have two ioctls, one to create 
>> the object
>> + * with gem_create, and another to apply various parameters, however 
>> this
>> + * creates some ambiguity for the params which are considered 
>> immutable. Also in
>> + * general we're phasing out the various SET/GET ioctls.
>> + */
>> +struct __drm_i915_gem_create_ext {
>> +    /**
>> +     * @size: Requested size for the object.
>> +     *
>> +     * The (page-aligned) allocated size for the object will be 
>> returned.
>> +     *
>> +     * Note that for some devices we have might have further minimum
>> +     * page-size restrictions(larger than 4K), like for device 
>> local-memory.
>> +     * However in general the final size here should always reflect any
>> +     * rounding up, if for example using the 
>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>> +     * extension to place the object in device local-memory.
>> +     */
>> +    __u64 size;
>> +    /**
>> +     * @handle: Returned handle for the object.
>> +     *
>> +     * Object handles are nonzero.
>> +     */
>> +    __u32 handle;
>> +    /**
>> +     * @flags: Optional flags.
>> +     *
>> +     * Supported values:
>> +     *
>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>> kernel that
>> +     * the object will need to be accessed via the CPU.
>> +     *
>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
>> +     * only strictly required on platforms where only some of the 
>> device
>> +     * memory is directly visible or mappable through the CPU, like 
>> on DG2+.
>> +     *
>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>> +     * ensure we can always spill the allocation to system memory, 
>> if we
>> +     * can't place the object in the mappable part of
>> +     * I915_MEMORY_CLASS_DEVICE.
>> +     *
>> +     * Note that since the kernel only supports flat-CCS on objects 
>> that can
>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>> +     * flat-CCS.
>> +     *
>> +     * Without this hint, the kernel will assume that non-mappable
>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>> that the
>> +     * kernel can still migrate the object to the mappable part, as 
>> a last
>> +     * resort, if userspace ever CPU faults this object, but this 
>> might be
>> +     * expensive, and so ideally should be avoided.
>> +     */
>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>> +    __u32 flags;
>> +    /**
>> +     * @extensions: The chain of extensions to apply to this object.
>> +     *
>> +     * This will be useful in the future when we need to support 
>> several
>> +     * different extensions, and we need to apply more than one when
>> +     * creating the object. See struct i915_user_extension.
>> +     *
>> +     * If we don't supply any extensions then we get the same old 
>> gem_create
>> +     * behaviour.
>> +     *
>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>> +     * struct drm_i915_gem_create_ext_memory_regions.
>> +     *
>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>> +     * struct drm_i915_gem_create_ext_protected_content.
>> +     */
>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>> +    __u64 extensions;
>> +};
>> +
>> +#define DRM_I915_QUERY_VMA_INFO    5
>> +
>> +/**
>> + * struct __drm_i915_query_vma_info
>> + *
>> + * Given a vm and GTT address, lookup the corresponding vma, 
>> returning its set
>> + * of attributes.
>> + *
>> + * .. code-block:: C
>> + *
>> + *    struct drm_i915_query_vma_info info = {};
>> + *    struct drm_i915_query_item item = {
>> + *        .data_ptr = (uintptr_t)&info,
>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>> + *    };
>> + *    struct drm_i915_query query = {
>> + *        .num_items = 1,
>> + *        .items_ptr = (uintptr_t)&item,
>> + *    };
>> + *    int err;
>> + *
>> + *    // Unlike some other types of queries, there is no need to 
>> first query
>> + *    // the size of the data_ptr blob here, since we already know 
>> ahead of
>> + *    // time how big this needs to be.
>> + *    item.length = sizeof(info);
>> + *
>> + *    // Next we fill in the vm_id and ppGTT address of the vma we wish
>> + *    // to query, before then firing off the query.
>> + *    info.vm_id = vm_id;
>> + *    info.offset = gtt_address;
>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>> + *    if (err || item.length < 0) ...
>> + *
>> + *    // If all went well we can now inspect the returned attributes.
>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>> + */
>> +struct __drm_i915_query_vma_info {
>> +    /**
>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>> value
>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>> +     * drm_i915_gem_vm_control.vm_id.
>> +     */
>> +    __u32 vm_id;
>> +    /** @pad: MBZ. */
>> +    __u32 pad;
>> +    /**
>> +     * @offset: The corresponding ppGTT address of the vma which the 
>> kernel
>> +     * will use to perform the lookup.
>> +     */
>> +    __u64 offset;
>> +    /**
>> +     * @attributes: The returned attributes for the given vma.
>> +     *
>> +     * Possible values:
>> +     *
>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>> backing the
>> +     * vma are currently CPU accessible. If this is not set then the 
>> vma is
>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>> the CPU
>> +     * cannot directly access(this is only possible on discrete 
>> devices with
>> +     * a small BAR). Attempting to MMAP and fault such an object will
>> +     * require the kernel first synchronising any GPU work tied to the
>> +     * object, before then migrating the pages, either to the CPU 
>> accessible
>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>> I915_MEMORY_CLASS_SYSTEM, if the
>> +     * placements permit it. See 
>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>> +     *
>> +     * Note that this is inherently racy.
>> +     */
>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>> +    __u64 attributes;
>> +    /** @rsvd: MBZ */
>> +    __u32 rsvd[4];
>> +};
>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>> b/Documentation/gpu/rfc/i915_small_bar.rst
>> new file mode 100644
>> index 000000000000..be3d9bcdd86d
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>> @@ -0,0 +1,58 @@
>> +==========================
>> +I915 Small BAR RFC Section
>> +==========================
>> +Starting from DG2 we will have resizable BAR support for device 
>> local-memory(i.e
>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>> might still be
>> +smaller than the total probed_size. In such cases, only some subset of
>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>> first 256M),
>> +while the remainder is only accessible via the GPU.
>> +
>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>> +----------------------------------------------
>> +New gem_create_ext flag to tell the kernel that a BO will require 
>> CPU access.
>> +This becomes important when placing an object in 
>> I915_MEMORY_CLASS_DEVICE, where
>> +underneath the device has a small BAR, meaning only some portion of 
>> it is CPU
>> +accessible. Without this flag the kernel will assume that CPU access 
>> is not
>> +required, and prioritize using the non-CPU visible portion of
>> +I915_MEMORY_CLASS_DEVICE.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_gem_create_ext
>> +
>> +probed_cpu_visible_size attribute
>> +---------------------------------
>> +New struct__drm_i915_memory_region attribute which returns the total 
>> size of the
>> +CPU accessible portion, for the particular region. This should only be
>> +applicable for I915_MEMORY_CLASS_DEVICE.
>> +
>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>> with the
>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>> visible portion,
>> +where the total size of the heap needs to be known.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_memory_region_info
>> +
>> +DRM_I915_QUERY_VMA_INFO query
>> +-----------------------------
>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>> +respective vma, and return its set of attributes. For now we only 
>> support
>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>> +currently placed in memory that is accessible by the CPU. This 
>> should always be
>> +set on devices where the CPU probed_cpu_visible_size of 
>> I915_MEMORY_CLASS_DEVICE
>> +matches the probed_size. If this is not set then CPU faulting the 
>> object will
>> +likely first require migrating the pages.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_query_vma_info
>> +
>> +Error Capture restrictions
>> +--------------------------
>> +With error capture we have two new restrictions:
>> +
>> +    1) Error capture is best effort on small BAR systems; if the 
>> pages are not
>> +    CPU accessible, at the time of capture, then the kernel is free 
>> to skip
>> +    trying to capture them.
>> +
>> +    2) On discrete we now reject error capture on recoverable 
>> contexts. In the
>> +    future the kernel may want to blit during error capture, when 
>> for example
>> +    something is not currently CPU accessible.
>> diff --git a/Documentation/gpu/rfc/index.rst 
>> b/Documentation/gpu/rfc/index.rst
>> index 91e93a705230..5a3bd3924ba6 100644
>> --- a/Documentation/gpu/rfc/index.rst
>> +++ b/Documentation/gpu/rfc/index.rst
>> @@ -23,3 +23,7 @@ host such documentation:
>>   .. toctree::
>>         i915_scheduler.rst
>> +
>> +.. toctree::
>> +
>> +    i915_small_bar.rst
>
>

Abodunrin, Akeem G May 2, 2022, 5:58 p.m. UTC | #16

> -----Original Message-----
> From: Landwerlin, Lionel G <[e-mail]>
> Sent: Monday, May 2, 2022 12:55 AM
> To: Auld, Matthew <[e-mail]>; [e-mail]
> Cc: [e-mail]; Thomas Hellström
> <[e-mail]>; Bloomfield, Jon
> <[e-mail]>; Daniel Vetter <[e-mail]>; Justen,
> Jordan L <[e-mail]>; Kenneth Graunke
> <[e-mail]>; Abodunrin, Akeem G
> <[e-mail]>; [e-mail]
> Subject: Re: [PATCH v2] drm/doc: add rfc section for small BAR uapi
> 
> On 20/04/2022 20:13, Matthew Auld wrote:
> > Add an entry for the new uapi needed for small BAR on DG2+.
> >
> > v2:
> >    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
> >    - Rework error capture interactions, including no longer needing
> >      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
> >    - Add probed_cpu_visible_size. (Lionel)
> >
> > Signed-off-by: Matthew Auld <[e-mail]>
> > Cc: Thomas Hellström <[e-mail]>
> > Cc: Lionel Landwerlin <[e-mail]>
> > Cc: Jon Bloomfield <[e-mail]>
> > Cc: Daniel Vetter <[e-mail]>
> > Cc: Jordan Justen <[e-mail]>
> > Cc: Kenneth Graunke <[e-mail]>
> > Cc: Akeem G Abodunrin <[e-mail]>
> > Cc: [e-mail]
> > ---
> >   Documentation/gpu/rfc/i915_small_bar.h   | 190
> +++++++++++++++++++++++
> >   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
> >   Documentation/gpu/rfc/index.rst          |   4 +
> >   3 files changed, 252 insertions(+)
> >   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
> >   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
> >
> > diff --git a/Documentation/gpu/rfc/i915_small_bar.h
> > b/Documentation/gpu/rfc/i915_small_bar.h
> > new file mode 100644
> > index 000000000000..7bfd0cf44d35
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/i915_small_bar.h
> > @@ -0,0 +1,190 @@
> > +/**
> > + * struct __drm_i915_memory_region_info - Describes one region as
> > +known to the
> > + * driver.
> > + *
> > + * Note this is using both struct drm_i915_query_item and struct
> drm_i915_query.
> > + * For this new query we are adding the new query id
> > +DRM_I915_QUERY_MEMORY_REGIONS
> > + * at &drm_i915_query_item.query_id.
> > + */
> > +struct __drm_i915_memory_region_info {
> > +	/** @region: The class:instance pair encoding */
> > +	struct drm_i915_gem_memory_class_instance region;
> > +
> > +	/** @rsvd0: MBZ */
> > +	__u32 rsvd0;
> > +
> > +	/** @probed_size: Memory probed by the driver (-1 = unknown) */
> > +	__u64 probed_size;
> > +
> > +	/** @unallocated_size: Estimate of memory remaining (-1 = unknown)
> */
> > +	__u64 unallocated_size;
> > +
> > +	union {
> > +		/** @rsvd1: MBZ */
> > +		__u64 rsvd1[8];
> > +		struct {
> > +			/**
> > +			 * @probed_cpu_visible_size: Memory probed by the
> driver
> > +			 * that is CPU accessible. (-1 = unknown).
> > +			 *
> > +			 * This will be always be <= @probed_size, and the
> > +			 * remainder(if there is any) will not be CPU
> > +			 * accessible.
> > +			 */
> > +			__u64 probed_cpu_visible_size;
> > +		};
> 
> 
> Trying to implement userspace support in Vulkan for this, I have an additional
> question about the value of probed_cpu_visible_size.
> 
> When is it set to -1?
I believe it is set to -1 if it is unknown, and/or not cpu accessible... 

Cheers!
~Akeem
> 
> I'm guessing before there is support for this value it'll be 0 (MBZ).
> 
> After after it should either be the entire lmem or something smaller.
> 
> 
> -Lionel
> 
> 
> > +	};
> > +};
> > +
> > +/**
> > + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour,
> > +with added
> > + * extension support using struct i915_user_extension.
> > + *
> > + * Note that new buffer flags should be added here, at least for the
> > +stuff that
> > + * is immutable. Previously we would have two ioctls, one to create
> > +the object
> > + * with gem_create, and another to apply various parameters, however
> > +this
> > + * creates some ambiguity for the params which are considered
> > +immutable. Also in
> > + * general we're phasing out the various SET/GET ioctls.
> > + */
> > +struct __drm_i915_gem_create_ext {
> > +	/**
> > +	 * @size: Requested size for the object.
> > +	 *
> > +	 * The (page-aligned) allocated size for the object will be returned.
> > +	 *
> > +	 * Note that for some devices we have might have further minimum
> > +	 * page-size restrictions(larger than 4K), like for device local-memory.
> > +	 * However in general the final size here should always reflect any
> > +	 * rounding up, if for example using the
> I915_GEM_CREATE_EXT_MEMORY_REGIONS
> > +	 * extension to place the object in device local-memory.
> > +	 */
> > +	__u64 size;
> > +	/**
> > +	 * @handle: Returned handle for the object.
> > +	 *
> > +	 * Object handles are nonzero.
> > +	 */
> > +	__u32 handle;
> > +	/**
> > +	 * @flags: Optional flags.
> > +	 *
> > +	 * Supported values:
> > +	 *
> > +	 * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the
> kernel that
> > +	 * the object will need to be accessed via the CPU.
> > +	 *
> > +	 * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE,
> and
> > +	 * only strictly required on platforms where only some of the device
> > +	 * memory is directly visible or mappable through the CPU, like on DG2+.
> > +	 *
> > +	 * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM,
> to
> > +	 * ensure we can always spill the allocation to system memory, if we
> > +	 * can't place the object in the mappable part of
> > +	 * I915_MEMORY_CLASS_DEVICE.
> > +	 *
> > +	 * Note that since the kernel only supports flat-CCS on objects that can
> > +	 * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore
> don't
> > +	 * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS
> together with
> > +	 * flat-CCS.
> > +	 *
> > +	 * Without this hint, the kernel will assume that non-mappable
> > +	 * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note that
> the
> > +	 * kernel can still migrate the object to the mappable part, as a last
> > +	 * resort, if userspace ever CPU faults this object, but this might be
> > +	 * expensive, and so ideally should be avoided.
> > +	 */
> > +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
> > +	__u32 flags;
> > +	/**
> > +	 * @extensions: The chain of extensions to apply to this object.
> > +	 *
> > +	 * This will be useful in the future when we need to support several
> > +	 * different extensions, and we need to apply more than one when
> > +	 * creating the object. See struct i915_user_extension.
> > +	 *
> > +	 * If we don't supply any extensions then we get the same old
> gem_create
> > +	 * behaviour.
> > +	 *
> > +	 * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
> > +	 * struct drm_i915_gem_create_ext_memory_regions.
> > +	 *
> > +	 * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
> > +	 * struct drm_i915_gem_create_ext_protected_content.
> > +	 */
> > +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0 #define
> > +I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
> > +	__u64 extensions;
> > +};
> > +
> > +#define DRM_I915_QUERY_VMA_INFO	5
> > +
> > +/**
> > + * struct __drm_i915_query_vma_info
> > + *
> > + * Given a vm and GTT address, lookup the corresponding vma,
> > +returning its set
> > + * of attributes.
> > + *
> > + * .. code-block:: C
> > + *
> > + *	struct drm_i915_query_vma_info info = {};
> > + *	struct drm_i915_query_item item = {
> > + *		.data_ptr = (uintptr_t)&info,
> > + *		.query_id = DRM_I915_QUERY_VMA_INFO,
> > + *	};
> > + *	struct drm_i915_query query = {
> > + *		.num_items = 1,
> > + *		.items_ptr = (uintptr_t)&item,
> > + *	};
> > + *	int err;
> > + *
> > + *	// Unlike some other types of queries, there is no need to first query
> > + *	// the size of the data_ptr blob here, since we already know ahead of
> > + *	// time how big this needs to be.
> > + *	item.length = sizeof(info);
> > + *
> > + *	// Next we fill in the vm_id and ppGTT address of the vma we wish
> > + *	// to query, before then firing off the query.
> > + *	info.vm_id = vm_id;
> > + *	info.offset = gtt_address;
> > + *	err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
> > + *	if (err || item.length < 0) ...
> > + *
> > + *	// If all went well we can now inspect the returned attributes.
> > + *	if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
> > + */
> > +struct __drm_i915_query_vma_info {
> > +	/**
> > +	 * @vm_id: The given vm id that contains the vma. The id is the value
> > +	 * returned by the DRM_I915_GEM_VM_CREATE. See struct
> > +	 * drm_i915_gem_vm_control.vm_id.
> > +	 */
> > +	__u32 vm_id;
> > +	/** @pad: MBZ. */
> > +	__u32 pad;
> > +	/**
> > +	 * @offset: The corresponding ppGTT address of the vma which the
> kernel
> > +	 * will use to perform the lookup.
> > +	 */
> > +	__u64 offset;
> > +	/**
> > +	 * @attributes: The returned attributes for the given vma.
> > +	 *
> > +	 * Possible values:
> > +	 *
> > +	 * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages
> backing the
> > +	 * vma are currently CPU accessible. If this is not set then the vma is
> > +	 * currently backed by I915_MEMORY_CLASS_DEVICE memory, which
> the CPU
> > +	 * cannot directly access(this is only possible on discrete devices with
> > +	 * a small BAR). Attempting to MMAP and fault such an object will
> > +	 * require the kernel first synchronising any GPU work tied to the
> > +	 * object, before then migrating the pages, either to the CPU accessible
> > +	 * part of I915_MEMORY_CLASS_DEVICE, or
> I915_MEMORY_CLASS_SYSTEM, if the
> > +	 * placements permit it. See
> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
> > +	 *
> > +	 * Note that this is inherently racy.
> > +	 */
> > +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
> > +	__u64 attributes;
> > +	/** @rsvd: MBZ */
> > +	__u32 rsvd[4];
> > +};
> > diff --git a/Documentation/gpu/rfc/i915_small_bar.rst
> > b/Documentation/gpu/rfc/i915_small_bar.rst
> > new file mode 100644
> > index 000000000000..be3d9bcdd86d
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/i915_small_bar.rst
> > @@ -0,0 +1,58 @@
> > +==========================
> > +I915 Small BAR RFC Section
> > +==========================
> > +Starting from DG2 we will have resizable BAR support for device
> > +local-memory(i.e I915_MEMORY_CLASS_DEVICE), but in some cases the
> > +final BAR size might still be smaller than the total probed_size. In
> > +such cases, only some subset of I915_MEMORY_CLASS_DEVICE will be CPU
> > +accessible(for example the first 256M), while the remainder is only accessible
> via the GPU.
> > +
> > +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
> > +----------------------------------------------
> > +New gem_create_ext flag to tell the kernel that a BO will require CPU access.
> > +This becomes important when placing an object in
> > +I915_MEMORY_CLASS_DEVICE, where underneath the device has a small
> > +BAR, meaning only some portion of it is CPU accessible. Without this
> > +flag the kernel will assume that CPU access is not required, and
> > +prioritize using the non-CPU visible portion of
> I915_MEMORY_CLASS_DEVICE.
> > +
> > +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> > +   :functions: __drm_i915_gem_create_ext
> > +
> > +probed_cpu_visible_size attribute
> > +---------------------------------
> > +New struct__drm_i915_memory_region attribute which returns the total
> > +size of the CPU accessible portion, for the particular region. This
> > +should only be applicable for I915_MEMORY_CLASS_DEVICE.
> > +
> > +Vulkan will need this as part of creating a separate VkMemoryHeap
> > +with the VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the
> > +CPU visible portion, where the total size of the heap needs to be known.
> > +
> > +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> > +   :functions: __drm_i915_memory_region_info
> > +
> > +DRM_I915_QUERY_VMA_INFO query
> > +-----------------------------
> > +Query the attributes of some vma. Given a vm and GTT offset, find the
> > +respective vma, and return its set of attributes. For now we only
> > +support DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the
> > +object/vma is currently placed in memory that is accessible by the
> > +CPU. This should always be set on devices where the CPU
> > +probed_cpu_visible_size of I915_MEMORY_CLASS_DEVICE matches the
> > +probed_size. If this is not set then CPU faulting the object will likely first
> require migrating the pages.
> > +
> > +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
> > +   :functions: __drm_i915_query_vma_info
> > +
> > +Error Capture restrictions
> > +--------------------------
> > +With error capture we have two new restrictions:
> > +
> > +    1) Error capture is best effort on small BAR systems; if the pages are not
> > +    CPU accessible, at the time of capture, then the kernel is free to skip
> > +    trying to capture them.
> > +
> > +    2) On discrete we now reject error capture on recoverable contexts. In the
> > +    future the kernel may want to blit during error capture, when for example
> > +    something is not currently CPU accessible.
> > diff --git a/Documentation/gpu/rfc/index.rst
> > b/Documentation/gpu/rfc/index.rst index 91e93a705230..5a3bd3924ba6
> > 100644
> > --- a/Documentation/gpu/rfc/index.rst
> > +++ b/Documentation/gpu/rfc/index.rst
> > @@ -23,3 +23,7 @@ host such documentation:
> >   .. toctree::
> >
> >       i915_scheduler.rst
> > +
> > +.. toctree::
> > +
> > +    i915_small_bar.rst
>

Lionel Landwerlin May 2, 2022, 6:03 p.m. UTC | #17

On 02/05/2022 20:58, Abodunrin, Akeem G wrote:
>
>> -----Original Message-----
>> From: Landwerlin, Lionel G <[e-mail]>
>> Sent: Monday, May 2, 2022 12:55 AM
>> To: Auld, Matthew <[e-mail]>; [e-mail]
>> Cc: [e-mail]; Thomas Hellström
>> <[e-mail]>; Bloomfield, Jon
>> <[e-mail]>; Daniel Vetter <[e-mail]>; Justen,
>> Jordan L <[e-mail]>; Kenneth Graunke
>> <[e-mail]>; Abodunrin, Akeem G
>> <[e-mail]>; [e-mail]
>> Subject: Re: [PATCH v2] drm/doc: add rfc section for small BAR uapi
>>
>> On 20/04/2022 20:13, Matthew Auld wrote:
>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>
>>> v2:
>>>     - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>     - Rework error capture interactions, including no longer needing
>>>       NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>     - Add probed_cpu_visible_size. (Lionel)
>>>
>>> Signed-off-by: Matthew Auld <[e-mail]>
>>> Cc: Thomas Hellström <[e-mail]>
>>> Cc: Lionel Landwerlin <[e-mail]>
>>> Cc: Jon Bloomfield <[e-mail]>
>>> Cc: Daniel Vetter <[e-mail]>
>>> Cc: Jordan Justen <[e-mail]>
>>> Cc: Kenneth Graunke <[e-mail]>
>>> Cc: Akeem G Abodunrin <[e-mail]>
>>> Cc: [e-mail]
>>> ---
>>>    Documentation/gpu/rfc/i915_small_bar.h   | 190
>> +++++++++++++++++++++++
>>>    Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>    Documentation/gpu/rfc/index.rst          |   4 +
>>>    3 files changed, 252 insertions(+)
>>>    create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>    create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h
>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>> new file mode 100644
>>> index 000000000000..7bfd0cf44d35
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>> @@ -0,0 +1,190 @@
>>> +/**
>>> + * struct __drm_i915_memory_region_info - Describes one region as
>>> +known to the
>>> + * driver.
>>> + *
>>> + * Note this is using both struct drm_i915_query_item and struct
>> drm_i915_query.
>>> + * For this new query we are adding the new query id
>>> +DRM_I915_QUERY_MEMORY_REGIONS
>>> + * at &drm_i915_query_item.query_id.
>>> + */
>>> +struct __drm_i915_memory_region_info {
>>> +   /** @region: The class:instance pair encoding */
>>> +   struct drm_i915_gem_memory_class_instance region;
>>> +
>>> +   /** @rsvd0: MBZ */
>>> +   __u32 rsvd0;
>>> +
>>> +   /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>> +   __u64 probed_size;
>>> +
>>> +   /** @unallocated_size: Estimate of memory remaining (-1 = unknown)
>> */
>>> +   __u64 unallocated_size;
>>> +
>>> +   union {
>>> +           /** @rsvd1: MBZ */
>>> +           __u64 rsvd1[8];
>>> +           struct {
>>> +                   /**
>>> +                    * @probed_cpu_visible_size: Memory probed by the
>> driver
>>> +                    * that is CPU accessible. (-1 = unknown).
>>> +                    *
>>> +                    * This will be always be <= @probed_size, and the
>>> +                    * remainder(if there is any) will not be CPU
>>> +                    * accessible.
>>> +                    */
>>> +                   __u64 probed_cpu_visible_size;
>>> +           };
>>
>> Trying to implement userspace support in Vulkan for this, I have an additional
>> question about the value of probed_cpu_visible_size.
>>
>> When is it set to -1?
> I believe it is set to -1 if it is unknown, and/or not cpu accessible...
>
> Cheers!
> ~Akeem


So what should I expect on system memory?

What value is returned when all of probed_size is CPU visible on local 
memory?


Thanks,


-Lionel


>> I'm guessing before there is support for this value it'll be 0 (MBZ).
>>
>> After after it should either be the entire lmem or something smaller.
>>
>>
>> -Lionel
>>
>>
>>> +   };
>>> +};
>>> +
>>> +/**
>>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour,
>>> +with added
>>> + * extension support using struct i915_user_extension.
>>> + *
>>> + * Note that new buffer flags should be added here, at least for the
>>> +stuff that
>>> + * is immutable. Previously we would have two ioctls, one to create
>>> +the object
>>> + * with gem_create, and another to apply various parameters, however
>>> +this
>>> + * creates some ambiguity for the params which are considered
>>> +immutable. Also in
>>> + * general we're phasing out the various SET/GET ioctls.
>>> + */
>>> +struct __drm_i915_gem_create_ext {
>>> +   /**
>>> +    * @size: Requested size for the object.
>>> +    *
>>> +    * The (page-aligned) allocated size for the object will be returned.
>>> +    *
>>> +    * Note that for some devices we have might have further minimum
>>> +    * page-size restrictions(larger than 4K), like for device local-memory.
>>> +    * However in general the final size here should always reflect any
>>> +    * rounding up, if for example using the
>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>> +    * extension to place the object in device local-memory.
>>> +    */
>>> +   __u64 size;
>>> +   /**
>>> +    * @handle: Returned handle for the object.
>>> +    *
>>> +    * Object handles are nonzero.
>>> +    */
>>> +   __u32 handle;
>>> +   /**
>>> +    * @flags: Optional flags.
>>> +    *
>>> +    * Supported values:
>>> +    *
>>> +    * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the
>> kernel that
>>> +    * the object will need to be accessed via the CPU.
>>> +    *
>>> +    * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE,
>> and
>>> +    * only strictly required on platforms where only some of the device
>>> +    * memory is directly visible or mappable through the CPU, like on DG2+.
>>> +    *
>>> +    * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM,
>> to
>>> +    * ensure we can always spill the allocation to system memory, if we
>>> +    * can't place the object in the mappable part of
>>> +    * I915_MEMORY_CLASS_DEVICE.
>>> +    *
>>> +    * Note that since the kernel only supports flat-CCS on objects that can
>>> +    * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore
>> don't
>>> +    * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS
>> together with
>>> +    * flat-CCS.
>>> +    *
>>> +    * Without this hint, the kernel will assume that non-mappable
>>> +    * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note that
>> the
>>> +    * kernel can still migrate the object to the mappable part, as a last
>>> +    * resort, if userspace ever CPU faults this object, but this might be
>>> +    * expensive, and so ideally should be avoided.
>>> +    */
>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>> +   __u32 flags;
>>> +   /**
>>> +    * @extensions: The chain of extensions to apply to this object.
>>> +    *
>>> +    * This will be useful in the future when we need to support several
>>> +    * different extensions, and we need to apply more than one when
>>> +    * creating the object. See struct i915_user_extension.
>>> +    *
>>> +    * If we don't supply any extensions then we get the same old
>> gem_create
>>> +    * behaviour.
>>> +    *
>>> +    * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>> +    * struct drm_i915_gem_create_ext_memory_regions.
>>> +    *
>>> +    * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>> +    * struct drm_i915_gem_create_ext_protected_content.
>>> +    */
>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0 #define
>>> +I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>> +   __u64 extensions;
>>> +};
>>> +
>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>> +
>>> +/**
>>> + * struct __drm_i915_query_vma_info
>>> + *
>>> + * Given a vm and GTT address, lookup the corresponding vma,
>>> +returning its set
>>> + * of attributes.
>>> + *
>>> + * .. code-block:: C
>>> + *
>>> + * struct drm_i915_query_vma_info info = {};
>>> + * struct drm_i915_query_item item = {
>>> + *         .data_ptr = (uintptr_t)&info,
>>> + *         .query_id = DRM_I915_QUERY_VMA_INFO,
>>> + * };
>>> + * struct drm_i915_query query = {
>>> + *         .num_items = 1,
>>> + *         .items_ptr = (uintptr_t)&item,
>>> + * };
>>> + * int err;
>>> + *
>>> + * // Unlike some other types of queries, there is no need to first query
>>> + * // the size of the data_ptr blob here, since we already know ahead of
>>> + * // time how big this needs to be.
>>> + * item.length = sizeof(info);
>>> + *
>>> + * // Next we fill in the vm_id and ppGTT address of the vma we wish
>>> + * // to query, before then firing off the query.
>>> + * info.vm_id = vm_id;
>>> + * info.offset = gtt_address;
>>> + * err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>> + * if (err || item.length < 0) ...
>>> + *
>>> + * // If all went well we can now inspect the returned attributes.
>>> + * if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>> + */
>>> +struct __drm_i915_query_vma_info {
>>> +   /**
>>> +    * @vm_id: The given vm id that contains the vma. The id is the value
>>> +    * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>> +    * drm_i915_gem_vm_control.vm_id.
>>> +    */
>>> +   __u32 vm_id;
>>> +   /** @pad: MBZ. */
>>> +   __u32 pad;
>>> +   /**
>>> +    * @offset: The corresponding ppGTT address of the vma which the
>> kernel
>>> +    * will use to perform the lookup.
>>> +    */
>>> +   __u64 offset;
>>> +   /**
>>> +    * @attributes: The returned attributes for the given vma.
>>> +    *
>>> +    * Possible values:
>>> +    *
>>> +    * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages
>> backing the
>>> +    * vma are currently CPU accessible. If this is not set then the vma is
>>> +    * currently backed by I915_MEMORY_CLASS_DEVICE memory, which
>> the CPU
>>> +    * cannot directly access(this is only possible on discrete devices with
>>> +    * a small BAR). Attempting to MMAP and fault such an object will
>>> +    * require the kernel first synchronising any GPU work tied to the
>>> +    * object, before then migrating the pages, either to the CPU accessible
>>> +    * part of I915_MEMORY_CLASS_DEVICE, or
>> I915_MEMORY_CLASS_SYSTEM, if the
>>> +    * placements permit it. See
>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>> +    *
>>> +    * Note that this is inherently racy.
>>> +    */
>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>> +   __u64 attributes;
>>> +   /** @rsvd: MBZ */
>>> +   __u32 rsvd[4];
>>> +};
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst
>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>> new file mode 100644
>>> index 000000000000..be3d9bcdd86d
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>> @@ -0,0 +1,58 @@
>>> +==========================
>>> +I915 Small BAR RFC Section
>>> +==========================
>>> +Starting from DG2 we will have resizable BAR support for device
>>> +local-memory(i.e I915_MEMORY_CLASS_DEVICE), but in some cases the
>>> +final BAR size might still be smaller than the total probed_size. In
>>> +such cases, only some subset of I915_MEMORY_CLASS_DEVICE will be CPU
>>> +accessible(for example the first 256M), while the remainder is only accessible
>> via the GPU.
>>> +
>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>> +----------------------------------------------
>>> +New gem_create_ext flag to tell the kernel that a BO will require CPU access.
>>> +This becomes important when placing an object in
>>> +I915_MEMORY_CLASS_DEVICE, where underneath the device has a small
>>> +BAR, meaning only some portion of it is CPU accessible. Without this
>>> +flag the kernel will assume that CPU access is not required, and
>>> +prioritize using the non-CPU visible portion of
>> I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_gem_create_ext
>>> +
>>> +probed_cpu_visible_size attribute
>>> +---------------------------------
>>> +New struct__drm_i915_memory_region attribute which returns the total
>>> +size of the CPU accessible portion, for the particular region. This
>>> +should only be applicable for I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +Vulkan will need this as part of creating a separate VkMemoryHeap
>>> +with the VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the
>>> +CPU visible portion, where the total size of the heap needs to be known.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_memory_region_info
>>> +
>>> +DRM_I915_QUERY_VMA_INFO query
>>> +-----------------------------
>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>> +respective vma, and return its set of attributes. For now we only
>>> +support DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the
>>> +object/vma is currently placed in memory that is accessible by the
>>> +CPU. This should always be set on devices where the CPU
>>> +probed_cpu_visible_size of I915_MEMORY_CLASS_DEVICE matches the
>>> +probed_size. If this is not set then CPU faulting the object will likely first
>> require migrating the pages.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_query_vma_info
>>> +
>>> +Error Capture restrictions
>>> +--------------------------
>>> +With error capture we have two new restrictions:
>>> +
>>> +    1) Error capture is best effort on small BAR systems; if the pages are not
>>> +    CPU accessible, at the time of capture, then the kernel is free to skip
>>> +    trying to capture them.
>>> +
>>> +    2) On discrete we now reject error capture on recoverable contexts. In the
>>> +    future the kernel may want to blit during error capture, when for example
>>> +    something is not currently CPU accessible.
>>> diff --git a/Documentation/gpu/rfc/index.rst
>>> b/Documentation/gpu/rfc/index.rst index 91e93a705230..5a3bd3924ba6
>>> 100644
>>> --- a/Documentation/gpu/rfc/index.rst
>>> +++ b/Documentation/gpu/rfc/index.rst
>>> @@ -23,3 +23,7 @@ host such documentation:
>>>    .. toctree::
>>>
>>>        i915_scheduler.rst
>>> +
>>> +.. toctree::
>>> +
>>> +    i915_small_bar.rst

Matthew Auld May 3, 2022, 9:01 a.m. UTC | #18

On 02/05/2022 08:54, Lionel Landwerlin wrote:
> On 20/04/2022 20:13, Matthew Auld wrote:
>> Add an entry for the new uapi needed for small BAR on DG2+.
>>
>> v2:
>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>    - Rework error capture interactions, including no longer needing
>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>    - Add probed_cpu_visible_size. (Lionel)
>>
>> Signed-off-by: Matthew Auld <[e-mail]>
>> Cc: Thomas Hellström <[e-mail]>
>> Cc: Lionel Landwerlin <[e-mail]>
>> Cc: Jon Bloomfield <[e-mail]>
>> Cc: Daniel Vetter <[e-mail]>
>> Cc: Jordan Justen <[e-mail]>
>> Cc: Kenneth Graunke <[e-mail]>
>> Cc: Akeem G Abodunrin <[e-mail]>
>> Cc: [e-mail]
>> ---
>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>   Documentation/gpu/rfc/index.rst          |   4 +
>>   3 files changed, 252 insertions(+)
>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>
>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>> b/Documentation/gpu/rfc/i915_small_bar.h
>> new file mode 100644
>> index 000000000000..7bfd0cf44d35
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>> @@ -0,0 +1,190 @@
>> +/**
>> + * struct __drm_i915_memory_region_info - Describes one region as 
>> known to the
>> + * driver.
>> + *
>> + * Note this is using both struct drm_i915_query_item and struct 
>> drm_i915_query.
>> + * For this new query we are adding the new query id 
>> DRM_I915_QUERY_MEMORY_REGIONS
>> + * at &drm_i915_query_item.query_id.
>> + */
>> +struct __drm_i915_memory_region_info {
>> +    /** @region: The class:instance pair encoding */
>> +    struct drm_i915_gem_memory_class_instance region;
>> +
>> +    /** @rsvd0: MBZ */
>> +    __u32 rsvd0;
>> +
>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>> +    __u64 probed_size;
>> +
>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>> unknown) */
>> +    __u64 unallocated_size;
>> +
>> +    union {
>> +        /** @rsvd1: MBZ */
>> +        __u64 rsvd1[8];
>> +        struct {
>> +            /**
>> +             * @probed_cpu_visible_size: Memory probed by the driver
>> +             * that is CPU accessible. (-1 = unknown).
>> +             *
>> +             * This will be always be <= @probed_size, and the
>> +             * remainder(if there is any) will not be CPU
>> +             * accessible.
>> +             */
>> +            __u64 probed_cpu_visible_size;
>> +        };
> 
> 
> Trying to implement userspace support in Vulkan for this, I have an 
> additional question about the value of probed_cpu_visible_size.
> 
> When is it set to -1?

I don't anything is currently using -1, for any of these fields.

> 
> I'm guessing before there is support for this value it'll be 0 (MBZ).
> 
> After after it should either be the entire lmem or something smaller.

Yup.

> 
> 
> -Lionel
> 
> 
>> +    };
>> +};
>> +
>> +/**
>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, 
>> with added
>> + * extension support using struct i915_user_extension.
>> + *
>> + * Note that new buffer flags should be added here, at least for the 
>> stuff that
>> + * is immutable. Previously we would have two ioctls, one to create 
>> the object
>> + * with gem_create, and another to apply various parameters, however 
>> this
>> + * creates some ambiguity for the params which are considered 
>> immutable. Also in
>> + * general we're phasing out the various SET/GET ioctls.
>> + */
>> +struct __drm_i915_gem_create_ext {
>> +    /**
>> +     * @size: Requested size for the object.
>> +     *
>> +     * The (page-aligned) allocated size for the object will be 
>> returned.
>> +     *
>> +     * Note that for some devices we have might have further minimum
>> +     * page-size restrictions(larger than 4K), like for device 
>> local-memory.
>> +     * However in general the final size here should always reflect any
>> +     * rounding up, if for example using the 
>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>> +     * extension to place the object in device local-memory.
>> +     */
>> +    __u64 size;
>> +    /**
>> +     * @handle: Returned handle for the object.
>> +     *
>> +     * Object handles are nonzero.
>> +     */
>> +    __u32 handle;
>> +    /**
>> +     * @flags: Optional flags.
>> +     *
>> +     * Supported values:
>> +     *
>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>> kernel that
>> +     * the object will need to be accessed via the CPU.
>> +     *
>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
>> +     * only strictly required on platforms where only some of the device
>> +     * memory is directly visible or mappable through the CPU, like 
>> on DG2+.
>> +     *
>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>> +     * ensure we can always spill the allocation to system memory, if we
>> +     * can't place the object in the mappable part of
>> +     * I915_MEMORY_CLASS_DEVICE.
>> +     *
>> +     * Note that since the kernel only supports flat-CCS on objects 
>> that can
>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>> +     * flat-CCS.
>> +     *
>> +     * Without this hint, the kernel will assume that non-mappable
>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>> that the
>> +     * kernel can still migrate the object to the mappable part, as a 
>> last
>> +     * resort, if userspace ever CPU faults this object, but this 
>> might be
>> +     * expensive, and so ideally should be avoided.
>> +     */
>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>> +    __u32 flags;
>> +    /**
>> +     * @extensions: The chain of extensions to apply to this object.
>> +     *
>> +     * This will be useful in the future when we need to support several
>> +     * different extensions, and we need to apply more than one when
>> +     * creating the object. See struct i915_user_extension.
>> +     *
>> +     * If we don't supply any extensions then we get the same old 
>> gem_create
>> +     * behaviour.
>> +     *
>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>> +     * struct drm_i915_gem_create_ext_memory_regions.
>> +     *
>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>> +     * struct drm_i915_gem_create_ext_protected_content.
>> +     */
>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>> +    __u64 extensions;
>> +};
>> +
>> +#define DRM_I915_QUERY_VMA_INFO    5
>> +
>> +/**
>> + * struct __drm_i915_query_vma_info
>> + *
>> + * Given a vm and GTT address, lookup the corresponding vma, 
>> returning its set
>> + * of attributes.
>> + *
>> + * .. code-block:: C
>> + *
>> + *    struct drm_i915_query_vma_info info = {};
>> + *    struct drm_i915_query_item item = {
>> + *        .data_ptr = (uintptr_t)&info,
>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>> + *    };
>> + *    struct drm_i915_query query = {
>> + *        .num_items = 1,
>> + *        .items_ptr = (uintptr_t)&item,
>> + *    };
>> + *    int err;
>> + *
>> + *    // Unlike some other types of queries, there is no need to 
>> first query
>> + *    // the size of the data_ptr blob here, since we already know 
>> ahead of
>> + *    // time how big this needs to be.
>> + *    item.length = sizeof(info);
>> + *
>> + *    // Next we fill in the vm_id and ppGTT address of the vma we wish
>> + *    // to query, before then firing off the query.
>> + *    info.vm_id = vm_id;
>> + *    info.offset = gtt_address;
>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>> + *    if (err || item.length < 0) ...
>> + *
>> + *    // If all went well we can now inspect the returned attributes.
>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>> + */
>> +struct __drm_i915_query_vma_info {
>> +    /**
>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>> value
>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>> +     * drm_i915_gem_vm_control.vm_id.
>> +     */
>> +    __u32 vm_id;
>> +    /** @pad: MBZ. */
>> +    __u32 pad;
>> +    /**
>> +     * @offset: The corresponding ppGTT address of the vma which the 
>> kernel
>> +     * will use to perform the lookup.
>> +     */
>> +    __u64 offset;
>> +    /**
>> +     * @attributes: The returned attributes for the given vma.
>> +     *
>> +     * Possible values:
>> +     *
>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages backing 
>> the
>> +     * vma are currently CPU accessible. If this is not set then the 
>> vma is
>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which the 
>> CPU
>> +     * cannot directly access(this is only possible on discrete 
>> devices with
>> +     * a small BAR). Attempting to MMAP and fault such an object will
>> +     * require the kernel first synchronising any GPU work tied to the
>> +     * object, before then migrating the pages, either to the CPU 
>> accessible
>> +     * part of I915_MEMORY_CLASS_DEVICE, or I915_MEMORY_CLASS_SYSTEM, 
>> if the
>> +     * placements permit it. See 
>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>> +     *
>> +     * Note that this is inherently racy.
>> +     */
>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>> +    __u64 attributes;
>> +    /** @rsvd: MBZ */
>> +    __u32 rsvd[4];
>> +};
>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>> b/Documentation/gpu/rfc/i915_small_bar.rst
>> new file mode 100644
>> index 000000000000..be3d9bcdd86d
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>> @@ -0,0 +1,58 @@
>> +==========================
>> +I915 Small BAR RFC Section
>> +==========================
>> +Starting from DG2 we will have resizable BAR support for device 
>> local-memory(i.e
>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size might 
>> still be
>> +smaller than the total probed_size. In such cases, only some subset of
>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the first 
>> 256M),
>> +while the remainder is only accessible via the GPU.
>> +
>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>> +----------------------------------------------
>> +New gem_create_ext flag to tell the kernel that a BO will require CPU 
>> access.
>> +This becomes important when placing an object in 
>> I915_MEMORY_CLASS_DEVICE, where
>> +underneath the device has a small BAR, meaning only some portion of 
>> it is CPU
>> +accessible. Without this flag the kernel will assume that CPU access 
>> is not
>> +required, and prioritize using the non-CPU visible portion of
>> +I915_MEMORY_CLASS_DEVICE.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_gem_create_ext
>> +
>> +probed_cpu_visible_size attribute
>> +---------------------------------
>> +New struct__drm_i915_memory_region attribute which returns the total 
>> size of the
>> +CPU accessible portion, for the particular region. This should only be
>> +applicable for I915_MEMORY_CLASS_DEVICE.
>> +
>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>> with the
>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU visible 
>> portion,
>> +where the total size of the heap needs to be known.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_memory_region_info
>> +
>> +DRM_I915_QUERY_VMA_INFO query
>> +-----------------------------
>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>> +respective vma, and return its set of attributes. For now we only 
>> support
>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>> +currently placed in memory that is accessible by the CPU. This should 
>> always be
>> +set on devices where the CPU probed_cpu_visible_size of 
>> I915_MEMORY_CLASS_DEVICE
>> +matches the probed_size. If this is not set then CPU faulting the 
>> object will
>> +likely first require migrating the pages.
>> +
>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>> +   :functions: __drm_i915_query_vma_info
>> +
>> +Error Capture restrictions
>> +--------------------------
>> +With error capture we have two new restrictions:
>> +
>> +    1) Error capture is best effort on small BAR systems; if the 
>> pages are not
>> +    CPU accessible, at the time of capture, then the kernel is free 
>> to skip
>> +    trying to capture them.
>> +
>> +    2) On discrete we now reject error capture on recoverable 
>> contexts. In the
>> +    future the kernel may want to blit during error capture, when for 
>> example
>> +    something is not currently CPU accessible.
>> diff --git a/Documentation/gpu/rfc/index.rst 
>> b/Documentation/gpu/rfc/index.rst
>> index 91e93a705230..5a3bd3924ba6 100644
>> --- a/Documentation/gpu/rfc/index.rst
>> +++ b/Documentation/gpu/rfc/index.rst
>> @@ -23,3 +23,7 @@ host such documentation:
>>   .. toctree::
>>       i915_scheduler.rst
>> +
>> +.. toctree::
>> +
>> +    i915_small_bar.rst
> 
>

Matthew Auld May 3, 2022, 9:07 a.m. UTC | #19

On 02/05/2022 19:03, Lionel Landwerlin wrote:
> On 02/05/2022 20:58, Abodunrin, Akeem G wrote:
>>
>>> -----Original Message-----
>>> From: Landwerlin, Lionel G <[e-mail]>
>>> Sent: Monday, May 2, 2022 12:55 AM
>>> To: Auld, Matthew <[e-mail]>; 
>>> [e-mail]
>>> Cc: [e-mail]; Thomas Hellström
>>> <[e-mail]>; Bloomfield, Jon
>>> <[e-mail]>; Daniel Vetter <[e-mail]>; 
>>> Justen,
>>> Jordan L <[e-mail]>; Kenneth Graunke
>>> <[e-mail]>; Abodunrin, Akeem G
>>> <[e-mail]>; [e-mail]
>>> Subject: Re: [PATCH v2] drm/doc: add rfc section for small BAR uapi
>>>
>>> On 20/04/2022 20:13, Matthew Auld wrote:
>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>
>>>> v2:
>>>>     - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>     - Rework error capture interactions, including no longer needing
>>>>       NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>     - Add probed_cpu_visible_size. (Lionel)
>>>>
>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>> Cc: Thomas Hellström <[e-mail]>
>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>> Cc: Jon Bloomfield <[e-mail]>
>>>> Cc: Daniel Vetter <[e-mail]>
>>>> Cc: Jordan Justen <[e-mail]>
>>>> Cc: Kenneth Graunke <[e-mail]>
>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>> Cc: [e-mail]
>>>> ---
>>>>    Documentation/gpu/rfc/i915_small_bar.h   | 190
>>> +++++++++++++++++++++++
>>>>    Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>    Documentation/gpu/rfc/index.rst          |   4 +
>>>>    3 files changed, 252 insertions(+)
>>>>    create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>    create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h
>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>> new file mode 100644
>>>> index 000000000000..7bfd0cf44d35
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>> @@ -0,0 +1,190 @@
>>>> +/**
>>>> + * struct __drm_i915_memory_region_info - Describes one region as
>>>> +known to the
>>>> + * driver.
>>>> + *
>>>> + * Note this is using both struct drm_i915_query_item and struct
>>> drm_i915_query.
>>>> + * For this new query we are adding the new query id
>>>> +DRM_I915_QUERY_MEMORY_REGIONS
>>>> + * at &drm_i915_query_item.query_id.
>>>> + */
>>>> +struct __drm_i915_memory_region_info {
>>>> +   /** @region: The class:instance pair encoding */
>>>> +   struct drm_i915_gem_memory_class_instance region;
>>>> +
>>>> +   /** @rsvd0: MBZ */
>>>> +   __u32 rsvd0;
>>>> +
>>>> +   /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>> +   __u64 probed_size;
>>>> +
>>>> +   /** @unallocated_size: Estimate of memory remaining (-1 = unknown)
>>> */
>>>> +   __u64 unallocated_size;
>>>> +
>>>> +   union {
>>>> +           /** @rsvd1: MBZ */
>>>> +           __u64 rsvd1[8];
>>>> +           struct {
>>>> +                   /**
>>>> +                    * @probed_cpu_visible_size: Memory probed by the
>>> driver
>>>> +                    * that is CPU accessible. (-1 = unknown).
>>>> +                    *
>>>> +                    * This will be always be <= @probed_size, and the
>>>> +                    * remainder(if there is any) will not be CPU
>>>> +                    * accessible.
>>>> +                    */
>>>> +                   __u64 probed_cpu_visible_size;
>>>> +           };
>>>
>>> Trying to implement userspace support in Vulkan for this, I have an 
>>> additional
>>> question about the value of probed_cpu_visible_size.
>>>
>>> When is it set to -1?
>> I believe it is set to -1 if it is unknown, and/or not cpu accessible...
>>
>> Cheers!
>> ~Akeem
> 
> 
> So what should I expect on system memory?

I guess just probed_cpu_visible_size == probed_size. Or maybe we can 
just use -1 here?

> 
> What value is returned when all of probed_size is CPU visible on local 
> memory?

probed_size == probed_cpu_visible_size.

> 
> 
> Thanks,
> 
> 
> -Lionel
> 
> 
>>> I'm guessing before there is support for this value it'll be 0 (MBZ).
>>>
>>> After after it should either be the entire lmem or something smaller.
>>>
>>>
>>> -Lionel
>>>
>>>
>>>> +   };
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour,
>>>> +with added
>>>> + * extension support using struct i915_user_extension.
>>>> + *
>>>> + * Note that new buffer flags should be added here, at least for the
>>>> +stuff that
>>>> + * is immutable. Previously we would have two ioctls, one to create
>>>> +the object
>>>> + * with gem_create, and another to apply various parameters, however
>>>> +this
>>>> + * creates some ambiguity for the params which are considered
>>>> +immutable. Also in
>>>> + * general we're phasing out the various SET/GET ioctls.
>>>> + */
>>>> +struct __drm_i915_gem_create_ext {
>>>> +   /**
>>>> +    * @size: Requested size for the object.
>>>> +    *
>>>> +    * The (page-aligned) allocated size for the object will be 
>>>> returned.
>>>> +    *
>>>> +    * Note that for some devices we have might have further minimum
>>>> +    * page-size restrictions(larger than 4K), like for device 
>>>> local-memory.
>>>> +    * However in general the final size here should always reflect any
>>>> +    * rounding up, if for example using the
>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>> +    * extension to place the object in device local-memory.
>>>> +    */
>>>> +   __u64 size;
>>>> +   /**
>>>> +    * @handle: Returned handle for the object.
>>>> +    *
>>>> +    * Object handles are nonzero.
>>>> +    */
>>>> +   __u32 handle;
>>>> +   /**
>>>> +    * @flags: Optional flags.
>>>> +    *
>>>> +    * Supported values:
>>>> +    *
>>>> +    * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the
>>> kernel that
>>>> +    * the object will need to be accessed via the CPU.
>>>> +    *
>>>> +    * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE,
>>> and
>>>> +    * only strictly required on platforms where only some of the 
>>>> device
>>>> +    * memory is directly visible or mappable through the CPU, like 
>>>> on DG2+.
>>>> +    *
>>>> +    * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM,
>>> to
>>>> +    * ensure we can always spill the allocation to system memory, 
>>>> if we
>>>> +    * can't place the object in the mappable part of
>>>> +    * I915_MEMORY_CLASS_DEVICE.
>>>> +    *
>>>> +    * Note that since the kernel only supports flat-CCS on objects 
>>>> that can
>>>> +    * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore
>>> don't
>>>> +    * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS
>>> together with
>>>> +    * flat-CCS.
>>>> +    *
>>>> +    * Without this hint, the kernel will assume that non-mappable
>>>> +    * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note that
>>> the
>>>> +    * kernel can still migrate the object to the mappable part, as 
>>>> a last
>>>> +    * resort, if userspace ever CPU faults this object, but this 
>>>> might be
>>>> +    * expensive, and so ideally should be avoided.
>>>> +    */
>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>> +   __u32 flags;
>>>> +   /**
>>>> +    * @extensions: The chain of extensions to apply to this object.
>>>> +    *
>>>> +    * This will be useful in the future when we need to support 
>>>> several
>>>> +    * different extensions, and we need to apply more than one when
>>>> +    * creating the object. See struct i915_user_extension.
>>>> +    *
>>>> +    * If we don't supply any extensions then we get the same old
>>> gem_create
>>>> +    * behaviour.
>>>> +    *
>>>> +    * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>>> +    * struct drm_i915_gem_create_ext_memory_regions.
>>>> +    *
>>>> +    * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>> +    * struct drm_i915_gem_create_ext_protected_content.
>>>> +    */
>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0 #define
>>>> +I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>> +   __u64 extensions;
>>>> +};
>>>> +
>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_query_vma_info
>>>> + *
>>>> + * Given a vm and GTT address, lookup the corresponding vma,
>>>> +returning its set
>>>> + * of attributes.
>>>> + *
>>>> + * .. code-block:: C
>>>> + *
>>>> + * struct drm_i915_query_vma_info info = {};
>>>> + * struct drm_i915_query_item item = {
>>>> + *         .data_ptr = (uintptr_t)&info,
>>>> + *         .query_id = DRM_I915_QUERY_VMA_INFO,
>>>> + * };
>>>> + * struct drm_i915_query query = {
>>>> + *         .num_items = 1,
>>>> + *         .items_ptr = (uintptr_t)&item,
>>>> + * };
>>>> + * int err;
>>>> + *
>>>> + * // Unlike some other types of queries, there is no need to first 
>>>> query
>>>> + * // the size of the data_ptr blob here, since we already know 
>>>> ahead of
>>>> + * // time how big this needs to be.
>>>> + * item.length = sizeof(info);
>>>> + *
>>>> + * // Next we fill in the vm_id and ppGTT address of the vma we wish
>>>> + * // to query, before then firing off the query.
>>>> + * info.vm_id = vm_id;
>>>> + * info.offset = gtt_address;
>>>> + * err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>> + * if (err || item.length < 0) ...
>>>> + *
>>>> + * // If all went well we can now inspect the returned attributes.
>>>> + * if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>> + */
>>>> +struct __drm_i915_query_vma_info {
>>>> +   /**
>>>> +    * @vm_id: The given vm id that contains the vma. The id is the 
>>>> value
>>>> +    * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>> +    * drm_i915_gem_vm_control.vm_id.
>>>> +    */
>>>> +   __u32 vm_id;
>>>> +   /** @pad: MBZ. */
>>>> +   __u32 pad;
>>>> +   /**
>>>> +    * @offset: The corresponding ppGTT address of the vma which the
>>> kernel
>>>> +    * will use to perform the lookup.
>>>> +    */
>>>> +   __u64 offset;
>>>> +   /**
>>>> +    * @attributes: The returned attributes for the given vma.
>>>> +    *
>>>> +    * Possible values:
>>>> +    *
>>>> +    * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages
>>> backing the
>>>> +    * vma are currently CPU accessible. If this is not set then the 
>>>> vma is
>>>> +    * currently backed by I915_MEMORY_CLASS_DEVICE memory, which
>>> the CPU
>>>> +    * cannot directly access(this is only possible on discrete 
>>>> devices with
>>>> +    * a small BAR). Attempting to MMAP and fault such an object will
>>>> +    * require the kernel first synchronising any GPU work tied to the
>>>> +    * object, before then migrating the pages, either to the CPU 
>>>> accessible
>>>> +    * part of I915_MEMORY_CLASS_DEVICE, or
>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>> +    * placements permit it. See
>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>> +    *
>>>> +    * Note that this is inherently racy.
>>>> +    */
>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>> +   __u64 attributes;
>>>> +   /** @rsvd: MBZ */
>>>> +   __u32 rsvd[4];
>>>> +};
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst
>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> new file mode 100644
>>>> index 000000000000..be3d9bcdd86d
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> @@ -0,0 +1,58 @@
>>>> +==========================
>>>> +I915 Small BAR RFC Section
>>>> +==========================
>>>> +Starting from DG2 we will have resizable BAR support for device
>>>> +local-memory(i.e I915_MEMORY_CLASS_DEVICE), but in some cases the
>>>> +final BAR size might still be smaller than the total probed_size. In
>>>> +such cases, only some subset of I915_MEMORY_CLASS_DEVICE will be CPU
>>>> +accessible(for example the first 256M), while the remainder is only 
>>>> accessible
>>> via the GPU.
>>>> +
>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>> +----------------------------------------------
>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>> CPU access.
>>>> +This becomes important when placing an object in
>>>> +I915_MEMORY_CLASS_DEVICE, where underneath the device has a small
>>>> +BAR, meaning only some portion of it is CPU accessible. Without this
>>>> +flag the kernel will assume that CPU access is not required, and
>>>> +prioritize using the non-CPU visible portion of
>>> I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_gem_create_ext
>>>> +
>>>> +probed_cpu_visible_size attribute
>>>> +---------------------------------
>>>> +New struct__drm_i915_memory_region attribute which returns the total
>>>> +size of the CPU accessible portion, for the particular region. This
>>>> +should only be applicable for I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap
>>>> +with the VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the
>>>> +CPU visible portion, where the total size of the heap needs to be 
>>>> known.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_memory_region_info
>>>> +
>>>> +DRM_I915_QUERY_VMA_INFO query
>>>> +-----------------------------
>>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>>> +respective vma, and return its set of attributes. For now we only
>>>> +support DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the
>>>> +object/vma is currently placed in memory that is accessible by the
>>>> +CPU. This should always be set on devices where the CPU
>>>> +probed_cpu_visible_size of I915_MEMORY_CLASS_DEVICE matches the
>>>> +probed_size. If this is not set then CPU faulting the object will 
>>>> likely first
>>> require migrating the pages.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_query_vma_info
>>>> +
>>>> +Error Capture restrictions
>>>> +--------------------------
>>>> +With error capture we have two new restrictions:
>>>> +
>>>> +    1) Error capture is best effort on small BAR systems; if the 
>>>> pages are not
>>>> +    CPU accessible, at the time of capture, then the kernel is free 
>>>> to skip
>>>> +    trying to capture them.
>>>> +
>>>> +    2) On discrete we now reject error capture on recoverable 
>>>> contexts. In the
>>>> +    future the kernel may want to blit during error capture, when 
>>>> for example
>>>> +    something is not currently CPU accessible.
>>>> diff --git a/Documentation/gpu/rfc/index.rst
>>>> b/Documentation/gpu/rfc/index.rst index 91e93a705230..5a3bd3924ba6
>>>> 100644
>>>> --- a/Documentation/gpu/rfc/index.rst
>>>> +++ b/Documentation/gpu/rfc/index.rst
>>>> @@ -23,3 +23,7 @@ host such documentation:
>>>>    .. toctree::
>>>>
>>>>        i915_scheduler.rst
>>>> +
>>>> +.. toctree::
>>>> +
>>>> +    i915_small_bar.rst
> 
>

Lionel Landwerlin May 3, 2022, 9:15 a.m. UTC | #20

On 03/05/2022 12:07, Matthew Auld wrote:
> On 02/05/2022 19:03, Lionel Landwerlin wrote:
>> On 02/05/2022 20:58, Abodunrin, Akeem G wrote:
>>>
>>>> -----Original Message-----
>>>> From: Landwerlin, Lionel G <[e-mail]>
>>>> Sent: Monday, May 2, 2022 12:55 AM
>>>> To: Auld, Matthew <[e-mail]>; 
>>>> [e-mail]
>>>> Cc: [e-mail]; Thomas Hellström
>>>> <[e-mail]>; Bloomfield, Jon
>>>> <[e-mail]>; Daniel Vetter <[e-mail]>; 
>>>> Justen,
>>>> Jordan L <[e-mail]>; Kenneth Graunke
>>>> <[e-mail]>; Abodunrin, Akeem G
>>>> <[e-mail]>; [e-mail]
>>>> Subject: Re: [PATCH v2] drm/doc: add rfc section for small BAR uapi
>>>>
>>>> On 20/04/2022 20:13, Matthew Auld wrote:
>>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>>
>>>>> v2:
>>>>>     - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>>     - Rework error capture interactions, including no longer needing
>>>>>       NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>>     - Add probed_cpu_visible_size. (Lionel)
>>>>>
>>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>>> Cc: Thomas Hellström <[e-mail]>
>>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>>> Cc: Jon Bloomfield <[e-mail]>
>>>>> Cc: Daniel Vetter <[e-mail]>
>>>>> Cc: Jordan Justen <[e-mail]>
>>>>> Cc: Kenneth Graunke <[e-mail]>
>>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>>> Cc: [e-mail]
>>>>> ---
>>>>>    Documentation/gpu/rfc/i915_small_bar.h   | 190
>>>> +++++++++++++++++++++++
>>>>> Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>>    Documentation/gpu/rfc/index.rst          |   4 +
>>>>>    3 files changed, 252 insertions(+)
>>>>>    create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>>    create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>>
>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h
>>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>>> new file mode 100644
>>>>> index 000000000000..7bfd0cf44d35
>>>>> --- /dev/null
>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>>> @@ -0,0 +1,190 @@
>>>>> +/**
>>>>> + * struct __drm_i915_memory_region_info - Describes one region as
>>>>> +known to the
>>>>> + * driver.
>>>>> + *
>>>>> + * Note this is using both struct drm_i915_query_item and struct
>>>> drm_i915_query.
>>>>> + * For this new query we are adding the new query id
>>>>> +DRM_I915_QUERY_MEMORY_REGIONS
>>>>> + * at &drm_i915_query_item.query_id.
>>>>> + */
>>>>> +struct __drm_i915_memory_region_info {
>>>>> +   /** @region: The class:instance pair encoding */
>>>>> +   struct drm_i915_gem_memory_class_instance region;
>>>>> +
>>>>> +   /** @rsvd0: MBZ */
>>>>> +   __u32 rsvd0;
>>>>> +
>>>>> +   /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>>> +   __u64 probed_size;
>>>>> +
>>>>> +   /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>>> unknown)
>>>> */
>>>>> +   __u64 unallocated_size;
>>>>> +
>>>>> +   union {
>>>>> +           /** @rsvd1: MBZ */
>>>>> +           __u64 rsvd1[8];
>>>>> +           struct {
>>>>> +                   /**
>>>>> +                    * @probed_cpu_visible_size: Memory probed by the
>>>> driver
>>>>> +                    * that is CPU accessible. (-1 = unknown).
>>>>> +                    *
>>>>> +                    * This will be always be <= @probed_size, and 
>>>>> the
>>>>> +                    * remainder(if there is any) will not be CPU
>>>>> +                    * accessible.
>>>>> +                    */
>>>>> +                   __u64 probed_cpu_visible_size;
>>>>> +           };
>>>>
>>>> Trying to implement userspace support in Vulkan for this, I have an 
>>>> additional
>>>> question about the value of probed_cpu_visible_size.
>>>>
>>>> When is it set to -1?
>>> I believe it is set to -1 if it is unknown, and/or not cpu 
>>> accessible...
>>>
>>> Cheers!
>>> ~Akeem
>>
>>
>> So what should I expect on system memory?
>
> I guess just probed_cpu_visible_size == probed_size. Or maybe we can 
> just use -1 here?
>
>>
>> What value is returned when all of probed_size is CPU visible on 
>> local memory?
>
> probed_size == probed_cpu_visible_size.


Thanks, looks good to me.

Then maybe we should update the comment to say that.

Looks like there are no cases where we'll get -1.


-Lionel


>
>>
>>
>> Thanks,
>>
>>
>> -Lionel
>>
>>
>>>> I'm guessing before there is support for this value it'll be 0 (MBZ).
>>>>
>>>> After after it should either be the entire lmem or something smaller.
>>>>
>>>>
>>>> -Lionel
>>>>
>>>>
>>>>> +   };
>>>>> +};
>>>>> +
>>>>> +/**
>>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour,
>>>>> +with added
>>>>> + * extension support using struct i915_user_extension.
>>>>> + *
>>>>> + * Note that new buffer flags should be added here, at least for the
>>>>> +stuff that
>>>>> + * is immutable. Previously we would have two ioctls, one to create
>>>>> +the object
>>>>> + * with gem_create, and another to apply various parameters, however
>>>>> +this
>>>>> + * creates some ambiguity for the params which are considered
>>>>> +immutable. Also in
>>>>> + * general we're phasing out the various SET/GET ioctls.
>>>>> + */
>>>>> +struct __drm_i915_gem_create_ext {
>>>>> +   /**
>>>>> +    * @size: Requested size for the object.
>>>>> +    *
>>>>> +    * The (page-aligned) allocated size for the object will be 
>>>>> returned.
>>>>> +    *
>>>>> +    * Note that for some devices we have might have further minimum
>>>>> +    * page-size restrictions(larger than 4K), like for device 
>>>>> local-memory.
>>>>> +    * However in general the final size here should always 
>>>>> reflect any
>>>>> +    * rounding up, if for example using the
>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>>> +    * extension to place the object in device local-memory.
>>>>> +    */
>>>>> +   __u64 size;
>>>>> +   /**
>>>>> +    * @handle: Returned handle for the object.
>>>>> +    *
>>>>> +    * Object handles are nonzero.
>>>>> +    */
>>>>> +   __u32 handle;
>>>>> +   /**
>>>>> +    * @flags: Optional flags.
>>>>> +    *
>>>>> +    * Supported values:
>>>>> +    *
>>>>> +    * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the
>>>> kernel that
>>>>> +    * the object will need to be accessed via the CPU.
>>>>> +    *
>>>>> +    * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE,
>>>> and
>>>>> +    * only strictly required on platforms where only some of the 
>>>>> device
>>>>> +    * memory is directly visible or mappable through the CPU, 
>>>>> like on DG2+.
>>>>> +    *
>>>>> +    * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM,
>>>> to
>>>>> +    * ensure we can always spill the allocation to system memory, 
>>>>> if we
>>>>> +    * can't place the object in the mappable part of
>>>>> +    * I915_MEMORY_CLASS_DEVICE.
>>>>> +    *
>>>>> +    * Note that since the kernel only supports flat-CCS on 
>>>>> objects that can
>>>>> +    * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore
>>>> don't
>>>>> +    * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS
>>>> together with
>>>>> +    * flat-CCS.
>>>>> +    *
>>>>> +    * Without this hint, the kernel will assume that non-mappable
>>>>> +    * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>>>> that
>>>> the
>>>>> +    * kernel can still migrate the object to the mappable part, 
>>>>> as a last
>>>>> +    * resort, if userspace ever CPU faults this object, but this 
>>>>> might be
>>>>> +    * expensive, and so ideally should be avoided.
>>>>> +    */
>>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>>> +   __u32 flags;
>>>>> +   /**
>>>>> +    * @extensions: The chain of extensions to apply to this object.
>>>>> +    *
>>>>> +    * This will be useful in the future when we need to support 
>>>>> several
>>>>> +    * different extensions, and we need to apply more than one when
>>>>> +    * creating the object. See struct i915_user_extension.
>>>>> +    *
>>>>> +    * If we don't supply any extensions then we get the same old
>>>> gem_create
>>>>> +    * behaviour.
>>>>> +    *
>>>>> +    * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>>>> +    * struct drm_i915_gem_create_ext_memory_regions.
>>>>> +    *
>>>>> +    * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>>> +    * struct drm_i915_gem_create_ext_protected_content.
>>>>> +    */
>>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0 #define
>>>>> +I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>>> +   __u64 extensions;
>>>>> +};
>>>>> +
>>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>>> +
>>>>> +/**
>>>>> + * struct __drm_i915_query_vma_info
>>>>> + *
>>>>> + * Given a vm and GTT address, lookup the corresponding vma,
>>>>> +returning its set
>>>>> + * of attributes.
>>>>> + *
>>>>> + * .. code-block:: C
>>>>> + *
>>>>> + * struct drm_i915_query_vma_info info = {};
>>>>> + * struct drm_i915_query_item item = {
>>>>> + *         .data_ptr = (uintptr_t)&info,
>>>>> + *         .query_id = DRM_I915_QUERY_VMA_INFO,
>>>>> + * };
>>>>> + * struct drm_i915_query query = {
>>>>> + *         .num_items = 1,
>>>>> + *         .items_ptr = (uintptr_t)&item,
>>>>> + * };
>>>>> + * int err;
>>>>> + *
>>>>> + * // Unlike some other types of queries, there is no need to 
>>>>> first query
>>>>> + * // the size of the data_ptr blob here, since we already know 
>>>>> ahead of
>>>>> + * // time how big this needs to be.
>>>>> + * item.length = sizeof(info);
>>>>> + *
>>>>> + * // Next we fill in the vm_id and ppGTT address of the vma we wish
>>>>> + * // to query, before then firing off the query.
>>>>> + * info.vm_id = vm_id;
>>>>> + * info.offset = gtt_address;
>>>>> + * err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>>> + * if (err || item.length < 0) ...
>>>>> + *
>>>>> + * // If all went well we can now inspect the returned attributes.
>>>>> + * if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>>> + */
>>>>> +struct __drm_i915_query_vma_info {
>>>>> +   /**
>>>>> +    * @vm_id: The given vm id that contains the vma. The id is 
>>>>> the value
>>>>> +    * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>>> +    * drm_i915_gem_vm_control.vm_id.
>>>>> +    */
>>>>> +   __u32 vm_id;
>>>>> +   /** @pad: MBZ. */
>>>>> +   __u32 pad;
>>>>> +   /**
>>>>> +    * @offset: The corresponding ppGTT address of the vma which the
>>>> kernel
>>>>> +    * will use to perform the lookup.
>>>>> +    */
>>>>> +   __u64 offset;
>>>>> +   /**
>>>>> +    * @attributes: The returned attributes for the given vma.
>>>>> +    *
>>>>> +    * Possible values:
>>>>> +    *
>>>>> +    * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages
>>>> backing the
>>>>> +    * vma are currently CPU accessible. If this is not set then 
>>>>> the vma is
>>>>> +    * currently backed by I915_MEMORY_CLASS_DEVICE memory, which
>>>> the CPU
>>>>> +    * cannot directly access(this is only possible on discrete 
>>>>> devices with
>>>>> +    * a small BAR). Attempting to MMAP and fault such an object will
>>>>> +    * require the kernel first synchronising any GPU work tied to 
>>>>> the
>>>>> +    * object, before then migrating the pages, either to the CPU 
>>>>> accessible
>>>>> +    * part of I915_MEMORY_CLASS_DEVICE, or
>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>>> +    * placements permit it. See
>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>>> +    *
>>>>> +    * Note that this is inherently racy.
>>>>> +    */
>>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>>> +   __u64 attributes;
>>>>> +   /** @rsvd: MBZ */
>>>>> +   __u32 rsvd[4];
>>>>> +};
>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst
>>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>> new file mode 100644
>>>>> index 000000000000..be3d9bcdd86d
>>>>> --- /dev/null
>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>> @@ -0,0 +1,58 @@
>>>>> +==========================
>>>>> +I915 Small BAR RFC Section
>>>>> +==========================
>>>>> +Starting from DG2 we will have resizable BAR support for device
>>>>> +local-memory(i.e I915_MEMORY_CLASS_DEVICE), but in some cases the
>>>>> +final BAR size might still be smaller than the total probed_size. In
>>>>> +such cases, only some subset of I915_MEMORY_CLASS_DEVICE will be CPU
>>>>> +accessible(for example the first 256M), while the remainder is 
>>>>> only accessible
>>>> via the GPU.
>>>>> +
>>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>>> +----------------------------------------------
>>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>>> CPU access.
>>>>> +This becomes important when placing an object in
>>>>> +I915_MEMORY_CLASS_DEVICE, where underneath the device has a small
>>>>> +BAR, meaning only some portion of it is CPU accessible. Without this
>>>>> +flag the kernel will assume that CPU access is not required, and
>>>>> +prioritize using the non-CPU visible portion of
>>>> I915_MEMORY_CLASS_DEVICE.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_gem_create_ext
>>>>> +
>>>>> +probed_cpu_visible_size attribute
>>>>> +---------------------------------
>>>>> +New struct__drm_i915_memory_region attribute which returns the total
>>>>> +size of the CPU accessible portion, for the particular region. This
>>>>> +should only be applicable for I915_MEMORY_CLASS_DEVICE.
>>>>> +
>>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap
>>>>> +with the VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the
>>>>> +CPU visible portion, where the total size of the heap needs to be 
>>>>> known.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_memory_region_info
>>>>> +
>>>>> +DRM_I915_QUERY_VMA_INFO query
>>>>> +-----------------------------
>>>>> +Query the attributes of some vma. Given a vm and GTT offset, find 
>>>>> the
>>>>> +respective vma, and return its set of attributes. For now we only
>>>>> +support DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the
>>>>> +object/vma is currently placed in memory that is accessible by the
>>>>> +CPU. This should always be set on devices where the CPU
>>>>> +probed_cpu_visible_size of I915_MEMORY_CLASS_DEVICE matches the
>>>>> +probed_size. If this is not set then CPU faulting the object will 
>>>>> likely first
>>>> require migrating the pages.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_query_vma_info
>>>>> +
>>>>> +Error Capture restrictions
>>>>> +--------------------------
>>>>> +With error capture we have two new restrictions:
>>>>> +
>>>>> +    1) Error capture is best effort on small BAR systems; if the 
>>>>> pages are not
>>>>> +    CPU accessible, at the time of capture, then the kernel is 
>>>>> free to skip
>>>>> +    trying to capture them.
>>>>> +
>>>>> +    2) On discrete we now reject error capture on recoverable 
>>>>> contexts. In the
>>>>> +    future the kernel may want to blit during error capture, when 
>>>>> for example
>>>>> +    something is not currently CPU accessible.
>>>>> diff --git a/Documentation/gpu/rfc/index.rst
>>>>> b/Documentation/gpu/rfc/index.rst index 91e93a705230..5a3bd3924ba6
>>>>> 100644
>>>>> --- a/Documentation/gpu/rfc/index.rst
>>>>> +++ b/Documentation/gpu/rfc/index.rst
>>>>> @@ -23,3 +23,7 @@ host such documentation:
>>>>>    .. toctree::
>>>>>
>>>>>        i915_scheduler.rst
>>>>> +
>>>>> +.. toctree::
>>>>> +
>>>>> +    i915_small_bar.rst
>>
>>

Matthew Auld May 3, 2022, 10:22 a.m. UTC | #21

On 02/05/2022 09:53, Lionel Landwerlin wrote:
> On 02/05/2022 10:54, Lionel Landwerlin wrote:
>> On 20/04/2022 20:13, Matthew Auld wrote:
>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>
>>> v2:
>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>    - Rework error capture interactions, including no longer needing
>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>    - Add probed_cpu_visible_size. (Lionel)
>>>
>>> Signed-off-by: Matthew Auld <[e-mail]>
>>> Cc: Thomas Hellström <[e-mail]>
>>> Cc: Lionel Landwerlin <[e-mail]>
>>> Cc: Jon Bloomfield <[e-mail]>
>>> Cc: Daniel Vetter <[e-mail]>
>>> Cc: Jordan Justen <[e-mail]>
>>> Cc: Kenneth Graunke <[e-mail]>
>>> Cc: Akeem G Abodunrin <[e-mail]>
>>> Cc: [e-mail]
>>> ---
>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 +++++++++++++++++++++++
>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>   3 files changed, 252 insertions(+)
>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>> new file mode 100644
>>> index 000000000000..7bfd0cf44d35
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>> @@ -0,0 +1,190 @@
>>> +/**
>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>> known to the
>>> + * driver.
>>> + *
>>> + * Note this is using both struct drm_i915_query_item and struct 
>>> drm_i915_query.
>>> + * For this new query we are adding the new query id 
>>> DRM_I915_QUERY_MEMORY_REGIONS
>>> + * at &drm_i915_query_item.query_id.
>>> + */
>>> +struct __drm_i915_memory_region_info {
>>> +    /** @region: The class:instance pair encoding */
>>> +    struct drm_i915_gem_memory_class_instance region;
>>> +
>>> +    /** @rsvd0: MBZ */
>>> +    __u32 rsvd0;
>>> +
>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>> +    __u64 probed_size;
>>> +
>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>> unknown) */
>>> +    __u64 unallocated_size;
>>> +
>>> +    union {
>>> +        /** @rsvd1: MBZ */
>>> +        __u64 rsvd1[8];
>>> +        struct {
>>> +            /**
>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>> +             * that is CPU accessible. (-1 = unknown).
>>> +             *
>>> +             * This will be always be <= @probed_size, and the
>>> +             * remainder(if there is any) will not be CPU
>>> +             * accessible.
>>> +             */
>>> +            __u64 probed_cpu_visible_size;
>>> +        };
>>
>>
>> Trying to implement userspace support in Vulkan for this, I have an 
>> additional question about the value of probed_cpu_visible_size.
>>
>> When is it set to -1?
>>
>> I'm guessing before there is support for this value it'll be 0 (MBZ).
>>
>> After after it should either be the entire lmem or something smaller.
>>
>>
>> -Lionel
> 
> 
> Other pain point of this new uAPI, previously we could query the 
> unallocated size for each heap.

unallocated_size should always give the same value as probed_size. We 
have the avail tracking, but we don't currently expose that through 
unallocated_size, due to lack of real userspace/user etc.

> 
> Now lmem is effectively divided into 2 heaps, but unallocated_size is 
> tracking allocation from both parts of lmem.

Yeah, if we ever properly expose the unallocated_size, then we could 
also just add unallocated_cpu_visible_size.

> 
> Is adding new I915_MEMORY_CLASS_DEVICE_NON_MAPPABLE out of question?

I don't think it's out of the question...

I guess user-space should be able to get the current flag behaviour just 
by specifying: device, system. And it does give more flexibly to allow 
something like: device, device-nm, smem.

We can also drop the probed_cpu_visible_size, which would now just be 
the probed_size with device/device-nm. And if we lack device-nm, then 
the entire thing must be CPU mappable.

One of the downsides though, is that we can no longer easily mix object 
pages from both device + device-nm, which we could previously do when we 
didn't specify the flag. At least according to the current 
design/behaviour for @regions that would not be allowed. I guess some 
kind of new flag like ALLOC_MIXED or so? Although currently that is only 
possible with device + device-nm in ttm/i915.

> 
> 
> -Lionel
> 
> 
>>
>>
>>> +    };
>>> +};
>>> +
>>> +/**
>>> + * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, 
>>> with added
>>> + * extension support using struct i915_user_extension.
>>> + *
>>> + * Note that new buffer flags should be added here, at least for the 
>>> stuff that
>>> + * is immutable. Previously we would have two ioctls, one to create 
>>> the object
>>> + * with gem_create, and another to apply various parameters, however 
>>> this
>>> + * creates some ambiguity for the params which are considered 
>>> immutable. Also in
>>> + * general we're phasing out the various SET/GET ioctls.
>>> + */
>>> +struct __drm_i915_gem_create_ext {
>>> +    /**
>>> +     * @size: Requested size for the object.
>>> +     *
>>> +     * The (page-aligned) allocated size for the object will be 
>>> returned.
>>> +     *
>>> +     * Note that for some devices we have might have further minimum
>>> +     * page-size restrictions(larger than 4K), like for device 
>>> local-memory.
>>> +     * However in general the final size here should always reflect any
>>> +     * rounding up, if for example using the 
>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>> +     * extension to place the object in device local-memory.
>>> +     */
>>> +    __u64 size;
>>> +    /**
>>> +     * @handle: Returned handle for the object.
>>> +     *
>>> +     * Object handles are nonzero.
>>> +     */
>>> +    __u32 handle;
>>> +    /**
>>> +     * @flags: Optional flags.
>>> +     *
>>> +     * Supported values:
>>> +     *
>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>> kernel that
>>> +     * the object will need to be accessed via the CPU.
>>> +     *
>>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
>>> +     * only strictly required on platforms where only some of the 
>>> device
>>> +     * memory is directly visible or mappable through the CPU, like 
>>> on DG2+.
>>> +     *
>>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>>> +     * ensure we can always spill the allocation to system memory, 
>>> if we
>>> +     * can't place the object in the mappable part of
>>> +     * I915_MEMORY_CLASS_DEVICE.
>>> +     *
>>> +     * Note that since the kernel only supports flat-CCS on objects 
>>> that can
>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>>> +     * flat-CCS.
>>> +     *
>>> +     * Without this hint, the kernel will assume that non-mappable
>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>> that the
>>> +     * kernel can still migrate the object to the mappable part, as 
>>> a last
>>> +     * resort, if userspace ever CPU faults this object, but this 
>>> might be
>>> +     * expensive, and so ideally should be avoided.
>>> +     */
>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>> +    __u32 flags;
>>> +    /**
>>> +     * @extensions: The chain of extensions to apply to this object.
>>> +     *
>>> +     * This will be useful in the future when we need to support 
>>> several
>>> +     * different extensions, and we need to apply more than one when
>>> +     * creating the object. See struct i915_user_extension.
>>> +     *
>>> +     * If we don't supply any extensions then we get the same old 
>>> gem_create
>>> +     * behaviour.
>>> +     *
>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>> +     *
>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>> +     */
>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>> +    __u64 extensions;
>>> +};
>>> +
>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>> +
>>> +/**
>>> + * struct __drm_i915_query_vma_info
>>> + *
>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>> returning its set
>>> + * of attributes.
>>> + *
>>> + * .. code-block:: C
>>> + *
>>> + *    struct drm_i915_query_vma_info info = {};
>>> + *    struct drm_i915_query_item item = {
>>> + *        .data_ptr = (uintptr_t)&info,
>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>> + *    };
>>> + *    struct drm_i915_query query = {
>>> + *        .num_items = 1,
>>> + *        .items_ptr = (uintptr_t)&item,
>>> + *    };
>>> + *    int err;
>>> + *
>>> + *    // Unlike some other types of queries, there is no need to 
>>> first query
>>> + *    // the size of the data_ptr blob here, since we already know 
>>> ahead of
>>> + *    // time how big this needs to be.
>>> + *    item.length = sizeof(info);
>>> + *
>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we wish
>>> + *    // to query, before then firing off the query.
>>> + *    info.vm_id = vm_id;
>>> + *    info.offset = gtt_address;
>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>> + *    if (err || item.length < 0) ...
>>> + *
>>> + *    // If all went well we can now inspect the returned attributes.
>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>> + */
>>> +struct __drm_i915_query_vma_info {
>>> +    /**
>>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>>> value
>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>> +     * drm_i915_gem_vm_control.vm_id.
>>> +     */
>>> +    __u32 vm_id;
>>> +    /** @pad: MBZ. */
>>> +    __u32 pad;
>>> +    /**
>>> +     * @offset: The corresponding ppGTT address of the vma which the 
>>> kernel
>>> +     * will use to perform the lookup.
>>> +     */
>>> +    __u64 offset;
>>> +    /**
>>> +     * @attributes: The returned attributes for the given vma.
>>> +     *
>>> +     * Possible values:
>>> +     *
>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>> backing the
>>> +     * vma are currently CPU accessible. If this is not set then the 
>>> vma is
>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>>> the CPU
>>> +     * cannot directly access(this is only possible on discrete 
>>> devices with
>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>> +     * require the kernel first synchronising any GPU work tied to the
>>> +     * object, before then migrating the pages, either to the CPU 
>>> accessible
>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>> I915_MEMORY_CLASS_SYSTEM, if the
>>> +     * placements permit it. See 
>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>> +     *
>>> +     * Note that this is inherently racy.
>>> +     */
>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>> +    __u64 attributes;
>>> +    /** @rsvd: MBZ */
>>> +    __u32 rsvd[4];
>>> +};
>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>> new file mode 100644
>>> index 000000000000..be3d9bcdd86d
>>> --- /dev/null
>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>> @@ -0,0 +1,58 @@
>>> +==========================
>>> +I915 Small BAR RFC Section
>>> +==========================
>>> +Starting from DG2 we will have resizable BAR support for device 
>>> local-memory(i.e
>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>> might still be
>>> +smaller than the total probed_size. In such cases, only some subset of
>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>> first 256M),
>>> +while the remainder is only accessible via the GPU.
>>> +
>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>> +----------------------------------------------
>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>> CPU access.
>>> +This becomes important when placing an object in 
>>> I915_MEMORY_CLASS_DEVICE, where
>>> +underneath the device has a small BAR, meaning only some portion of 
>>> it is CPU
>>> +accessible. Without this flag the kernel will assume that CPU access 
>>> is not
>>> +required, and prioritize using the non-CPU visible portion of
>>> +I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_gem_create_ext
>>> +
>>> +probed_cpu_visible_size attribute
>>> +---------------------------------
>>> +New struct__drm_i915_memory_region attribute which returns the total 
>>> size of the
>>> +CPU accessible portion, for the particular region. This should only be
>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>> +
>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>> with the
>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>> visible portion,
>>> +where the total size of the heap needs to be known.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_memory_region_info
>>> +
>>> +DRM_I915_QUERY_VMA_INFO query
>>> +-----------------------------
>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>> +respective vma, and return its set of attributes. For now we only 
>>> support
>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>>> +currently placed in memory that is accessible by the CPU. This 
>>> should always be
>>> +set on devices where the CPU probed_cpu_visible_size of 
>>> I915_MEMORY_CLASS_DEVICE
>>> +matches the probed_size. If this is not set then CPU faulting the 
>>> object will
>>> +likely first require migrating the pages.
>>> +
>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>> +   :functions: __drm_i915_query_vma_info
>>> +
>>> +Error Capture restrictions
>>> +--------------------------
>>> +With error capture we have two new restrictions:
>>> +
>>> +    1) Error capture is best effort on small BAR systems; if the 
>>> pages are not
>>> +    CPU accessible, at the time of capture, then the kernel is free 
>>> to skip
>>> +    trying to capture them.
>>> +
>>> +    2) On discrete we now reject error capture on recoverable 
>>> contexts. In the
>>> +    future the kernel may want to blit during error capture, when 
>>> for example
>>> +    something is not currently CPU accessible.
>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>> b/Documentation/gpu/rfc/index.rst
>>> index 91e93a705230..5a3bd3924ba6 100644
>>> --- a/Documentation/gpu/rfc/index.rst
>>> +++ b/Documentation/gpu/rfc/index.rst
>>> @@ -23,3 +23,7 @@ host such documentation:
>>>   .. toctree::
>>>         i915_scheduler.rst
>>> +
>>> +.. toctree::
>>> +
>>> +    i915_small_bar.rst
>>
>>
>

Lionel Landwerlin May 3, 2022, 10:39 a.m. UTC | #22

On 03/05/2022 13:22, Matthew Auld wrote:
> On 02/05/2022 09:53, Lionel Landwerlin wrote:
>> On 02/05/2022 10:54, Lionel Landwerlin wrote:
>>> On 20/04/2022 20:13, Matthew Auld wrote:
>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>
>>>> v2:
>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>    - Rework error capture interactions, including no longer needing
>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>
>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>> Cc: Thomas Hellström <[e-mail]>
>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>> Cc: Jon Bloomfield <[e-mail]>
>>>> Cc: Daniel Vetter <[e-mail]>
>>>> Cc: Jordan Justen <[e-mail]>
>>>> Cc: Kenneth Graunke <[e-mail]>
>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>> Cc: [e-mail]
>>>> ---
>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>> +++++++++++++++++++++++
>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>   3 files changed, 252 insertions(+)
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>> new file mode 100644
>>>> index 000000000000..7bfd0cf44d35
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>> @@ -0,0 +1,190 @@
>>>> +/**
>>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>>> known to the
>>>> + * driver.
>>>> + *
>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>> drm_i915_query.
>>>> + * For this new query we are adding the new query id 
>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>> + * at &drm_i915_query_item.query_id.
>>>> + */
>>>> +struct __drm_i915_memory_region_info {
>>>> +    /** @region: The class:instance pair encoding */
>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>> +
>>>> +    /** @rsvd0: MBZ */
>>>> +    __u32 rsvd0;
>>>> +
>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>> +    __u64 probed_size;
>>>> +
>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>> unknown) */
>>>> +    __u64 unallocated_size;
>>>> +
>>>> +    union {
>>>> +        /** @rsvd1: MBZ */
>>>> +        __u64 rsvd1[8];
>>>> +        struct {
>>>> +            /**
>>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>>> +             * that is CPU accessible. (-1 = unknown).
>>>> +             *
>>>> +             * This will be always be <= @probed_size, and the
>>>> +             * remainder(if there is any) will not be CPU
>>>> +             * accessible.
>>>> +             */
>>>> +            __u64 probed_cpu_visible_size;
>>>> +        };
>>>
>>>
>>> Trying to implement userspace support in Vulkan for this, I have an 
>>> additional question about the value of probed_cpu_visible_size.
>>>
>>> When is it set to -1?
>>>
>>> I'm guessing before there is support for this value it'll be 0 (MBZ).
>>>
>>> After after it should either be the entire lmem or something smaller.
>>>
>>>
>>> -Lionel
>>
>>
>> Other pain point of this new uAPI, previously we could query the 
>> unallocated size for each heap.
>
> unallocated_size should always give the same value as probed_size. We 
> have the avail tracking, but we don't currently expose that through 
> unallocated_size, due to lack of real userspace/user etc.
>
>>
>> Now lmem is effectively divided into 2 heaps, but unallocated_size is 
>> tracking allocation from both parts of lmem.
>
> Yeah, if we ever properly expose the unallocated_size, then we could 
> also just add unallocated_cpu_visible_size.
>
>>
>> Is adding new I915_MEMORY_CLASS_DEVICE_NON_MAPPABLE out of question?
>
> I don't think it's out of the question...
>
> I guess user-space should be able to get the current flag behaviour 
> just by specifying: device, system. And it does give more flexibly to 
> allow something like: device, device-nm, smem.
>
> We can also drop the probed_cpu_visible_size, which would now just be 
> the probed_size with device/device-nm. And if we lack device-nm, then 
> the entire thing must be CPU mappable.
>
> One of the downsides though, is that we can no longer easily mix 
> object pages from both device + device-nm, which we could previously 
> do when we didn't specify the flag. At least according to the current 
> design/behaviour for @regions that would not be allowed. I guess some 
> kind of new flag like ALLOC_MIXED or so? Although currently that is 
> only possible with device + device-nm in ttm/i915.


Thanks, I wasn't aware of the restrictions.

Adding unallocated_cpu_visible_size would be great.


-Lionel


>
>>
>>
>> -Lionel
>>
>>
>>>
>>>
>>>> +    };
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>> behaviour, with added
>>>> + * extension support using struct i915_user_extension.
>>>> + *
>>>> + * Note that new buffer flags should be added here, at least for 
>>>> the stuff that
>>>> + * is immutable. Previously we would have two ioctls, one to 
>>>> create the object
>>>> + * with gem_create, and another to apply various parameters, 
>>>> however this
>>>> + * creates some ambiguity for the params which are considered 
>>>> immutable. Also in
>>>> + * general we're phasing out the various SET/GET ioctls.
>>>> + */
>>>> +struct __drm_i915_gem_create_ext {
>>>> +    /**
>>>> +     * @size: Requested size for the object.
>>>> +     *
>>>> +     * The (page-aligned) allocated size for the object will be 
>>>> returned.
>>>> +     *
>>>> +     * Note that for some devices we have might have further minimum
>>>> +     * page-size restrictions(larger than 4K), like for device 
>>>> local-memory.
>>>> +     * However in general the final size here should always 
>>>> reflect any
>>>> +     * rounding up, if for example using the 
>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>> +     * extension to place the object in device local-memory.
>>>> +     */
>>>> +    __u64 size;
>>>> +    /**
>>>> +     * @handle: Returned handle for the object.
>>>> +     *
>>>> +     * Object handles are nonzero.
>>>> +     */
>>>> +    __u32 handle;
>>>> +    /**
>>>> +     * @flags: Optional flags.
>>>> +     *
>>>> +     * Supported values:
>>>> +     *
>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>> kernel that
>>>> +     * the object will need to be accessed via the CPU.
>>>> +     *
>>>> +     * Only valid when placing objects in 
>>>> I915_MEMORY_CLASS_DEVICE, and
>>>> +     * only strictly required on platforms where only some of the 
>>>> device
>>>> +     * memory is directly visible or mappable through the CPU, 
>>>> like on DG2+.
>>>> +     *
>>>> +     * One of the placements MUST also be 
>>>> I915_MEMORY_CLASS_SYSTEM, to
>>>> +     * ensure we can always spill the allocation to system memory, 
>>>> if we
>>>> +     * can't place the object in the mappable part of
>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>> +     *
>>>> +     * Note that since the kernel only supports flat-CCS on 
>>>> objects that can
>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>>> don't
>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together 
>>>> with
>>>> +     * flat-CCS.
>>>> +     *
>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>>> that the
>>>> +     * kernel can still migrate the object to the mappable part, 
>>>> as a last
>>>> +     * resort, if userspace ever CPU faults this object, but this 
>>>> might be
>>>> +     * expensive, and so ideally should be avoided.
>>>> +     */
>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>> +    __u32 flags;
>>>> +    /**
>>>> +     * @extensions: The chain of extensions to apply to this object.
>>>> +     *
>>>> +     * This will be useful in the future when we need to support 
>>>> several
>>>> +     * different extensions, and we need to apply more than one when
>>>> +     * creating the object. See struct i915_user_extension.
>>>> +     *
>>>> +     * If we don't supply any extensions then we get the same old 
>>>> gem_create
>>>> +     * behaviour.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>> +     */
>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>> +    __u64 extensions;
>>>> +};
>>>> +
>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_query_vma_info
>>>> + *
>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>> returning its set
>>>> + * of attributes.
>>>> + *
>>>> + * .. code-block:: C
>>>> + *
>>>> + *    struct drm_i915_query_vma_info info = {};
>>>> + *    struct drm_i915_query_item item = {
>>>> + *        .data_ptr = (uintptr_t)&info,
>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>> + *    };
>>>> + *    struct drm_i915_query query = {
>>>> + *        .num_items = 1,
>>>> + *        .items_ptr = (uintptr_t)&item,
>>>> + *    };
>>>> + *    int err;
>>>> + *
>>>> + *    // Unlike some other types of queries, there is no need to 
>>>> first query
>>>> + *    // the size of the data_ptr blob here, since we already know 
>>>> ahead of
>>>> + *    // time how big this needs to be.
>>>> + *    item.length = sizeof(info);
>>>> + *
>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we 
>>>> wish
>>>> + *    // to query, before then firing off the query.
>>>> + *    info.vm_id = vm_id;
>>>> + *    info.offset = gtt_address;
>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>> + *    if (err || item.length < 0) ...
>>>> + *
>>>> + *    // If all went well we can now inspect the returned attributes.
>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>> + */
>>>> +struct __drm_i915_query_vma_info {
>>>> +    /**
>>>> +     * @vm_id: The given vm id that contains the vma. The id is 
>>>> the value
>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>> +     */
>>>> +    __u32 vm_id;
>>>> +    /** @pad: MBZ. */
>>>> +    __u32 pad;
>>>> +    /**
>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>> the kernel
>>>> +     * will use to perform the lookup.
>>>> +     */
>>>> +    __u64 offset;
>>>> +    /**
>>>> +     * @attributes: The returned attributes for the given vma.
>>>> +     *
>>>> +     * Possible values:
>>>> +     *
>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>>> backing the
>>>> +     * vma are currently CPU accessible. If this is not set then 
>>>> the vma is
>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>>>> the CPU
>>>> +     * cannot directly access(this is only possible on discrete 
>>>> devices with
>>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>>> +     * require the kernel first synchronising any GPU work tied to 
>>>> the
>>>> +     * object, before then migrating the pages, either to the CPU 
>>>> accessible
>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>> +     * placements permit it. See 
>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>> +     *
>>>> +     * Note that this is inherently racy.
>>>> +     */
>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>> +    __u64 attributes;
>>>> +    /** @rsvd: MBZ */
>>>> +    __u32 rsvd[4];
>>>> +};
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> new file mode 100644
>>>> index 000000000000..be3d9bcdd86d
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> @@ -0,0 +1,58 @@
>>>> +==========================
>>>> +I915 Small BAR RFC Section
>>>> +==========================
>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>> local-memory(i.e
>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>> might still be
>>>> +smaller than the total probed_size. In such cases, only some 
>>>> subset of
>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>> first 256M),
>>>> +while the remainder is only accessible via the GPU.
>>>> +
>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>> +----------------------------------------------
>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>> CPU access.
>>>> +This becomes important when placing an object in 
>>>> I915_MEMORY_CLASS_DEVICE, where
>>>> +underneath the device has a small BAR, meaning only some portion 
>>>> of it is CPU
>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>> access is not
>>>> +required, and prioritize using the non-CPU visible portion of
>>>> +I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_gem_create_ext
>>>> +
>>>> +probed_cpu_visible_size attribute
>>>> +---------------------------------
>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>> total size of the
>>>> +CPU accessible portion, for the particular region. This should 
>>>> only be
>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>>> with the
>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>> visible portion,
>>>> +where the total size of the heap needs to be known.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_memory_region_info
>>>> +
>>>> +DRM_I915_QUERY_VMA_INFO query
>>>> +-----------------------------
>>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>>> +respective vma, and return its set of attributes. For now we only 
>>>> support
>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the 
>>>> object/vma is
>>>> +currently placed in memory that is accessible by the CPU. This 
>>>> should always be
>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>> I915_MEMORY_CLASS_DEVICE
>>>> +matches the probed_size. If this is not set then CPU faulting the 
>>>> object will
>>>> +likely first require migrating the pages.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_query_vma_info
>>>> +
>>>> +Error Capture restrictions
>>>> +--------------------------
>>>> +With error capture we have two new restrictions:
>>>> +
>>>> +    1) Error capture is best effort on small BAR systems; if the 
>>>> pages are not
>>>> +    CPU accessible, at the time of capture, then the kernel is 
>>>> free to skip
>>>> +    trying to capture them.
>>>> +
>>>> +    2) On discrete we now reject error capture on recoverable 
>>>> contexts. In the
>>>> +    future the kernel may want to blit during error capture, when 
>>>> for example
>>>> +    something is not currently CPU accessible.
>>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>>> b/Documentation/gpu/rfc/index.rst
>>>> index 91e93a705230..5a3bd3924ba6 100644
>>>> --- a/Documentation/gpu/rfc/index.rst
>>>> +++ b/Documentation/gpu/rfc/index.rst
>>>> @@ -23,3 +23,7 @@ host such documentation:
>>>>   .. toctree::
>>>>         i915_scheduler.rst
>>>> +
>>>> +.. toctree::
>>>> +
>>>> +    i915_small_bar.rst
>>>
>>>
>>

Tvrtko Ursulin May 3, 2022, 11:06 a.m. UTC | #23

On 03/05/2022 11:22, Matthew Auld wrote:
> On 02/05/2022 09:53, Lionel Landwerlin wrote:
>> On 02/05/2022 10:54, Lionel Landwerlin wrote:
>>> On 20/04/2022 20:13, Matthew Auld wrote:
>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>
>>>> v2:
>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>    - Rework error capture interactions, including no longer needing
>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>
>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>> Cc: Thomas Hellström <[e-mail]>
>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>> Cc: Jon Bloomfield <[e-mail]>
>>>> Cc: Daniel Vetter <[e-mail]>
>>>> Cc: Jordan Justen <[e-mail]>
>>>> Cc: Kenneth Graunke <[e-mail]>
>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>> Cc: [e-mail]
>>>> ---
>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>> +++++++++++++++++++++++
>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>   3 files changed, 252 insertions(+)
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>> new file mode 100644
>>>> index 000000000000..7bfd0cf44d35
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>> @@ -0,0 +1,190 @@
>>>> +/**
>>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>>> known to the
>>>> + * driver.
>>>> + *
>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>> drm_i915_query.
>>>> + * For this new query we are adding the new query id 
>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>> + * at &drm_i915_query_item.query_id.
>>>> + */
>>>> +struct __drm_i915_memory_region_info {
>>>> +    /** @region: The class:instance pair encoding */
>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>> +
>>>> +    /** @rsvd0: MBZ */
>>>> +    __u32 rsvd0;
>>>> +
>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>> +    __u64 probed_size;
>>>> +
>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>> unknown) */
>>>> +    __u64 unallocated_size;
>>>> +
>>>> +    union {
>>>> +        /** @rsvd1: MBZ */
>>>> +        __u64 rsvd1[8];
>>>> +        struct {
>>>> +            /**
>>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>>> +             * that is CPU accessible. (-1 = unknown).
>>>> +             *
>>>> +             * This will be always be <= @probed_size, and the
>>>> +             * remainder(if there is any) will not be CPU
>>>> +             * accessible.
>>>> +             */
>>>> +            __u64 probed_cpu_visible_size;
>>>> +        };
>>>
>>>
>>> Trying to implement userspace support in Vulkan for this, I have an 
>>> additional question about the value of probed_cpu_visible_size.
>>>
>>> When is it set to -1?
>>>
>>> I'm guessing before there is support for this value it'll be 0 (MBZ).
>>>
>>> After after it should either be the entire lmem or something smaller.
>>>
>>>
>>> -Lionel
>>
>>
>> Other pain point of this new uAPI, previously we could query the 
>> unallocated size for each heap.
> 
> unallocated_size should always give the same value as probed_size. We 
> have the avail tracking, but we don't currently expose that through 
> unallocated_size, due to lack of real userspace/user etc.
> 
>>
>> Now lmem is effectively divided into 2 heaps, but unallocated_size is 
>> tracking allocation from both parts of lmem.
> 
> Yeah, if we ever properly expose the unallocated_size, then we could 
> also just add unallocated_cpu_visible_size.
> 
>>
>> Is adding new I915_MEMORY_CLASS_DEVICE_NON_MAPPABLE out of question?
> 
> I don't think it's out of the question...
> 
> I guess user-space should be able to get the current flag behaviour just 
> by specifying: device, system. And it does give more flexibly to allow 
> something like: device, device-nm, smem.

I was also thinking about that option, albeit both regions under the 
existing class just separate instances with "capability" flags differing.

Downsides I thought were a) it does not really match the underlying 
resource, which is one and not two from the backing storage POV, and b) 
it allows userspace to do potentially do too much of restrictive 
regions=device-mappable,system  (even if only an innocent mistake); 
disallowing i915 to manage the space better in cases where multiple 
clients happen to fight over it.

The last part is going back to what I was commenting earlier, where I 
though migrating objects which had cpu-access set to any device should 
be allowed. (When mappable region is over subscribed.)

Regards,

Tvrtko

> We can also drop the probed_cpu_visible_size, which would now just be 
> the probed_size with device/device-nm. And if we lack device-nm, then 
> the entire thing must be CPU mappable.
> 
> One of the downsides though, is that we can no longer easily mix object 
> pages from both device + device-nm, which we could previously do when we 
> didn't specify the flag. At least according to the current 
> design/behaviour for @regions that would not be allowed. I guess some 
> kind of new flag like ALLOC_MIXED or so? Although currently that is only 
> possible with device + device-nm in ttm/i915.
> 
>>
>>
>> -Lionel
>>
>>
>>>
>>>
>>>> +    };
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>> behaviour, with added
>>>> + * extension support using struct i915_user_extension.
>>>> + *
>>>> + * Note that new buffer flags should be added here, at least for 
>>>> the stuff that
>>>> + * is immutable. Previously we would have two ioctls, one to create 
>>>> the object
>>>> + * with gem_create, and another to apply various parameters, 
>>>> however this
>>>> + * creates some ambiguity for the params which are considered 
>>>> immutable. Also in
>>>> + * general we're phasing out the various SET/GET ioctls.
>>>> + */
>>>> +struct __drm_i915_gem_create_ext {
>>>> +    /**
>>>> +     * @size: Requested size for the object.
>>>> +     *
>>>> +     * The (page-aligned) allocated size for the object will be 
>>>> returned.
>>>> +     *
>>>> +     * Note that for some devices we have might have further minimum
>>>> +     * page-size restrictions(larger than 4K), like for device 
>>>> local-memory.
>>>> +     * However in general the final size here should always reflect 
>>>> any
>>>> +     * rounding up, if for example using the 
>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>> +     * extension to place the object in device local-memory.
>>>> +     */
>>>> +    __u64 size;
>>>> +    /**
>>>> +     * @handle: Returned handle for the object.
>>>> +     *
>>>> +     * Object handles are nonzero.
>>>> +     */
>>>> +    __u32 handle;
>>>> +    /**
>>>> +     * @flags: Optional flags.
>>>> +     *
>>>> +     * Supported values:
>>>> +     *
>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>> kernel that
>>>> +     * the object will need to be accessed via the CPU.
>>>> +     *
>>>> +     * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, 
>>>> and
>>>> +     * only strictly required on platforms where only some of the 
>>>> device
>>>> +     * memory is directly visible or mappable through the CPU, like 
>>>> on DG2+.
>>>> +     *
>>>> +     * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
>>>> +     * ensure we can always spill the allocation to system memory, 
>>>> if we
>>>> +     * can't place the object in the mappable part of
>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>> +     *
>>>> +     * Note that since the kernel only supports flat-CCS on objects 
>>>> that can
>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>>> don't
>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
>>>> +     * flat-CCS.
>>>> +     *
>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>>> that the
>>>> +     * kernel can still migrate the object to the mappable part, as 
>>>> a last
>>>> +     * resort, if userspace ever CPU faults this object, but this 
>>>> might be
>>>> +     * expensive, and so ideally should be avoided.
>>>> +     */
>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>> +    __u32 flags;
>>>> +    /**
>>>> +     * @extensions: The chain of extensions to apply to this object.
>>>> +     *
>>>> +     * This will be useful in the future when we need to support 
>>>> several
>>>> +     * different extensions, and we need to apply more than one when
>>>> +     * creating the object. See struct i915_user_extension.
>>>> +     *
>>>> +     * If we don't supply any extensions then we get the same old 
>>>> gem_create
>>>> +     * behaviour.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>> +     *
>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>> +     */
>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>> +    __u64 extensions;
>>>> +};
>>>> +
>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>> +
>>>> +/**
>>>> + * struct __drm_i915_query_vma_info
>>>> + *
>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>> returning its set
>>>> + * of attributes.
>>>> + *
>>>> + * .. code-block:: C
>>>> + *
>>>> + *    struct drm_i915_query_vma_info info = {};
>>>> + *    struct drm_i915_query_item item = {
>>>> + *        .data_ptr = (uintptr_t)&info,
>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>> + *    };
>>>> + *    struct drm_i915_query query = {
>>>> + *        .num_items = 1,
>>>> + *        .items_ptr = (uintptr_t)&item,
>>>> + *    };
>>>> + *    int err;
>>>> + *
>>>> + *    // Unlike some other types of queries, there is no need to 
>>>> first query
>>>> + *    // the size of the data_ptr blob here, since we already know 
>>>> ahead of
>>>> + *    // time how big this needs to be.
>>>> + *    item.length = sizeof(info);
>>>> + *
>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we 
>>>> wish
>>>> + *    // to query, before then firing off the query.
>>>> + *    info.vm_id = vm_id;
>>>> + *    info.offset = gtt_address;
>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>> + *    if (err || item.length < 0) ...
>>>> + *
>>>> + *    // If all went well we can now inspect the returned attributes.
>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>> + */
>>>> +struct __drm_i915_query_vma_info {
>>>> +    /**
>>>> +     * @vm_id: The given vm id that contains the vma. The id is the 
>>>> value
>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>> +     */
>>>> +    __u32 vm_id;
>>>> +    /** @pad: MBZ. */
>>>> +    __u32 pad;
>>>> +    /**
>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>> the kernel
>>>> +     * will use to perform the lookup.
>>>> +     */
>>>> +    __u64 offset;
>>>> +    /**
>>>> +     * @attributes: The returned attributes for the given vma.
>>>> +     *
>>>> +     * Possible values:
>>>> +     *
>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>>> backing the
>>>> +     * vma are currently CPU accessible. If this is not set then 
>>>> the vma is
>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>>>> the CPU
>>>> +     * cannot directly access(this is only possible on discrete 
>>>> devices with
>>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>>> +     * require the kernel first synchronising any GPU work tied to the
>>>> +     * object, before then migrating the pages, either to the CPU 
>>>> accessible
>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>> +     * placements permit it. See 
>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>> +     *
>>>> +     * Note that this is inherently racy.
>>>> +     */
>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>> +    __u64 attributes;
>>>> +    /** @rsvd: MBZ */
>>>> +    __u32 rsvd[4];
>>>> +};
>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> new file mode 100644
>>>> index 000000000000..be3d9bcdd86d
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>> @@ -0,0 +1,58 @@
>>>> +==========================
>>>> +I915 Small BAR RFC Section
>>>> +==========================
>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>> local-memory(i.e
>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>> might still be
>>>> +smaller than the total probed_size. In such cases, only some subset of
>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>> first 256M),
>>>> +while the remainder is only accessible via the GPU.
>>>> +
>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>> +----------------------------------------------
>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>> CPU access.
>>>> +This becomes important when placing an object in 
>>>> I915_MEMORY_CLASS_DEVICE, where
>>>> +underneath the device has a small BAR, meaning only some portion of 
>>>> it is CPU
>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>> access is not
>>>> +required, and prioritize using the non-CPU visible portion of
>>>> +I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_gem_create_ext
>>>> +
>>>> +probed_cpu_visible_size attribute
>>>> +---------------------------------
>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>> total size of the
>>>> +CPU accessible portion, for the particular region. This should only be
>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>> +
>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>>> with the
>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>> visible portion,
>>>> +where the total size of the heap needs to be known.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_memory_region_info
>>>> +
>>>> +DRM_I915_QUERY_VMA_INFO query
>>>> +-----------------------------
>>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>>> +respective vma, and return its set of attributes. For now we only 
>>>> support
>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
>>>> +currently placed in memory that is accessible by the CPU. This 
>>>> should always be
>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>> I915_MEMORY_CLASS_DEVICE
>>>> +matches the probed_size. If this is not set then CPU faulting the 
>>>> object will
>>>> +likely first require migrating the pages.
>>>> +
>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>> +   :functions: __drm_i915_query_vma_info
>>>> +
>>>> +Error Capture restrictions
>>>> +--------------------------
>>>> +With error capture we have two new restrictions:
>>>> +
>>>> +    1) Error capture is best effort on small BAR systems; if the 
>>>> pages are not
>>>> +    CPU accessible, at the time of capture, then the kernel is free 
>>>> to skip
>>>> +    trying to capture them.
>>>> +
>>>> +    2) On discrete we now reject error capture on recoverable 
>>>> contexts. In the
>>>> +    future the kernel may want to blit during error capture, when 
>>>> for example
>>>> +    something is not currently CPU accessible.
>>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>>> b/Documentation/gpu/rfc/index.rst
>>>> index 91e93a705230..5a3bd3924ba6 100644
>>>> --- a/Documentation/gpu/rfc/index.rst
>>>> +++ b/Documentation/gpu/rfc/index.rst
>>>> @@ -23,3 +23,7 @@ host such documentation:
>>>>   .. toctree::
>>>>         i915_scheduler.rst
>>>> +
>>>> +.. toctree::
>>>> +
>>>> +    i915_small_bar.rst
>>>
>>>
>>

Matthew Auld May 3, 2022, 2:27 p.m. UTC | #24

On 03/05/2022 11:39, Lionel Landwerlin wrote:
> On 03/05/2022 13:22, Matthew Auld wrote:
>> On 02/05/2022 09:53, Lionel Landwerlin wrote:
>>> On 02/05/2022 10:54, Lionel Landwerlin wrote:
>>>> On 20/04/2022 20:13, Matthew Auld wrote:
>>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>>
>>>>> v2:
>>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>>    - Rework error capture interactions, including no longer needing
>>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>>
>>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>>> Cc: Thomas Hellström <[e-mail]>
>>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>>> Cc: Jon Bloomfield <[e-mail]>
>>>>> Cc: Daniel Vetter <[e-mail]>
>>>>> Cc: Jordan Justen <[e-mail]>
>>>>> Cc: Kenneth Graunke <[e-mail]>
>>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>>> Cc: [e-mail]
>>>>> ---
>>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>>> +++++++++++++++++++++++
>>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>>   3 files changed, 252 insertions(+)
>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>>
>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>>> new file mode 100644
>>>>> index 000000000000..7bfd0cf44d35
>>>>> --- /dev/null
>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>>> @@ -0,0 +1,190 @@
>>>>> +/**
>>>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>>>> known to the
>>>>> + * driver.
>>>>> + *
>>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>>> drm_i915_query.
>>>>> + * For this new query we are adding the new query id 
>>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>>> + * at &drm_i915_query_item.query_id.
>>>>> + */
>>>>> +struct __drm_i915_memory_region_info {
>>>>> +    /** @region: The class:instance pair encoding */
>>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>>> +
>>>>> +    /** @rsvd0: MBZ */
>>>>> +    __u32 rsvd0;
>>>>> +
>>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>>> +    __u64 probed_size;
>>>>> +
>>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>>> unknown) */
>>>>> +    __u64 unallocated_size;
>>>>> +
>>>>> +    union {
>>>>> +        /** @rsvd1: MBZ */
>>>>> +        __u64 rsvd1[8];
>>>>> +        struct {
>>>>> +            /**
>>>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>>>> +             * that is CPU accessible. (-1 = unknown).
>>>>> +             *
>>>>> +             * This will be always be <= @probed_size, and the
>>>>> +             * remainder(if there is any) will not be CPU
>>>>> +             * accessible.
>>>>> +             */
>>>>> +            __u64 probed_cpu_visible_size;
>>>>> +        };
>>>>
>>>>
>>>> Trying to implement userspace support in Vulkan for this, I have an 
>>>> additional question about the value of probed_cpu_visible_size.
>>>>
>>>> When is it set to -1?
>>>>
>>>> I'm guessing before there is support for this value it'll be 0 (MBZ).
>>>>
>>>> After after it should either be the entire lmem or something smaller.
>>>>
>>>>
>>>> -Lionel
>>>
>>>
>>> Other pain point of this new uAPI, previously we could query the 
>>> unallocated size for each heap.
>>
>> unallocated_size should always give the same value as probed_size. We 
>> have the avail tracking, but we don't currently expose that through 
>> unallocated_size, due to lack of real userspace/user etc.
>>
>>>
>>> Now lmem is effectively divided into 2 heaps, but unallocated_size is 
>>> tracking allocation from both parts of lmem.
>>
>> Yeah, if we ever properly expose the unallocated_size, then we could 
>> also just add unallocated_cpu_visible_size.
>>
>>>
>>> Is adding new I915_MEMORY_CLASS_DEVICE_NON_MAPPABLE out of question?
>>
>> I don't think it's out of the question...
>>
>> I guess user-space should be able to get the current flag behaviour 
>> just by specifying: device, system. And it does give more flexibly to 
>> allow something like: device, device-nm, smem.
>>
>> We can also drop the probed_cpu_visible_size, which would now just be 
>> the probed_size with device/device-nm. And if we lack device-nm, then 
>> the entire thing must be CPU mappable.
>>
>> One of the downsides though, is that we can no longer easily mix 
>> object pages from both device + device-nm, which we could previously 
>> do when we didn't specify the flag. At least according to the current 
>> design/behaviour for @regions that would not be allowed. I guess some 
>> kind of new flag like ALLOC_MIXED or so? Although currently that is 
>> only possible with device + device-nm in ttm/i915.
> 
> 
> Thanks, I wasn't aware of the restrictions.
> 
> Adding unallocated_cpu_visible_size would be great.

So do we want this in the next version? i.e we already have a current 
real use case in mind for unallocated_size where probed_size is not good 
enough?

> 
> 
> -Lionel
> 
> 
>>
>>>
>>>
>>> -Lionel
>>>
>>>
>>>>
>>>>
>>>>> +    };
>>>>> +};
>>>>> +
>>>>> +/**
>>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>>> behaviour, with added
>>>>> + * extension support using struct i915_user_extension.
>>>>> + *
>>>>> + * Note that new buffer flags should be added here, at least for 
>>>>> the stuff that
>>>>> + * is immutable. Previously we would have two ioctls, one to 
>>>>> create the object
>>>>> + * with gem_create, and another to apply various parameters, 
>>>>> however this
>>>>> + * creates some ambiguity for the params which are considered 
>>>>> immutable. Also in
>>>>> + * general we're phasing out the various SET/GET ioctls.
>>>>> + */
>>>>> +struct __drm_i915_gem_create_ext {
>>>>> +    /**
>>>>> +     * @size: Requested size for the object.
>>>>> +     *
>>>>> +     * The (page-aligned) allocated size for the object will be 
>>>>> returned.
>>>>> +     *
>>>>> +     * Note that for some devices we have might have further minimum
>>>>> +     * page-size restrictions(larger than 4K), like for device 
>>>>> local-memory.
>>>>> +     * However in general the final size here should always 
>>>>> reflect any
>>>>> +     * rounding up, if for example using the 
>>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>>> +     * extension to place the object in device local-memory.
>>>>> +     */
>>>>> +    __u64 size;
>>>>> +    /**
>>>>> +     * @handle: Returned handle for the object.
>>>>> +     *
>>>>> +     * Object handles are nonzero.
>>>>> +     */
>>>>> +    __u32 handle;
>>>>> +    /**
>>>>> +     * @flags: Optional flags.
>>>>> +     *
>>>>> +     * Supported values:
>>>>> +     *
>>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>>> kernel that
>>>>> +     * the object will need to be accessed via the CPU.
>>>>> +     *
>>>>> +     * Only valid when placing objects in 
>>>>> I915_MEMORY_CLASS_DEVICE, and
>>>>> +     * only strictly required on platforms where only some of the 
>>>>> device
>>>>> +     * memory is directly visible or mappable through the CPU, 
>>>>> like on DG2+.
>>>>> +     *
>>>>> +     * One of the placements MUST also be 
>>>>> I915_MEMORY_CLASS_SYSTEM, to
>>>>> +     * ensure we can always spill the allocation to system memory, 
>>>>> if we
>>>>> +     * can't place the object in the mappable part of
>>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>>> +     *
>>>>> +     * Note that since the kernel only supports flat-CCS on 
>>>>> objects that can
>>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>>>> don't
>>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together 
>>>>> with
>>>>> +     * flat-CCS.
>>>>> +     *
>>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note 
>>>>> that the
>>>>> +     * kernel can still migrate the object to the mappable part, 
>>>>> as a last
>>>>> +     * resort, if userspace ever CPU faults this object, but this 
>>>>> might be
>>>>> +     * expensive, and so ideally should be avoided.
>>>>> +     */
>>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>>> +    __u32 flags;
>>>>> +    /**
>>>>> +     * @extensions: The chain of extensions to apply to this object.
>>>>> +     *
>>>>> +     * This will be useful in the future when we need to support 
>>>>> several
>>>>> +     * different extensions, and we need to apply more than one when
>>>>> +     * creating the object. See struct i915_user_extension.
>>>>> +     *
>>>>> +     * If we don't supply any extensions then we get the same old 
>>>>> gem_create
>>>>> +     * behaviour.
>>>>> +     *
>>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>>> +     *
>>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>>> +     */
>>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>>> +    __u64 extensions;
>>>>> +};
>>>>> +
>>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>>> +
>>>>> +/**
>>>>> + * struct __drm_i915_query_vma_info
>>>>> + *
>>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>>> returning its set
>>>>> + * of attributes.
>>>>> + *
>>>>> + * .. code-block:: C
>>>>> + *
>>>>> + *    struct drm_i915_query_vma_info info = {};
>>>>> + *    struct drm_i915_query_item item = {
>>>>> + *        .data_ptr = (uintptr_t)&info,
>>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>>> + *    };
>>>>> + *    struct drm_i915_query query = {
>>>>> + *        .num_items = 1,
>>>>> + *        .items_ptr = (uintptr_t)&item,
>>>>> + *    };
>>>>> + *    int err;
>>>>> + *
>>>>> + *    // Unlike some other types of queries, there is no need to 
>>>>> first query
>>>>> + *    // the size of the data_ptr blob here, since we already know 
>>>>> ahead of
>>>>> + *    // time how big this needs to be.
>>>>> + *    item.length = sizeof(info);
>>>>> + *
>>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma we 
>>>>> wish
>>>>> + *    // to query, before then firing off the query.
>>>>> + *    info.vm_id = vm_id;
>>>>> + *    info.offset = gtt_address;
>>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>>> + *    if (err || item.length < 0) ...
>>>>> + *
>>>>> + *    // If all went well we can now inspect the returned attributes.
>>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>>> + */
>>>>> +struct __drm_i915_query_vma_info {
>>>>> +    /**
>>>>> +     * @vm_id: The given vm id that contains the vma. The id is 
>>>>> the value
>>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>>> +     */
>>>>> +    __u32 vm_id;
>>>>> +    /** @pad: MBZ. */
>>>>> +    __u32 pad;
>>>>> +    /**
>>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>>> the kernel
>>>>> +     * will use to perform the lookup.
>>>>> +     */
>>>>> +    __u64 offset;
>>>>> +    /**
>>>>> +     * @attributes: The returned attributes for the given vma.
>>>>> +     *
>>>>> +     * Possible values:
>>>>> +     *
>>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>>>> backing the
>>>>> +     * vma are currently CPU accessible. If this is not set then 
>>>>> the vma is
>>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, which 
>>>>> the CPU
>>>>> +     * cannot directly access(this is only possible on discrete 
>>>>> devices with
>>>>> +     * a small BAR). Attempting to MMAP and fault such an object will
>>>>> +     * require the kernel first synchronising any GPU work tied to 
>>>>> the
>>>>> +     * object, before then migrating the pages, either to the CPU 
>>>>> accessible
>>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>>> +     * placements permit it. See 
>>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>>> +     *
>>>>> +     * Note that this is inherently racy.
>>>>> +     */
>>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>>> +    __u64 attributes;
>>>>> +    /** @rsvd: MBZ */
>>>>> +    __u32 rsvd[4];
>>>>> +};
>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>> new file mode 100644
>>>>> index 000000000000..be3d9bcdd86d
>>>>> --- /dev/null
>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>> @@ -0,0 +1,58 @@
>>>>> +==========================
>>>>> +I915 Small BAR RFC Section
>>>>> +==========================
>>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>>> local-memory(i.e
>>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>>> might still be
>>>>> +smaller than the total probed_size. In such cases, only some 
>>>>> subset of
>>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>>> first 256M),
>>>>> +while the remainder is only accessible via the GPU.
>>>>> +
>>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>>> +----------------------------------------------
>>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>>> CPU access.
>>>>> +This becomes important when placing an object in 
>>>>> I915_MEMORY_CLASS_DEVICE, where
>>>>> +underneath the device has a small BAR, meaning only some portion 
>>>>> of it is CPU
>>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>>> access is not
>>>>> +required, and prioritize using the non-CPU visible portion of
>>>>> +I915_MEMORY_CLASS_DEVICE.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_gem_create_ext
>>>>> +
>>>>> +probed_cpu_visible_size attribute
>>>>> +---------------------------------
>>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>>> total size of the
>>>>> +CPU accessible portion, for the particular region. This should 
>>>>> only be
>>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>>> +
>>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>>>> with the
>>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>>> visible portion,
>>>>> +where the total size of the heap needs to be known.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_memory_region_info
>>>>> +
>>>>> +DRM_I915_QUERY_VMA_INFO query
>>>>> +-----------------------------
>>>>> +Query the attributes of some vma. Given a vm and GTT offset, find the
>>>>> +respective vma, and return its set of attributes. For now we only 
>>>>> support
>>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the 
>>>>> object/vma is
>>>>> +currently placed in memory that is accessible by the CPU. This 
>>>>> should always be
>>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>>> I915_MEMORY_CLASS_DEVICE
>>>>> +matches the probed_size. If this is not set then CPU faulting the 
>>>>> object will
>>>>> +likely first require migrating the pages.
>>>>> +
>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>> +   :functions: __drm_i915_query_vma_info
>>>>> +
>>>>> +Error Capture restrictions
>>>>> +--------------------------
>>>>> +With error capture we have two new restrictions:
>>>>> +
>>>>> +    1) Error capture is best effort on small BAR systems; if the 
>>>>> pages are not
>>>>> +    CPU accessible, at the time of capture, then the kernel is 
>>>>> free to skip
>>>>> +    trying to capture them.
>>>>> +
>>>>> +    2) On discrete we now reject error capture on recoverable 
>>>>> contexts. In the
>>>>> +    future the kernel may want to blit during error capture, when 
>>>>> for example
>>>>> +    something is not currently CPU accessible.
>>>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>>>> b/Documentation/gpu/rfc/index.rst
>>>>> index 91e93a705230..5a3bd3924ba6 100644
>>>>> --- a/Documentation/gpu/rfc/index.rst
>>>>> +++ b/Documentation/gpu/rfc/index.rst
>>>>> @@ -23,3 +23,7 @@ host such documentation:
>>>>>   .. toctree::
>>>>>         i915_scheduler.rst
>>>>> +
>>>>> +.. toctree::
>>>>> +
>>>>> +    i915_small_bar.rst
>>>>
>>>>
>>>
>

Lionel Landwerlin May 3, 2022, 2:37 p.m. UTC | #25

On 03/05/2022 17:27, Matthew Auld wrote:
> On 03/05/2022 11:39, Lionel Landwerlin wrote:
>> On 03/05/2022 13:22, Matthew Auld wrote:
>>> On 02/05/2022 09:53, Lionel Landwerlin wrote:
>>>> On 02/05/2022 10:54, Lionel Landwerlin wrote:
>>>>> On 20/04/2022 20:13, Matthew Auld wrote:
>>>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>>>
>>>>>> v2:
>>>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>>>    - Rework error capture interactions, including no longer needing
>>>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>>>
>>>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>>>> Cc: Thomas Hellström <[e-mail]>
>>>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>>>> Cc: Jon Bloomfield <[e-mail]>
>>>>>> Cc: Daniel Vetter <[e-mail]>
>>>>>> Cc: Jordan Justen <[e-mail]>
>>>>>> Cc: Kenneth Graunke <[e-mail]>
>>>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>>>> Cc: [e-mail]
>>>>>> ---
>>>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>>>> +++++++++++++++++++++++
>>>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>>>   3 files changed, 252 insertions(+)
>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>>>
>>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>>>> new file mode 100644
>>>>>> index 000000000000..7bfd0cf44d35
>>>>>> --- /dev/null
>>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>>>> @@ -0,0 +1,190 @@
>>>>>> +/**
>>>>>> + * struct __drm_i915_memory_region_info - Describes one region 
>>>>>> as known to the
>>>>>> + * driver.
>>>>>> + *
>>>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>>>> drm_i915_query.
>>>>>> + * For this new query we are adding the new query id 
>>>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>>>> + * at &drm_i915_query_item.query_id.
>>>>>> + */
>>>>>> +struct __drm_i915_memory_region_info {
>>>>>> +    /** @region: The class:instance pair encoding */
>>>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>>>> +
>>>>>> +    /** @rsvd0: MBZ */
>>>>>> +    __u32 rsvd0;
>>>>>> +
>>>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>>>> +    __u64 probed_size;
>>>>>> +
>>>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>>>> unknown) */
>>>>>> +    __u64 unallocated_size;
>>>>>> +
>>>>>> +    union {
>>>>>> +        /** @rsvd1: MBZ */
>>>>>> +        __u64 rsvd1[8];
>>>>>> +        struct {
>>>>>> +            /**
>>>>>> +             * @probed_cpu_visible_size: Memory probed by the 
>>>>>> driver
>>>>>> +             * that is CPU accessible. (-1 = unknown).
>>>>>> +             *
>>>>>> +             * This will be always be <= @probed_size, and the
>>>>>> +             * remainder(if there is any) will not be CPU
>>>>>> +             * accessible.
>>>>>> +             */
>>>>>> +            __u64 probed_cpu_visible_size;
>>>>>> +        };
>>>>>
>>>>>
>>>>> Trying to implement userspace support in Vulkan for this, I have 
>>>>> an additional question about the value of probed_cpu_visible_size.
>>>>>
>>>>> When is it set to -1?
>>>>>
>>>>> I'm guessing before there is support for this value it'll be 0 (MBZ).
>>>>>
>>>>> After after it should either be the entire lmem or something smaller.
>>>>>
>>>>>
>>>>> -Lionel
>>>>
>>>>
>>>> Other pain point of this new uAPI, previously we could query the 
>>>> unallocated size for each heap.
>>>
>>> unallocated_size should always give the same value as probed_size. 
>>> We have the avail tracking, but we don't currently expose that 
>>> through unallocated_size, due to lack of real userspace/user etc.
>>>
>>>>
>>>> Now lmem is effectively divided into 2 heaps, but unallocated_size 
>>>> is tracking allocation from both parts of lmem.
>>>
>>> Yeah, if we ever properly expose the unallocated_size, then we could 
>>> also just add unallocated_cpu_visible_size.
>>>
>>>>
>>>> Is adding new I915_MEMORY_CLASS_DEVICE_NON_MAPPABLE out of question?
>>>
>>> I don't think it's out of the question...
>>>
>>> I guess user-space should be able to get the current flag behaviour 
>>> just by specifying: device, system. And it does give more flexibly 
>>> to allow something like: device, device-nm, smem.
>>>
>>> We can also drop the probed_cpu_visible_size, which would now just 
>>> be the probed_size with device/device-nm. And if we lack device-nm, 
>>> then the entire thing must be CPU mappable.
>>>
>>> One of the downsides though, is that we can no longer easily mix 
>>> object pages from both device + device-nm, which we could previously 
>>> do when we didn't specify the flag. At least according to the 
>>> current design/behaviour for @regions that would not be allowed. I 
>>> guess some kind of new flag like ALLOC_MIXED or so? Although 
>>> currently that is only possible with device + device-nm in ttm/i915.
>>
>>
>> Thanks, I wasn't aware of the restrictions.
>>
>> Adding unallocated_cpu_visible_size would be great.
>
> So do we want this in the next version? i.e we already have a current 
> real use case in mind for unallocated_size where probed_size is not 
> good enough?


Yeah in the  next iteration.

We're using unallocated_size to implement VK_EXT_memory_budget and since 
I'm going to expose lmem mappable/unmappable as 2 different heaps on 
Vulkan, I would use that there too.


-Lionel


>
>>
>>
>> -Lionel
>>
>>
>>>
>>>>
>>>>
>>>> -Lionel
>>>>
>>>>
>>>>>
>>>>>
>>>>>> +    };
>>>>>> +};
>>>>>> +
>>>>>> +/**
>>>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>>>> behaviour, with added
>>>>>> + * extension support using struct i915_user_extension.
>>>>>> + *
>>>>>> + * Note that new buffer flags should be added here, at least for 
>>>>>> the stuff that
>>>>>> + * is immutable. Previously we would have two ioctls, one to 
>>>>>> create the object
>>>>>> + * with gem_create, and another to apply various parameters, 
>>>>>> however this
>>>>>> + * creates some ambiguity for the params which are considered 
>>>>>> immutable. Also in
>>>>>> + * general we're phasing out the various SET/GET ioctls.
>>>>>> + */
>>>>>> +struct __drm_i915_gem_create_ext {
>>>>>> +    /**
>>>>>> +     * @size: Requested size for the object.
>>>>>> +     *
>>>>>> +     * The (page-aligned) allocated size for the object will be 
>>>>>> returned.
>>>>>> +     *
>>>>>> +     * Note that for some devices we have might have further 
>>>>>> minimum
>>>>>> +     * page-size restrictions(larger than 4K), like for device 
>>>>>> local-memory.
>>>>>> +     * However in general the final size here should always 
>>>>>> reflect any
>>>>>> +     * rounding up, if for example using the 
>>>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>>>> +     * extension to place the object in device local-memory.
>>>>>> +     */
>>>>>> +    __u64 size;
>>>>>> +    /**
>>>>>> +     * @handle: Returned handle for the object.
>>>>>> +     *
>>>>>> +     * Object handles are nonzero.
>>>>>> +     */
>>>>>> +    __u32 handle;
>>>>>> +    /**
>>>>>> +     * @flags: Optional flags.
>>>>>> +     *
>>>>>> +     * Supported values:
>>>>>> +     *
>>>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>>>> kernel that
>>>>>> +     * the object will need to be accessed via the CPU.
>>>>>> +     *
>>>>>> +     * Only valid when placing objects in 
>>>>>> I915_MEMORY_CLASS_DEVICE, and
>>>>>> +     * only strictly required on platforms where only some of 
>>>>>> the device
>>>>>> +     * memory is directly visible or mappable through the CPU, 
>>>>>> like on DG2+.
>>>>>> +     *
>>>>>> +     * One of the placements MUST also be 
>>>>>> I915_MEMORY_CLASS_SYSTEM, to
>>>>>> +     * ensure we can always spill the allocation to system 
>>>>>> memory, if we
>>>>>> +     * can't place the object in the mappable part of
>>>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>>>> +     *
>>>>>> +     * Note that since the kernel only supports flat-CCS on 
>>>>>> objects that can
>>>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we 
>>>>>> therefore don't
>>>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS 
>>>>>> together with
>>>>>> +     * flat-CCS.
>>>>>> +     *
>>>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. 
>>>>>> Note that the
>>>>>> +     * kernel can still migrate the object to the mappable part, 
>>>>>> as a last
>>>>>> +     * resort, if userspace ever CPU faults this object, but 
>>>>>> this might be
>>>>>> +     * expensive, and so ideally should be avoided.
>>>>>> +     */
>>>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>>>> +    __u32 flags;
>>>>>> +    /**
>>>>>> +     * @extensions: The chain of extensions to apply to this 
>>>>>> object.
>>>>>> +     *
>>>>>> +     * This will be useful in the future when we need to support 
>>>>>> several
>>>>>> +     * different extensions, and we need to apply more than one 
>>>>>> when
>>>>>> +     * creating the object. See struct i915_user_extension.
>>>>>> +     *
>>>>>> +     * If we don't supply any extensions then we get the same 
>>>>>> old gem_create
>>>>>> +     * behaviour.
>>>>>> +     *
>>>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
>>>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>>>> +     *
>>>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>>>> +     */
>>>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>>>> +    __u64 extensions;
>>>>>> +};
>>>>>> +
>>>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>>>> +
>>>>>> +/**
>>>>>> + * struct __drm_i915_query_vma_info
>>>>>> + *
>>>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>>>> returning its set
>>>>>> + * of attributes.
>>>>>> + *
>>>>>> + * .. code-block:: C
>>>>>> + *
>>>>>> + *    struct drm_i915_query_vma_info info = {};
>>>>>> + *    struct drm_i915_query_item item = {
>>>>>> + *        .data_ptr = (uintptr_t)&info,
>>>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>>>> + *    };
>>>>>> + *    struct drm_i915_query query = {
>>>>>> + *        .num_items = 1,
>>>>>> + *        .items_ptr = (uintptr_t)&item,
>>>>>> + *    };
>>>>>> + *    int err;
>>>>>> + *
>>>>>> + *    // Unlike some other types of queries, there is no need to 
>>>>>> first query
>>>>>> + *    // the size of the data_ptr blob here, since we already 
>>>>>> know ahead of
>>>>>> + *    // time how big this needs to be.
>>>>>> + *    item.length = sizeof(info);
>>>>>> + *
>>>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma 
>>>>>> we wish
>>>>>> + *    // to query, before then firing off the query.
>>>>>> + *    info.vm_id = vm_id;
>>>>>> + *    info.offset = gtt_address;
>>>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>>>> + *    if (err || item.length < 0) ...
>>>>>> + *
>>>>>> + *    // If all went well we can now inspect the returned 
>>>>>> attributes.
>>>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) 
>>>>>> ...
>>>>>> + */
>>>>>> +struct __drm_i915_query_vma_info {
>>>>>> +    /**
>>>>>> +     * @vm_id: The given vm id that contains the vma. The id is 
>>>>>> the value
>>>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>>>> +     */
>>>>>> +    __u32 vm_id;
>>>>>> +    /** @pad: MBZ. */
>>>>>> +    __u32 pad;
>>>>>> +    /**
>>>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>>>> the kernel
>>>>>> +     * will use to perform the lookup.
>>>>>> +     */
>>>>>> +    __u64 offset;
>>>>>> +    /**
>>>>>> +     * @attributes: The returned attributes for the given vma.
>>>>>> +     *
>>>>>> +     * Possible values:
>>>>>> +     *
>>>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages 
>>>>>> backing the
>>>>>> +     * vma are currently CPU accessible. If this is not set then 
>>>>>> the vma is
>>>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICE memory, 
>>>>>> which the CPU
>>>>>> +     * cannot directly access(this is only possible on discrete 
>>>>>> devices with
>>>>>> +     * a small BAR). Attempting to MMAP and fault such an object 
>>>>>> will
>>>>>> +     * require the kernel first synchronising any GPU work tied 
>>>>>> to the
>>>>>> +     * object, before then migrating the pages, either to the 
>>>>>> CPU accessible
>>>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>>>> +     * placements permit it. See 
>>>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>>>> +     *
>>>>>> +     * Note that this is inherently racy.
>>>>>> +     */
>>>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>>>> +    __u64 attributes;
>>>>>> +    /** @rsvd: MBZ */
>>>>>> +    __u32 rsvd[4];
>>>>>> +};
>>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>>> new file mode 100644
>>>>>> index 000000000000..be3d9bcdd86d
>>>>>> --- /dev/null
>>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>>> @@ -0,0 +1,58 @@
>>>>>> +==========================
>>>>>> +I915 Small BAR RFC Section
>>>>>> +==========================
>>>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>>>> local-memory(i.e
>>>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>>>> might still be
>>>>>> +smaller than the total probed_size. In such cases, only some 
>>>>>> subset of
>>>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>>>> first 256M),
>>>>>> +while the remainder is only accessible via the GPU.
>>>>>> +
>>>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>>>> +----------------------------------------------
>>>>>> +New gem_create_ext flag to tell the kernel that a BO will 
>>>>>> require CPU access.
>>>>>> +This becomes important when placing an object in 
>>>>>> I915_MEMORY_CLASS_DEVICE, where
>>>>>> +underneath the device has a small BAR, meaning only some portion 
>>>>>> of it is CPU
>>>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>>>> access is not
>>>>>> +required, and prioritize using the non-CPU visible portion of
>>>>>> +I915_MEMORY_CLASS_DEVICE.
>>>>>> +
>>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>>> +   :functions: __drm_i915_gem_create_ext
>>>>>> +
>>>>>> +probed_cpu_visible_size attribute
>>>>>> +---------------------------------
>>>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>>>> total size of the
>>>>>> +CPU accessible portion, for the particular region. This should 
>>>>>> only be
>>>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>>>> +
>>>>>> +Vulkan will need this as part of creating a separate 
>>>>>> VkMemoryHeap with the
>>>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>>>> visible portion,
>>>>>> +where the total size of the heap needs to be known.
>>>>>> +
>>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>>> +   :functions: __drm_i915_memory_region_info
>>>>>> +
>>>>>> +DRM_I915_QUERY_VMA_INFO query
>>>>>> +-----------------------------
>>>>>> +Query the attributes of some vma. Given a vm and GTT offset, 
>>>>>> find the
>>>>>> +respective vma, and return its set of attributes. For now we 
>>>>>> only support
>>>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the 
>>>>>> object/vma is
>>>>>> +currently placed in memory that is accessible by the CPU. This 
>>>>>> should always be
>>>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>>>> I915_MEMORY_CLASS_DEVICE
>>>>>> +matches the probed_size. If this is not set then CPU faulting 
>>>>>> the object will
>>>>>> +likely first require migrating the pages.
>>>>>> +
>>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>>> +   :functions: __drm_i915_query_vma_info
>>>>>> +
>>>>>> +Error Capture restrictions
>>>>>> +--------------------------
>>>>>> +With error capture we have two new restrictions:
>>>>>> +
>>>>>> +    1) Error capture is best effort on small BAR systems; if the 
>>>>>> pages are not
>>>>>> +    CPU accessible, at the time of capture, then the kernel is 
>>>>>> free to skip
>>>>>> +    trying to capture them.
>>>>>> +
>>>>>> +    2) On discrete we now reject error capture on recoverable 
>>>>>> contexts. In the
>>>>>> +    future the kernel may want to blit during error capture, 
>>>>>> when for example
>>>>>> +    something is not currently CPU accessible.
>>>>>> diff --git a/Documentation/gpu/rfc/index.rst 
>>>>>> b/Documentation/gpu/rfc/index.rst
>>>>>> index 91e93a705230..5a3bd3924ba6 100644
>>>>>> --- a/Documentation/gpu/rfc/index.rst
>>>>>> +++ b/Documentation/gpu/rfc/index.rst
>>>>>> @@ -23,3 +23,7 @@ host such documentation:
>>>>>>   .. toctree::
>>>>>>         i915_scheduler.rst
>>>>>> +
>>>>>> +.. toctree::
>>>>>> +
>>>>>> +    i915_small_bar.rst
>>>>>
>>>>>
>>>>
>>

Matthew Auld May 3, 2022, 2:40 p.m. UTC | #26

On 28/04/2022 12:11, Tvrtko Ursulin wrote:
> 
> On 28/04/2022 11:25, Matthew Auld wrote:
>> On 28/04/2022 09:55, Tvrtko Ursulin wrote:
>>>
>>> On 27/04/2022 18:36, Matthew Auld wrote:
>>>> On 27/04/2022 09:36, Tvrtko Ursulin wrote:
>>>>>
>>>>> On 20/04/2022 18:13, Matthew Auld wrote:
>>>>>> Add an entry for the new uapi needed for small BAR on DG2+.
>>>>>>
>>>>>> v2:
>>>>>>    - Some spelling fixes and other small tweaks. (Akeem & Thomas)
>>>>>>    - Rework error capture interactions, including no longer needing
>>>>>>      NEEDS_CPU_ACCESS for objects marked for capture. (Thomas)
>>>>>>    - Add probed_cpu_visible_size. (Lionel)
>>>>>>
>>>>>> Signed-off-by: Matthew Auld <[e-mail]>
>>>>>> Cc: Thomas Hellström <[e-mail]>
>>>>>> Cc: Lionel Landwerlin <[e-mail]>
>>>>>> Cc: Jon Bloomfield <[e-mail]>
>>>>>> Cc: Daniel Vetter <[e-mail]>
>>>>>> Cc: Jordan Justen <[e-mail]>
>>>>>> Cc: Kenneth Graunke <[e-mail]>
>>>>>> Cc: Akeem G Abodunrin <[e-mail]>
>>>>>> Cc: [e-mail]
>>>>>> ---
>>>>>>   Documentation/gpu/rfc/i915_small_bar.h   | 190 
>>>>>> +++++++++++++++++++++++
>>>>>>   Documentation/gpu/rfc/i915_small_bar.rst |  58 +++++++
>>>>>>   Documentation/gpu/rfc/index.rst          |   4 +
>>>>>>   3 files changed, 252 insertions(+)
>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.h
>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_small_bar.rst
>>>>>>
>>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.h 
>>>>>> b/Documentation/gpu/rfc/i915_small_bar.h
>>>>>> new file mode 100644
>>>>>> index 000000000000..7bfd0cf44d35
>>>>>> --- /dev/null
>>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.h
>>>>>> @@ -0,0 +1,190 @@
>>>>>> +/**
>>>>>> + * struct __drm_i915_memory_region_info - Describes one region as 
>>>>>> known to the
>>>>>> + * driver.
>>>>>> + *
>>>>>> + * Note this is using both struct drm_i915_query_item and struct 
>>>>>> drm_i915_query.
>>>>>> + * For this new query we are adding the new query id 
>>>>>> DRM_I915_QUERY_MEMORY_REGIONS
>>>>>> + * at &drm_i915_query_item.query_id.
>>>>>> + */
>>>>>> +struct __drm_i915_memory_region_info {
>>>>>> +    /** @region: The class:instance pair encoding */
>>>>>> +    struct drm_i915_gem_memory_class_instance region;
>>>>>> +
>>>>>> +    /** @rsvd0: MBZ */
>>>>>> +    __u32 rsvd0;
>>>>>> +
>>>>>> +    /** @probed_size: Memory probed by the driver (-1 = unknown) */
>>>>>> +    __u64 probed_size;
>>>>>> +
>>>>>> +    /** @unallocated_size: Estimate of memory remaining (-1 = 
>>>>>> unknown) */
>>>>>> +    __u64 unallocated_size;
>>>>>> +
>>>>>> +    union {
>>>>>> +        /** @rsvd1: MBZ */
>>>>>> +        __u64 rsvd1[8];
>>>>>> +        struct {
>>>>>> +            /**
>>>>>> +             * @probed_cpu_visible_size: Memory probed by the driver
>>>>>> +             * that is CPU accessible. (-1 = unknown).
>>>>>> +             *
>>>>>> +             * This will be always be <= @probed_size, and the
>>>>>> +             * remainder(if there is any) will not be CPU
>>>>>> +             * accessible.
>>>>>> +             */
>>>>>> +            __u64 probed_cpu_visible_size;
>>>>>
>>>>> Would unallocated_cpu_visible_size be useful, to follow the total 
>>>>> unallocated_size?
>>>>
>>>> Make sense. But I don't think unallocated_size has actually been 
>>>> properly wired up yet. It still just gives the same value as 
>>>> probed_size. IIRC for unallocated_size we still need a real 
>>>> user/usecase/umd, before wiring that up for real with the existing 
>>>> avail tracking. Once we have that we can also add 
>>>> unallocated_cpu_visible_size.
>>>
>>> So this does nothing at the moment:
>>>
>>>   info.unallocated_size = mr->avail;
>>>
>>> Right, it is set to "mem->avail = mem->total;" at region init time 
>>> and I indeed can't find it ever getting modified. Okay.
>>>
>>>>> Btw, have we ever considered whether unallocated_size should 
>>>>> require CAP_SYS_ADMIN/PERFMON or something?
>>>>
>>>> Note sure. But just in case we do add it for real at some point, why 
>>>> the added restriction?
>>>
>>> To avoid a side channel, albeit perhaps a very weak one. For engine 
>>> utilization we require CAP_SYS_PERFMON, but that is implied by the 
>>> perf core API. It's open for discussion. I guess it may make sense to 
>>> limit it also because it is questionable the field(s) are even useful.
>>>
>>>>
>>>>>
>>>>>> +        };
>>>>>> +    };
>>>>>> +};
>>>>>> +
>>>>>> +/**
>>>>>> + * struct __drm_i915_gem_create_ext - Existing gem_create 
>>>>>> behaviour, with added
>>>>>> + * extension support using struct i915_user_extension.
>>>>>> + *
>>>>>> + * Note that new buffer flags should be added here, at least for 
>>>>>> the stuff that
>>>>>> + * is immutable. Previously we would have two ioctls, one to 
>>>>>> create the object
>>>>>> + * with gem_create, and another to apply various parameters, 
>>>>>> however this
>>>>>> + * creates some ambiguity for the params which are considered 
>>>>>> immutable. Also in
>>>>>> + * general we're phasing out the various SET/GET ioctls.
>>>>>> + */
>>>>>> +struct __drm_i915_gem_create_ext {
>>>>>> +    /**
>>>>>> +     * @size: Requested size for the object.
>>>>>> +     *
>>>>>> +     * The (page-aligned) allocated size for the object will be 
>>>>>> returned.
>>>>>> +     *
>>>>>> +     * Note that for some devices we have might have further minimum
>>>>>> +     * page-size restrictions(larger than 4K), likefor device 
>>>>>> local-memory.
>>>>>> +     * However in general the final size here should always 
>>>>>> reflect any
>>>>>> +     * rounding up, if for example using the 
>>>>>> I915_GEM_CREATE_EXT_MEMORY_REGIONS
>>>>>> +     * extension to place the object in device local-memory.
>>>>>> +     */
>>>>>> +    __u64 size;
>>>>>> +    /**
>>>>>> +     * @handle: Returned handle for the object.
>>>>>> +     *
>>>>>> +     * Object handles are nonzero.
>>>>>> +     */
>>>>>> +    __u32 handle;
>>>>>> +    /**
>>>>>> +     * @flags: Optional flags.
>>>>>> +     *
>>>>>> +     * Supported values:
>>>>>> +     *
>>>>>> +     * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the 
>>>>>> kernel that
>>>>>> +     * the object will need to be accessed via the CPU.
>>>>>> +     *
>>>>>> +     * Only valid when placing objects in 
>>>>>> I915_MEMORY_CLASS_DEVICE, and
>>>>>> +     * only strictly required on platforms where only some of the 
>>>>>> device
>>>>>> +     * memory is directly visible or mappable through the CPU, 
>>>>>> like on DG2+.
>>>>>> +     *
>>>>>> +     * One of the placements MUST also be 
>>>>>> I915_MEMORY_CLASS_SYSTEM, to
>>>>>> +     * ensure we can always spill the allocation tosystem memory, 
>>>>>> if we
>>>>>> +     * can't place the object in the mappable part of
>>>>>> +     * I915_MEMORY_CLASS_DEVICE.
>>>>>> +     *
>>>>>> +     * Note that since the kernel only supports flat-CCS on 
>>>>>> objects that can
>>>>>> +     * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore 
>>>>>> don't
>>>>>> +     * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together 
>>>>>> with
>>>>>> +     * flat-CCS.
>>>>>> +     *
>>>>>> +     * Without this hint, the kernel will assume that non-mappable
>>>>>> +     * I915_MEMORY_CLASS_DEVICE is preferred for this object. 
>>>>>> Note that the
>>>>>> +     * kernel can still migrate the object to the mappable part, 
>>>>>> as a last
>>>>>> +     * resort, if userspace ever CPU faults this object, but this 
>>>>>> might be
>>>>>> +     * expensive, and so ideally should be avoided.
>>>>>> +     */
>>>>>
>>>>> So "needs_cpu_access" flag could almost be viewed as a sub-region 
>>>>> placement priority? What I mean is this:
>>>>>
>>>>> 1)
>>>>> placements=device,system flags=
>>>>>
>>>>> This results in placement priorities: device, device_cpu_mappable, 
>>>>> system.
>>>>
>>>> Yup.
>>>>
>>>>>
>>>>> 2)
>>>>> placements=device,system flags=needs_cpu_access
>>>>>
>>>>> This results in placement priorities: device_cpu_mappable, device, 
>>>>> system.
>>>>
>>>> Here it would only be: device_cpu_mappable, system. We would 
>>>> completely ignore "device" in this case.
>>>>
>>>>>
>>>>> Is this correct?
>>>>>
>>>>> The benefit of the flag is that i915 can place the object to the 
>>>>> right place from the start instead of on the first CPU access? Is 
>>>>> that worth it or is there more to it?
>>>>
>>>> Yeah, the object will only be placed somewhere that is also CPU 
>>>> mappable, with the flag set.
>>>
>>> Hm, wouldn't it be more efficient to be able to migrate it over to 
>>> non-mappable in cases when mappable is over-subscribed?
>>
>> Not sure. As an alternative strategy, I guess that might be 
>> interesting, and if userspace wants something like that we can always 
>> add a new flag I guess? It's a toss up whether just using system 
>> memory is better/worse than incurring an extra move at fault time?
> 
> For me it doesn't make sense to allow bos wo/ cpu mappable flag to use 
> the mappable section (albeit as 2nd priority) and not allowing the cpu 
> mappable ones to temporarily go anywhere in lmem.
> 
>  From i915 side it needs to know the allowed regions (for this argument 
> I speak of mappable / non-mappable as separate regions, even if uapi 
> does not expose them as separate memory regions). i915 does not know in 
> advance the exact usage pattern.
> 
> In case of multiple clients, one might touch a buffer from the CPU once 
> and then render with GPU many times. Another client might touch from CPU 
> a lot more. With mappable space contention this would cause the buffer 
> from the first client to constantly get migrated between smem and lmem, 
> while in reality it could have been migrated to non mappable lmem and 
> used by the GPU without any problems.
> 
> As some sort of diagram:
> 
>      Client A BO    Client B BO
>      --------------    --------------
>      CPU access
>              CPU access (Client A BO "evicted" to smem)
>      GPU access    GPU access
>      GPU access    CPU access
>      GPU access    GPU access
> 
> If we assume there is only space for one BO in mappable, allowing 
> non-mappable placement allows Client A to be unaffected by Client B 
> activity. While with the current proposal it needlessly takes a hit on 
> every, or every other GPU access.
> 
> So I think, if there isn't a fundamental reason to disallow it which I 
> am missing, not limiting the implied placement when cpu access flag is 
> given is beneficial to flexibility of migration decisions i915 can make.
> 

Jon/Daniel, any thoughts/inputs here?


> Regards,
> 
> Tvrtko
> 
>>
>>>
>>>>>> +#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
>>>>>> +    __u32 flags;
>>>>>> +    /**
>>>>>> +     * @extensions: The chain of extensions to apply to this object.
>>>>>> +     *
>>>>>> +     * This will be useful in the future when we need to support 
>>>>>> several
>>>>>> +     * different extensions, and we need to apply more than one when
>>>>>> +     * creating the object. See struct i915_user_extension.
>>>>>> +     *
>>>>>> +     * If we don't supply any extensions then we get the same old 
>>>>>> gem_create
>>>>>> +     * behaviour.
>>>>>> +     *
>>>>>> +     * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usagesee
>>>>>> +     * struct drm_i915_gem_create_ext_memory_regions.
>>>>>> +     *
>>>>>> +     * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
>>>>>> +     * struct drm_i915_gem_create_ext_protected_content.
>>>>>> +     */
>>>>>> +#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
>>>>>> +#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
>>>>>> +    __u64 extensions;
>>>>>> +};
>>>>>> +
>>>>>> +#define DRM_I915_QUERY_VMA_INFO    5
>>>>>> +
>>>>>> +/**
>>>>>> + * struct __drm_i915_query_vma_info
>>>>>> + *
>>>>>> + * Given a vm and GTT address, lookup the corresponding vma, 
>>>>>> returning its set
>>>>>> + * of attributes.
>>>>>> + *
>>>>>> + * .. code-block:: C
>>>>>> + *
>>>>>> + *    struct drm_i915_query_vma_info info = {};
>>>>>> + *    struct drm_i915_query_item item = {
>>>>>> + *        .data_ptr = (uintptr_t)&info,
>>>>>> + *        .query_id = DRM_I915_QUERY_VMA_INFO,
>>>>>> + *    };
>>>>>> + *    struct drm_i915_query query = {
>>>>>> + *        .num_items = 1,
>>>>>> + *        .items_ptr = (uintptr_t)&item,
>>>>>> + *    };
>>>>>> + *    int err;
>>>>>> + *
>>>>>> + *    // Unlike some other types of queries, there is noneed to 
>>>>>> first query
>>>>>> + *    // the size of the data_ptr blob here, since we already 
>>>>>> know ahead of
>>>>>> + *    // time how big this needs to be.
>>>>>> + *    item.length = sizeof(info);
>>>>>> + *
>>>>>> + *    // Next we fill in the vm_id and ppGTT address of the vma 
>>>>>> we wish
>>>>>> + *    // to query, before then firing off the query.
>>>>>> + *    info.vm_id = vm_id;
>>>>>> + *    info.offset = gtt_address;
>>>>>> + *    err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
>>>>>> + *    if (err || item.length < 0) ...
>>>>>> + *
>>>>>> + *    // If all went well we can now inspect the returned 
>>>>>> attributes.
>>>>>> + *    if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
>>>>>> + */
>>>>>> +struct __drm_i915_query_vma_info {
>>>>>> +    /**
>>>>>> +     * @vm_id: The given vm id that contains the vma. The id is 
>>>>>> the value
>>>>>> +     * returned by the DRM_I915_GEM_VM_CREATE. See struct
>>>>>> +     * drm_i915_gem_vm_control.vm_id.
>>>>>> +     */
>>>>>> +    __u32 vm_id;
>>>>>> +    /** @pad: MBZ. */
>>>>>> +    __u32 pad;
>>>>>> +    /**
>>>>>> +     * @offset: The corresponding ppGTT address of the vma which 
>>>>>> the kernel
>>>>>> +     * will use to perform the lookup.
>>>>>> +     */
>>>>>> +    __u64 offset;
>>>>>> +    /**
>>>>>> +     * @attributes: The returned attributes for thegiven vma.
>>>>>> +     *
>>>>>> +     * Possible values:
>>>>>> +     *
>>>>>> +     * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set ifthe pages 
>>>>>> backing the
>>>>>> +     * vma are currently CPU accessible. If this isnot set then 
>>>>>> the vma is
>>>>>> +     * currently backed by I915_MEMORY_CLASS_DEVICEmemory, which 
>>>>>> the CPU
>>>>>> +     * cannot directly access(this is only possibleon discrete 
>>>>>> devices with
>>>>>> +     * a small BAR). Attempting to MMAP and fault such an object 
>>>>>> will
>>>>>> +     * require the kernel first synchronising any GPU work tied 
>>>>>> to the
>>>>>> +     * object, before then migrating the pages, either to the CPU 
>>>>>> accessible
>>>>>> +     * part of I915_MEMORY_CLASS_DEVICE, or 
>>>>>> I915_MEMORY_CLASS_SYSTEM, if the
>>>>>> +     * placements permit it. See 
>>>>>> I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
>>>>>> +     *
>>>>>> +     * Note that this is inherently racy.
>>>>>> +     */
>>>>>> +#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
>>>>>> +    __u64 attributes;
>>>>>> +    /** @rsvd: MBZ */
>>>>>> +    __u32 rsvd[4];
>>>>>> +};
>>>>>> diff --git a/Documentation/gpu/rfc/i915_small_bar.rst 
>>>>>> b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>>> new file mode 100644
>>>>>> index 000000000000..be3d9bcdd86d
>>>>>> --- /dev/null
>>>>>> +++ b/Documentation/gpu/rfc/i915_small_bar.rst
>>>>>> @@ -0,0 +1,58 @@
>>>>>> +==========================
>>>>>> +I915 Small BAR RFC Section
>>>>>> +==========================
>>>>>> +Starting from DG2 we will have resizable BAR support for device 
>>>>>> local-memory(i.e
>>>>>> +I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size 
>>>>>> might still be
>>>>>> +smaller than the total probed_size. In such cases, only some 
>>>>>> subset of
>>>>>> +I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the 
>>>>>> first 256M),
>>>>>> +while the remainder is only accessible via the GPU.
>>>>>> +
>>>>>> +I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
>>>>>> +----------------------------------------------
>>>>>> +New gem_create_ext flag to tell the kernel that a BO will require 
>>>>>> CPU access.
>>>>>> +This becomes important when placing an object in 
>>>>>> I915_MEMORY_CLASS_DEVICE, where
>>>>>> +underneath the device has a small BAR, meaning only some portion 
>>>>>> of it is CPU
>>>>>> +accessible. Without this flag the kernel will assume that CPU 
>>>>>> access is not
>>>>>> +required, and prioritize using the non-CPU visible portion of
>>>>>> +I915_MEMORY_CLASS_DEVICE.
>>>>>> +
>>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>>> +   :functions: __drm_i915_gem_create_ext
>>>>>> +
>>>>>> +probed_cpu_visible_size attribute
>>>>>> +---------------------------------
>>>>>> +New struct__drm_i915_memory_region attribute which returns the 
>>>>>> total size of the
>>>>>> +CPU accessible portion, for the particular region. This should 
>>>>>> only be
>>>>>> +applicable for I915_MEMORY_CLASS_DEVICE.
>>>>>> +
>>>>>> +Vulkan will need this as part of creating a separate VkMemoryHeap 
>>>>>> with the
>>>>>> +VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU 
>>>>>> visible portion,
>>>>>> +where the total size of the heap needs to be known.
>>>>>> +
>>>>>> +.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
>>>>>> +   :functions: __drm_i915_memory_region_info
>>>>>> +
>>>>>> +DRM_I915_QUERY_VMA_INFO query
>>>>>> +-----------------------------
>>>>>> +Query the attributes of some vma. Given a vm and GTT offset, find 
>>>>>> the
>>>>>> +respective vma, and return its set of attributes. For now we only 
>>>>>> support
>>>>>> +DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the 
>>>>>> object/vma is
>>>>>> +currently placed in memory that is accessible by the CPU. This 
>>>>>> should always be
>>>>>> +set on devices where the CPU probed_cpu_visible_size of 
>>>>>> I915_MEMORY_CLASS_DEVICE
>>>>>> +matches the probed_size. If this is not set then CPU faulting the 
>>>>>> object will
>>>>>> +likely first require migrating the pages.
>>>>>
>>>>> I think there should be justification for the new query documented 
>>>>> as well. (Why on top of what.)
>>>>
>>>> Yeah, I'm wondering now if we can just drop this part of the uapi, 
>>>> for now at least, and focus on landing the new flag stuff first.
>>>>
>>>>>
>>>>> Without it personally I can't immediately understand why the 
>>>>> disconnect between the object based and VMA based API. Userspace 
>>>>> has to do some intervening operations like either execbuf, or vm 
>>>>> bind in the future, to make this query usable after object 
>>>>> creation. So question is why wouldn't it know already which 
>>>>> placements it allowed and so would i915 auto-migrate or not for 
>>>>> this particular object. No? Or in other words why this wouldn't be 
>>>>> an object based query since the question it is answering is about 
>>>>> the object backing store and not the VMA.
>>>>
>>>> Yeah, just using the object handle or so I guess would also work. 
>>>> Thanks for the comments.
>>>
>>> I saw other folks have said the same so omitting for now sounds good 
>>> to me indeed.
>>>
>>> Regards,
>>>
>>> Tvrtko

Patch

diff --git a/Documentation/gpu/rfc/i915_small_bar.h b/Documentation/gpu/rfc/i915_small_bar.h
new file mode 100644
index 000000000000..7bfd0cf44d35
--- /dev/null
+++ b/Documentation/gpu/rfc/i915_small_bar.h
@@ -0,0 +1,190 @@ 
+/**
+ * struct __drm_i915_memory_region_info - Describes one region as known to the
+ * driver.
+ *
+ * Note this is using both struct drm_i915_query_item and struct drm_i915_query.
+ * For this new query we are adding the new query id DRM_I915_QUERY_MEMORY_REGIONS
+ * at &drm_i915_query_item.query_id.
+ */
+struct __drm_i915_memory_region_info {
+	/** @region: The class:instance pair encoding */
+	struct drm_i915_gem_memory_class_instance region;
+
+	/** @rsvd0: MBZ */
+	__u32 rsvd0;
+
+	/** @probed_size: Memory probed by the driver (-1 = unknown) */
+	__u64 probed_size;
+
+	/** @unallocated_size: Estimate of memory remaining (-1 = unknown) */
+	__u64 unallocated_size;
+
+	union {
+		/** @rsvd1: MBZ */
+		__u64 rsvd1[8];
+		struct {
+			/**
+			 * @probed_cpu_visible_size: Memory probed by the driver
+			 * that is CPU accessible. (-1 = unknown).
+			 *
+			 * This will be always be <= @probed_size, and the
+			 * remainder(if there is any) will not be CPU
+			 * accessible.
+			 */
+			__u64 probed_cpu_visible_size;
+		};
+	};
+};
+
+/**
+ * struct __drm_i915_gem_create_ext - Existing gem_create behaviour, with added
+ * extension support using struct i915_user_extension.
+ *
+ * Note that new buffer flags should be added here, at least for the stuff that
+ * is immutable. Previously we would have two ioctls, one to create the object
+ * with gem_create, and another to apply various parameters, however this
+ * creates some ambiguity for the params which are considered immutable. Also in
+ * general we're phasing out the various SET/GET ioctls.
+ */
+struct __drm_i915_gem_create_ext {
+	/**
+	 * @size: Requested size for the object.
+	 *
+	 * The (page-aligned) allocated size for the object will be returned.
+	 *
+	 * Note that for some devices we have might have further minimum
+	 * page-size restrictions(larger than 4K), like for device local-memory.
+	 * However in general the final size here should always reflect any
+	 * rounding up, if for example using the I915_GEM_CREATE_EXT_MEMORY_REGIONS
+	 * extension to place the object in device local-memory.
+	 */
+	__u64 size;
+	/**
+	 * @handle: Returned handle for the object.
+	 *
+	 * Object handles are nonzero.
+	 */
+	__u32 handle;
+	/**
+	 * @flags: Optional flags.
+	 *
+	 * Supported values:
+	 *
+	 * I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS - Signal to the kernel that
+	 * the object will need to be accessed via the CPU.
+	 *
+	 * Only valid when placing objects in I915_MEMORY_CLASS_DEVICE, and
+	 * only strictly required on platforms where only some of the device
+	 * memory is directly visible or mappable through the CPU, like on DG2+.
+	 *
+	 * One of the placements MUST also be I915_MEMORY_CLASS_SYSTEM, to
+	 * ensure we can always spill the allocation to system memory, if we
+	 * can't place the object in the mappable part of
+	 * I915_MEMORY_CLASS_DEVICE.
+	 *
+	 * Note that since the kernel only supports flat-CCS on objects that can
+	 * *only* be placed in I915_MEMORY_CLASS_DEVICE, we therefore don't
+	 * support I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS together with
+	 * flat-CCS.
+	 *
+	 * Without this hint, the kernel will assume that non-mappable
+	 * I915_MEMORY_CLASS_DEVICE is preferred for this object. Note that the
+	 * kernel can still migrate the object to the mappable part, as a last
+	 * resort, if userspace ever CPU faults this object, but this might be
+	 * expensive, and so ideally should be avoided.
+	 */
+#define I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS (1 << 0)
+	__u32 flags;
+	/**
+	 * @extensions: The chain of extensions to apply to this object.
+	 *
+	 * This will be useful in the future when we need to support several
+	 * different extensions, and we need to apply more than one when
+	 * creating the object. See struct i915_user_extension.
+	 *
+	 * If we don't supply any extensions then we get the same old gem_create
+	 * behaviour.
+	 *
+	 * For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
+	 * struct drm_i915_gem_create_ext_memory_regions.
+	 *
+	 * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
+	 * struct drm_i915_gem_create_ext_protected_content.
+	 */
+#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
+#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
+	__u64 extensions;
+};
+
+#define DRM_I915_QUERY_VMA_INFO	5
+
+/**
+ * struct __drm_i915_query_vma_info
+ *
+ * Given a vm and GTT address, lookup the corresponding vma, returning its set
+ * of attributes.
+ *
+ * .. code-block:: C
+ *
+ *	struct drm_i915_query_vma_info info = {};
+ *	struct drm_i915_query_item item = {
+ *		.data_ptr = (uintptr_t)&info,
+ *		.query_id = DRM_I915_QUERY_VMA_INFO,
+ *	};
+ *	struct drm_i915_query query = {
+ *		.num_items = 1,
+ *		.items_ptr = (uintptr_t)&item,
+ *	};
+ *	int err;
+ *
+ *	// Unlike some other types of queries, there is no need to first query
+ *	// the size of the data_ptr blob here, since we already know ahead of
+ *	// time how big this needs to be.
+ *	item.length = sizeof(info);
+ *
+ *	// Next we fill in the vm_id and ppGTT address of the vma we wish
+ *	// to query, before then firing off the query.
+ *	info.vm_id = vm_id;
+ *	info.offset = gtt_address;
+ *	err = ioctl(fd, DRM_IOCTL_I915_QUERY, &query);
+ *	if (err || item.length < 0) ...
+ *
+ *	// If all went well we can now inspect the returned attributes.
+ *	if (info.attributes & DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE) ...
+ */
+struct __drm_i915_query_vma_info {
+	/**
+	 * @vm_id: The given vm id that contains the vma. The id is the value
+	 * returned by the DRM_I915_GEM_VM_CREATE. See struct
+	 * drm_i915_gem_vm_control.vm_id.
+	 */
+	__u32 vm_id;
+	/** @pad: MBZ. */
+	__u32 pad;
+	/**
+	 * @offset: The corresponding ppGTT address of the vma which the kernel
+	 * will use to perform the lookup.
+	 */
+	__u64 offset;
+	/**
+	 * @attributes: The returned attributes for the given vma.
+	 *
+	 * Possible values:
+	 *
+	 * DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE - Set if the pages backing the
+	 * vma are currently CPU accessible. If this is not set then the vma is
+	 * currently backed by I915_MEMORY_CLASS_DEVICE memory, which the CPU
+	 * cannot directly access(this is only possible on discrete devices with
+	 * a small BAR). Attempting to MMAP and fault such an object will
+	 * require the kernel first synchronising any GPU work tied to the
+	 * object, before then migrating the pages, either to the CPU accessible
+	 * part of I915_MEMORY_CLASS_DEVICE, or I915_MEMORY_CLASS_SYSTEM, if the
+	 * placements permit it. See I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS.
+	 *
+	 * Note that this is inherently racy.
+	 */
+#define DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE (1<<0)
+	__u64 attributes;
+	/** @rsvd: MBZ */
+	__u32 rsvd[4];
+};
diff --git a/Documentation/gpu/rfc/i915_small_bar.rst b/Documentation/gpu/rfc/i915_small_bar.rst
new file mode 100644
index 000000000000..be3d9bcdd86d
--- /dev/null
+++ b/Documentation/gpu/rfc/i915_small_bar.rst
@@ -0,0 +1,58 @@ 
+==========================
+I915 Small BAR RFC Section
+==========================
+Starting from DG2 we will have resizable BAR support for device local-memory(i.e
+I915_MEMORY_CLASS_DEVICE), but in some cases the final BAR size might still be
+smaller than the total probed_size. In such cases, only some subset of
+I915_MEMORY_CLASS_DEVICE will be CPU accessible(for example the first 256M),
+while the remainder is only accessible via the GPU.
+
+I915_GEM_CREATE_EXT_FLAG_NEEDS_CPU_ACCESS flag
+----------------------------------------------
+New gem_create_ext flag to tell the kernel that a BO will require CPU access.
+This becomes important when placing an object in I915_MEMORY_CLASS_DEVICE, where
+underneath the device has a small BAR, meaning only some portion of it is CPU
+accessible. Without this flag the kernel will assume that CPU access is not
+required, and prioritize using the non-CPU visible portion of
+I915_MEMORY_CLASS_DEVICE.
+
+.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
+   :functions: __drm_i915_gem_create_ext
+
+probed_cpu_visible_size attribute
+---------------------------------
+New struct__drm_i915_memory_region attribute which returns the total size of the
+CPU accessible portion, for the particular region. This should only be
+applicable for I915_MEMORY_CLASS_DEVICE.
+
+Vulkan will need this as part of creating a separate VkMemoryHeap with the
+VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT set, to represent the CPU visible portion,
+where the total size of the heap needs to be known.
+
+.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
+   :functions: __drm_i915_memory_region_info
+
+DRM_I915_QUERY_VMA_INFO query
+-----------------------------
+Query the attributes of some vma. Given a vm and GTT offset, find the
+respective vma, and return its set of attributes. For now we only support
+DRM_I915_QUERY_VMA_INFO_CPU_VISIBLE, which is set if the object/vma is
+currently placed in memory that is accessible by the CPU. This should always be
+set on devices where the CPU probed_cpu_visible_size of I915_MEMORY_CLASS_DEVICE
+matches the probed_size. If this is not set then CPU faulting the object will
+likely first require migrating the pages.
+
+.. kernel-doc:: Documentation/gpu/rfc/i915_small_bar.h
+   :functions: __drm_i915_query_vma_info
+
+Error Capture restrictions
+--------------------------
+With error capture we have two new restrictions:
+
+    1) Error capture is best effort on small BAR systems; if the pages are not
+    CPU accessible, at the time of capture, then the kernel is free to skip
+    trying to capture them.
+
+    2) On discrete we now reject error capture on recoverable contexts. In the
+    future the kernel may want to blit during error capture, when for example
+    something is not currently CPU accessible.
diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
index 91e93a705230..5a3bd3924ba6 100644
--- a/Documentation/gpu/rfc/index.rst
+++ b/Documentation/gpu/rfc/index.rst
@@ -23,3 +23,7 @@  host such documentation:
 .. toctree::
 
     i915_scheduler.rst
+
+.. toctree::
+
+    i915_small_bar.rst