[RFC,53/60] drm/i915: Create stolen memory region from local memory

Commit Message

Matthew Auld July 10, 2020, 11:57 a.m. UTC

From: CQ Tang <[e-mail]>

Add "REGION_STOLEN" device info to dg1, create stolen memory
region from upper portion of local device memory, starting
from DSMBASE.

The memory region is marked with "is_devmem=true".

Cc: Joonas Lahtinen <[e-mail]>
Cc: Matthew Auld <[e-mail]>
Cc: Abdiel Janulgue <[e-mail]>
Cc: Chris P Wilson <[e-mail]>
Cc: Balestrieri, Francesco <[e-mail]>
Cc: Niranjana Vishwanathapura <[e-mail]>
Cc: Venkata S Dhanalakota <[e-mail]>
Cc: Neel Desai <[e-mail]>
Cc: Matthew Brost <[e-mail]>
Cc: Sudeep Dutt <[e-mail]>
Cc: Lucas De Marchi <[e-mail]>
Signed-off-by: CQ Tang <[e-mail]>
---
 drivers/gpu/drm/i915/gem/i915_gem_lmem.c   | 12 ++---
 drivers/gpu/drm/i915/gem/i915_gem_lmem.h   |  7 +++
 drivers/gpu/drm/i915/gem/i915_gem_stolen.c | 56 +++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_pci.c            |  2 +-
 drivers/gpu/drm/i915/i915_reg.h            |  1 +
 drivers/gpu/drm/i915/intel_memory_region.c |  5 ++
 drivers/gpu/drm/i915/intel_memory_region.h |  2 +-
 7 files changed, 75 insertions(+), 10 deletions(-)

Comments

Dave Airlie July 13, 2020, 4:48 a.m. UTC | #1

On Fri, 10 Jul 2020 at 22:01, Matthew Auld <[e-mail]> wrote:
>
> From: CQ Tang <[e-mail]>
>
> Add "REGION_STOLEN" device info to dg1, create stolen memory
> region from upper portion of local device memory, starting
> from DSMBASE.
>
> The memory region is marked with "is_devmem=true".

So is stolen fake on LMEM devices? The concept of stolen doesn't seem
to make much sense with VRAM, so please enlighten me.

Dave.

Matthew Auld July 14, 2020, 3:01 p.m. UTC | #2

On 13/07/2020 05:48, Dave Airlie wrote:
> On Fri, 10 Jul 2020 at 22:01, Matthew Auld <[e-mail]> wrote:
>>
>> From: CQ Tang <[e-mail]>
>>
>> Add "REGION_STOLEN" device info to dg1, create stolen memory
>> region from upper portion of local device memory, starting
>> from DSMBASE.
>>
>> The memory region is marked with "is_devmem=true".
> 
> So is stolen fake on LMEM devices? The concept of stolen doesn't seem
> to make much sense with VRAM, so please enlighten me.

CQ, do we actually need an explicit stolen LMEM region? The idea of 
having a DSM like stolen region for LMEM does sound strange(outside of 
the usual reserved portions which are for HW use etc), since the driver 
has complete control over LMEM. Is it just a convenience thing to keep 
things working as-is for fbc, initial fb, etc. or is there more to it? 
There is buddy_alloc_range() for LMEM which we could potentially use to 
wrap an object around for things like the initial fb or similar.

> 
> Dave.
>

Tang, CQ July 14, 2020, 4:57 p.m. UTC | #3

> -----Original Message-----
> From: Auld, Matthew <[e-mail]>
> Sent: Tuesday, July 14, 2020 8:02 AM
> To: Dave Airlie <[e-mail]>
> Cc: Intel Graphics Development <[e-mail]>; Tang, CQ
> <[e-mail]>; Joonas Lahtinen <[e-mail]>;
> Abdiel Janulgue <[e-mail]>; Wilson, Chris P
> <[e-mail]>; Balestrieri, Francesco
> <[e-mail]>; Vishwanathapura, Niranjana
> <[e-mail]>; Dhanalakota, Venkata S
> <[e-mail]>; Neel Desai <[e-mail]>;
> Brost, Matthew <[e-mail]>; Dutt, Sudeep
> <[e-mail]>; De Marchi, Lucas <[e-mail]>
> Subject: Re: [RFC 53/60] drm/i915: Create stolen memory region from local
> memory
> 
> On 13/07/2020 05:48, Dave Airlie wrote:
> > On Fri, 10 Jul 2020 at 22:01, Matthew Auld <[e-mail]>
> wrote:
> >>
> >> From: CQ Tang <[e-mail]>
> >>
> >> Add "REGION_STOLEN" device info to dg1, create stolen memory region
> >> from upper portion of local device memory, starting from DSMBASE.
> >>
> >> The memory region is marked with "is_devmem=true".
> >
> > So is stolen fake on LMEM devices? The concept of stolen doesn't seem
> > to make much sense with VRAM, so please enlighten me.
> 
> CQ, do we actually need an explicit stolen LMEM region? The idea of having a
> DSM like stolen region for LMEM does sound strange(outside of the usual
> reserved portions which are for HW use etc), since the driver has complete
> control over LMEM. Is it just a convenience thing to keep things working as-is
> for fbc, initial fb, etc. or is there more to it?
> There is buddy_alloc_range() for LMEM which we could potentially use to
> wrap an object around for things like the initial fb or similar.

This is a natural extension from IGT stolen memory region into DGT, we want to allocate objects from stolen area. In DGT, we have one stolen area per tile so we create one region in each of these area. Using memory region is easier to manage objects allocation and free. Other than fbc and rc6, we have gt/ring allocate stolen memory objects when without LMEM, so only apply to IGT case:

display/intel_display.c:	obj = i915_gem_object_create_stolen_for_preallocated(dev_priv,
display/intel_fbdev.c:			obj = i915_gem_object_create_stolen(dev_priv, size);
display/intel_overlay.c:	obj = i915_gem_object_create_stolen(i915, PAGE_SIZE);
intel_rc6.c:		pctx = i915_gem_object_create_stolen_for_preallocated(i915,
intel_rc6.c:	pctx = i915_gem_object_create_stolen(i915, pctx_size);

intel_ring.c:			obj = intel_gt_object_create_stolen(ggtt->vm.gt, size);
intel_gt.c:		obj = intel_gt_object_create_stolen(gt, size);

For some reason, we don't use buddy allocator to manage the stolen memory, instead, we use drm_mm_node allocator directly, we have one-to-one mapping between drm_mm address space to dma address of the stolen memory. We also use contiguous allocation where an object always get a single contiguous block of pages.

So fundamentally, we want to use the same code to work on both IGT stolen memory and DGT stolen memory.

--CQ


> 
> >
> > Dave.
> >

Ville Syrjala July 14, 2020, 5:39 p.m. UTC | #4

On Tue, Jul 14, 2020 at 04:01:31PM +0100, Matthew Auld wrote:
> On 13/07/2020 05:48, Dave Airlie wrote:
> > On Fri, 10 Jul 2020 at 22:01, Matthew Auld <[e-mail]> wrote:
> >>
> >> From: CQ Tang <[e-mail]>
> >>
> >> Add "REGION_STOLEN" device info to dg1, create stolen memory
> >> region from upper portion of local device memory, starting
> >> from DSMBASE.
> >>
> >> The memory region is marked with "is_devmem=true".
> > 
> > So is stolen fake on LMEM devices? The concept of stolen doesn't seem
> > to make much sense with VRAM, so please enlighten me.
> 
> CQ, do we actually need an explicit stolen LMEM region? The idea of 
> having a DSM like stolen region for LMEM does sound strange(outside of 
> the usual reserved portions which are for HW use etc), since the driver 
> has complete control over LMEM. Is it just a convenience thing to keep 
> things working as-is for fbc, initial fb, etc. or is there more to it? 
> There is buddy_alloc_range() for LMEM which we could potentially use to 
> wrap an object around for things like the initial fb or similar.

For some reason the FBC hardware itself will treat the CFB base
as an offset into stolen memory. So assuming they didn't change
how the FBC hardware works we do still need the stolen
base set up in some fashion. Also the CFB base register is
32bits wide so without the stolen base it wouldn't be able
to address more than 4GiB or memory. Hmm, actually the
docs say the register only accepts 28bit offsets so only
256MiB in fact. I'll have to double check if that's true and
whether we are currently at risk of going past that limit...

I think there are some other magic things in the hardware that
also use the stolen base for something.

I do agree that having stolen inside lmem is a bit odd. That's
not how i810 worked IIRC. Back then you either had lmem or
you had stolen. But I guess all the i810 people retired by now
so there was no one to tell the new guys how to design this
stuff :P

Dave Airlie July 14, 2020, 7:26 p.m. UTC | #5

On Wed, 15 Jul 2020 at 02:57, Tang, CQ <[e-mail]> wrote:
>
>
>
> > -----Original Message-----
> > From: Auld, Matthew <[e-mail]>
> > Sent: Tuesday, July 14, 2020 8:02 AM
> > To: Dave Airlie <[e-mail]>
> > Cc: Intel Graphics Development <[e-mail]>; Tang, CQ
> > <[e-mail]>; Joonas Lahtinen <[e-mail]>;
> > Abdiel Janulgue <[e-mail]>; Wilson, Chris P
> > <[e-mail]>; Balestrieri, Francesco
> > <[e-mail]>; Vishwanathapura, Niranjana
> > <[e-mail]>; Dhanalakota, Venkata S
> > <[e-mail]>; Neel Desai <[e-mail]>;
> > Brost, Matthew <[e-mail]>; Dutt, Sudeep
> > <[e-mail]>; De Marchi, Lucas <[e-mail]>
> > Subject: Re: [RFC 53/60] drm/i915: Create stolen memory region from local
> > memory
> >
> > On 13/07/2020 05:48, Dave Airlie wrote:
> > > On Fri, 10 Jul 2020 at 22:01, Matthew Auld <[e-mail]>
> > wrote:
> > >>
> > >> From: CQ Tang <[e-mail]>
> > >>
> > >> Add "REGION_STOLEN" device info to dg1, create stolen memory region
> > >> from upper portion of local device memory, starting from DSMBASE.
> > >>
> > >> The memory region is marked with "is_devmem=true".
> > >
> > > So is stolen fake on LMEM devices? The concept of stolen doesn't seem
> > > to make much sense with VRAM, so please enlighten me.
> >
> > CQ, do we actually need an explicit stolen LMEM region? The idea of having a
> > DSM like stolen region for LMEM does sound strange(outside of the usual
> > reserved portions which are for HW use etc), since the driver has complete
> > control over LMEM. Is it just a convenience thing to keep things working as-is
> > for fbc, initial fb, etc. or is there more to it?
> > There is buddy_alloc_range() for LMEM which we could potentially use to
> > wrap an object around for things like the initial fb or similar.
>
> This is a natural extension from IGT stolen memory region into DGT, we want to allocate objects from stolen area. In DGT, we have one stolen area per tile so we create one region in each of these area. Using memory region is easier to manage objects allocation and free. Other than fbc and rc6, we have gt/ring allocate stolen memory objects when without LMEM, so only apply to IGT case:
>
> display/intel_display.c:        obj = i915_gem_object_create_stolen_for_preallocated(dev_priv,
> display/intel_fbdev.c:                  obj = i915_gem_object_create_stolen(dev_priv, size);
> display/intel_overlay.c:        obj = i915_gem_object_create_stolen(i915, PAGE_SIZE);
> intel_rc6.c:            pctx = i915_gem_object_create_stolen_for_preallocated(i915,
> intel_rc6.c:    pctx = i915_gem_object_create_stolen(i915, pctx_size);
>
> intel_ring.c:                   obj = intel_gt_object_create_stolen(ggtt->vm.gt, size);
> intel_gt.c:             obj = intel_gt_object_create_stolen(gt, size);
>
> For some reason, we don't use buddy allocator to manage the stolen memory, instead, we use drm_mm_node allocator directly, we have one-to-one mapping between drm_mm address space to dma address of the stolen memory. We also use contiguous allocation where an object always get a single contiguous block of pages.
>
> So fundamentally, we want to use the same code to work on both IGT stolen memory and DGT stolen memory.

If this is fundamentally a software construct then it's horrible, if
the HW has a stolen base like Ville said, and it needs to be in a
chunk of VRAM, how do you go about sizing that, and carving it out
from the user?

I don't think wanting to share the same codepaths here is an
acceptable answer, just fix the code to handle LMEM vs stolen, but if
there are hw reasons on why this is required it would be good to
enumerate exactly what they are and document them.

Dave.

Joonas Lahtinen Aug. 7, 2020, 9:38 a.m. UTC | #6

Quoting Dave Airlie (2020-07-14 22:26:16)
> On Wed, 15 Jul 2020 at 02:57, Tang, CQ <[e-mail]> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Auld, Matthew <[e-mail]>
> > > Sent: Tuesday, July 14, 2020 8:02 AM
> > > To: Dave Airlie <[e-mail]>
> > > Cc: Intel Graphics Development <[e-mail]>; Tang, CQ
> > > <[e-mail]>; Joonas Lahtinen <[e-mail]>;
> > > Abdiel Janulgue <[e-mail]>; Wilson, Chris P
> > > <[e-mail]>; Balestrieri, Francesco
> > > <[e-mail]>; Vishwanathapura, Niranjana
> > > <[e-mail]>; Dhanalakota, Venkata S
> > > <[e-mail]>; Neel Desai <[e-mail]>;
> > > Brost, Matthew <[e-mail]>; Dutt, Sudeep
> > > <[e-mail]>; De Marchi, Lucas <[e-mail]>
> > > Subject: Re: [RFC 53/60] drm/i915: Create stolen memory region from local
> > > memory
> > >
> > > On 13/07/2020 05:48, Dave Airlie wrote:
> > > > On Fri, 10 Jul 2020 at 22:01, Matthew Auld <[e-mail]>
> > > wrote:
> > > >>
> > > >> From: CQ Tang <[e-mail]>
> > > >>
> > > >> Add "REGION_STOLEN" device info to dg1, create stolen memory region
> > > >> from upper portion of local device memory, starting from DSMBASE.
> > > >>
> > > >> The memory region is marked with "is_devmem=true".
> > > >
> > > > So is stolen fake on LMEM devices? The concept of stolen doesn't seem
> > > > to make much sense with VRAM, so please enlighten me.
> > >
> > > CQ, do we actually need an explicit stolen LMEM region? The idea of having a
> > > DSM like stolen region for LMEM does sound strange(outside of the usual
> > > reserved portions which are for HW use etc), since the driver has complete
> > > control over LMEM. Is it just a convenience thing to keep things working as-is
> > > for fbc, initial fb, etc. or is there more to it?
> > > There is buddy_alloc_range() for LMEM which we could potentially use to
> > > wrap an object around for things like the initial fb or similar.
> >
> > This is a natural extension from IGT stolen memory region into DGT, we want to allocate objects from stolen area. In DGT, we have one stolen area per tile so we create one region in each of these area. Using memory region is easier to manage objects allocation and free. Other than fbc and rc6, we have gt/ring allocate stolen memory objects when without LMEM, so only apply to IGT case:
> >
> > display/intel_display.c:        obj = i915_gem_object_create_stolen_for_preallocated(dev_priv,
> > display/intel_fbdev.c:                  obj = i915_gem_object_create_stolen(dev_priv, size);
> > display/intel_overlay.c:        obj = i915_gem_object_create_stolen(i915, PAGE_SIZE);
> > intel_rc6.c:            pctx = i915_gem_object_create_stolen_for_preallocated(i915,
> > intel_rc6.c:    pctx = i915_gem_object_create_stolen(i915, pctx_size);
> >
> > intel_ring.c:                   obj = intel_gt_object_create_stolen(ggtt->vm.gt, size);
> > intel_gt.c:             obj = intel_gt_object_create_stolen(gt, size);
> >
> > For some reason, we don't use buddy allocator to manage the stolen memory, instead, we use drm_mm_node allocator directly, we have one-to-one mapping between drm_mm address space to dma address of the stolen memory. We also use contiguous allocation where an object always get a single contiguous block of pages.
> >
> > So fundamentally, we want to use the same code to work on both IGT stolen memory and DGT stolen memory.
> 
> If this is fundamentally a software construct then it's horrible, if
> the HW has a stolen base like Ville said, and it needs to be in a
> chunk of VRAM, how do you go about sizing that, and carving it out
> from the user?
> 
> I don't think wanting to share the same codepaths here is an
> acceptable answer, just fix the code to handle LMEM vs stolen, but if
> there are hw reasons on why this is required it would be good to
> enumerate exactly what they are and document them.

The boot firmware does configure the stolen region, so it's not fake.

But there's no need to expose the stolen memory to userspace, so these
patches will be dropped.

Regards, Joonas

Tang, CQ Aug. 7, 2020, 4:24 p.m. UTC | #7

> -----Original Message-----
> From: Joonas Lahtinen <[e-mail]>
> Sent: Friday, August 7, 2020 2:39 AM
> To: Tang, CQ <[e-mail]>; Dave Airlie <[e-mail]>
> Cc: Auld, Matthew <[e-mail]>; Intel Graphics Development
> <[e-mail]>; Abdiel Janulgue
> <[e-mail]>; Wilson, Chris P
> <[e-mail]>; Balestrieri, Francesco
> <[e-mail]>; Vishwanathapura, Niranjana
> <[e-mail]>; Dhanalakota, Venkata S
> <[e-mail]>; Neel Desai <[e-mail]>;
> Brost, Matthew <[e-mail]>; Dutt, Sudeep
> <[e-mail]>; De Marchi, Lucas <[e-mail]>
> Subject: Re: [RFC 53/60] drm/i915: Create stolen memory region from local
> memory
> 
> Quoting Dave Airlie (2020-07-14 22:26:16)
> > On Wed, 15 Jul 2020 at 02:57, Tang, CQ <[e-mail]> wrote:
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Auld, Matthew <[e-mail]>
> > > > Sent: Tuesday, July 14, 2020 8:02 AM
> > > > To: Dave Airlie <[e-mail]>
> > > > Cc: Intel Graphics Development <[e-mail]>;
> > > > Tang, CQ <[e-mail]>; Joonas Lahtinen
> > > > <[e-mail]>;
> > > > Abdiel Janulgue <[e-mail]>; Wilson, Chris P
> > > > <[e-mail]>; Balestrieri, Francesco
> > > > <[e-mail]>; Vishwanathapura, Niranjana
> > > > <[e-mail]>; Dhanalakota, Venkata S
> > > > <[e-mail]>; Neel Desai
> > > > <[e-mail]>; Brost, Matthew <[e-mail]>;
> > > > Dutt, Sudeep <[e-mail]>; De Marchi, Lucas
> > > > <[e-mail]>
> > > > Subject: Re: [RFC 53/60] drm/i915: Create stolen memory region
> > > > from local memory
> > > >
> > > > On 13/07/2020 05:48, Dave Airlie wrote:
> > > > > On Fri, 10 Jul 2020 at 22:01, Matthew Auld
> > > > > <[e-mail]>
> > > > wrote:
> > > > >>
> > > > >> From: CQ Tang <[e-mail]>
> > > > >>
> > > > >> Add "REGION_STOLEN" device info to dg1, create stolen memory
> > > > >> region from upper portion of local device memory, starting from
> DSMBASE.
> > > > >>
> > > > >> The memory region is marked with "is_devmem=true".
> > > > >
> > > > > So is stolen fake on LMEM devices? The concept of stolen doesn't
> > > > > seem to make much sense with VRAM, so please enlighten me.
> > > >
> > > > CQ, do we actually need an explicit stolen LMEM region? The idea
> > > > of having a DSM like stolen region for LMEM does sound
> > > > strange(outside of the usual reserved portions which are for HW
> > > > use etc), since the driver has complete control over LMEM. Is it
> > > > just a convenience thing to keep things working as-is for fbc, initial fb,
> etc. or is there more to it?
> > > > There is buddy_alloc_range() for LMEM which we could potentially
> > > > use to wrap an object around for things like the initial fb or similar.
> > >
> > > This is a natural extension from IGT stolen memory region into DGT, we
> want to allocate objects from stolen area. In DGT, we have one stolen area
> per tile so we create one region in each of these area. Using memory region
> is easier to manage objects allocation and free. Other than fbc and rc6, we
> have gt/ring allocate stolen memory objects when without LMEM, so only
> apply to IGT case:
> > >
> > > display/intel_display.c:        obj =
> i915_gem_object_create_stolen_for_preallocated(dev_priv,
> > > display/intel_fbdev.c:                  obj =
> i915_gem_object_create_stolen(dev_priv, size);
> > > display/intel_overlay.c:        obj = i915_gem_object_create_stolen(i915,
> PAGE_SIZE);
> > > intel_rc6.c:            pctx =
> i915_gem_object_create_stolen_for_preallocated(i915,
> > > intel_rc6.c:    pctx = i915_gem_object_create_stolen(i915, pctx_size);
> > >
> > > intel_ring.c:                   obj = intel_gt_object_create_stolen(ggtt->vm.gt,
> size);
> > > intel_gt.c:             obj = intel_gt_object_create_stolen(gt, size);
> > >
> > > For some reason, we don't use buddy allocator to manage the stolen
> memory, instead, we use drm_mm_node allocator directly, we have one-to-
> one mapping between drm_mm address space to dma address of the stolen
> memory. We also use contiguous allocation where an object always get a
> single contiguous block of pages.
> > >
> > > So fundamentally, we want to use the same code to work on both IGT
> stolen memory and DGT stolen memory.
> >
> > If this is fundamentally a software construct then it's horrible, if
> > the HW has a stolen base like Ville said, and it needs to be in a
> > chunk of VRAM, how do you go about sizing that, and carving it out
> > from the user?
> >
> > I don't think wanting to share the same codepaths here is an
> > acceptable answer, just fix the code to handle LMEM vs stolen, but if
> > there are hw reasons on why this is required it would be good to
> > enumerate exactly what they are and document them.
> 
> The boot firmware does configure the stolen region, so it's not fake.
> 
> But there's no need to expose the stolen memory to userspace, so these
> patches will be dropped.

Yes, I confirmed that on discrete GPU with display, firmware does reserve a portion of device local memory as stolen memory region, and used by device hardware, the usage of this local stolen memory is the same as system stolen memory on integrated GPU.

So we need to continue to create this local stolen memory region in driver. However, because this stolen memory region is only used by driver and hardware, there is no need to expose the stolen memory region to userspace.

--CQ

> 
> Regards, Joonas

Patch

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index 423662cec5da..456b304b244e 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -8,8 +8,8 @@ 
 #include "gem/i915_gem_lmem.h"
 #include "i915_drv.h"
 
-static int lmem_pread(struct drm_i915_gem_object *obj,
-		      const struct drm_i915_gem_pread *arg)
+int i915_gem_object_lmem_pread(struct drm_i915_gem_object *obj,
+			       const struct drm_i915_gem_pread *arg)
 {
 	struct drm_i915_private *i915 = to_i915(obj->base.dev);
 	struct intel_runtime_pm *rpm = &i915->runtime_pm;
@@ -94,8 +94,8 @@  static int lmem_pread(struct drm_i915_gem_object *obj,
 	return ret;
 }
 
-static int lmem_pwrite(struct drm_i915_gem_object *obj,
-		       const struct drm_i915_gem_pwrite *arg)
+int i915_gem_object_lmem_pwrite(struct drm_i915_gem_object *obj,
+				const struct drm_i915_gem_pwrite *arg)
 {
 	struct drm_i915_private *i915 = to_i915(obj->base.dev);
 	struct intel_runtime_pm *rpm = &i915->runtime_pm;
@@ -187,8 +187,8 @@  const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops = {
 	.put_pages = i915_gem_object_put_pages_buddy,
 	.release = i915_gem_object_release_memory_region,
 
-	.pread = lmem_pread,
-	.pwrite = lmem_pwrite,
+	.pread = i915_gem_object_lmem_pread,
+	.pwrite = i915_gem_object_lmem_pwrite,
 };
 
 void __iomem *
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
index e11e0545e39c..c59aa6c014c7 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
@@ -11,9 +11,16 @@ 
 struct drm_i915_private;
 struct drm_i915_gem_object;
 struct intel_memory_region;
+struct drm_i915_gem_pread;
+struct drm_i915_gem_pwrite;
 
 extern const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops;
 
+int i915_gem_object_lmem_pread(struct drm_i915_gem_object *obj,
+			       const struct drm_i915_gem_pread *args);
+int i915_gem_object_lmem_pwrite(struct drm_i915_gem_object *obj,
+				const struct drm_i915_gem_pwrite *args);
+
 void __iomem *
 i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
 			    unsigned long n,
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
index fee5ff1a1863..0bcfe4464fa1 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
@@ -10,6 +10,7 @@ 
 #include <drm/drm_mm.h>
 #include <drm/i915_drm.h>
 
+#include "gem/i915_gem_lmem.h"
 #include "gem/i915_gem_region.h"
 #include "i915_drv.h"
 #include "i915_gem_stolen.h"
@@ -119,6 +120,14 @@  static int i915_adjust_stolen(struct drm_i915_private *i915,
 		}
 	}
 
+	/*
+	 * With device local memory, we don't need to check the address range,
+	 * this is device memory physical address, could overlap with system
+	 * memory.
+	 */
+	if (HAS_LMEM(i915))
+		return 0;
+
 	/*
 	 * Verify that nothing else uses this physical address. Stolen
 	 * memory should be reserved by the BIOS and hidden from the
@@ -558,7 +567,7 @@  static void i915_gem_object_put_pages_stolen(struct drm_i915_gem_object *obj,
 	kfree(pages);
 }
 
-static const struct drm_i915_gem_object_ops i915_gem_object_stolen_ops = {
+static struct drm_i915_gem_object_ops i915_gem_object_stolen_ops = {
 	.name = "i915_gem_object_stolen",
 	.get_pages = i915_gem_object_get_pages_stolen,
 	.put_pages = i915_gem_object_put_pages_stolen,
@@ -661,7 +670,19 @@  i915_gem_object_create_stolen(struct drm_i915_private *i915,
 
 static int init_stolen(struct intel_memory_region *mem)
 {
-	intel_memory_region_set_name(mem, "stolen");
+	if (mem->type == INTEL_MEMORY_STOLEN_SYSTEM)
+		intel_memory_region_set_name(mem, "stolen-system");
+	else
+		intel_memory_region_set_name(mem, "stolen-local");
+
+	if (HAS_LMEM(mem->i915)) {
+		i915_gem_object_stolen_ops.pread = i915_gem_object_lmem_pread;
+		i915_gem_object_stolen_ops.pwrite = i915_gem_object_lmem_pwrite;
+		if (!io_mapping_init_wc(&mem->iomap,
+					mem->io_start,
+					resource_size(&mem->region)))
+			return -EIO;
+	}
 
 	/*
 	 * Initialise stolen early so that we may reserve preallocated
@@ -681,8 +702,39 @@  static const struct intel_memory_region_ops i915_region_stolen_ops = {
 	.create_object = i915_gem_object_create_stolen_region,
 };
 
+static
+struct intel_memory_region *setup_lmem_stolen(struct drm_i915_private *i915)
+{
+	struct intel_uncore *uncore = &i915->uncore;
+	struct pci_dev *pdev = i915->drm.pdev;
+	struct intel_memory_region *mem;
+	resource_size_t io_start;
+	resource_size_t lmem_size;
+	u64 lmem_base;
+
+	lmem_base = intel_uncore_read64(uncore, GEN12_DSMBASE);
+	lmem_size = pci_resource_len(pdev, 2) - lmem_base;
+	io_start = pci_resource_start(pdev, 2) + lmem_base;
+
+	mem = intel_memory_region_create(i915, lmem_base, lmem_size,
+					 I915_GTT_PAGE_SIZE_4K, io_start,
+					 &i915_region_stolen_ops);
+	if (!IS_ERR(mem)) {
+		DRM_INFO("Intel graphics stolen LMEM: %pR\n", &mem->region);
+		DRM_INFO("Intel graphics stolen LMEM IO start: %llx\n",
+			 (u64)mem->io_start);
+		/* this is real device memory */
+		mem->is_devmem = true;
+	}
+
+	return mem;
+}
+
 struct intel_memory_region *i915_gem_stolen_setup(struct drm_i915_private *i915)
 {
+	if (HAS_LMEM(i915))
+		return setup_lmem_stolen(i915);
+
 	return intel_memory_region_create(i915,
 					  intel_graphics_stolen_res.start,
 					  resource_size(&intel_graphics_stolen_res),
diff --git a/drivers/gpu/drm/i915/i915_pci.c b/drivers/gpu/drm/i915/i915_pci.c
index e132fdffa432..fbb7ac96d881 100644
--- a/drivers/gpu/drm/i915/i915_pci.c
+++ b/drivers/gpu/drm/i915/i915_pci.c
@@ -897,7 +897,7 @@  static const struct intel_device_info rkl_info = {
 
 #define GEN12_DGFX_FEATURES \
 	GEN12_FEATURES, \
-	.memory_regions = REGION_SMEM | REGION_LMEM, \
+	.memory_regions = REGION_SMEM | REGION_LMEM | REGION_STOLEN_LMEM, \
 	.has_master_unit_irq = 1, \
 	.has_llc = 0, \
 	.has_snoop = 1, \
diff --git a/drivers/gpu/drm/i915/i915_reg.h b/drivers/gpu/drm/i915/i915_reg.h
index 6e67c4ee09c2..7ee322728a65 100644
--- a/drivers/gpu/drm/i915/i915_reg.h
+++ b/drivers/gpu/drm/i915/i915_reg.h
@@ -12039,6 +12039,7 @@  enum skl_power_gate {
 #define   LMEM_ENABLE			(1 << 31)
 
 #define GEN12_GSMBASE			_MMIO(0x108100)
+#define GEN12_DSMBASE			_MMIO(0x1080C0)
 
 /* gamt regs */
 #define GEN8_L3_LRA_1_GPGPU _MMIO(0x4dd4)
diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c
index d564b596efda..905cdfa6e3e2 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/intel_memory_region.c
@@ -19,6 +19,10 @@  const struct intel_memory_region_info intel_region_map[] = {
                .class = INTEL_MEMORY_STOLEN_SYSTEM,
                .instance = 0,
        },
+       [INTEL_REGION_STOLEN_LMEM] = {
+               .class = INTEL_MEMORY_STOLEN_LOCAL,
+               .instance = 0,
+       },
 };
 
 struct intel_memory_region *
@@ -305,6 +309,7 @@  int intel_memory_regions_hw_probe(struct drm_i915_private *i915)
 		case INTEL_MEMORY_SYSTEM:
 			mem = i915_gem_shmem_setup(i915);
 			break;
+		case INTEL_MEMORY_STOLEN_LOCAL: /* fallthrough */
 		case INTEL_MEMORY_STOLEN_SYSTEM:
 			mem = i915_gem_stolen_setup(i915);
 			break;
diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h
index a08cbae18979..8f9c943a16af 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.h
+++ b/drivers/gpu/drm/i915/intel_memory_region.h
@@ -93,7 +93,7 @@  struct intel_memory_region {
 	u16 type;
 	u16 instance;
 	enum intel_region_id id;
-	char name[8];
+	char name[16];
 	bool is_devmem;	/* true for device memory */
 
 	dma_addr_t remap_addr;