[3/3] e820: Add the unknown-12 Memory type (DDR3-NvDIMM)
diff mbox

Message ID 54F82ED1.8030900@plexistor.com
State New, archived
Headers show

Commit Message

Boaz Harrosh March 5, 2015, 10:24 a.m. UTC
There are multiple vendors of DDR3 NvDIMMs out in the market today.
At various stages of development/production. It is estimated that
there are already more the 100ds of thousands chips sold to
testers and sites.

All the BIOS vendors I know of, tagged these chips at e820 table
as type-12 memory.

Now the ACPI comity, as far as I know, did not yet define a
standard type for NvDIMM. Also, as far as I know any NvDIMM
standard will only be defined for DDR4. So DDR3 NvDIMM is
probably stuck with this none STD type.

I Wish and call the ACPI comity to Define that NvDIMM is type-12.
Also for DDR4

The motivation of this patch is to be able to differentiate
this NvDIMM type from a real future "unknown-reserved" type.

In this patch I name type-12 "unknown-12". This is because of
ACPI politics that refuse to reserve type-12 as DDR3-NvDIMM
and members keep saying:
	"What if ACPI assigns type-12 for something else in future"

[And I say: Then just don't. Please?]

Signed-off-by: Boaz Harrosh <boaz@plexistor.com>
---
 arch/x86/kernel/e820.c | 9 +++++++++
 1 file changed, 9 insertions(+)

Comments

Dan Williams March 5, 2015, 8:56 p.m. UTC | #1
On Thu, Mar 5, 2015 at 2:24 AM, Boaz Harrosh <boaz@plexistor.com> wrote:
>
> There are multiple vendors of DDR3 NvDIMMs out in the market today.
> At various stages of development/production. It is estimated that
> there are already more the 100ds of thousands chips sold to
> testers and sites.
>
> All the BIOS vendors I know of, tagged these chips at e820 table
> as type-12 memory.
>
> Now the ACPI comity, as far as I know, did not yet define a
> standard type for NvDIMM. Also, as far as I know any NvDIMM
> standard will only be defined for DDR4. So DDR3 NvDIMM is
> probably stuck with this none STD type.

There's no relation between E820 types and DDR technology revisions.

> I Wish and call the ACPI comity to Define that NvDIMM is type-12.
> Also for DDR4
>
> The motivation of this patch is to be able to differentiate
> this NvDIMM type from a real future "unknown-reserved" type.
>
> In this patch I name type-12 "unknown-12". This is because of
> ACPI politics that refuse to reserve type-12 as DDR3-NvDIMM

It's not "politics".  Setting standards takes time and the platforms
in question simply jumped the gun to enable a proof-of-concept.

> and members keep saying:
>         "What if ACPI assigns type-12 for something else in future"
>
> [And I say: Then just don't. Please?]

Once a standard number is assigned, platform firmwares can update
type-12 to that number.  We might consider a compile time override for
these niche/pre-standard systems that can't/won't update, but it's not
clear to me that we even need to go that far.
Andy Lutomirski March 5, 2015, 11:09 p.m. UTC | #2
On Thu, Mar 5, 2015 at 12:56 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Thu, Mar 5, 2015 at 2:24 AM, Boaz Harrosh <boaz@plexistor.com> wrote:
>>
>> There are multiple vendors of DDR3 NvDIMMs out in the market today.
>> At various stages of development/production. It is estimated that
>> there are already more the 100ds of thousands chips sold to
>> testers and sites.
>>
>> All the BIOS vendors I know of, tagged these chips at e820 table
>> as type-12 memory.
>>
>> Now the ACPI comity, as far as I know, did not yet define a
>> standard type for NvDIMM. Also, as far as I know any NvDIMM
>> standard will only be defined for DDR4. So DDR3 NvDIMM is
>> probably stuck with this none STD type.
>
> There's no relation between E820 types and DDR technology revisions.
>
>> I Wish and call the ACPI comity to Define that NvDIMM is type-12.
>> Also for DDR4
>>
>> The motivation of this patch is to be able to differentiate
>> this NvDIMM type from a real future "unknown-reserved" type.
>>
>> In this patch I name type-12 "unknown-12". This is because of
>> ACPI politics that refuse to reserve type-12 as DDR3-NvDIMM
>
> It's not "politics".  Setting standards takes time and the platforms
> in question simply jumped the gun to enable a proof-of-concept.
>
>> and members keep saying:
>>         "What if ACPI assigns type-12 for something else in future"
>>
>> [And I say: Then just don't. Please?]
>
> Once a standard number is assigned, platform firmwares can update
> type-12 to that number.  We might consider a compile time override for
> these niche/pre-standard systems that can't/won't update, but it's not
> clear to me that we even need to go that far.

I will be shocked if a standard of this form ever appears.  Modern
systems *don't have e820*.  The BIOSes that are using this type 12
hack are awful throwbacks.

--Andy
Boaz Harrosh March 9, 2015, 11:19 a.m. UTC | #3
On 03/05/2015 10:56 PM, Dan Williams wrote:
> On Thu, Mar 5, 2015 at 2:24 AM, Boaz Harrosh <boaz@plexistor.com> wrote:
>>
<>
>> Now the ACPI comity, as far as I know, did not yet define a
>> standard type for NvDIMM. Also, as far as I know any NvDIMM
>> standard will only be defined for DDR4. So DDR3 NvDIMM is
>> probably stuck with this none STD type.
> 
> There's no relation between E820 types and DDR technology revisions.
> 

Yes and no, I mean the DDR4 has extra legs and signals defined
for NvDIMM. So DDR3 will always mean different style of NvDIMM.

You tell me. Say the standard finally comes out. Will I have a
new bios from Intel for my DDR3 system here in the lab that will
report the new STD type ?

What I meant is that DDR3 is too old for the proposed STD and probably
only DDR4 NvDIMMs will be supported in systems. The way the STD defined
it.

<>
>> In this patch I name type-12 "unknown-12". This is because of
>> ACPI politics that refuse to reserve type-12 as DDR3-NvDIMM
> 
> It's not "politics".  Setting standards takes time and the platforms
> in question simply jumped the gun to enable a proof-of-concept.
> 

So ye, but once you have 100,000 devices out there, then the dichotomy
between standards-takes-time vs proof-of-concept, becomes politics.

This is the definition of politics, when life moves faster than some
"body", the "body" stands on its back feet and shoots fire from
his head.

>> and members keep saying:
>>         "What if ACPI assigns type-12 for something else in future"
>>
>> [And I say: Then just don't. Please?]
> 
> Once a standard number is assigned, platform firmwares can update
> type-12 to that number.  We might consider a compile time override for
> these niche/pre-standard systems that can't/won't update, but it's not
> clear to me that we even need to go that far.
> 

OK, so I do not understand what you want. Yes or No to this patch?

This patch with unknown-12 is for NOW. For systems already running.

So we can differentiate between  reserved-unknown which might mean
type-13 and this here bastard type-12 which we know is NvDIMM but
for future sake we do not call by name?

Or maybe we should call it NVDIMM-12 ?

Thanks
Boaz
Boaz Harrosh March 9, 2015, 12:10 p.m. UTC | #4
On 03/06/2015 01:09 AM, Andy Lutomirski wrote:
<>
> 
> I will be shocked if a standard of this form ever appears.  Modern
> systems *don't have e820*.  The BIOSes that are using this type 12
> hack are awful throwbacks.

So far the systems we have, with DDR4 NvDIMM(s) (Actual chips arriving soon)
still have BIOS (On by default). The BIOS was yelled "Wolfe" for so long NOW ;-)
Again these are working system in the field, who will switch them all to UEFI?

How will the UEFI present them to the system? can you point me to the relevant
code? (Or did you mean BIOS but with a different communication path than e820?)

> 
> --Andy
> 

Thanks
Boaz
Dan Williams March 9, 2015, 2:44 p.m. UTC | #5
On Mon, Mar 9, 2015 at 7:19 AM, Boaz Harrosh <boaz@plexistor.com> wrote:
> On 03/05/2015 10:56 PM, Dan Williams wrote:
[..]
>> It's not "politics".  Setting standards takes time and the platforms
>> in question simply jumped the gun to enable a proof-of-concept.
>>
>
> So ye, but once you have 100,000 devices out there, then the dichotomy
> between standards-takes-time vs proof-of-concept, becomes politics.

Ok.

...although, I have a question about this "100,000 devices" you quote.
What's not clear to me is how many platforms are shipping with
type-12.  Certainly there are very many NVDIMMs on the market, but
which off-the-shelf systems can one obtain that include type-12
support?  Not that it would change the disposition of this patch, if
platforms are in the field they're "in the field!", just clarifying
that "100,000 devices" is NVDIMMs not type-12 platforms, right?
Andy Lutomirski March 9, 2015, 3:14 p.m. UTC | #6
On Mon, Mar 9, 2015 at 7:44 AM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Mon, Mar 9, 2015 at 7:19 AM, Boaz Harrosh <boaz@plexistor.com> wrote:
>> On 03/05/2015 10:56 PM, Dan Williams wrote:
> [..]
>>> It's not "politics".  Setting standards takes time and the platforms
>>> in question simply jumped the gun to enable a proof-of-concept.
>>>
>>
>> So ye, but once you have 100,000 devices out there, then the dichotomy
>> between standards-takes-time vs proof-of-concept, becomes politics.
>
> Ok.
>
> ...although, I have a question about this "100,000 devices" you quote.
> What's not clear to me is how many platforms are shipping with
> type-12.  Certainly there are very many NVDIMMs on the market, but
> which off-the-shelf systems can one obtain that include type-12
> support?  Not that it would change the disposition of this patch, if
> platforms are in the field they're "in the field!", just clarifying
> that "100,000 devices" is NVDIMMs not type-12 platforms, right?

This is a type-12 platform:

http://www.supermicro.com/products/motherboard/Xeon/C600/X9DRH-iF-NV.cfm

You can order one online, for example:

http://www.acmemicro.com/Product/11926/Supermicro+X9DRH-iF-NV-O+Server+Board+DP+Xeon+E5-2600+LGA2011+DDR3+SATA3+RAID+IPMI+GbE+PCIe+ATX+MBD-X9DRH-iF-NV

--Andy
Dan Williams March 9, 2015, 3:17 p.m. UTC | #7
On Mon, Mar 9, 2015 at 11:14 AM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Mon, Mar 9, 2015 at 7:44 AM, Dan Williams <dan.j.williams@intel.com> wrote:
>> On Mon, Mar 9, 2015 at 7:19 AM, Boaz Harrosh <boaz@plexistor.com> wrote:
>>> On 03/05/2015 10:56 PM, Dan Williams wrote:
>> [..]
>>>> It's not "politics".  Setting standards takes time and the platforms
>>>> in question simply jumped the gun to enable a proof-of-concept.
>>>>
>>>
>>> So ye, but once you have 100,000 devices out there, then the dichotomy
>>> between standards-takes-time vs proof-of-concept, becomes politics.
>>
>> Ok.
>>
>> ...although, I have a question about this "100,000 devices" you quote.
>> What's not clear to me is how many platforms are shipping with
>> type-12.  Certainly there are very many NVDIMMs on the market, but
>> which off-the-shelf systems can one obtain that include type-12
>> support?  Not that it would change the disposition of this patch, if
>> platforms are in the field they're "in the field!", just clarifying
>> that "100,000 devices" is NVDIMMs not type-12 platforms, right?
>
> This is a type-12 platform:
>
> http://www.supermicro.com/products/motherboard/Xeon/C600/X9DRH-iF-NV.cfm
>
> You can order one online, for example:
>
> http://www.acmemicro.com/Product/11926/Supermicro+X9DRH-iF-NV-O+Server+Board+DP+Xeon+E5-2600+LGA2011+DDR3+SATA3+RAID+IPMI+GbE+PCIe+ATX+MBD-X9DRH-iF-NV
>

Ok, the gig is up :)
Joey Lee March 10, 2015, 5:11 a.m. UTC | #8
Hi, 

On Mon, Mar 09, 2015 at 02:10:37PM +0200, Boaz Harrosh wrote:
> On 03/06/2015 01:09 AM, Andy Lutomirski wrote:
> <>
> > 
> > I will be shocked if a standard of this form ever appears.  Modern
> > systems *don't have e820*.  The BIOSes that are using this type 12
> > hack are awful throwbacks.
> 
> So far the systems we have, with DDR4 NvDIMM(s) (Actual chips arriving soon)
> still have BIOS (On by default). The BIOS was yelled "Wolfe" for so long NOW ;-)
> Again these are working system in the field, who will switch them all to UEFI?
> 
> How will the UEFI present them to the system? can you point me to the relevant
> code? (Or did you mean BIOS but with a different communication path than e820?)
>

Per my understand...

With EFI Boot Stub, there have setup_e820() codes in arch/x86/boot/compressed/eboot.c
that used to transfer EFI memmap to e820 entries. Currently doesn't have any
EFI_MEMORY_TYPE reflects to NvDIMM that will map to e820_type. 

I wonder what kind of EFI_MEMORY_TYPE reported by UEFI BIOS on those "shipped
NvDIMMs motherboards", like supermicro X9DRH-iF-NV. Then we may need add code
to setup_e820() for mapping the efi memory type to e820 type 12 region.

> > 
> > --Andy
> > 
> 
> Thanks
> Boaz
> 

Thanks a lot!
Joey Lee
Boaz Harrosh March 10, 2015, 8:47 a.m. UTC | #9
On 03/09/2015 05:17 PM, Dan Williams wrote:
> On Mon, Mar 9, 2015 at 11:14 AM, Andy Lutomirski <luto@amacapital.net> wrote:
>> On Mon, Mar 9, 2015 at 7:44 AM, Dan Williams <dan.j.williams@intel.com> wrote:
>>> On Mon, Mar 9, 2015 at 7:19 AM, Boaz Harrosh <boaz@plexistor.com> wrote:
>>>> On 03/05/2015 10:56 PM, Dan Williams wrote:
>>> [..]
>>>>> It's not "politics".  Setting standards takes time and the platforms
>>>>> in question simply jumped the gun to enable a proof-of-concept.
>>>>>
>>>>
>>>> So ye, but once you have 100,000 devices out there, then the dichotomy
>>>> between standards-takes-time vs proof-of-concept, becomes politics.
>>>
>>> Ok.
>>>
>>> ...although, I have a question about this "100,000 devices" you quote.
>>> What's not clear to me is how many platforms are shipping with
>>> type-12.  Certainly there are very many NVDIMMs on the market, but
>>> which off-the-shelf systems can one obtain that include type-12
>>> support?  Not that it would change the disposition of this patch, if
>>> platforms are in the field they're "in the field!", just clarifying
>>> that "100,000 devices" is NVDIMMs not type-12 platforms, right?
>>
>> This is a type-12 platform:
>>
>> http://www.supermicro.com/products/motherboard/Xeon/C600/X9DRH-iF-NV.cfm
>>
>> You can order one online, for example:
>>
>> http://www.acmemicro.com/Product/11926/Supermicro+X9DRH-iF-NV-O+Server+Board+DP+Xeon+E5-2600+LGA2011+DDR3+SATA3+RAID+IPMI+GbE+PCIe+ATX+MBD-X9DRH-iF-NV
>>
> 
> Ok, the gig is up :)
> 

Hi Dan.

Sigh, yes. These systems are out there. lots of them, and not only at developers.

So far all the system I had a hands on and all the people I've asked all reported
type-12, if the DIMM was reported at all (or even booted). With various success rate
of actual NVIDIMM functionality.

I call to all developers that are working on NvDIMM. What are the methods you see
that these chips are presented to the system? Are you using BIOS or UEFI?

(From what I understood, there is actually only one family of motherboards/firmware
 That actually boot NVDIMMs, and it all originated from Intel. I might be off by a
 mile, I have not signed any NDA and no one told me so, just my personal follow the
 bread crumbs)

Could we please reconsider and name those in /proc/iomem as NVDIMM-12 instead of unknow-12
just as the nobleness obliges? We are tainting the Kernel and complaining at driver use of
them, that will not change, just the name?

Thanks
Boaz
Boaz Harrosh March 10, 2015, 8:56 a.m. UTC | #10
On 03/10/2015 07:11 AM, joeyli wrote:
<>
> 
> Per my understand...
> 
> With EFI Boot Stub, there have setup_e820() codes in arch/x86/boot/compressed/eboot.c
> that used to transfer EFI memmap to e820 entries. Currently doesn't have any
> EFI_MEMORY_TYPE reflects to NvDIMM that will map to e820_type. 
> 
> I wonder what kind of EFI_MEMORY_TYPE reported by UEFI BIOS on those "shipped
> NvDIMMs motherboards", like supermicro X9DRH-iF-NV. Then we may need add code
> to setup_e820() for mapping the efi memory type to e820 type 12 region.
> 

Thanks Joey. This is tremendous help. (Just saved me days of research)

Our DDR4 NvDIMM chips arrive this week I will try and debug all this, also
booting EFI.

If you have any chance to experiments in your lab, and compare notes it
would be great.

<>
> 
> Thanks a lot!
> Joey Lee
> 

Thanks
Boaz
Andy Lutomirski March 10, 2015, 1:19 p.m. UTC | #11
On Mar 10, 2015 1:12 AM, "joeyli" <jlee@suse.com> wrote:
>
> Hi,
>
> On Mon, Mar 09, 2015 at 02:10:37PM +0200, Boaz Harrosh wrote:
> > On 03/06/2015 01:09 AM, Andy Lutomirski wrote:
> > <>
> > >
> > > I will be shocked if a standard of this form ever appears.  Modern
> > > systems *don't have e820*.  The BIOSes that are using this type 12
> > > hack are awful throwbacks.
> >
> > So far the systems we have, with DDR4 NvDIMM(s) (Actual chips arriving soon)
> > still have BIOS (On by default). The BIOS was yelled "Wolfe" for so long NOW ;-)
> > Again these are working system in the field, who will switch them all to UEFI?
> >
> > How will the UEFI present them to the system? can you point me to the relevant
> > code? (Or did you mean BIOS but with a different communication path than e820?)
> >
>
> Per my understand...
>
> With EFI Boot Stub, there have setup_e820() codes in arch/x86/boot/compressed/eboot.c
> that used to transfer EFI memmap to e820 entries. Currently doesn't have any
> EFI_MEMORY_TYPE reflects to NvDIMM that will map to e820_type.
>
> I wonder what kind of EFI_MEMORY_TYPE reported by UEFI BIOS on those "shipped
> NvDIMMs motherboards", like supermicro X9DRH-iF-NV. Then we may need add code
> to setup_e820() for mapping the efi memory type to e820 type 12 region.

Nothing useful.  You can't detect the NvDIMM in EFI mode on my board
as far as I know.

Rumor has it that we'll learn more about the EFI case very soon.

--Andy

>
> > >
> > > --Andy
> > >
> >
> > Thanks
> > Boaz
> >
>
> Thanks a lot!
> Joey Lee

Patch
diff mbox

diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c
index c2f2da2..4bf574a 100644
--- a/arch/x86/kernel/e820.c
+++ b/arch/x86/kernel/e820.c
@@ -25,6 +25,11 @@ 
 #include <asm/proto.h>
 #include <asm/setup.h>
 
+/* These are nonstandard Memory types that we do not want in the exported
+ * header.
+ */
+#define E820_UNKNOWN_12 12	/* This is the unofficial DDR3-NVDIMM */
+
 /*
  * The e820 map is the map that gets modified e.g. with command line parameters
  * and that is also registered with modifications in the kernel resource tree
@@ -169,6 +174,9 @@  static void __init e820_print_type(u32 type)
 	case E820_UNUSABLE:
 		printk(KERN_CONT "unusable");
 		break;
+	case E820_UNKNOWN_12:
+		printk(KERN_CONT "unknown-12");
+		break;
 	default:
 		printk(KERN_CONT "type %u", type);
 		break;
@@ -928,6 +936,7 @@  static inline const char *e820_type_to_string(int e820_type)
 	case E820_NVS:	return "ACPI Non-volatile Storage";
 	case E820_UNUSABLE:	return "Unusable memory";
 	case E820_RESERVED:	return "reserved";
+	case E820_UNKNOWN_12:	return "unknown-12";
 	default:		return "reserved-unknown";
 	}
 }