diff mbox series

x86/mce: count the number of occurrences of each MCE severity

Message ID 20240618124331.12729-1-qirui.001@bytedance.com (mailing list archive)
State New
Headers show
Series x86/mce: count the number of occurrences of each MCE severity | expand

Commit Message

Rui Qi June 18, 2024, 12:43 p.m. UTC
From: Rui Qi <qirui.001@bytedance.com>

In the current implementation, we can only know whether each MCE
severity has occurred, and cannot know how many times it has occurred
accurately. This submission supports viewing how many times each MCE
severity has occurred.

Due to the limitation of char type, the maximum supported statistics are
currently 255 times

Signed-off-by: Rui Qi<qirui.001@bytedance.com>
---
 arch/x86/kernel/cpu/mce/severity.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

Tony Luck June 18, 2024, 5:35 p.m. UTC | #1
> In the current implementation, we can only know whether each MCE
> severity has occurred, and cannot know how many times it has occurred
> accurately. This submission supports viewing how many times each MCE
> severity has occurred.

Is know how many times each case was hit useful? The original commit
for this code said it was just to check coverage for the mce-test suite.

4611a6fa4b37 ("x86, mce: export MCE severities coverage via debugfs")

So you either covered a case in the severities table, or you didn't. Does it
help to know that you covered a case multiple times?

> Due to the limitation of char type, the maximum supported statistics are
> currently 255 times
>
> Signed-off-by: Rui Qi<qirui.001@bytedance.com>
> ---
>  arch/x86/kernel/cpu/mce/severity.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/x86/kernel/cpu/mce/severity.c b/arch/x86/kernel/cpu/mce/severity.c
> index dac4d64dfb2a..a81e34c6e3ee 100644
> --- a/arch/x86/kernel/cpu/mce/severity.c
> +++ b/arch/x86/kernel/cpu/mce/severity.c
> @@ -405,7 +405,7 @@ static noinstr int mce_severity_intel(struct mce *m, struct pt_regs *regs, char
>                       continue;
>               if (msg)
>                       *msg = s->msg;
> -             s->covered = 1;
> +             s->covered++;

Wraparound sets this back to zero. Should this be:

	if (s->covered < 255)
		s->covered++;

[Is there a #define for max value in an unsigned char? I could find one. If there is,
then use that instead of hard coded 255]

>
>               return s->sev;
>       }-- 

-Tony
Rui Qi June 20, 2024, 7:01 a.m. UTC | #2
On 6/19/24 上午1:35, Luck, Tony wrote:
> 
>> In the current implementation, we can only know whether each MCE
>> severity has occurred, and cannot know how many times it has occurred
>> accurately. This submission supports viewing how many times each MCE
>> severity has occurred.
> 
> Is know how many times each case was hit useful? The original commit
> for this code said it was just to check coverage for the mce-test suite.
> 
> 4611a6fa4b37 ("x86, mce: export MCE severities coverage via debugfs")
> 
> So you either covered a case in the severities table, or you didn't. Does it
> help to know that you covered a case multiple times?
> 

In the fault injection test in the laboratory, we inject errors multiple
times and need a counter to tell us how many times each case has
occurred and compare it with the expected number to determine the test
results

In the production environment, the counter can reflect the actual number
of times each MCE error type occurs, which can help us detect the MCE
error distribution of large-scale Data center infrastructure 

>> Due to the limitation of char type, the maximum supported statistics are
>> currently 255 times
>>

How about changing char to u64, which is enough for real-world
situations and won't waste a lot of memory? 

>> Signed-off-by: Rui Qi<qirui.001@bytedance.com>
>> ---
>>   arch/x86/kernel/cpu/mce/severity.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/arch/x86/kernel/cpu/mce/severity.c b/arch/x86/kernel/cpu/mce/severity.c
>> index dac4d64dfb2a..a81e34c6e3ee 100644
>> --- a/arch/x86/kernel/cpu/mce/severity.c
>> +++ b/arch/x86/kernel/cpu/mce/severity.c
>> @@ -405,7 +405,7 @@ static noinstr int mce_severity_intel(struct mce *m, struct pt_regs *regs, char
>>                        continue;
>>                if (msg)
>>                        *msg = s->msg;
>> -             s->covered = 1;
>> +             s->covered++;
> 
> Wraparound sets this back to zero. Should this be:
> 
> 	if (s->covered < 255)
> 		s->covered++;
> 
> [Is there a #define for max value in an unsigned char? I could find one. If there is,
> then use that instead of hard coded 255]
> 
>>
>>                return s->sev;
>>        }--
> 
> -Tony
Tony Luck June 20, 2024, 4:20 p.m. UTC | #3
You seem to have problems with the e-mail infrastructure. I got a few extra copies
of this in HTML format. This one is in plain text, but the From: header says "$(name)"


> > So you either covered a case in the severities table, or you didn't. Does it
> > help to know that you covered a case multiple times?
> >
>
> In the fault injection test in the laboratory, we inject errors multiple
> times and need a counter to tell us how many times each case has
> occurred and compare it with the expected number to determine the test
> results

In my testing on Intel/x86 I don't always see a 1:1 mapping between my
test, and the severities rule. This is because of a h/w race between the
memory controller reporting the error when it sees an uncorrectable ECC
issue, and the core trying to consume the poisoned data. If the memory
controller signal wins the race, Linux takes the page offline and there isn't
a poison consumption error, just a page fault.

> In the production environment, the counter can reflect the actual number
> of times each MCE error type occurs, which can help us detect the MCE
> error distribution of large-scale Data center infrastructure

That could be useful.

> >> Due to the limitation of char type, the maximum supported statistics are
> >> currently 255 times
> >>
>
> How about changing char to u64, which is enough for real-world
> situations and won't waste a lot of memory?

u64 seems like serious overkill. A change from "unsigned char" to "unsigned int"
would keep track of 4 billion errors. That seems like plenty :-)

-Tony
Rui Qi June 24, 2024, 4:48 a.m. UTC | #4
From: Rui Qi <qirui.001@bytedance.com>

Hi Tony,

> You seem to have problems with the e-mail infrastructure. I got a few extra copies
> of this in HTML format. This one is in plain text, but the From: header says "$(name)"
> 

Sorry, some problem with my thunderbird mail agent. I will use git send-mail instead from now on.
> 
>>> So you either covered a case in the severities table, or you didn't. Does it
>>> help to know that you covered a case multiple times?
>>>
>>
>> In the fault injection test in the laboratory, we inject errors multiple
>> times and need a counter to tell us how many times each case has
>> occurred and compare it with the expected number to determine the test
>> results
> 
> In my testing on Intel/x86 I don't always see a 1:1 mapping between my
> test, and the severities rule. This is because of a h/w race between the
> memory controller reporting the error when it sees an uncorrectable ECC
> issue, and the core trying to consume the poisoned data. If the memory
> controller signal wins the race, Linux takes the page offline and there isn't
> a poison consumption error, just a page fault.
> 
>> In the production environment, the counter can reflect the actual number
>> of times each MCE error type occurs, which can help us detect the MCE
>> error distribution of large-scale Data center infrastructure
> 
> That could be useful.
>
Thank you for your expertise!

>>>> Due to the limitation of char type, the maximum supported statistics are
>>>> currently 255 times
>>>>
>>
>> How about changing char to u64, which is enough for real-world
>> situations and won't waste a lot of memory?
> 
> u64 seems like serious overkill. A change from "unsigned char" to "unsigned int"
> would keep track of 4 billion errors. That seems like plenty :-)
>> -Tony
Yes, unsinged int is far enough.

BTW, if you dont mind, I will send a V2 patch based on our discussion.
diff mbox series

Patch

diff --git a/arch/x86/kernel/cpu/mce/severity.c b/arch/x86/kernel/cpu/mce/severity.c
index dac4d64dfb2a..a81e34c6e3ee 100644
--- a/arch/x86/kernel/cpu/mce/severity.c
+++ b/arch/x86/kernel/cpu/mce/severity.c
@@ -405,7 +405,7 @@  static noinstr int mce_severity_intel(struct mce *m, struct pt_regs *regs, char
 			continue;
 		if (msg)
 			*msg = s->msg;
-		s->covered = 1;
+		s->covered++;
 
 		return s->sev;
 	}