| Message ID | 20240611215544.2105970-4-jiaqiyan@google.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | Userspace controls soft-offline pages | expand |
On Tue, 11 Jun 2024, Jiaqi Yan wrote: > @@ -267,6 +268,20 @@ used:: > These are informational only. They do not mean that anything is wrong > with your system. To disable them, echo 4 (bit 2) into drop_caches. > > +enable_soft_offline > +=================== > +Control whether to soft offline memory pages that have (excessive) correctable > +memory errors. It is your call to choose between reliability (stay away from > +fragile physical memory) vs performance (brought by HugeTLB or transparent > +hugepages). > + Could you expand upon the relevance of HugeTLB or THP in this documentation? I understand the need in some cases to soft offline memory after a number of correctable memory errors, but it's not clear how the performance implications plays into this. The paragraph below goes into a difference in the splitting behavior, are hugepage users the only ones that should be concerned with this? > +When setting to 1, kernel attempts to soft offline the page when it thinks > +needed. For in-use page, page content will be migrated to a new page. If > +the oringinal hugepage is a HugeTLB hugepage, regardless of in-use or free, s/oringinal/original/ > +it will be dissolved into raw pages, and the capacity of the HugeTLB pool > +will reduce by 1. If the original hugepage is a transparent hugepage, it > +will be split into raw pages. When setting to 0, kernel won't attempt to > +soft offline the page. Its default value is 1. > This behavior is the same for all architectures?
Thanks for your questions, David! On Tue, Jun 11, 2024 at 5:25 PM David Rientjes <rientjes@google.com> wrote: > > On Tue, 11 Jun 2024, Jiaqi Yan wrote: > > > @@ -267,6 +268,20 @@ used:: > > These are informational only. They do not mean that anything is wrong > > with your system. To disable them, echo 4 (bit 2) into drop_caches. > > > > +enable_soft_offline > > +=================== > > +Control whether to soft offline memory pages that have (excessive) correctable > > +memory errors. It is your call to choose between reliability (stay away from > > +fragile physical memory) vs performance (brought by HugeTLB or transparent > > +hugepages). > > + > > Could you expand upon the relevance of HugeTLB or THP in this > documentation? I understand the need in some cases to soft offline memory > after a number of correctable memory errors, but it's not clear how the > performance implications plays into this. The paragraph below goes into a To be accurate, I should say soft offlining transparent hugepage impacts performance, and soft offlining hugetlb hugepage impacts capacity. It may be clearer to first explain soft-offline's behaviors and implications, so that user knows what is the cost of soft-offline, then talks about the behavior of enable_soft_offline: Correctable memory errors are very common on servers. Soft-offline is kernel's handling for memory pages having (excessive) corrected memory errors. For different types of page, soft-offline has different behaviors / costs. - For a raw error page, soft-offline migrates the in-use page's content to a new raw page. - For a page that is part of a transparent hugepage, soft-offline splits the transparent hugepage into raw pages, then migrates only the raw error page. As a result, user is transparently backed by 1 less hugepage, impacting memory access performance. - For a page that is part of a HugeTLB hugepage, soft-offline first migrates the entire HugeTLB hugepage, during which a free hugepage will be consumed as migration target. Then the original hugepage is dissolved into raw pages without compensation, reducing the capacity of the HugeTLB pool by 1. It is user's call to choose between reliability (staying away from fragile physical memory) vs performance / capacity implications in transparent and HugeTLB cases. > difference in the splitting behavior, are hugepage users the only ones > that should be concerned with this? If the cost of migrating a raw page is negligible, then yes, only hugepage users should be concerned and think about should they disable soft offline. > > > +When setting to 1, kernel attempts to soft offline the page when it thinks > > +needed. For in-use page, page content will be migrated to a new page. If > > +the oringinal hugepage is a HugeTLB hugepage, regardless of in-use or free, > > s/oringinal/original/ To fix in v3. > > > +it will be dissolved into raw pages, and the capacity of the HugeTLB pool > > +will reduce by 1. If the original hugepage is a transparent hugepage, it > > +will be split into raw pages. When setting to 0, kernel won't attempt to > > +soft offline the page. Its default value is 1. > > > > This behavior is the same for all architectures? > Yes, enable_soft_offline has the same behavior for all architectures, and default=1. It may be worth mentioning that setting enable_soft_offline to 0 means: - If RAS Correctable Errors Collector is running, its request to soft offline pages will be ignored. - On ARM, the request to soft offline pages from GHES driver will be ignored. - On PARISC, the request to soft offline pages from Page Deallocation Table will be ignored. I can add these clarifications in v3 if they are valuable.
diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index e86c968a7a0e..856bb17c07bc 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -36,6 +36,7 @@ Currently, these files are in /proc/sys/vm: - dirtytime_expire_seconds - dirty_writeback_centisecs - drop_caches +- enable_soft_offline - extfrag_threshold - highmem_is_dirtyable - hugetlb_shm_group @@ -267,6 +268,20 @@ used:: These are informational only. They do not mean that anything is wrong with your system. To disable them, echo 4 (bit 2) into drop_caches. +enable_soft_offline +=================== +Control whether to soft offline memory pages that have (excessive) correctable +memory errors. It is your call to choose between reliability (stay away from +fragile physical memory) vs performance (brought by HugeTLB or transparent +hugepages). + +When setting to 1, kernel attempts to soft offline the page when it thinks +needed. For in-use page, page content will be migrated to a new page. If +the oringinal hugepage is a HugeTLB hugepage, regardless of in-use or free, +it will be dissolved into raw pages, and the capacity of the HugeTLB pool +will reduce by 1. If the original hugepage is a transparent hugepage, it +will be split into raw pages. When setting to 0, kernel won't attempt to +soft offline the page. Its default value is 1. extfrag_threshold =================
Add the documentation for what enable_soft_offline sysctl is for. Signed-off-by: Jiaqi Yan <jiaqiyan@google.com> --- Documentation/admin-guide/sysctl/vm.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+)