From patchwork Fri Aug 7 06:17:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704679 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 852A514B7 for ; Fri, 7 Aug 2020 06:17:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4F90122C9F for ; Fri, 7 Aug 2020 06:17:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="N6VO3pLQ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F90122C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8B6D16B0002; Fri, 7 Aug 2020 02:17:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 83FBF6B0003; Fri, 7 Aug 2020 02:17:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E0686B0005; Fri, 7 Aug 2020 02:17:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0228.hostedemail.com [216.40.44.228]) by kanga.kvack.org (Postfix) with ESMTP id 55A286B0002 for ; Fri, 7 Aug 2020 02:17:09 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 12BF22C79 for ; Fri, 7 Aug 2020 06:17:09 +0000 (UTC) X-FDA: 77122764978.05.toad96_440fce526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id E323018029583 for ; Fri, 7 Aug 2020 06:17:08 +0000 (UTC) X-Spam-Summary: 1,0,0,f19da07425842373,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1543:1711:1730:1747:1777:1792:2393:2525:2553:2559:2564:2682:2685:2693:2859:2895:2901:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3151:3355:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4605:5007:6117:6119:6120:6261:6653:6737:7576:7901:7903:8603:8660:9025:9391:9545:10004:10913:11026:11473:11658:11914:12043:12048:12295:12296:12297:12438:12517:12519:12555:12679:12783:12986:13053:13148:13161:13221:13229:13230:13255:14181:14721:14849:21060:21080:21324:21433:21451:21600:21627:21819:21939:21987:21990:30003:30012:30054:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfnpy5591w4kwwic7fnk7qcixxhocd513xb8jbn8jferb1wigd7pcqxma5u9q.4noqe9r7ra6ssdr63n133xjwmut7rj3u98a3gqy568saxjxtnwqokzdm o6xnqh9. X-HE-Tag: toad96_440fce526fbe X-Filterd-Recvd-Size: 5208 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:08 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E93372177B; Fri, 7 Aug 2020 06:17:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781027; bh=MCnaqNJ6WIuDxo8PWl+PeGkryXJcTchOMpSH4xm+irQ=; h=Date:From:To:Subject:In-Reply-To:From; b=N6VO3pLQCxzag37HewKG+OIc9wWHES5HRHVGCIadcEQ/DDsdZO3fZu2oYxDIZ14bu Gb4QFBf6XIcWyfVcO5TnUspvPCpqMdiHypwkToaopU0hKqZHUGDDWHUKeKYYfZQKel CVUsOXK1POowhG5o9WEUJfl3ffNB2bvuwbk+V+Lc= Date: Thu, 06 Aug 2020 23:17:06 -0700 From: Andrew Morton To: akpm@linux-foundation.org, catalin.marinas@arm.com, hannes@cmpxchg.org, hdanton@sina.com, hughd@google.com, josef@toxicpanda.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, stable@vger.kernel.org, torvalds@linux-foundation.org, will.deacon@arm.com, willy@infradead.org, xuyu@linux.alibaba.com, yang.shi@linux.alibaba.com Subject: [patch 001/163] mm/memory.c: avoid access flag update TLB flush for retried page fault Message-ID: <20200807061706.unk5_0KtC%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E323018029583 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yang Shi Subject: mm/memory.c: avoid access flag update TLB flush for retried page fault Recently we found regression when running will_it_scale/page_fault3 test on ARM64. Over 70% down for the multi processes cases and over 20% down for the multi threads cases. It turns out the regression is caused by commit 89b15332af7c0312a41e50846819ca6613b58b4c ("mm: drop mmap_sem before calling balance_dirty_pages() in write fault"). The test mmaps a memory size file then write to the mapping, this would make all memory dirty and trigger dirty pages throttle, that upstream commit would release mmap_sem then retry the page fault. The retried page fault would see correct PTEs installed by the first try then update dirty bit and clear read-only bit and flush TLBs for ARM. The regression is caused by the excessive TLB flush. It is fine on x86 since x86 doesn't clear read-only bit so there is no need to flush TLB for this case. The page fault would be retried due to: 1. Waiting for page readahead 2. Waiting for page swapped in 3. Waiting for dirty pages throttling The first two cases don't have PTEs set up at all, so the retried page fault would install the PTEs, so they don't reach there. But the #3 case usually has PTEs installed, the retried page fault would reach the dirty bit and read-only bit update. But it seems not necessary to modify those bits again for #3 since they should be already set by the first page fault try. Of course the parallel page fault may set up PTEs, but we just need care about write fault. If the parallel page fault setup a writable and dirty PTE then the retried fault doesn't need do anything extra. If the parallel page fault setup a clean read-only PTE, the retried fault should just call do_wp_page() then return as the below code snippet shows: if (vmf->flags & FAULT_FLAG_WRITE) { if (!pte_write(entry)) return do_wp_page(vmf); } With this fix the test result get back to normal. [yang.shi@linux.alibaba.com: incorporate comment from Will Deacon, update commit log per discussion] Link: http://lkml.kernel.org/r/1594848990-55657-1-git-send-email-yang.shi@linux.alibaba.com Link: http://lkml.kernel.org/r/1594148072-91273-1-git-send-email-yang.shi@linux.alibaba.com Signed-off-by: Yang Shi Reported-by: Xu Yu Debugged-by: Xu Yu Tested-by: Xu Yu Cc: Johannes Weiner Cc: Matthew Wilcox (Oracle) Cc: Kirill A. Shutemov Cc: Josef Bacik Cc: Hillf Danton Cc: Hugh Dickins Cc: Catalin Marinas Cc: Will Deacon Cc: Signed-off-by: Andrew Morton --- mm/memory.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) --- a/mm/memory.c~mm-avoid-access-flag-update-tlb-flush-for-retried-page-fault +++ a/mm/memory.c @@ -4241,8 +4241,14 @@ static vm_fault_t handle_pte_fault(struc if (vmf->flags & FAULT_FLAG_WRITE) { if (!pte_write(entry)) return do_wp_page(vmf); - entry = pte_mkdirty(entry); } + + if (vmf->flags & FAULT_FLAG_TRIED) + goto unlock; + + if (vmf->flags & FAULT_FLAG_WRITE) + entry = pte_mkdirty(entry); + entry = pte_mkyoung(entry); if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry, vmf->flags & FAULT_FLAG_WRITE)) { From patchwork Fri Aug 7 06:17:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704681 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 88F9513B6 for ; Fri, 7 Aug 2020 06:17:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 53FC522C9F for ; Fri, 7 Aug 2020 06:17:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Zc/68k2i" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 53FC522C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7CC3B6B0003; Fri, 7 Aug 2020 02:17:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 755A28D0002; Fri, 7 Aug 2020 02:17:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 643726B0007; Fri, 7 Aug 2020 02:17:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0231.hostedemail.com [216.40.44.231]) by kanga.kvack.org (Postfix) with ESMTP id 4F6586B0003 for ; Fri, 7 Aug 2020 02:17:13 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 43FF83627 for ; Fri, 7 Aug 2020 06:17:12 +0000 (UTC) X-FDA: 77122765104.05.bomb84_0a0442b26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 180111802912B for ; Fri, 7 Aug 2020 06:17:12 +0000 (UTC) X-Spam-Summary: 1,0,0,fc33925e62a1a765,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1543:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6261:6653:7576:8599:8784:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13221:13229:13846:13870:13972:14181:14721:14849:21080:21451:21627:21939:21972:30003:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8dhiwtw91ph88qs8h63iy7w46hocuig89qaui54eqd8tbsi6u7bac6mzswk4.f1w9ejktgnth3xqr9zujcsu576tp6xsafr5oos63x637tnb5q16ji6xy5gxeqdt.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Cus tom_rule X-HE-Tag: bomb84_0a0442b26fbe X-Filterd-Recvd-Size: 4445 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:11 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5AF4B22CAF; Fri, 7 Aug 2020 06:17:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781030; bh=5G2wDP9QWKyn+JHhF726/Cc2UeOzBeoGwYmuq4sUMGI=; h=Date:From:To:Subject:In-Reply-To:From; b=Zc/68k2ibppvB0quoP+wRl+bclFk0Vro/2RW02m9XLtdZS6IsgD9mPN0I2u0f7XZe dw4qz161GCnZa/2/zC3uTC8xZdOIgtQUm+wc3WTfU29VgUPaXtjrsKKECdGK2Iw79I 4SmVl4tRbVyxXSJutS8+XCAFs5ROdqKttjsRxKnM= Date: Thu, 06 Aug 2020 23:17:09 -0700 From: Andrew Morton To: akpm@linux-foundation.org, hch@lst.de, jgg@mellanox.com, jglisse@redhat.com, jhubbard@nvidia.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, rcampbell@nvidia.com, rdunlap@infradead.org, torvalds@linux-foundation.org Subject: [patch 002/163] mm/migrate: fix migrate_pgmap_owner w/o CONFIG_MMU_NOTIFIER Message-ID: <20200807061709.xu5tflYEQ%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 180111802912B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Ralph Campbell Subject: mm/migrate: fix migrate_pgmap_owner w/o CONFIG_MMU_NOTIFIER On x86_64, when CONFIG_MMU_NOTIFIER is not set/enabled, there is a compiler error: ../mm/migrate.c: In function 'migrate_vma_collect': ../mm/migrate.c:2481:7: error: 'struct mmu_notifier_range' has no member named 'migrate_pgmap_owner' range.migrate_pgmap_owner = migrate->pgmap_owner; ^ Link: http://lkml.kernel.org/r/20200806193353.7124-1-rcampbell@nvidia.com Fixes: 998427b3ad2c ("mm/notifier: add migration invalidation type") Signed-off-by: Ralph Campbell Reported-by: Randy Dunlap Acked-by: Randy Dunlap Tested-by: Randy Dunlap Cc: Jerome Glisse Cc: John Hubbard Cc: Christoph Hellwig Cc: "Jason Gunthorpe" Signed-off-by: Andrew Morton --- include/linux/mmu_notifier.h | 13 +++++++++++++ mm/migrate.c | 6 +++--- 2 files changed, 16 insertions(+), 3 deletions(-) --- a/include/linux/mmu_notifier.h~mm-migrate-fix-migrate_pgmap_owner-w-o-config_mmu_notifier +++ a/include/linux/mmu_notifier.h @@ -521,6 +521,16 @@ static inline void mmu_notifier_range_in range->flags = flags; } +static inline void mmu_notifier_range_init_migrate( + struct mmu_notifier_range *range, unsigned int flags, + struct vm_area_struct *vma, struct mm_struct *mm, + unsigned long start, unsigned long end, void *pgmap) +{ + mmu_notifier_range_init(range, MMU_NOTIFY_MIGRATE, flags, vma, mm, + start, end); + range->migrate_pgmap_owner = pgmap; +} + #define ptep_clear_flush_young_notify(__vma, __address, __ptep) \ ({ \ int __young; \ @@ -645,6 +655,9 @@ static inline void _mmu_notifier_range_i #define mmu_notifier_range_init(range,event,flags,vma,mm,start,end) \ _mmu_notifier_range_init(range, start, end) +#define mmu_notifier_range_init_migrate(range, flags, vma, mm, start, end, \ + pgmap) \ + _mmu_notifier_range_init(range, start, end) static inline bool mmu_notifier_range_blockable(const struct mmu_notifier_range *range) --- a/mm/migrate.c~mm-migrate-fix-migrate_pgmap_owner-w-o-config_mmu_notifier +++ a/mm/migrate.c @@ -2386,9 +2386,9 @@ static void migrate_vma_collect(struct m * that the registered device driver can skip invalidating device * private page mappings that won't be migrated. */ - mmu_notifier_range_init(&range, MMU_NOTIFY_MIGRATE, 0, migrate->vma, - migrate->vma->vm_mm, migrate->start, migrate->end); - range.migrate_pgmap_owner = migrate->pgmap_owner; + mmu_notifier_range_init_migrate(&range, 0, migrate->vma, + migrate->vma->vm_mm, migrate->start, migrate->end, + migrate->pgmap_owner); mmu_notifier_invalidate_range_start(&range); walk_page_range(migrate->vma->vm_mm, migrate->start, migrate->end, From patchwork Fri Aug 7 06:17:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704683 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 55E6914B7 for ; Fri, 7 Aug 2020 06:17:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1FEBD22CAF for ; Fri, 7 Aug 2020 06:17:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="W5XaOy7e" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1FEBD22CAF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2E9248D0003; Fri, 7 Aug 2020 02:17:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 273718D0002; Fri, 7 Aug 2020 02:17:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13BF08D0003; Fri, 7 Aug 2020 02:17:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0107.hostedemail.com [216.40.44.107]) by kanga.kvack.org (Postfix) with ESMTP id EE0F08D0002 for ; Fri, 7 Aug 2020 02:17:16 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E3C1C181AEF10 for ; Fri, 7 Aug 2020 06:17:15 +0000 (UTC) X-FDA: 77122765230.12.pie58_1603ee526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id AE33C1801116C for ; Fri, 7 Aug 2020 06:17:15 +0000 (UTC) X-Spam-Summary: 1,0,0,3b67978bbd29b80b,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1543:1711:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2525:2559:2563:2682:2685:2693:2731:2741:2859:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6117:6119:6261:6653:6737:7514:7576:7903:8599:8957:9010:9025:9545:10004:10913:11026:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13255:13846:14093:14096:14181:14721:14754:14849:21063:21080:21094:21323:21451:21627:21939:21990:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yggp7zzrxc53quxp9hdiustijgpypt67euzhpnrk1jo3b9ntor1zkfzcsjxyk.34e63x5ij5qnpjicjjn9d691zdjrz5ys6jsh4iocpn3a6em1wg7ot3d1mt9nntn.a-lbl8.mailshell.net-223.23 8.255.10 X-HE-Tag: pie58_1603ee526fbe X-Filterd-Recvd-Size: 5145 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:15 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B2BE8221E5; Fri, 7 Aug 2020 06:17:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781034; bh=OVWvsSaIpqu3QJpZgPxOm+EG3djcXSl2jWK7tt7skCc=; h=Date:From:To:Subject:In-Reply-To:From; b=W5XaOy7e8UMN+AXcaZp4uK7XILFl6Hz+VBo+fOn7tNQlNuOCQ6VU1o1S6uyTXVQeS wqOj/3im35gr42zzC4RZE1lDJXCdxlMSwZu8lXBQCPfrQ42RxLkGQfTqQs3evQhDuR cPYrVRd+TWFBxUXlZeyfBeeR/dBuxcllYB5fSmMo= Date: Thu, 06 Aug 2020 23:17:13 -0700 From: Andrew Morton To: akpm@linux-foundation.org, dan.j.williams@intel.com, david@redhat.com, hannes@cmpxchg.org, linux-mm@kvack.org, mgorman@techsingularity.net, mhocko@suse.com, minchan@kernel.org, mm-commits@vger.kernel.org, richard.weiyang@gmail.com, richard.weiyang@linux.alibaba.com, stable@vger.kernel.org, torvalds@linux-foundation.org, ying.huang@intel.com Subject: [patch 003/163] mm/shuffle: don't move pages between zones and don't read garbage memmaps Message-ID: <20200807061713.iMOMr3Cb5%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: AE33C1801116C X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: David Hildenbrand Subject: mm/shuffle: don't move pages between zones and don't read garbage memmaps Especially with memory hotplug, we can have offline sections (with a garbage memmap) and overlapping zones. We have to make sure to only touch initialized memmaps (online sections managed by the buddy) and that the zone matches, to not move pages between zones. To test if this can actually happen, I added a simple BUG_ON(page_zone(page_i) != page_zone(page_j)); right before the swap. When hotplugging a 256M DIMM to a 4G x86-64 VM and onlining the first memory block "online_movable" and the second memory block "online_kernel", it will trigger the BUG, as both zones (NORMAL and MOVABLE) overlap. This might result in all kinds of weird situations (e.g., double allocations, list corruptions, unmovable allocations ending up in the movable zone). Link: http://lkml.kernel.org/r/20200624094741.9918-2-david@redhat.com Fixes: e900a918b098 ("mm: shuffle initial free memory to improve memory-side-cache utilization") Signed-off-by: David Hildenbrand Reviewed-by: Wei Yang Acked-by: Michal Hocko Acked-by: Dan Williams Cc: Andrew Morton Cc: Johannes Weiner Cc: Michal Hocko Cc: Minchan Kim Cc: Huang Ying Cc: Wei Yang Cc: Mel Gorman Cc: [5.2+] Signed-off-by: Andrew Morton --- mm/shuffle.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) --- a/mm/shuffle.c~mm-shuffle-dont-move-pages-between-zones-and-dont-read-garbage-memmaps +++ a/mm/shuffle.c @@ -58,25 +58,25 @@ module_param_call(shuffle, shuffle_store * For two pages to be swapped in the shuffle, they must be free (on a * 'free_area' lru), have the same order, and have the same migratetype. */ -static struct page * __meminit shuffle_valid_page(unsigned long pfn, int order) +static struct page * __meminit shuffle_valid_page(struct zone *zone, + unsigned long pfn, int order) { - struct page *page; + struct page *page = pfn_to_online_page(pfn); /* * Given we're dealing with randomly selected pfns in a zone we * need to ask questions like... */ - /* ...is the pfn even in the memmap? */ - if (!pfn_valid_within(pfn)) + /* ... is the page managed by the buddy? */ + if (!page) return NULL; - /* ...is the pfn in a present section or a hole? */ - if (!pfn_in_present_section(pfn)) + /* ... is the page assigned to the same zone? */ + if (page_zone(page) != zone) return NULL; /* ...is the page free and currently on a free_area list? */ - page = pfn_to_page(pfn); if (!PageBuddy(page)) return NULL; @@ -123,7 +123,7 @@ void __meminit __shuffle_zone(struct zon * page_j randomly selected in the span @zone_start_pfn to * @spanned_pages. */ - page_i = shuffle_valid_page(i, order); + page_i = shuffle_valid_page(z, i, order); if (!page_i) continue; @@ -137,7 +137,7 @@ void __meminit __shuffle_zone(struct zon j = z->zone_start_pfn + ALIGN_DOWN(get_random_long() % z->spanned_pages, order_pages); - page_j = shuffle_valid_page(j, order); + page_j = shuffle_valid_page(z, j, order); if (page_j && page_j != page_i) break; } From patchwork Fri Aug 7 06:17:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704685 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EEEC813B6 for ; Fri, 7 Aug 2020 06:17:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C3BBB221E5 for ; Fri, 7 Aug 2020 06:17:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="LiR/rG0Y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C3BBB221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 890208D0005; Fri, 7 Aug 2020 02:17:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 844DD8D0002; Fri, 7 Aug 2020 02:17:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70B9D8D0005; Fri, 7 Aug 2020 02:17:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id 573578D0002 for ; Fri, 7 Aug 2020 02:17:19 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 1B2093644 for ; Fri, 7 Aug 2020 06:17:19 +0000 (UTC) X-FDA: 77122765398.30.clam03_461239f26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id E8E6C180B3AA7 for ; Fri, 7 Aug 2020 06:17:18 +0000 (UTC) X-Spam-Summary: 1,0,0,c2e41a19636e4b9b,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1712:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3868:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4605:5007:6117:6119:6120:6261:6653:6737:7514:7576:7875:7901:7903:8599:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13221:13229:14096:14181:14721:14849:21063:21080:21324:21451:21627:21939:21966:30054:30055,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8zjoqe5ey751or3niocawegrg1ycw5z5qw6s76kfquonfn8bym386wi49bi6.kijyj8jmtsrw96rk4c6yfwpp9oi933f9pw8n7chbitn5d6ba77q3u8zaksh567d.k-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,Doma inCache: X-HE-Tag: clam03_461239f26fbe X-Filterd-Recvd-Size: 4456 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:18 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2A1E72177B; Fri, 7 Aug 2020 06:17:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781037; bh=Qt1PcoJNJQycui0r/yXB5pNj6s8z3BlaXbgYR0ObX4g=; h=Date:From:To:Subject:In-Reply-To:From; b=LiR/rG0YPlk2vr+rAOhGWbsIYfsp/d+QouqBGOTjN/0bikSPNXa4wtLmX4p484Ti8 02Gdk/S3ZZBOLON/Ef5FKAndMjSca3rXkiKWEGuyiuz5y+XofpWQFoeN+LOMWP1lu5 U8REiVn21Xc0uldeDIg1O7HJDGKSq8ce8LA5OHG4= Date: Thu, 06 Aug 2020 23:17:16 -0700 From: Andrew Morton To: akpm@linux-foundation.org, axboe@kernel.dk, hch@lst.de, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, luto@amacapital.net, mathieu.desnoyers@efficios.com, mm-commits@vger.kernel.org, npiggin@gmail.com, peterz@infradead.org, stable@vger.kernel.org, torvalds@linux-foundation.org, will@kernel.org Subject: [patch 004/163] mm: fix kthread_use_mm() vs TLB invalidate Message-ID: <20200807061716.0Q5HJvMJ8%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E8E6C180B3AA7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Peter Zijlstra Subject: mm: fix kthread_use_mm() vs TLB invalidate For SMP systems using IPI based TLB invalidation, looking at current->active_mm is entirely reasonable. This then presents the following race condition: CPU0 CPU1 flush_tlb_mm(mm) use_mm(mm) tsk->active_mm = mm; if (tsk->active_mm == mm) // flush TLBs switch_mm(old_mm,mm,tsk); Where it is possible the IPI flushed the TLBs for @old_mm, not @mm, because the IPI lands before we actually switched. Avoid this by disabling IRQs across changing ->active_mm and switch_mm(). Of the (SMP) architectures that have IPI based TLB invalidate: Alpha - checks active_mm ARC - ASID specific IA64 - checks active_mm MIPS - ASID specific flush OpenRISC - shoots down world PARISC - shoots down world SH - ASID specific SPARC - ASID specific x86 - N/A xtensa - checks active_mm So at the very least Alpha, IA64 and Xtensa are suspect. On top of this, for scheduler consistency we need at least preemption disabled across changing tsk->mm and doing switch_mm(), which is currently provided by task_lock(), but that's not sufficient for PREEMPT_RT. [akpm@linux-foundation.org: add comment] Link: http://lkml.kernel.org/r/20200721154106.GE10769@hirez.programming.kicks-ass.net Signed-off-by: Peter Zijlstra (Intel) Reported-by: Andy Lutomirski Cc: Nicholas Piggin Cc: Jens Axboe Cc: Kees Cook Cc: Jann Horn Cc: Will Deacon Cc: Christoph Hellwig Cc: Nicholas Piggin Cc: Mathieu Desnoyers Cc: Signed-off-by: Andrew Morton --- kernel/kthread.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) --- a/kernel/kthread.c~mm-fix-kthread_use_mm-vs-tlb-invalidate +++ a/kernel/kthread.c @@ -1241,13 +1241,16 @@ void kthread_use_mm(struct mm_struct *mm WARN_ON_ONCE(tsk->mm); task_lock(tsk); + /* Hold off tlb flush IPIs while switching mm's */ + local_irq_disable(); active_mm = tsk->active_mm; if (active_mm != mm) { mmgrab(mm); tsk->active_mm = mm; } tsk->mm = mm; - switch_mm(active_mm, mm, tsk); + switch_mm_irqs_off(active_mm, mm, tsk); + local_irq_enable(); task_unlock(tsk); #ifdef finish_arch_post_lock_switch finish_arch_post_lock_switch(); @@ -1276,9 +1279,11 @@ void kthread_unuse_mm(struct mm_struct * task_lock(tsk); sync_mm_rss(mm); + local_irq_disable(); tsk->mm = NULL; /* active_mm is still 'mm' */ enter_lazy_tlb(mm, tsk); + local_irq_enable(); task_unlock(tsk); } EXPORT_SYMBOL_GPL(kthread_unuse_mm); From patchwork Fri Aug 7 06:17:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704687 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB71A14B7 for ; Fri, 7 Aug 2020 06:17:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C09E02177B for ; Fri, 7 Aug 2020 06:17:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="CFFCKCYH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C09E02177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 55A9D8D0006; Fri, 7 Aug 2020 02:17:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 50C3D8D0002; Fri, 7 Aug 2020 02:17:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D4418D0006; Fri, 7 Aug 2020 02:17:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0217.hostedemail.com [216.40.44.217]) by kanga.kvack.org (Postfix) with ESMTP id 225D28D0002 for ; Fri, 7 Aug 2020 02:17:22 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CBF3A3627 for ; Fri, 7 Aug 2020 06:17:21 +0000 (UTC) X-FDA: 77122765482.04.sleet94_600a1b926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id A13EA800ABAA for ; Fri, 7 Aug 2020 06:17:21 +0000 (UTC) X-Spam-Summary: 1,0,0,534162d81ae46075,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:968:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3867:3868:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7514:7576:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13069:13161:13229:13311:13357:13846:14181:14384:14721:14849:21080:21451:21627:21939:30012:30029:30030:30034:30054,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygydgkbqgdxsxiq81ygxy5i6hquycbdufmz3kbj3ec3wiy18mdzfc43c9xpau.6i8wwg4q6p6mznf3o4r11fe1tcqpc6e7tgarnpyx8ebacb9hgzznycd7dnwnq6c.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache: 0,MSF:no X-HE-Tag: sleet94_600a1b926fbe X-Filterd-Recvd-Size: 2630 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:21 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5EFAF22C9F; Fri, 7 Aug 2020 06:17:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781040; bh=2BFBHw3XZvwXwNRkBzrcop/NSxWNyJBMp5dQKM4eftE=; h=Date:From:To:Subject:In-Reply-To:From; b=CFFCKCYHzzen0Wiyu6XcPm1JKb8s8bZblQCcMcfnGTtnXuwNSidvQCgaW8MeM3HQ7 f0cWwr4p0anyuGM9mvO+D3+p73nQxIm16Twv2OJygn/U35ndQBwtc8053nyxwX28By O5szumLg89lGiwTE3JPEPvhLiOqR7wSTCK2al11k= Date: Thu, 06 Aug 2020 23:17:19 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, pmladek@suse.com, stamatis.iliass@gmail.com, torvalds@linux-foundation.org Subject: [patch 005/163] kthread: remove incorrect comment in kthread_create_on_cpu() Message-ID: <20200807061719.Bc1q_uR0m%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: A13EA800ABAA X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Ilias Stamatis Subject: kthread: remove incorrect comment in kthread_create_on_cpu() Originally kthread_create_on_cpu() parked and woke up the new thread. However, since commit a65d40961dc7 ("kthread/smpboot: do not park in kthread_create_on_cpu()") this is no longer the case. This patch removes the comment that has been left behind and is now incorrect / stale. Link: http://lkml.kernel.org/r/20200611135920.240551-1-stamatis.iliass@gmail.com Fixes: a65d40961dc7 ("kthread/smpboot: do not park in kthread_create_on_cpu()") Signed-off-by: Ilias Stamatis Reviewed-by: Petr Mladek Signed-off-by: Andrew Morton --- kernel/kthread.c | 1 - 1 file changed, 1 deletion(-) --- a/kernel/kthread.c~kthread-remove-incorrect-comment-in-kthread_create_on_cpu +++ a/kernel/kthread.c @@ -480,7 +480,6 @@ EXPORT_SYMBOL(kthread_bind); * to "name.*%u". Code fills in cpu number. * * Description: This helper function creates and names a kernel thread - * The thread will be woken and put into park mode. */ struct task_struct *kthread_create_on_cpu(int (*threadfn)(void *data), void *data, unsigned int cpu, From patchwork Fri Aug 7 06:17:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704689 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C08E814B7 for ; Fri, 7 Aug 2020 06:17:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9592F2177B for ; Fri, 7 Aug 2020 06:17:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="m1MRVAM3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9592F2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6E0ED8D0007; Fri, 7 Aug 2020 02:17:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 66B868D0002; Fri, 7 Aug 2020 02:17:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 538438D0007; Fri, 7 Aug 2020 02:17:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0152.hostedemail.com [216.40.44.152]) by kanga.kvack.org (Postfix) with ESMTP id 3BA9F8D0002 for ; Fri, 7 Aug 2020 02:17:25 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id ED63C180AD801 for ; Fri, 7 Aug 2020 06:17:24 +0000 (UTC) X-FDA: 77122765608.17.magic73_1302e7a26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id C15F9180D0180 for ; Fri, 7 Aug 2020 06:17:24 +0000 (UTC) X-Spam-Summary: 50,0,0,a253e9191c113c41,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:979:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1543:1711:1730:1747:1777:1792:1801:1981:2194:2199:2393:2525:2538:2568:2632:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4383:4605:5007:6117:6261:6653:7576:7875:7901:8599:8829:9025:9388:9545:10004:10049:10913:11026:11257:11473:11658:11914:12043:12048:12295:12296:12297:12517:12519:12555:12679:12776:12783:12986:13437:13846:14181:14721:14849:21080:21450:21451:21627:21781:21939:30054:30056:30067:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrcgcm3zmeaw8ibq8tiz564zyuxocjhh3igombbsiurncbm6ogdke54mnhuw8.ynrodujddqx5xrnu5yat1wpx5omzy3xx9w79k9hndhpamgymnde6mjr5a6itw84.h-lbl8.mailshell.net-223.238.255.100,CacheIP:n one,Baye X-HE-Tag: magic73_1302e7a26fbe X-Filterd-Recvd-Size: 4335 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:24 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6481C221E5; Fri, 7 Aug 2020 06:17:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781043; bh=iOQku6WUzYr21BCqT4QFwL8gAQwzMMrfzFVs36xdQbc=; h=Date:From:To:Subject:In-Reply-To:From; b=m1MRVAM35M3BXouHY6LXeWcoJn4Sgx51e8pDwd+sPQtnhVMYwF8+8y9nnarcpjyTb ZfN/iPsFdR09vf5W7chUjS4mpS5pklSo+0nroGn9LECgDjFrpC2VP9ToTT4cAS+uOY Ii9XPynue982rxp1gWzydffITR6eeSLHy3XFkaD0= Date: Thu, 06 Aug 2020 23:17:22 -0700 From: Andrew Morton To: akpm@linux-foundation.org, grandmaster@al2klimov.de, linux-mm@kvack.org, mm-commits@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 006/163] tools/: replace HTTP links with HTTPS ones Message-ID: <20200807061722.Z9iuNCxnG%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: C15F9180D0180 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Alexander A. Klimov" Subject: tools/: replace HTTP links with HTTPS ones Rationale: Reduces attack surface on kernel devs opening the links for MITM as HTTPS traffic is much harder to manipulate. Link: http://lkml.kernel.org/r/20200726120752.16768-1-grandmaster@al2klimov.de Signed-off-by: Alexander A. Klimov Signed-off-by: Andrew Morton --- tools/include/linux/jhash.h | 2 +- tools/lib/rbtree.c | 2 +- tools/lib/traceevent/event-parse.h | 2 +- tools/testing/ktest/examples/README | 2 +- tools/testing/ktest/examples/crosstests.conf | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) --- a/tools/include/linux/jhash.h~tools-replace-http-links-with-https-ones +++ a/tools/include/linux/jhash.h @@ -5,7 +5,7 @@ * * Copyright (C) 2006. Bob Jenkins (bob_jenkins@burtleburtle.net) * - * http://burtleburtle.net/bob/hash/ + * https://burtleburtle.net/bob/hash/ * * These are the credits from Bob's sources: * --- a/tools/lib/rbtree.c~tools-replace-http-links-with-https-ones +++ a/tools/lib/rbtree.c @@ -13,7 +13,7 @@ #include /* - * red-black trees properties: http://en.wikipedia.org/wiki/Rbtree + * red-black trees properties: https://en.wikipedia.org/wiki/Rbtree * * 1) A node is either red or black * 2) The root is black --- a/tools/lib/traceevent/event-parse.h~tools-replace-http-links-with-https-ones +++ a/tools/lib/traceevent/event-parse.h @@ -379,7 +379,7 @@ enum tep_errno { * errno since SUS requires the errno has distinct positive values. * See 'Issue 6' in the link below. * - * http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html + * https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html */ __TEP_ERRNO__START = -100000, --- a/tools/testing/ktest/examples/crosstests.conf~tools-replace-http-links-with-https-ones +++ a/tools/testing/ktest/examples/crosstests.conf @@ -3,7 +3,7 @@ # # In this config, it is expected that the tool chains from: # -# http://kernel.org/pub/tools/crosstool/files/bin/x86_64/ +# https://kernel.org/pub/tools/crosstool/files/bin/x86_64/ # # running on a x86_64 system have been downloaded and installed into: # --- a/tools/testing/ktest/examples/README~tools-replace-http-links-with-https-ones +++ a/tools/testing/ktest/examples/README @@ -11,7 +11,7 @@ crosstests.conf - this config shows an e lots of different architectures. It only does build tests, but makes it easy to compile test different archs. You can download the arch cross compilers from: - http://kernel.org/pub/tools/crosstool/files/bin/x86_64/ + https://kernel.org/pub/tools/crosstool/files/bin/x86_64/ test.conf - A generic example of a config. This is based on an actual config used to perform real testing. From patchwork Fri Aug 7 06:17:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704691 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B119314B7 for ; Fri, 7 Aug 2020 06:17:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 850CE221E5 for ; Fri, 7 Aug 2020 06:17:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="mdSCBu1G" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 850CE221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 55C4A8D0008; Fri, 7 Aug 2020 02:17:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4BBE48D0002; Fri, 7 Aug 2020 02:17:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AC678D0008; Fri, 7 Aug 2020 02:17:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0220.hostedemail.com [216.40.44.220]) by kanga.kvack.org (Postfix) with ESMTP id 25AD18D0002 for ; Fri, 7 Aug 2020 02:17:29 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 1DAEE3641 for ; Fri, 7 Aug 2020 06:17:28 +0000 (UTC) X-FDA: 77122765776.01.bee07_0808ea326fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id E83611005147B for ; Fri, 7 Aug 2020 06:17:27 +0000 (UTC) X-Spam-Summary: 1,0,0,69ff6c7a4f06af08,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:5007:6261:6653:6737:7514:7576:8599:8603:9025:9545:10004:10913:11026:11658:11914:12043:12048:12297:12517:12519:12555:12679:12783:12986:13069:13255:13311:13357:13846:14181:14384:14721:14849:21080:21451:21627:21939:21990:30012:30054:30056:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8c916rnqpwg7btrys4bzihu7syopt1ii7co3q53qmextde1mrqt5ybht41m7.3hu5zyxjrdwcmyw934yw7u4q6o31z7ksu61zchgu1ro6rq3fcbmkpgh9ywygo9p.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Cust om_rules X-HE-Tag: bee07_0808ea326fbe X-Filterd-Recvd-Size: 2663 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:27 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5CFFC22C9F; Fri, 7 Aug 2020 06:17:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781046; bh=xzzxhU/8SiRsBrTPD9nwPf1PmQq1mdHEor5/lZyDKIs=; h=Date:From:To:Subject:In-Reply-To:From; b=mdSCBu1GrFNJGlkgjqRrkwUYsbUbI70jxufGxBdzhM5OCTQHrdHD1V0aNkPrqrLmV XCJml3xkGDstDFAX9Frg6ak1nWJB2tEToOGPTFh2tOdJvNBRxcCsBVotWalTwVi6ny Z2lIx2lRqfd6KD6YUcaTEcXaodTX+ZgFICbJXk+U= Date: Thu, 06 Aug 2020 23:17:25 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, christian.brauner@ubuntu.com, gaurav1086@gmail.com, guro@fb.com, linux-mm@kvack.org, mkoutny@suse.com, mm-commits@vger.kernel.org, shuah@kernel.org, tj@kernel.org, torvalds@linux-foundation.org Subject: [patch 007/163] tools/testing/selftests/cgroup/cgroup_util.c: cg_read_strcmp: fix null pointer dereference Message-ID: <20200807061725.WdE4MoFqu%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E83611005147B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Gaurav Singh Subject: tools/testing/selftests/cgroup/cgroup_util.c: cg_read_strcmp: fix null pointer dereference Haven't reproduced this issue. This PR is does a minor code cleanup. Link: http://lkml.kernel.org/r/20200726013808.22242-1-gaurav1086@gmail.com Signed-off-by: Gaurav Singh Reviewed-by: Andrew Morton Cc: Shuah Khan Cc: Tejun Heo Cc: Michal Koutn Cc: Roman Gushchin Cc: Christian Brauner Cc: Chris Down Signed-off-by: Andrew Morton --- tools/testing/selftests/cgroup/cgroup_util.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/tools/testing/selftests/cgroup/cgroup_util.c~cg_read_strcmp-fix-null-pointer-dereference +++ a/tools/testing/selftests/cgroup/cgroup_util.c @@ -106,7 +106,7 @@ int cg_read_strcmp(const char *cgroup, c /* Handle the case of comparing against empty string */ if (!expected) - size = 32; + return -1; else size = strlen(expected) + 1; From patchwork Fri Aug 7 06:17:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704693 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7767F14B7 for ; Fri, 7 Aug 2020 06:17:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 42B1B22C9F for ; Fri, 7 Aug 2020 06:17:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="siHRymgq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 42B1B22C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DE7D18D0009; Fri, 7 Aug 2020 02:17:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D6F058D0002; Fri, 7 Aug 2020 02:17:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C170A8D0009; Fri, 7 Aug 2020 02:17:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0174.hostedemail.com [216.40.44.174]) by kanga.kvack.org (Postfix) with ESMTP id A735E8D0002 for ; Fri, 7 Aug 2020 02:17:31 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 65FA3181AEF10 for ; Fri, 7 Aug 2020 06:17:31 +0000 (UTC) X-FDA: 77122765902.18.coal34_22113cb26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 41F8F100EC676 for ; Fri, 7 Aug 2020 06:17:31 +0000 (UTC) X-Spam-Summary: 1,0,0,4d7828e3123f1de9,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2197:2199:2393:2525:2559:2564:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3653:3867:3870:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:5007:6119:6261:6653:6691:7576:7903:8599:9025:9545:10004:11026:11257:11473:11658:11914:12043:12048:12297:12438:12517:12519:12679:12783:13069:13311:13357:14181:14384:14721:14849:21080:21221:21324:21451:21622:21939:30029:30054,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yft3dtzkc6fgs4x76tstjif6ghmopbw66acad6sy6b4tijura3ypujcnzbsyw.s6k345if9dgmhuzpzcgpcaypi1jmbea55xoukcub7f71dhia1gcppkb6ewjsmsk.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rul es:0:0:0 X-HE-Tag: coal34_22113cb26fbe X-Filterd-Recvd-Size: 2987 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:30 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 914772177B; Fri, 7 Aug 2020 06:17:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781050; bh=0J8e97K1uRmENcQYlqcoiBYrSlHuFNGoRr9491BvVWU=; h=Date:From:To:Subject:In-Reply-To:From; b=siHRymgqrQI3/p432q2Wba/dxbv7a/wUhFwpAnwMvW3zqYZcAClxLtdU469p6ZqOj NvUbfOnpuME8/H/fumfWAdmQv5JBZZJosgXuyqRsNycRYIxyzJKou6YziEov9c9KSc Hs6jHzmMwHZ7mbOQjlFPzRotGxHMuJ9m9JR4fYec= Date: Thu, 06 Aug 2020 23:17:29 -0700 From: Andrew Morton To: akpm@linux-foundation.org, corbet@lwn.net, gregkh@linuxfoundation.org, joe@perches.com, linux-mm@kvack.org, masahiroy@kernel.org, mchehab+huawei@kernel.org, mm-commits@vger.kernel.org, torvalds@linux-foundation.org, xujialu@vimux.org Subject: [patch 008/163] scripts/tags.sh: collect compiled source precisely Message-ID: <20200807061729.OFb0Kw9sM%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 41F8F100EC676 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jialu Xu Subject: scripts/tags.sh: collect compiled source precisely Parse compiled source from *.cmd but don't 'find' too many files that are not related to compilation. [xujialu@vimux.org: don't expand symlinks by add option -s for realpath] Link: http://lkml.kernel.org/r/5efc5bfb.1c69fb81.41bf5.7131SMTPIN_ADDED_MISSING@mx.google.com Link: http://lkml.kernel.org/r/5ee5d8e3.1c69fb81.9b804.47b2SMTPIN_ADDED_MISSING@mx.google.com Signed-off-by: Jialu Xu Cc: Greg Kroah-Hartman Cc: Masahiro Yamada Cc: Mauro Carvalho Chehab Cc: Jonathan Corbet Cc: Joe Perches Signed-off-by: Andrew Morton --- scripts/tags.sh | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) --- a/scripts/tags.sh~scripts-tagssh-collect-compiled-source-precisely +++ a/scripts/tags.sh @@ -91,20 +91,10 @@ all_sources() all_compiled_sources() { - for i in $(all_sources); do - case "$i" in - *.[cS]) - j=${i/\.[cS]/\.o} - j="${j#$tree}" - if [ -e $j ]; then - echo $i - fi - ;; - *) - echo $i - ;; - esac - done + realpath -es $([ -z "$KBUILD_ABS_SRCTREE" ] && echo --relative-to=.) \ + include/generated/autoconf.h $(find -name "*.cmd" -exec \ + grep -Poh '(?(?=^source_.* \K).*|(?=^ \K\S).*(?= \\))' {} \+ | + awk '!a[$0]++') | sort -u } all_target_sources() From patchwork Fri Aug 7 06:17:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704695 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 19EEE13B6 for ; Fri, 7 Aug 2020 06:17:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E2F74221E5 for ; Fri, 7 Aug 2020 06:17:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="IZXoWJ+b" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E2F74221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7A9E88D000A; Fri, 7 Aug 2020 02:17:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 708CD8D0002; Fri, 7 Aug 2020 02:17:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5AA968D000A; Fri, 7 Aug 2020 02:17:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0199.hostedemail.com [216.40.44.199]) by kanga.kvack.org (Postfix) with ESMTP id 441688D0002 for ; Fri, 7 Aug 2020 02:17:34 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0D41C82499B9 for ; Fri, 7 Aug 2020 06:17:34 +0000 (UTC) X-FDA: 77122766028.14.book16_4901abf26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id D6EB218229818 for ; Fri, 7 Aug 2020 06:17:33 +0000 (UTC) X-Spam-Summary: 1,0,0,dd45d0c4361c72a1,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:305:334:355:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3167:3352:3866:3870:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6119:6261:6653:7576:7903:7904:8599:8603:9025:9121:9545:10004:10903:10913:11658:11914:12043:12048:12296:12297:12517:12519:12555:12679:12783:12986:13069:13311:13357:13846:14094:14181:14384:14721:14849:21080:21451:21627:21939:30029:30054:30062:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrimx6rdnnunh8t77syyhm3tm8uocfoyfgw6n1kk17war1mietpx9y6h8oib7.tfnxhr4zu7bzi4moupieh3b11y7xho8u3m3msjtx4ie1oxakt1e7tc4mg4wsu6s.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:f p,MSBL:0 X-HE-Tag: book16_4901abf26fbe X-Filterd-Recvd-Size: 2586 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:33 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AA498221E5; Fri, 7 Aug 2020 06:17:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781052; bh=c7aI5CAH4MJlD2dGZs2NQx1mLlEILlnOB/MkFg9trag=; h=Date:From:To:Subject:In-Reply-To:From; b=IZXoWJ+bpAJ7DPs3laGkU7B1mjJ087S2/0G33JvN1X6QkIKiQcA3sogyD3orzkJV3 q6NTod296MTWV3L0BONW/U4zzBK1Vei2Ar73OYmL9l4yFOqa3PUlywb08KZYVvTQmu Vpv5VdkE0yZPLxKSI79//OOZzEAKFbHVAm9d9EJs= Date: Thu, 06 Aug 2020 23:17:32 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, nborisov@suse.com, torvalds@linux-foundation.org Subject: [patch 009/163] scripts/bloat-o-meter: Support comparing library archives Message-ID: <20200807061732.dXHNntoww%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: D6EB218229818 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nikolay Borisov Subject: scripts/bloat-o-meter: Support comparing library archives Library archives (.a) usually contain multiple object files so their output of nm --size-sort contains lines like: 00000000000003a8 t run_test extent-map-tests.o: bloat-o-meter currently doesn't handle them which results in errors when calling .split() on them. Fix this by simply ignoring them. This enables diffing subsystems which generate built-in.a files. Link: http://lkml.kernel.org/r/20200603103513.3712-1-nborisov@suse.com Signed-off-by: Nikolay Borisov Signed-off-by: Andrew Morton --- scripts/bloat-o-meter | 2 ++ 1 file changed, 2 insertions(+) --- a/scripts/bloat-o-meter~bloat-o-meter-support-comparing-library-archives +++ a/scripts/bloat-o-meter @@ -26,6 +26,8 @@ def getsizes(file, format): sym = {} with os.popen("nm --size-sort " + file) as f: for line in f: + if line.startswith("\n") or ":" in line: + continue size, type, name = line.split() if type in format: # strip generated symbols From patchwork Fri Aug 7 06:17:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704697 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3DAEB13B6 for ; Fri, 7 Aug 2020 06:17:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 124B2221E5 for ; Fri, 7 Aug 2020 06:17:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="PKsFyyON" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 124B2221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8B6B88D000B; Fri, 7 Aug 2020 02:17:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8661A8D0002; Fri, 7 Aug 2020 02:17:37 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 755B98D000B; Fri, 7 Aug 2020 02:17:37 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0167.hostedemail.com [216.40.44.167]) by kanga.kvack.org (Postfix) with ESMTP id 6008F8D0002 for ; Fri, 7 Aug 2020 02:17:37 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 22A4F82499B9 for ; Fri, 7 Aug 2020 06:17:37 +0000 (UTC) X-FDA: 77122766154.06.dust41_5b0938f26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id E7907100513F1 for ; Fri, 7 Aug 2020 06:17:36 +0000 (UTC) X-Spam-Summary: 1,0,0,03c91764c2462318,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2197:2199:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3308:3352:3653:3865:3866:3867:3868:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6119:6261:6653:7576:7904:8599:9025:9207:9545:10004:10913:11473:11658:11914:12043:12048:12114:12297:12517:12519:12555:12679:12783:12986:13069:13161:13229:13311:13357:14181:14384:14721:14849:14915:21080:21221:21451:21627:21939:21990:30003:30029:30054,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8mxoocdzm96aszhqr6fr89afk5yptnmy4az8zzqezmf77r6931gcxbruyjk9.ngfctse7jtbykmh7ycogiyzjrwue9eujocfkzekmr6a59swtfpesnike98m1e1e.k-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not b ulk,SPF: X-HE-Tag: dust41_5b0938f26fbe X-Filterd-Recvd-Size: 2861 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:36 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8CAF822C9F; Fri, 7 Aug 2020 06:17:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781055; bh=D0gGW7zN6emYU3ITObP13wLCAJvpfiCO+L9kgcJ1wCg=; h=Date:From:To:Subject:In-Reply-To:From; b=PKsFyyONcslj0FyYxSkmE+8iGdg1BuDwH7mqGX8SpL44WatKNLzCazBBy3miGxwPW LmBeAufIk5Xp7hr0PrYoc5SlPyFkvvqJVPnu1DXP/Q2jThsXIGujfK0J7pMyR9WKLj MX4aiJqrSjcdUdqsUwh5M/coLMGhZAT9xe5RwpqM= Date: Thu, 06 Aug 2020 23:17:35 -0700 From: Andrew Morton To: akpm@linux-foundation.org, khlebnikov@yandex-team.ru, linux-mm@kvack.org, mm-commits@vger.kernel.org, sashal@kernel.org, torvalds@linux-foundation.org Subject: [patch 010/163] scripts/decode_stacktrace.sh: skip missing symbols Message-ID: <20200807061735.tb6sJM1z7%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E7907100513F1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Konstantin Khlebnikov Subject: scripts/decode_stacktrace.sh: skip missing symbols For now script turns missing symbols into '0' and make bogus decode. Skip them instead. Also simplify parsing output of 'nm'. Before: $ echo 'xxx+0x0/0x0' | ./scripts/decode_stacktrace.sh vmlinux "" xxx (home/khlebnikov/src/linux/./arch/x86/include/asm/processor.h:398) After: $ echo 'xxx+0x0/0x0' | ./scripts/decode_stacktrace.sh vmlinux "" xxx+0x0/0x0 Link: http://lkml.kernel.org/r/159282922499.248444.4883465570858385250.stgit@buzz Signed-off-by: Konstantin Khlebnikov Cc: Sasha Levin Signed-off-by: Andrew Morton --- scripts/decode_stacktrace.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) --- a/scripts/decode_stacktrace.sh~scripts-decode_stacktrace-skip-missing-symbols +++ a/scripts/decode_stacktrace.sh @@ -56,7 +56,11 @@ parse_symbol() { if [[ "${cache[$module,$name]+isset}" == "isset" ]]; then local base_addr=${cache[$module,$name]} else - local base_addr=$(nm "$objfile" | grep -i ' t ' | awk "/ $name\$/ {print \$1}" | head -n1) + local base_addr=$(nm "$objfile" | awk '$3 == "'$name'" && ($2 == "t" || $2 == "T") {print $1; exit}') + if [[ $base_addr == "" ]] ; then + # address not found + return + fi cache[$module,$name]="$base_addr" fi # Let's start doing the math to get the exact address into the From patchwork Fri Aug 7 06:17:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704705 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35FEC13B6 for ; Fri, 7 Aug 2020 06:17:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0104122C9F for ; Fri, 7 Aug 2020 06:17:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="K5fzZ13D" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0104122C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0A15A8D0002; Fri, 7 Aug 2020 02:17:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0018E8D000F; Fri, 7 Aug 2020 02:17:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBE468D0002; Fri, 7 Aug 2020 02:17:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0158.hostedemail.com [216.40.44.158]) by kanga.kvack.org (Postfix) with ESMTP id AAD338D000F for ; Fri, 7 Aug 2020 02:17:49 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 769FB3641 for ; Fri, 7 Aug 2020 06:17:49 +0000 (UTC) X-FDA: 77122766658.13.stove90_4e05a2426fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id C227B18140B82 for ; Fri, 7 Aug 2020 06:17:39 +0000 (UTC) X-Spam-Summary: 1,0,0,a993235e2c190727,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:334:355:368:369:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1714:1730:1747:1777:1792:2197:2199:2393:2525:2538:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3350:3653:3865:3866:3867:3870:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:8599:9025:9207:9545:10004:10913:11473:11658:11914:12043:12048:12114:12296:12297:12517:12519:12555:12679:12783:12986:13069:13311:13357:14181:14384:14721:14849:14915:21080:21221:21451:21627:21939:30003:30025:30054:30064:30089,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygttb991qmti53ah1opo65xc37joc7pjezxomcz5sxakr8crmj7ri55jshyfr.kmhecz719t778feaassjjd7qpcn66z583zhzzw4mce777jkkxtojbj5mwz3itgr.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MS F:not bu X-HE-Tag: stove90_4e05a2426fbe X-Filterd-Recvd-Size: 3091 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:39 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 73F1C2177B; Fri, 7 Aug 2020 06:17:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781058; bh=BzNFnikdwUm2ueekbCaXzR1+hwYi6tFIhMeqZpbBYv4=; h=Date:From:To:Subject:In-Reply-To:From; b=K5fzZ13D+CFYGWqhZ+oV1ehQKNTRTD1UDrz7a8S5CgAVu5ENDffsNDAx7xZum1HxE dLWemmuT8z9iDUqjCZG50QtKtR2GD8x3q3LZDOkEQX+4luVC+vpFsfsziF3EvgK8g5 6kTupA0Cn6b6kuG8D5uKrl9umrCruX5S9Z8AWB8s= Date: Thu, 06 Aug 2020 23:17:38 -0700 From: Andrew Morton To: akpm@linux-foundation.org, khlebnikov@yandex-team.ru, linux-mm@kvack.org, mm-commits@vger.kernel.org, sashal@kernel.org, torvalds@linux-foundation.org Subject: [patch 011/163] scripts/decode_stacktrace.sh: guess basepath if not specified Message-ID: <20200807061738.5uS4tlDwj%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: C227B18140B82 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Konstantin Khlebnikov Subject: scripts/decode_stacktrace.sh: guess basepath if not specified Guess path to kernel sources using known location of symbol "kernel_init". Make basepath argument optional. Before: $ echo 'vfs_open+0x0/0x0' | ./scripts/decode_stacktrace.sh vmlinux "" vfs_open (home/khlebnikov/src/linux/fs/open.c:912) After: $ echo 'vfs_open+0x0/0x0' | ./scripts/decode_stacktrace.sh vmlinux vfs_open (fs/open.c:912) Link: http://lkml.kernel.org/r/159282922803.248444.2379229451667913634.stgit@buzz Signed-off-by: Konstantin Khlebnikov Cc: Sasha Levin Signed-off-by: Andrew Morton --- scripts/decode_stacktrace.sh | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) --- a/scripts/decode_stacktrace.sh~scripts-decode_stacktrace-guess-basepath-if-not-specified +++ a/scripts/decode_stacktrace.sh @@ -3,14 +3,14 @@ # (c) 2014, Sasha Levin #set -x -if [[ $# < 2 ]]; then +if [[ $# < 1 ]]; then echo "Usage:" - echo " $0 [vmlinux] [base path] [modules path]" + echo " $0 [base path] [modules path]" exit 1 fi vmlinux=$1 -basepath=$2 +basepath=${2-auto} modpath=$3 declare -A cache declare -A modcache @@ -152,6 +152,14 @@ handle_line() { echo "${words[@]}" "$symbol $module" } +if [[ $basepath == "auto" ]] ; then + module="" + symbol="kernel_init+0x0/0x0" + parse_symbol + basepath=${symbol#kernel_init (} + basepath=${basepath%/init/main.c:*)} +fi + while read line; do # Let's see if we have an address in the line if [[ $line =~ \[\<([^]]+)\>\] ]] || From patchwork Fri Aug 7 06:17:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704699 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E75E214B7 for ; Fri, 7 Aug 2020 06:17:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B249B22C9F for ; Fri, 7 Aug 2020 06:17:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="fF4RKBA/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B249B22C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A3F858D000C; Fri, 7 Aug 2020 02:17:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9C5FC8D0002; Fri, 7 Aug 2020 02:17:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B62A8D000C; Fri, 7 Aug 2020 02:17:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 725838D0002 for ; Fri, 7 Aug 2020 02:17:43 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 37CCB82499B9 for ; Fri, 7 Aug 2020 06:17:43 +0000 (UTC) X-FDA: 77122766406.19.route06_3500f8e26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id DA1A01AD31E for ; Fri, 7 Aug 2020 06:17:42 +0000 (UTC) X-Spam-Summary: 1,0,0,fdd5edfdab68db28,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:334:355:368:369:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:1981:2194:2197:2199:2200:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3653:3865:3866:3867:3870:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6119:6261:6653:7576:7903:7974:8599:9025:9040:9207:9545:10004:10913:11026:11473:11658:11914:12043:12048:12114:12291:12296:12297:12517:12519:12555:12679:12683:12783:12895:12986:14181:14721:14849:14915:21080:21221:21324:21325:21451:21611:21627:21939:21987:21990:30029:30054:30089,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8f47tg54u57mmxqeht6ywgojzyopftngeekw9aw4z7xtjb9nx55xratbz5bp.htxoej9b8wbi8ioucj6siksryjktkbmcgkhze16u6o8ypoxh5fb1zef1txz46mf.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bay esian:0. X-HE-Tag: route06_3500f8e26fbe X-Filterd-Recvd-Size: 3943 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:42 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 675BC221E5; Fri, 7 Aug 2020 06:17:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781061; bh=3lygZn61skO3nENgvbN9CEK39S+HwKM4TZIeg/IYHsI=; h=Date:From:To:Subject:In-Reply-To:From; b=fF4RKBA/Dw5gVi1GAWOD5pgAYZqucoU+6Vpnw7HdYzPup4RdNl5a6dj5/OqdSAvYH vqMS6HqM9PqRBXUf/wlUrBgJD9S46Q9uST0yfWFEb3Kewma0+p/hmmDee5QuNMLzc8 8cmVwZDtrTjXRQh1jDDWpJhD08Zc2rLSmFUj7VMg= Date: Thu, 06 Aug 2020 23:17:41 -0700 From: Andrew Morton To: akpm@linux-foundation.org, khlebnikov@yandex-team.ru, linux-mm@kvack.org, mm-commits@vger.kernel.org, sashal@kernel.org, torvalds@linux-foundation.org Subject: [patch 012/163] scripts/decode_stacktrace.sh: guess path to modules Message-ID: <20200807061741.ZN18xMsId%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: DA1A01AD31E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Konstantin Khlebnikov Subject: scripts/decode_stacktrace.sh: guess path to modules Try to find module in directory with vmlinux (for fresh build). Then try standard paths where debuginfo are usually placed. Pick first file which have elf section '.debug_line'. Before: $ echo 'tap_open+0x0/0x0 [tap]' | ./scripts/decode_stacktrace.sh /usr/lib/debug/boot/vmlinux-5.4.0-37-generic WARNING! Modules path isn't set, but is needed to parse this symbol tap_open+0x0/0x0 tap After: $ echo 'tap_open+0x0/0x0 [tap]' | ./scripts/decode_stacktrace.sh /usr/lib/debug/boot/vmlinux-5.4.0-37-generic tap_open (drivers/net/tap.c:502) tap Link: http://lkml.kernel.org/r/159282923068.248444.5461337458421616083.stgit@buzz Signed-off-by: Konstantin Khlebnikov Cc: Sasha Levin Signed-off-by: Andrew Morton --- scripts/decode_stacktrace.sh | 36 ++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) --- a/scripts/decode_stacktrace.sh~scripts-decode_stacktrace-guess-path-to-modules +++ a/scripts/decode_stacktrace.sh @@ -12,9 +12,40 @@ fi vmlinux=$1 basepath=${2-auto} modpath=$3 +release="" + declare -A cache declare -A modcache +find_module() { + if [[ "$modpath" != "" ]] ; then + for fn in $(find "$modpath" -name "${module//_/[-_]}.ko*") ; do + if readelf -WS "$fn" | grep -qwF .debug_line ; then + echo $fn + return + fi + done + return 1 + fi + + modpath=$(dirname "$vmlinux") + find_module && return + + if [[ $release == "" ]] ; then + release=$(gdb -ex 'print init_uts_ns.name.release' -ex 'quit' -quiet -batch "$vmlinux" | sed -n 's/\$1 = "\(.*\)".*/\1/p') + fi + + for dn in {/usr/lib/debug,}/lib/modules/$release ; do + if [ -e "$dn" ] ; then + modpath="$dn" + find_module && return + fi + done + + modpath="" + return 1 +} + parse_symbol() { # The structure of symbol at this point is: # ([name]+[offset]/[total length]) @@ -27,12 +58,11 @@ parse_symbol() { elif [[ "${modcache[$module]+isset}" == "isset" ]]; then local objfile=${modcache[$module]} else - if [[ $modpath == "" ]]; then + local objfile=$(find_module) + if [[ $objfile == "" ]] ; then echo "WARNING! Modules path isn't set, but is needed to parse this symbol" >&2 return fi - local objfile=$(find "$modpath" -name "${module//_/[-_]}.ko*" -print -quit) - [[ $objfile == "" ]] && return modcache[$module]=$objfile fi From patchwork Fri Aug 7 06:17:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704701 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0CEC14B7 for ; Fri, 7 Aug 2020 06:17:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8540622CF6 for ; Fri, 7 Aug 2020 06:17:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="uHBlGl0o" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8540622CF6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 49BAC8D000D; Fri, 7 Aug 2020 02:17:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3D4FB8D0002; Fri, 7 Aug 2020 02:17:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24CD48D000D; Fri, 7 Aug 2020 02:17:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0159.hostedemail.com [216.40.44.159]) by kanga.kvack.org (Postfix) with ESMTP id 0BF358D0002 for ; Fri, 7 Aug 2020 02:17:46 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CCDD13627 for ; Fri, 7 Aug 2020 06:17:45 +0000 (UTC) X-FDA: 77122766490.25.jeans16_310392426fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id A4AC01804E3A0 for ; Fri, 7 Aug 2020 06:17:45 +0000 (UTC) X-Spam-Summary: 1,0,0,e7fb7fbf71af93a5,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:1981:2194:2197:2199:2200:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3653:3867:3868:3870:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:7901:7903:8599:9025:9040:9207:9545:10004:10913:11473:11657:11658:11914:12043:12048:12114:12291:12296:12297:12517:12519:12555:12679:12783:12895:12986:13069:13255:13311:13357:14181:14384:14721:14849:14915:21080:21221:21325:21451:21611:21627:21939:21987:30029:30054,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8uxeozqf4exn41k66a1g4keu3wocxsozyjqpzmronuqzwwt5wjo9s98z9dgc.ethdo9fjgnq54rn7ymr1qny7tr7rame966oymhbn7nhg7ykkabk79w6ux43mxgi.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0. 5,Netche X-HE-Tag: jeans16_310392426fbe X-Filterd-Recvd-Size: 3118 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:45 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 668D22177B; Fri, 7 Aug 2020 06:17:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781064; bh=ghCvHaU0MhNYqjK58MzNNFkEn04Nm53tON39oKwbNH8=; h=Date:From:To:Subject:In-Reply-To:From; b=uHBlGl0ocHXyugJCbYfP/UhjwsqDuMhNcz7oAES/mnEG9pOfS9lIJeJ0+jyAV8FTI 4fGBrLuB2IfucjC0edC7Jn2NV0hhcJ6p46DHXvAph47bMpa+CyDq6tf24WSWmCzIz3 0mHd3WHrgSHIsrc+23ucdTadbRJySzr71lBRD0KA= Date: Thu, 06 Aug 2020 23:17:43 -0700 From: Andrew Morton To: akpm@linux-foundation.org, khlebnikov@yandex-team.ru, linux-mm@kvack.org, mm-commits@vger.kernel.org, sashal@kernel.org, torvalds@linux-foundation.org Subject: [patch 013/163] scripts/decode_stacktrace.sh: guess path to vmlinux by release name Message-ID: <20200807061743.TkcOXWDd2%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: A4AC01804E3A0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Konstantin Khlebnikov Subject: scripts/decode_stacktrace.sh: guess path to vmlinux by release name Add option decode_stacktrace -r to specify only release name. This is enough to guess standard paths to vmlinux and modules: $ echo -e 'schedule+0x0/0x0 tap_open+0x0/0x0 [tap]' | ./scripts/decode_stacktrace.sh -r 5.4.0-37-generic schedule (kernel/sched/core.c:4138) tap_open (drivers/net/tap.c:502) tap Link: http://lkml.kernel.org/r/159282923334.248444.2399153100007347838.stgit@buzz Signed-off-by: Konstantin Khlebnikov Cc: Sasha Levin Signed-off-by: Andrew Morton --- scripts/decode_stacktrace.sh | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) --- a/scripts/decode_stacktrace.sh~scripts-decode_stacktrace-guess-path-to-vmlinux-by-release-name +++ a/scripts/decode_stacktrace.sh @@ -5,14 +5,33 @@ if [[ $# < 1 ]]; then echo "Usage:" - echo " $0 [base path] [modules path]" + echo " $0 -r | [base path] [modules path]" exit 1 fi -vmlinux=$1 -basepath=${2-auto} -modpath=$3 -release="" +if [[ $1 == "-r" ]] ; then + vmlinux="" + basepath="auto" + modpath="" + release=$2 + + for fn in {,/usr/lib/debug}/boot/vmlinux-$release{,.debug} /lib/modules/$release{,/build}/vmlinux ; do + if [ -e "$fn" ] ; then + vmlinux=$fn + break + fi + done + + if [[ $vmlinux == "" ]] ; then + echo "ERROR! vmlinux image for release $release is not found" >&2 + exit 2 + fi +else + vmlinux=$1 + basepath=${2-auto} + modpath=$3 + release="" +fi declare -A cache declare -A modcache From patchwork Fri Aug 7 06:17:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704703 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0E57413B6 for ; Fri, 7 Aug 2020 06:17:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D734D221E5 for ; Fri, 7 Aug 2020 06:17:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="d2MwzfCi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D734D221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9C9648D000E; Fri, 7 Aug 2020 02:17:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 950AC8D0002; Fri, 7 Aug 2020 02:17:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 840648D000E; Fri, 7 Aug 2020 02:17:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0045.hostedemail.com [216.40.44.45]) by kanga.kvack.org (Postfix) with ESMTP id 694278D0002 for ; Fri, 7 Aug 2020 02:17:49 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2E653180AD801 for ; Fri, 7 Aug 2020 06:17:49 +0000 (UTC) X-FDA: 77122766658.14.basin10_4011f6a26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id E9AEB18229835 for ; Fri, 7 Aug 2020 06:17:48 +0000 (UTC) X-Spam-Summary: 1,0,0,9326285f4b299089,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1568:1711:1714:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3867:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6642:6653:6737:7514:7576:9010:9025:9545:10004:11026:11257:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:13069:13311:13357:13846:14181:14384:14721:14849:21080:21451:21627:21939:30054,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrp31pgnbqxawnn6741ey9q3kmzycj63enfphncokgdqidn6ciyscsu75fjkw.ef97ob7p5srb3sqy5u3dnzq8cgbhwu3yqo8x9e8w9bwbzuykdf7jms3p8gfs8cb.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24 ,LUA_SUM X-HE-Tag: basin10_4011f6a26fbe X-Filterd-Recvd-Size: 2503 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:48 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5A5F9221E5; Fri, 7 Aug 2020 06:17:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781067; bh=3SUE6uPfKgAVqijImsS7I/IPEr0raTLy/24Tx7gc+uA=; h=Date:From:To:Subject:In-Reply-To:From; b=d2MwzfCi9blh0ovXU5OCEFMPy6Kr2Nw7t7Zl/pejto4J5LEw8Xaubs1PC+4KpHKdY bDhkhSeRMTX5vrOpRVO7DHi1tcSIVm8AVWS6coMTujO1wI1z3JQhq9DUzmxlRIZvVd vDpc5MsM86JUadq5SPm1kNMTSGg1fHbWEpaY1ATY= Date: Thu, 06 Aug 2020 23:17:46 -0700 From: Andrew Morton To: akpm@linux-foundation.org, bleung@chromium.org, broonie@kernel.org, enric.balletbo@collabora.com, groeck@chromium.org, joe@perches.com, lgirdwood@gmail.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, pihsun@chromium.org, rikard.falkeborn@gmail.com, torvalds@linux-foundation.org Subject: [patch 014/163] const_structs.checkpatch: add regulator_ops Message-ID: <20200807061746.Xki5F1zJ5%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E9AEB18229835 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joe Perches Subject: const_structs.checkpatch: add regulator_ops Add regulator_ops to expected to be const list. Link: http://lkml.kernel.org/r/dab1ba1aa03a8236933cfb7a28937efb0b808f13.camel@perches.com Signed-off-by: Joe Perches Cc: Pi-Hsun Shih Cc: Liam Girdwood Cc: Mark Brown Cc: Benson Leung Cc: Enric Balletbo i Serra Cc: Guenter Roeck Cc: Rikard Falkeborn Signed-off-by: Andrew Morton --- scripts/const_structs.checkpatch | 1 + 1 file changed, 1 insertion(+) --- a/scripts/const_structs.checkpatch~const_structscheckpatch-add-regulator_ops +++ a/scripts/const_structs.checkpatch @@ -44,6 +44,7 @@ platform_hibernation_ops platform_suspend_ops proto_ops regmap_access_table +regulator_ops rpc_pipe_ops rtc_class_ops sd_desc From patchwork Fri Aug 7 06:17:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704707 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B7DA114B7 for ; Fri, 7 Aug 2020 06:17:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8BB2522CF6 for ; Fri, 7 Aug 2020 06:17:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="XMXBlydw" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8BB2522CF6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8DFF08D0010; Fri, 7 Aug 2020 02:17:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 817C98D000F; Fri, 7 Aug 2020 02:17:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E0B28D0010; Fri, 7 Aug 2020 02:17:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id 533F48D000F for ; Fri, 7 Aug 2020 02:17:53 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1E2C9180AD801 for ; Fri, 7 Aug 2020 06:17:53 +0000 (UTC) X-FDA: 77122766826.20.corn70_2e0763d26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id E94FD180C07A3 for ; Fri, 7 Aug 2020 06:17:51 +0000 (UTC) X-Spam-Summary: 1,0,0,3f462097d72b1b7c,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1544:1711:1730:1747:1777:1792:2198:2199:2393:2525:2553:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6119:6261:6653:7576:7974:8599:9025:9545:10004:10913:11658:11914:12043:12048:12297:12517:12519:12555:12679:12783:12986:13846:14096:14181:14721:14849:21080:21324:21451:21627:21939:30054:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yr437ujieqeq5ccfjk79fxrygqfopq8yqpb8ku8g4qq1fbqdsoarsjkwcp7bi.4ni1u7ot141s3etgufge6w4wrtg91k3gwzraxnqcb55f3icaqndthcnhzg8oeq3.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFt ime:24,L X-HE-Tag: corn70_2e0763d26fbe X-Filterd-Recvd-Size: 5492 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:51 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 90E792177B; Fri, 7 Aug 2020 06:17:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781070; bh=LSyFib3IiHYu93c9FA0ZSkQ8Yvfds6bzoSpjQCBx5V0=; h=Date:From:To:Subject:In-Reply-To:From; b=XMXBlydw49+AyCfrp32mQg6bEtqkzCT/XzNfBQ4Uzgz6hqdLFX23TuOKFuRW9R+jK 4cIrLKwnYNsktEy3AfTwLTK8jEUZ+na9hhxwsOKipfm1ZZ3+Bs9QQF+lfJHnHvLZs5 UZ/+XnQ7IHJJeXrmlbMOYDbXxJNa3XOG+XmEaVfo= Date: Thu, 06 Aug 2020 23:17:50 -0700 From: Andrew Morton To: akpm@linux-foundation.org, colin.king@canonical.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 015/163] scripts/spelling.txt: add more spellings to spelling.txt Message-ID: <20200807061750.QNVnq0Omj%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E94FD180C07A3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Colin Ian King Subject: scripts/spelling.txt: add more spellings to spelling.txt Here are some of the more common spelling mistakes and typos that I've found while fixing up spelling mistakes in the kernel since April 2020. Link: http://lkml.kernel.org/r/20200714092837.173796-1-colin.king@canonical.com Signed-off-by: Colin Ian King Signed-off-by: Andrew Morton --- scripts/spelling.txt | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) --- a/scripts/spelling.txt~scripts-spellingtxt-add-more-spellings-to-spellingtxt +++ a/scripts/spelling.txt @@ -149,6 +149,7 @@ arbitary||arbitrary architechture||architecture arguement||argument arguements||arguments +arithmatic||arithmetic aritmetic||arithmetic arne't||aren't arraival||arrival @@ -454,6 +455,7 @@ destorys||destroys destroied||destroyed detabase||database deteced||detected +detectt||detect develope||develop developement||development developped||developed @@ -545,6 +547,7 @@ entires||entries entites||entities entrys||entries enocded||encoded +enought||enough enterily||entirely enviroiment||environment enviroment||environment @@ -556,11 +559,14 @@ equivelant||equivalent equivilant||equivalent eror||error errorr||error +errror||error estbalishment||establishment etsablishment||establishment etsbalishment||establishment +evalution||evaluation excecutable||executable exceded||exceeded +exceds||exceeds exceeed||exceed excellant||excellent execeeded||exceeded @@ -583,6 +589,7 @@ explictly||explicitly expresion||expression exprimental||experimental extened||extended +exteneded||extended||extended extensability||extensibility extention||extension extenstion||extension @@ -610,10 +617,12 @@ feautures||features fetaure||feature fetaures||features fileystem||filesystem +fimrware||firmware fimware||firmware firmare||firmware firmaware||firmware firware||firmware +firwmare||firmware finanize||finalize findn||find finilizes||finalizes @@ -661,6 +670,7 @@ globel||global grabing||grabbing grahical||graphical grahpical||graphical +granularty||granularity grapic||graphic grranted||granted guage||gauge @@ -906,6 +916,7 @@ miximum||maximum mmnemonic||mnemonic mnay||many modfiy||modify +modifer||modifier modulues||modules momery||memory memomry||memory @@ -915,6 +926,7 @@ monochromo||monochrome monocrome||monochrome mopdule||module mroe||more +multipler||multiplier mulitplied||multiplied multidimensionnal||multidimensional multipe||multiple @@ -952,6 +964,7 @@ occassionally||occasionally occationally||occasionally occurance||occurrence occurances||occurrences +occurd||occurred occured||occurred occurence||occurrence occure||occurred @@ -1058,6 +1071,7 @@ precission||precision preemptable||preemptible prefered||preferred prefferably||preferably +prefitler||prefilter premption||preemption prepaired||prepared preperation||preparation @@ -1101,6 +1115,7 @@ pronunce||pronounce propery||property propigate||propagate propigation||propagation +propogation||propagation propogate||propagate prosess||process protable||portable @@ -1316,6 +1331,7 @@ sturcture||structure subdirectoires||subdirectories suble||subtle substract||subtract +submited||submitted submition||submission suceed||succeed succesfully||successfully @@ -1324,6 +1340,7 @@ successed||succeeded successfull||successful successfuly||successfully sucessfully||successfully +sucessful||successful sucess||success superflous||superfluous superseeded||superseded @@ -1409,6 +1426,7 @@ transormed||transformed trasfer||transfer trasmission||transmission treshold||threshold +triggerd||triggered trigerred||triggered trigerring||triggering trun||turn @@ -1421,6 +1439,7 @@ uknown||unknown usccess||success usupported||unsupported uncommited||uncommitted +uncompatible||incompatible unconditionaly||unconditionally undeflow||underflow underun||underrun From patchwork Fri Aug 7 06:17:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704709 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4F6C316B1 for ; Fri, 7 Aug 2020 06:17:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1A2F92177B for ; Fri, 7 Aug 2020 06:17:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="J5ZRRNKK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1A2F92177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CD63F8D0011; Fri, 7 Aug 2020 02:17:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C60DC8D000F; Fri, 7 Aug 2020 02:17:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8A118D0011; Fri, 7 Aug 2020 02:17:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0135.hostedemail.com [216.40.44.135]) by kanga.kvack.org (Postfix) with ESMTP id 8EC6D8D000F for ; Fri, 7 Aug 2020 02:17:55 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4C175249B for ; Fri, 7 Aug 2020 06:17:55 +0000 (UTC) X-FDA: 77122766910.30.ink14_590bbe426fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 2610B180B3C85 for ; Fri, 7 Aug 2020 06:17:55 +0000 (UTC) X-Spam-Summary: 10,1,0,8c7e5b695bc6cb1e,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2731:2859:2894:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4049:4119:4250:4321:5007:6117:6119:6261:6653:7514:7576:7875:7903:8599:8660:9025:9545:9592:10004:10913:11026:11657:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13148:13230:13255:13846:21080:21324:21433:21451:21627:21740:21891:21939:21990:30012:30029:30054,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfachnfr5eempxesixonjmbf3t4ycibi87u3i3bkdkxmgt8jdfxnaeqnytjnj.rndu5wtszh1yiqwc8fzdh83iaraypn5boncg15fjg3s7kdsgc5a5qbfmhmzrbj6.g-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0. 5,0.5,0. X-HE-Tag: ink14_590bbe426fbe X-Filterd-Recvd-Size: 8128 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:54 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9DA8D22C9F; Fri, 7 Aug 2020 06:17:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781074; bh=OvKO/4xCG99Mbv4yLmm86L5dPLMNG7wWuKmY/YE1hKQ=; h=Date:From:To:Subject:In-Reply-To:From; b=J5ZRRNKKkJ4rhwLJgpQSUP/FG2kuDsR8FKvM9gAuD+RmkyHeoiqB7p0ZgyUCvfCVZ JhaxSkXFiiyhXcyg34nnC1G9Rn6io7GhGbVPKuaa2M1BDRIxVjalswqfbW2loqFNVa UU9422/sdjtSxyxrLujq7GBZbFKhnSi5V9ivvqsY= Date: Thu, 06 Aug 2020 23:17:53 -0700 From: Andrew Morton To: akpm@linux-foundation.org, anton@tuxera.com, linux-mm@kvack.org, luca.stefani.ge1@gmail.com, michalechner92@googlemail.com, mm-commits@vger.kernel.org, natechancellor@gmail.com, ndesaulniers@google.com, torvalds@linux-foundation.org Subject: [patch 016/163] ntfs: fix ntfs_test_inode and ntfs_init_locked_inode function type Message-ID: <20200807061753.64bvhlRIv%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 2610B180B3C85 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Luca Stefani Subject: ntfs: fix ntfs_test_inode and ntfs_init_locked_inode function type Clang's Control Flow Integrity (CFI) is a security mechanism that can help prevent JOP chains, deployed extensively in downstream kernels used in Android. Its deployment is hindered by mismatches in function signatures. For this case, we make callbacks match their intended function signature, and cast parameters within them rather than casting the callback when passed as a parameter. When running `mount -t ntfs ...` we observe the following trace: Call trace: __cfi_check_fail+0x1c/0x24 name_to_dev_t+0x0/0x404 iget5_locked+0x594/0x5e8 ntfs_fill_super+0xbfc/0x43ec mount_bdev+0x30c/0x3cc ntfs_mount+0x18/0x24 mount_fs+0x1b0/0x380 vfs_kern_mount+0x90/0x398 do_mount+0x5d8/0x1a10 SyS_mount+0x108/0x144 el0_svc_naked+0x34/0x38 Link: http://lkml.kernel.org/r/20200718112513.533800-1-luca.stefani.ge1@gmail.com Signed-off-by: Luca Stefani Tested-by: freak07 Acked-by: Anton Altaparmakov Reviewed-by: Nick Desaulniers Reviewed-by: Nathan Chancellor Signed-off-by: Andrew Morton --- fs/ntfs/dir.c | 2 +- fs/ntfs/inode.c | 27 ++++++++++++++------------- fs/ntfs/inode.h | 4 +--- fs/ntfs/mft.c | 4 ++-- 4 files changed, 18 insertions(+), 19 deletions(-) --- a/fs/ntfs/dir.c~ntfs-fix-ntfs_test_inode-and-ntfs_init_locked_inode-function-type +++ a/fs/ntfs/dir.c @@ -1504,7 +1504,7 @@ static int ntfs_dir_fsync(struct file *f na.type = AT_BITMAP; na.name = I30; na.name_len = 4; - bmp_vi = ilookup5(vi->i_sb, vi->i_ino, (test_t)ntfs_test_inode, &na); + bmp_vi = ilookup5(vi->i_sb, vi->i_ino, ntfs_test_inode, &na); if (bmp_vi) { write_inode_now(bmp_vi, !datasync); iput(bmp_vi); --- a/fs/ntfs/inode.c~ntfs-fix-ntfs_test_inode-and-ntfs_init_locked_inode-function-type +++ a/fs/ntfs/inode.c @@ -30,10 +30,10 @@ /** * ntfs_test_inode - compare two (possibly fake) inodes for equality * @vi: vfs inode which to test - * @na: ntfs attribute which is being tested with + * @data: data which is being tested with * * Compare the ntfs attribute embedded in the ntfs specific part of the vfs - * inode @vi for equality with the ntfs attribute @na. + * inode @vi for equality with the ntfs attribute @data. * * If searching for the normal file/directory inode, set @na->type to AT_UNUSED. * @na->name and @na->name_len are then ignored. @@ -43,8 +43,9 @@ * NOTE: This function runs with the inode_hash_lock spin lock held so it is not * allowed to sleep. */ -int ntfs_test_inode(struct inode *vi, ntfs_attr *na) +int ntfs_test_inode(struct inode *vi, void *data) { + ntfs_attr *na = (ntfs_attr *)data; ntfs_inode *ni; if (vi->i_ino != na->mft_no) @@ -72,9 +73,9 @@ int ntfs_test_inode(struct inode *vi, nt /** * ntfs_init_locked_inode - initialize an inode * @vi: vfs inode to initialize - * @na: ntfs attribute which to initialize @vi to + * @data: data which to initialize @vi to * - * Initialize the vfs inode @vi with the values from the ntfs attribute @na in + * Initialize the vfs inode @vi with the values from the ntfs attribute @data in * order to enable ntfs_test_inode() to do its work. * * If initializing the normal file/directory inode, set @na->type to AT_UNUSED. @@ -87,8 +88,9 @@ int ntfs_test_inode(struct inode *vi, nt * NOTE: This function runs with the inode->i_lock spin lock held so it is not * allowed to sleep. (Hence the GFP_ATOMIC allocation.) */ -static int ntfs_init_locked_inode(struct inode *vi, ntfs_attr *na) +static int ntfs_init_locked_inode(struct inode *vi, void *data) { + ntfs_attr *na = (ntfs_attr *)data; ntfs_inode *ni = NTFS_I(vi); vi->i_ino = na->mft_no; @@ -131,7 +133,6 @@ static int ntfs_init_locked_inode(struct return 0; } -typedef int (*set_t)(struct inode *, void *); static int ntfs_read_locked_inode(struct inode *vi); static int ntfs_read_locked_attr_inode(struct inode *base_vi, struct inode *vi); static int ntfs_read_locked_index_inode(struct inode *base_vi, @@ -164,8 +165,8 @@ struct inode *ntfs_iget(struct super_blo na.name = NULL; na.name_len = 0; - vi = iget5_locked(sb, mft_no, (test_t)ntfs_test_inode, - (set_t)ntfs_init_locked_inode, &na); + vi = iget5_locked(sb, mft_no, ntfs_test_inode, + ntfs_init_locked_inode, &na); if (unlikely(!vi)) return ERR_PTR(-ENOMEM); @@ -225,8 +226,8 @@ struct inode *ntfs_attr_iget(struct inod na.name = name; na.name_len = name_len; - vi = iget5_locked(base_vi->i_sb, na.mft_no, (test_t)ntfs_test_inode, - (set_t)ntfs_init_locked_inode, &na); + vi = iget5_locked(base_vi->i_sb, na.mft_no, ntfs_test_inode, + ntfs_init_locked_inode, &na); if (unlikely(!vi)) return ERR_PTR(-ENOMEM); @@ -280,8 +281,8 @@ struct inode *ntfs_index_iget(struct ino na.name = name; na.name_len = name_len; - vi = iget5_locked(base_vi->i_sb, na.mft_no, (test_t)ntfs_test_inode, - (set_t)ntfs_init_locked_inode, &na); + vi = iget5_locked(base_vi->i_sb, na.mft_no, ntfs_test_inode, + ntfs_init_locked_inode, &na); if (unlikely(!vi)) return ERR_PTR(-ENOMEM); --- a/fs/ntfs/inode.h~ntfs-fix-ntfs_test_inode-and-ntfs_init_locked_inode-function-type +++ a/fs/ntfs/inode.h @@ -253,9 +253,7 @@ typedef struct { ATTR_TYPE type; } ntfs_attr; -typedef int (*test_t)(struct inode *, void *); - -extern int ntfs_test_inode(struct inode *vi, ntfs_attr *na); +extern int ntfs_test_inode(struct inode *vi, void *data); extern struct inode *ntfs_iget(struct super_block *sb, unsigned long mft_no); extern struct inode *ntfs_attr_iget(struct inode *base_vi, ATTR_TYPE type, --- a/fs/ntfs/mft.c~ntfs-fix-ntfs_test_inode-and-ntfs_init_locked_inode-function-type +++ a/fs/ntfs/mft.c @@ -958,7 +958,7 @@ bool ntfs_may_write_mft_record(ntfs_volu * dirty code path of the inode dirty code path when writing * $MFT occurs. */ - vi = ilookup5_nowait(sb, mft_no, (test_t)ntfs_test_inode, &na); + vi = ilookup5_nowait(sb, mft_no, ntfs_test_inode, &na); } if (vi) { ntfs_debug("Base inode 0x%lx is in icache.", mft_no); @@ -1019,7 +1019,7 @@ bool ntfs_may_write_mft_record(ntfs_volu vi = igrab(mft_vi); BUG_ON(vi != mft_vi); } else - vi = ilookup5_nowait(sb, na.mft_no, (test_t)ntfs_test_inode, + vi = ilookup5_nowait(sb, na.mft_no, ntfs_test_inode, &na); if (!vi) { /* From patchwork Fri Aug 7 06:17:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704711 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F39D013B6 for ; Fri, 7 Aug 2020 06:18:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C7D6222C9F for ; Fri, 7 Aug 2020 06:18:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="UigfX3xl" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7D6222C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 005AD8D0012; Fri, 7 Aug 2020 02:17:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E806C8D000F; Fri, 7 Aug 2020 02:17:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D71198D0012; Fri, 7 Aug 2020 02:17:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0152.hostedemail.com [216.40.44.152]) by kanga.kvack.org (Postfix) with ESMTP id B72038D000F for ; Fri, 7 Aug 2020 02:17:58 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8ADBE181AEF10 for ; Fri, 7 Aug 2020 06:17:58 +0000 (UTC) X-FDA: 77122767036.10.comb99_480aa5a26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 57C5316A0D2 for ; Fri, 7 Aug 2020 06:17:58 +0000 (UTC) X-Spam-Summary: 1,0,0,c1b7cd1a98651eeb,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2525:2553:2559:2563:2682:2685:2859:2892:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3150:3352:3865:3867:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6261:6653:6737:7576:8599:8603:9025:9545:10004:10913:11026:11658:11914:12043:12048:12297:12517:12519:12555:12679:12783:12986:13069:13161:13184:13229:13311:13357:13846:14181:14384:14721:14849:21080:21451:21627:21795:21939:21990:30030:30051:30054:30064:30070:30075:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygti9m9f4xhr57xbjkqwnypiskjocmfgbuuan9gjg38pw9wys15bqoz6uwr6c.iz6q3axgcq8kxqhrkj3wsqe56geisims1zrqbhjeszag13ty5gyjt3rznjryxhs.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,Doma inCache: X-HE-Tag: comb99_480aa5a26fbe X-Filterd-Recvd-Size: 2815 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:17:57 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id DEC15221E5; Fri, 7 Aug 2020 06:17:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781077; bh=pnN8QFT/hiZ+CQ+boSDBciJdH/VWXkOF1ATLoWC62xA=; h=Date:From:To:Subject:In-Reply-To:From; b=UigfX3xljNhcXG8NsD1KBsjCWv9n+BM7bRQmx7EZ28HCRs/uD86ZBsqw4B35i0GFR pB2XBZSaY9Pdwd0cPBKk7VZrlM7Cq0LYfXqutsFvP3SJf/lrcPc0GK8o2oo+Wht5wN wdMHSfPGb27IjdDEUGudwpsp2+3QyVmu0ZWATkW4= Date: Thu, 06 Aug 2020 23:17:56 -0700 From: Andrew Morton To: akpm@linux-foundation.org, gechangwei@live.cn, ghe@suse.com, jlbec@evilplan.org, joseph.qi@linux.alibaba.com, junxiao.bi@oracle.com, linux-mm@kvack.org, mark@fasheh.com, mm-commits@vger.kernel.org, piaojun@huawei.com, torvalds@linux-foundation.org Subject: [patch 017/163] ocfs2: fix remounting needed after setfacl command Message-ID: <20200807061756.lkEFtWo3s%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 57C5316A0D2 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Gang He Subject: ocfs2: fix remounting needed after setfacl command When use setfacl command to change a file's acl, the user cannot get the latest acl information from the file via getfacl command, until remounting the file system. e.g. setfacl -m u:ivan:rw /ocfs2/ivan getfacl /ocfs2/ivan getfacl: Removing leading '/' from absolute path names file: ocfs2/ivan owner: root group: root user::rw- group::r-- mask::r-- other::r-- The latest acl record("u:ivan:rw") cannot be returned via getfacl command until remounting. Link: http://lkml.kernel.org/r/20200717023751.9922-1-ghe@suse.com Signed-off-by: Gang He Reviewed-by: Joseph Qi Cc: Mark Fasheh Cc: Joel Becker Cc: Junxiao Bi Cc: Changwei Ge Cc: Jun Piao Signed-off-by: Andrew Morton --- fs/ocfs2/acl.c | 2 ++ 1 file changed, 2 insertions(+) --- a/fs/ocfs2/acl.c~ocfs2-fix-remounting-needed-after-setfacl-command +++ a/fs/ocfs2/acl.c @@ -256,6 +256,8 @@ static int ocfs2_set_acl(handle_t *handl ret = ocfs2_xattr_set(inode, name_index, "", value, size, 0); kfree(value); + if (!ret) + set_cached_acl(inode, type, acl); return ret; } From patchwork Fri Aug 7 06:17:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704713 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CD11813B6 for ; Fri, 7 Aug 2020 06:18:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9EAB322C9F for ; Fri, 7 Aug 2020 06:18:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="XvtMAZA1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9EAB322C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 204068D0013; Fri, 7 Aug 2020 02:18:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1611F8D000F; Fri, 7 Aug 2020 02:18:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 056068D0013; Fri, 7 Aug 2020 02:18:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0027.hostedemail.com [216.40.44.27]) by kanga.kvack.org (Postfix) with ESMTP id D783F8D000F for ; Fri, 7 Aug 2020 02:18:01 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 9D1CF181AEF10 for ; Fri, 7 Aug 2020 06:18:01 +0000 (UTC) X-FDA: 77122767162.19.hand64_5906b5626fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 7367B1AD1B3 for ; Fri, 7 Aug 2020 06:18:01 +0000 (UTC) X-Spam-Summary: 1,0,0,f8c7a3cd1a5cc3fd,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3867:3870:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:7875:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12783:12986:13069:13311:13357:13846:14181:14384:14721:14849:21080:21433:21451:21627:21939:30034:30054,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrkqaw71htwoxqeacsp8s6xopcfycjwpy1z3d45x9w5apb6rgomncgn9wcfws.etfujcfhadpkzxidsku5f3snos74pga7pxe7phyqf5t6atkuij58cc9ahqqarfk.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0 :0,LFtim X-HE-Tag: hand64_5906b5626fbe X-Filterd-Recvd-Size: 2434 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:00 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0E4492177B; Fri, 7 Aug 2020 06:18:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781080; bh=cx03duJfSEM3xER1ayQx0ZitLE6DIqG3as1jVLmYEOo=; h=Date:From:To:Subject:In-Reply-To:From; b=XvtMAZA1Bq82d+C6tHdR757PD9Yl2E15jGoEJw1Q2kCDXJTS/u7wDiydsTY1jEpti g16jRoWCXXEVLEAI1gMHspctF87LI2MMwisG8E3urzLGcN6vnFNkOpJ7E2TTAdggcL avFonacXYKptcpwvN3+7OVn2EbKHPPWzoVJUNmAY= Date: Thu, 06 Aug 2020 23:17:59 -0700 From: Andrew Morton To: akpm@linux-foundation.org, jlbec@evilplan.org, joseph.qi@linux.alibaba.com, linux-mm@kvack.org, mark@fasheh.com, mm-commits@vger.kernel.org, rdunlap@infradead.org, torvalds@linux-foundation.org Subject: [patch 018/163] ocfs2: suballoc.h: delete a duplicated word Message-ID: <20200807061759.oznPk9awL%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 7367B1AD1B3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Randy Dunlap Subject: ocfs2: suballoc.h: delete a duplicated word Drop the repeated word "is" in a comment. Link: http://lkml.kernel.org/r/20200720001421.28823-1-rdunlap@infradead.org Signed-off-by: Randy Dunlap Acked-by: Joseph Qi Cc: Mark Fasheh Cc: Joel Becker Cc: Joseph Qi Signed-off-by: Andrew Morton --- fs/ocfs2/suballoc.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/fs/ocfs2/suballoc.h~ocfs2-suballoch-delete-a-duplicated-word +++ a/fs/ocfs2/suballoc.h @@ -40,7 +40,7 @@ struct ocfs2_alloc_context { u64 ac_last_group; u64 ac_max_block; /* Highest block number to allocate. 0 is - is the same as ~0 - unlimited */ + the same as ~0 - unlimited */ int ac_find_loc_only; /* hack for reflink operation ordering */ struct ocfs2_suballoc_result *ac_find_loc_priv; /* */ From patchwork Fri Aug 7 06:18:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704715 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 052EE13B6 for ; Fri, 7 Aug 2020 06:18:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C172B22D03 for ; Fri, 7 Aug 2020 06:18:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="V6YFwy07" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C172B22D03 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9A0A28D0014; Fri, 7 Aug 2020 02:18:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 928ED8D000F; Fri, 7 Aug 2020 02:18:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 842E58D0014; Fri, 7 Aug 2020 02:18:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0113.hostedemail.com [216.40.44.113]) by kanga.kvack.org (Postfix) with ESMTP id 68AC08D000F for ; Fri, 7 Aug 2020 02:18:05 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2B8293641 for ; Fri, 7 Aug 2020 06:18:05 +0000 (UTC) X-FDA: 77122767330.15.mine88_44122c926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id F20DB1814B0C1 for ; Fri, 7 Aug 2020 06:18:04 +0000 (UTC) X-Spam-Summary: 1,0,0,fcf2c33db46d9d27,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:560:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2525:2553:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3150:3353:3865:3866:3867:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:6737:7576:8599:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13221:13229:13846:14181:14721:14849:21080:21451:21627:21939:21990:30034:30054:30064:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yr7c6gnmw1okfc836nge7kt5ui6ypg4mwpkhqhhi4kdka98ckrfqsh3zamecz.j4b4f3n48xwkp9eozwfp3rc6m35spukb41n3ariy99wrzp8espjx8hunt1mf1gr.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SP F:fp,MSB X-HE-Tag: mine88_44122c926fbe X-Filterd-Recvd-Size: 4329 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:04 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2CCA922CAE; Fri, 7 Aug 2020 06:18:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781083; bh=JEjT5MjxPtlExL/ndg5RGSqopRhTQSjzsLnn9XKylbs=; h=Date:From:To:Subject:In-Reply-To:From; b=V6YFwy07FiK8iF+0DsXVL4sLoeSuhRsp7Zvf9xoViRMScDvQRQanPIJTrH0DdrtqE w3kbUdrbfFazzee8l+1XlZSmnAHGKxFJ3LYXDebH05wSSpMHqFAtzEtuxpOruzj0HB OCEc05BKNamekNVzz0lnfnQRFAXgFBl7hTpjjOT4= Date: Thu, 06 Aug 2020 23:18:02 -0700 From: Andrew Morton To: akpm@linux-foundation.org, dan.carpenter@oracle.com, gechangwei@live.cn, ghe@suse.com, jlbec@evilplan.org, joseph.qi@linux.alibaba.com, junxiao.bi@oracle.com, linux-mm@kvack.org, mark@fasheh.com, mm-commits@vger.kernel.org, piaojun@huawei.com, stable@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 019/163] ocfs2: change slot number type s16 to u16 Message-ID: <20200807061802.UMoDJErtC%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: F20DB1814B0C1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Junxiao Bi Subject: ocfs2: change slot number type s16 to u16 Dan Carpenter reported the following static checker warning. fs/ocfs2/super.c:1269 ocfs2_parse_options() warn: '(-1)' 65535 can't fit into 32767 'mopt->slot' fs/ocfs2/suballoc.c:859 ocfs2_init_inode_steal_slot() warn: '(-1)' 65535 can't fit into 32767 'osb->s_inode_steal_slot' fs/ocfs2/suballoc.c:867 ocfs2_init_meta_steal_slot() warn: '(-1)' 65535 can't fit into 32767 'osb->s_meta_steal_slot' That's because OCFS2_INVALID_SLOT is (u16)-1. Slot number in ocfs2 can be never negative, so change s16 to u16. Link: http://lkml.kernel.org/r/20200627001259.19757-1-junxiao.bi@oracle.com Fixes: 9277f8334ffc ("ocfs2: fix value of OCFS2_INVALID_SLOT") Signed-off-by: Junxiao Bi Reported-by: Dan Carpenter Reviewed-by: Joseph Qi Reviewed-by: Gang He Cc: Mark Fasheh Cc: Joel Becker Cc: Junxiao Bi Cc: Changwei Ge Cc: Jun Piao Cc: Signed-off-by: Andrew Morton --- fs/ocfs2/ocfs2.h | 4 ++-- fs/ocfs2/suballoc.c | 4 ++-- fs/ocfs2/super.c | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) --- a/fs/ocfs2/ocfs2.h~ocfs2-change-slot-number-type-s16-to-u16 +++ a/fs/ocfs2/ocfs2.h @@ -327,8 +327,8 @@ struct ocfs2_super spinlock_t osb_lock; u32 s_next_generation; unsigned long osb_flags; - s16 s_inode_steal_slot; - s16 s_meta_steal_slot; + u16 s_inode_steal_slot; + u16 s_meta_steal_slot; atomic_t s_num_inodes_stolen; atomic_t s_num_meta_stolen; --- a/fs/ocfs2/suballoc.c~ocfs2-change-slot-number-type-s16-to-u16 +++ a/fs/ocfs2/suballoc.c @@ -879,9 +879,9 @@ static void __ocfs2_set_steal_slot(struc { spin_lock(&osb->osb_lock); if (type == INODE_ALLOC_SYSTEM_INODE) - osb->s_inode_steal_slot = slot; + osb->s_inode_steal_slot = (u16)slot; else if (type == EXTENT_ALLOC_SYSTEM_INODE) - osb->s_meta_steal_slot = slot; + osb->s_meta_steal_slot = (u16)slot; spin_unlock(&osb->osb_lock); } --- a/fs/ocfs2/super.c~ocfs2-change-slot-number-type-s16-to-u16 +++ a/fs/ocfs2/super.c @@ -78,7 +78,7 @@ struct mount_options unsigned long commit_interval; unsigned long mount_opt; unsigned int atime_quantum; - signed short slot; + unsigned short slot; int localalloc_opt; unsigned int resv_level; int dir_resv_level; @@ -1349,7 +1349,7 @@ static int ocfs2_parse_options(struct su goto bail; } if (option) - mopt->slot = (s16)option; + mopt->slot = (u16)option; break; case Opt_commit: if (match_int(&args[0], &option)) { From patchwork Fri Aug 7 06:18:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704717 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BC6BD14B7 for ; Fri, 7 Aug 2020 06:18:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7F8FB2177B for ; Fri, 7 Aug 2020 06:18:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="m644SMHR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7F8FB2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 496D78D0015; Fri, 7 Aug 2020 02:18:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 447138D000F; Fri, 7 Aug 2020 02:18:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3363A8D0015; Fri, 7 Aug 2020 02:18:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0232.hostedemail.com [216.40.44.232]) by kanga.kvack.org (Postfix) with ESMTP id 193288D000F for ; Fri, 7 Aug 2020 02:18:09 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D0FAF180AD804 for ; Fri, 7 Aug 2020 06:18:08 +0000 (UTC) X-FDA: 77122767456.11.jump13_5600e7d26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id AB577180F8B80 for ; Fri, 7 Aug 2020 06:18:08 +0000 (UTC) X-Spam-Summary: 50,0,0,9c047ea067ce22a9,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:12:41:355:379:800:960:967:973:983:988:989:1208:1260:1263:1345:1359:1381:1431:1437:1535:1543:1711:1730:1747:1777:1792:1801:2198:2199:2393:2525:2538:2553:2568:2682:2685:2687:2731:2859:2892:2902:2933:2937:2939:2942:2945:2947:2951:2954:3000:3022:3138:3139:3140:3141:3142:3150:3354:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4605:5007:6117:6119:6261:6653:6737:7576:7901:7904:8599:8784:8814:9025:9072:9121:9388:9545:10004:10049:10913:11232:11257:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12783:12986:13846:14181:14721:14849:21080:21433:21451:21627:21740:21781:21881:21939:21990:30012:30034:30054:30064:30067:30075:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y844pychm138mh6qsnxjs83wqbkoca9fg4oafsibqkgwayynua3bauare39ng.b48kdi7uefsxxxccpfms8efhoam8uqhyaz51g9nwr17uanxkou e37cmt7f X-HE-Tag: jump13_5600e7d26fbe X-Filterd-Recvd-Size: 5112 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:08 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B449D22CAF; Fri, 7 Aug 2020 06:18:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781087; bh=inwxXJOzW583L1TEnOmq/+82ALgllUdI9JsbQA1l81U=; h=Date:From:To:Subject:In-Reply-To:From; b=m644SMHR10wQKU+cYSNoMEdqdoZM80D1RU/8k0tMoEA9zevo+gNWwTf0SuTwhjKM+ +0u+tA0Bo+jjhOwqeXTrNoK3VhpskHIW2OHCNIwrcGIyr6bNYJAaIpS47Io0hJ+jgO 6bcoeNANxTznKc6cC8D4orTo+v+TxUyR1ynnWXsk= Date: Thu, 06 Aug 2020 23:18:06 -0700 From: Andrew Morton To: akpm@linux-foundation.org, gechangwei@live.cn, ghe@suse.com, grandmaster@al2klimov.de, jlbec@evilplan.org, joseph.qi@linux.alibaba.com, junxiao.bi@oracle.com, linux-mm@kvack.org, mark@fasheh.com, mm-commits@vger.kernel.org, piaojun@huawei.com, torvalds@linux-foundation.org Subject: [patch 020/163] ocfs2: replace HTTP links with HTTPS ones Message-ID: <20200807061806.uYzfSKgwM%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: AB577180F8B80 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000128, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Alexander A. Klimov" Subject: ocfs2: replace HTTP links with HTTPS ones Rationale: Reduces attack surface on kernel devs opening the links for MITM as HTTPS traffic is much harder to manipulate. Deterministic algorithm: For each file: If not .svg: For each line: If doesn't contain `xmlns`: For each link, `http://[^# ]*(?:\w|/)`: If neither `gnu\.org/license`, nor `mozilla\.org/MPL`: If both the HTTP and HTTPS versions return 200 OK and serve the same content: Replace HTTP with HTTPS. Link: http://lkml.kernel.org/r/20200713174456.36596-1-grandmaster@al2klimov.de Signed-off-by: Alexander A. Klimov Acked-by: Joseph Qi Cc: Mark Fasheh Cc: Joel Becker Cc: Junxiao Bi Cc: Changwei Ge Cc: Gang He Cc: Jun Piao Signed-off-by: Andrew Morton --- Documentation/filesystems/dlmfs.rst | 2 +- Documentation/filesystems/ocfs2.rst | 2 +- fs/ocfs2/Kconfig | 6 +++--- fs/ocfs2/blockcheck.c | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) --- a/Documentation/filesystems/dlmfs.rst~ocfs2-replace-http-links-with-https-ones +++ a/Documentation/filesystems/dlmfs.rst @@ -12,7 +12,7 @@ dlmfs is built with OCFS2 as it requires :Project web page: http://ocfs2.wiki.kernel.org :Tools web page: https://github.com/markfasheh/ocfs2-tools -:OCFS2 mailing lists: http://oss.oracle.com/projects/ocfs2/mailman/ +:OCFS2 mailing lists: https://oss.oracle.com/projects/ocfs2/mailman/ All code copyright 2005 Oracle except when otherwise noted. --- a/Documentation/filesystems/ocfs2.rst~ocfs2-replace-http-links-with-https-ones +++ a/Documentation/filesystems/ocfs2.rst @@ -14,7 +14,7 @@ get "mount.ocfs2" and "ocfs2_hb_ctl". Project web page: http://ocfs2.wiki.kernel.org Tools git tree: https://github.com/markfasheh/ocfs2-tools -OCFS2 mailing lists: http://oss.oracle.com/projects/ocfs2/mailman/ +OCFS2 mailing lists: https://oss.oracle.com/projects/ocfs2/mailman/ All code copyright 2005 Oracle except when otherwise noted. --- a/fs/ocfs2/blockcheck.c~ocfs2-replace-http-links-with-https-ones +++ a/fs/ocfs2/blockcheck.c @@ -124,7 +124,7 @@ u32 ocfs2_hamming_encode(u32 parity, voi * parity bits that are part of the bit number * representation. Huh? * - * + * * In other words, the parity bit at position 2^k * checks bits in positions having bit k set in * their binary representation. Conversely, for --- a/fs/ocfs2/Kconfig~ocfs2-replace-http-links-with-https-ones +++ a/fs/ocfs2/Kconfig @@ -16,9 +16,9 @@ config OCFS2_FS You'll want to install the ocfs2-tools package in order to at least get "mount.ocfs2". - Project web page: http://oss.oracle.com/projects/ocfs2 - Tools web page: http://oss.oracle.com/projects/ocfs2-tools - OCFS2 mailing lists: http://oss.oracle.com/projects/ocfs2/mailman/ + Project web page: https://oss.oracle.com/projects/ocfs2 + Tools web page: https://oss.oracle.com/projects/ocfs2-tools + OCFS2 mailing lists: https://oss.oracle.com/projects/ocfs2/mailman/ For more information on OCFS2, see the file . From patchwork Fri Aug 7 06:18:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704719 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4051F13B6 for ; Fri, 7 Aug 2020 06:18:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0ECA422D03 for ; Fri, 7 Aug 2020 06:18:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="D0QL0E51" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0ECA422D03 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D724B8D0016; Fri, 7 Aug 2020 02:18:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D21418D000F; Fri, 7 Aug 2020 02:18:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BEA748D0016; Fri, 7 Aug 2020 02:18:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0215.hostedemail.com [216.40.44.215]) by kanga.kvack.org (Postfix) with ESMTP id A638D8D000F for ; Fri, 7 Aug 2020 02:18:12 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 70F5A3641 for ; Fri, 7 Aug 2020 06:18:12 +0000 (UTC) X-FDA: 77122767624.04.chin25_4a1550e26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 4BA3B800DE0E for ; Fri, 7 Aug 2020 06:18:12 +0000 (UTC) X-Spam-Summary: 1,0,0,5d3ab7b9194e898b,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2553:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3150:3352:3871:3872:3874:3876:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6117:6119:6261:6653:6737:7576:7903:8599:9025:9389:9545:10004:10913:11026:11658:11914:12043:12048:12297:12517:12519:12555:12679:12696:12737:12783:12986:13069:13311:13357:14093:14181:14384:14721:14849:21080:21451:21627:21939:21990:30046:30054:30064:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygr11bgnuetb5twj7xyead7pu9iypsffdr5hmtrjz61q71z7tpr3hc3xd39pu.1ypcsxdz8zrixqsg8jxq6ey7hdmrk3f6y39drewsh1dzyykyiauhp3y63azkqsa.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp ,MSBL:0, X-HE-Tag: chin25_4a1550e26fbe X-Filterd-Recvd-Size: 2856 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:11 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 490D9221E5; Fri, 7 Aug 2020 06:18:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781090; bh=YB1Z+IkKR3faue2WGlQzvP5jU376aGWYOOMblVshJ3M=; h=Date:From:To:Subject:In-Reply-To:From; b=D0QL0E512urQsEFTPPvLjrkiKuzOn8jGDRcCJG54OI2evWIN7ReQTt/6eS2mVrfBS lNnU48zzoTR9SnwbuIY/BOO1mJZ0I+cW3/1r6x56sVdX+HzRtohSENIur0Ul+7h4Xn Ml3mM/mS+YwNKfyu5C4eqXzHWW5fyGOPEJ0TySR4= Date: Thu, 06 Aug 2020 23:18:09 -0700 From: Andrew Morton To: akpm@linux-foundation.org, gechangwei@live.cn, ghe@suse.com, jlbec@evilplan.org, joseph.qi@linux.alibaba.com, junxiao.bi@oracle.com, linux-mm@kvack.org, mark@fasheh.com, mm-commits@vger.kernel.org, pavel@denx.de, pavel@ucw.cz, piaojun@huawei.com, torvalds@linux-foundation.org Subject: [patch 021/163] ocfs2: fix unbalanced locking Message-ID: <20200807061809.D9W48nBXK%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 4BA3B800DE0E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Pavel Machek Subject: ocfs2: fix unbalanced locking Based on what fails, function can return with nfs_sync_rwlock either locked or unlocked. That can not be right. Always return with lock unlocked on error. Link: http://lkml.kernel.org/r/20200724124443.GA28164@duo.ucw.cz Fixes: 4cd9973f9ff6 ("ocfs2: avoid inode removal while nfsd is accessing it") Signed-off-by: Pavel Machek (CIP) Reviewed-by: Joseph Qi Reviewed-by: Andrew Morton Cc: Mark Fasheh Cc: Joel Becker Cc: Junxiao Bi Cc: Changwei Ge Cc: Gang He Cc: Jun Piao Signed-off-by: Andrew Morton --- fs/ocfs2/dlmglue.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) --- a/fs/ocfs2/dlmglue.c~ocfs2-fix-unbalanced-locking +++ a/fs/ocfs2/dlmglue.c @@ -2871,9 +2871,15 @@ int ocfs2_nfs_sync_lock(struct ocfs2_sup status = ocfs2_cluster_lock(osb, lockres, ex ? LKM_EXMODE : LKM_PRMODE, 0, 0); - if (status < 0) + if (status < 0) { mlog(ML_ERROR, "lock on nfs sync lock failed %d\n", status); + if (ex) + up_write(&osb->nfs_sync_rwlock); + else + up_read(&osb->nfs_sync_rwlock); + } + return status; } From patchwork Fri Aug 7 06:18:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704721 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B094314B7 for ; Fri, 7 Aug 2020 06:18:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 337EC22D0A for ; Fri, 7 Aug 2020 06:18:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="rKmNCYk1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 337EC22D0A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CD54C8D0017; Fri, 7 Aug 2020 02:18:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C5C398D000F; Fri, 7 Aug 2020 02:18:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB2DB8D0017; Fri, 7 Aug 2020 02:18:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0162.hostedemail.com [216.40.44.162]) by kanga.kvack.org (Postfix) with ESMTP id 844B28D000F for ; Fri, 7 Aug 2020 02:18:17 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 44FAA3644 for ; Fri, 7 Aug 2020 06:18:17 +0000 (UTC) X-FDA: 77122767834.30.pail10_150bdce26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 0AF92180B3AA7 for ; Fri, 7 Aug 2020 06:18:17 +0000 (UTC) X-Spam-Summary: 1,0,0,6a1bd3adf1c5c9a1,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:355:379:960:965:966:967:968:973:982:988:989:1252:1256:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1801:2194:2196:2199:2200:2393:2525:2538:2559:2563:2682:2685:2736:2859:2892:2894:2898:2902:2910:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4225:4250:4321:4385:4390:4395:4605:5007:6261:6653:6691:6737:7576:7774:7875:7903:7904:8557:8599:8603:8957:9010:9025:9121:9163:9545:10004:10044:10798:10913:11026:11233:11657:11914:12043:12048:12294:12296:12297:12438:12517:12519:12555:12679:12681:12702:12737:12783:12986:13149:13161:13229:13230:13846:13869:13972:14196:21080:21220:21433:21450:21451:21611:21622:21740:21795:21796:21939:21987:21990:30005:30012:30017:30029:30037:30045:30046:30051:30052:30054:30055:30062:30064:30070:30075:30080,0,RBL:198.145.29.99:@linux-foundation. org:.lbl X-HE-Tag: pail10_150bdce26fbe X-Filterd-Recvd-Size: 88872 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:15 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EE90522D00; Fri, 7 Aug 2020 06:18:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781094; bh=HDiryqwPtlS6DqbjxiFWAs4nGTNv6fRDnWEvUohcalU=; h=Date:From:To:Subject:In-Reply-To:From; b=rKmNCYk1msRStPHOOYcL85b1WNzppjZb8VN4b1YyOXA71IswD6G3gAj87k+rDhtX3 /6RHenSc5Wm0Hj2LM/rMJxGP8lXbDGGvDkflCUZaXqFKyDw/nEfxPSoB9LqwxUqPq8 Vwa8Tgdit8od820HFJ6vBCRfc7bTEkhoMsvLE/Fo= Date: Thu, 06 Aug 2020 23:18:13 -0700 From: Andrew Morton To: akpm@linux-foundation.org, dan.carpenter@oracle.com, dhowells@redhat.com, hannes@cmpxchg.org, jarkko.sakkinen@linux.intel.com, Jason@zx2c4.com, jmorris@namei.org, joe@perches.com, linux-mm@kvack.org, longman@redhat.com, mhocko@suse.com, mm-commits@vger.kernel.org, rientjes@google.com, serge@hallyn.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 022/163] mm, treewide: rename kzfree() to kfree_sensitive() Message-ID: <20200807061813.5vUCQzAOw%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 0AF92180B3AA7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Waiman Long Subject: mm, treewide: rename kzfree() to kfree_sensitive() As said by Linus: A symmetric naming is only helpful if it implies symmetries in use. Otherwise it's actively misleading. In "kzalloc()", the z is meaningful and an important part of what the caller wants. In "kzfree()", the z is actively detrimental, because maybe in the future we really _might_ want to use that "memfill(0xdeadbeef)" or something. The "zero" part of the interface isn't even _relevant_. The main reason that kzfree() exists is to clear sensitive information that should not be leaked to other future users of the same memory objects. Rename kzfree() to kfree_sensitive() to follow the example of the recently added kvfree_sensitive() and make the intention of the API more explicit. In addition, memzero_explicit() is used to clear the memory to make sure that it won't get optimized away by the compiler. The renaming is done by using the command sequence: git grep -w --name-only kzfree |\ xargs sed -i 's/kzfree/kfree_sensitive/' followed by some editing of the kfree_sensitive() kerneldoc and adding a kzfree backward compatibility macro in slab.h. [akpm@linux-foundation.org: fs/crypto/inline_crypt.c needs linux/slab.h] [akpm@linux-foundation.org: fix fs/crypto/inline_crypt.c some more] Link: http://lkml.kernel.org/r/20200616154311.12314-3-longman@redhat.com Suggested-by: Joe Perches Acked-by: David Howells Acked-by: Michal Hocko Acked-by: Johannes Weiner Signed-off-by: Waiman Long Cc: Jarkko Sakkinen Cc: James Morris Cc: "Serge E. Hallyn" Cc: Joe Perches Cc: Matthew Wilcox Cc: David Rientjes Cc: Dan Carpenter Cc: "Jason A . Donenfeld" Signed-off-by: Andrew Morton --- arch/s390/crypto/prng.c | 4 - arch/x86/power/hibernate.c | 2 crypto/adiantum.c | 2 crypto/ahash.c | 4 - crypto/api.c | 2 crypto/asymmetric_keys/verify_pefile.c | 4 - crypto/deflate.c | 2 crypto/drbg.c | 10 +- crypto/ecc.c | 8 +- crypto/ecdh.c | 2 crypto/gcm.c | 2 crypto/gf128mul.c | 4 - crypto/jitterentropy-kcapi.c | 2 crypto/rng.c | 2 crypto/rsa-pkcs1pad.c | 6 - crypto/seqiv.c | 2 crypto/shash.c | 2 crypto/skcipher.c | 2 crypto/testmgr.c | 6 - crypto/zstd.c | 2 drivers/crypto/allwinner/sun8i-ce/sun8i-ce-cipher.c | 2 drivers/crypto/allwinner/sun8i-ss/sun8i-ss-cipher.c | 2 drivers/crypto/amlogic/amlogic-gxl-cipher.c | 4 - drivers/crypto/atmel-ecc.c | 2 drivers/crypto/caam/caampkc.c | 28 ++++---- drivers/crypto/cavium/cpt/cptvf_main.c | 6 - drivers/crypto/cavium/cpt/cptvf_reqmanager.c | 12 +-- drivers/crypto/cavium/nitrox/nitrox_lib.c | 4 - drivers/crypto/cavium/zip/zip_crypto.c | 6 - drivers/crypto/ccp/ccp-crypto-rsa.c | 6 - drivers/crypto/ccree/cc_aead.c | 4 - drivers/crypto/ccree/cc_buffer_mgr.c | 4 - drivers/crypto/ccree/cc_cipher.c | 6 - drivers/crypto/ccree/cc_hash.c | 8 +- drivers/crypto/ccree/cc_request_mgr.c | 2 drivers/crypto/marvell/cesa/hash.c | 2 drivers/crypto/marvell/octeontx/otx_cptvf_main.c | 6 - drivers/crypto/marvell/octeontx/otx_cptvf_reqmgr.h | 2 drivers/crypto/nx/nx.c | 4 - drivers/crypto/virtio/virtio_crypto_algs.c | 12 +-- drivers/crypto/virtio/virtio_crypto_core.c | 2 drivers/md/dm-crypt.c | 32 ++++----- drivers/md/dm-integrity.c | 6 - drivers/misc/ibmvmc.c | 6 - drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mbx.c | 2 drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c | 6 - drivers/net/ppp/ppp_mppe.c | 6 - drivers/net/wireguard/noise.c | 4 - drivers/net/wireguard/peer.c | 2 drivers/net/wireless/intel/iwlwifi/pcie/rx.c | 2 drivers/net/wireless/intel/iwlwifi/pcie/tx-gen2.c | 6 - drivers/net/wireless/intel/iwlwifi/pcie/tx.c | 6 - drivers/net/wireless/intersil/orinoco/wext.c | 4 - drivers/s390/crypto/ap_bus.h | 4 - drivers/staging/ks7010/ks_hostif.c | 2 drivers/staging/rtl8723bs/core/rtw_security.c | 2 drivers/staging/wlan-ng/p80211netdev.c | 2 drivers/target/iscsi/iscsi_target_auth.c | 2 fs/cifs/cifsencrypt.c | 2 fs/cifs/connect.c | 10 +- fs/cifs/dfs_cache.c | 2 fs/cifs/misc.c | 8 +- fs/crypto/inline_crypt.c | 5 - fs/crypto/keyring.c | 6 - fs/crypto/keysetup_v1.c | 4 - fs/ecryptfs/keystore.c | 4 - fs/ecryptfs/messaging.c | 2 include/crypto/aead.h | 2 include/crypto/akcipher.h | 2 include/crypto/gf128mul.h | 2 include/crypto/hash.h | 2 include/crypto/internal/acompress.h | 2 include/crypto/kpp.h | 2 include/crypto/skcipher.h | 2 include/linux/slab.h | 4 - lib/mpi/mpiutil.c | 6 - lib/test_kasan.c | 6 - mm/slab_common.c | 8 +- net/atm/mpoa_caches.c | 4 - net/bluetooth/ecdh_helper.c | 6 - net/bluetooth/smp.c | 24 +++---- net/core/sock.c | 2 net/ipv4/tcp_fastopen.c | 2 net/mac80211/aead_api.c | 4 - net/mac80211/aes_gmac.c | 2 net/mac80211/key.c | 2 net/mac802154/llsec.c | 20 ++--- net/sctp/auth.c | 2 net/sunrpc/auth_gss/gss_krb5_crypto.c | 4 - net/sunrpc/auth_gss/gss_krb5_keys.c | 6 - net/sunrpc/auth_gss/gss_krb5_mech.c | 2 net/tipc/crypto.c | 10 +- net/wireless/core.c | 2 net/wireless/ibss.c | 4 - net/wireless/lib80211_crypt_tkip.c | 2 net/wireless/lib80211_crypt_wep.c | 2 net/wireless/nl80211.c | 24 +++---- net/wireless/sme.c | 6 - net/wireless/util.c | 2 net/wireless/wext-sme.c | 2 scripts/coccinelle/free/devm_free.cocci | 4 - scripts/coccinelle/free/ifnullfree.cocci | 4 - scripts/coccinelle/free/kfree.cocci | 6 - scripts/coccinelle/free/kfreeaddr.cocci | 2 security/apparmor/domain.c | 4 - security/apparmor/include/file.h | 2 security/apparmor/policy.c | 24 +++---- security/apparmor/policy_ns.c | 6 - security/apparmor/policy_unpack.c | 14 ++-- security/keys/big_key.c | 6 - security/keys/dh.c | 14 ++-- security/keys/encrypted-keys/encrypted.c | 14 ++-- security/keys/trusted-keys/trusted_tpm1.c | 34 +++++----- security/keys/user_defined.c | 6 - 114 files changed, 323 insertions(+), 320 deletions(-) --- a/arch/s390/crypto/prng.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/arch/s390/crypto/prng.c @@ -249,7 +249,7 @@ static void prng_tdes_deinstantiate(void { pr_debug("The prng module stopped " "after running in triple DES mode\n"); - kzfree(prng_data); + kfree_sensitive(prng_data); } @@ -442,7 +442,7 @@ outfree: static void prng_sha512_deinstantiate(void) { pr_debug("The prng module stopped after running in SHA-512 mode\n"); - kzfree(prng_data); + kfree_sensitive(prng_data); } --- a/arch/x86/power/hibernate.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/arch/x86/power/hibernate.c @@ -98,7 +98,7 @@ static int get_e820_md5(struct e820_tabl if (crypto_shash_digest(desc, (u8 *)table, size, buf)) ret = -EINVAL; - kzfree(desc); + kfree_sensitive(desc); free_tfm: crypto_free_shash(tfm); --- a/crypto/adiantum.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/adiantum.c @@ -177,7 +177,7 @@ static int adiantum_setkey(struct crypto keyp += NHPOLY1305_KEY_SIZE; WARN_ON(keyp != &data->derived_keys[ARRAY_SIZE(data->derived_keys)]); out: - kzfree(data); + kfree_sensitive(data); return err; } --- a/crypto/ahash.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/ahash.c @@ -183,7 +183,7 @@ static int ahash_setkey_unaligned(struct alignbuffer = (u8 *)ALIGN((unsigned long)buffer, alignmask + 1); memcpy(alignbuffer, key, keylen); ret = tfm->setkey(tfm, alignbuffer, keylen); - kzfree(buffer); + kfree_sensitive(buffer); return ret; } @@ -302,7 +302,7 @@ static void ahash_restore_req(struct aha req->priv = NULL; /* Free the req->priv.priv from the ADJUSTED request. */ - kzfree(priv); + kfree_sensitive(priv); } static void ahash_notify_einprogress(struct ahash_request *req) --- a/crypto/api.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/api.c @@ -571,7 +571,7 @@ void crypto_destroy_tfm(void *mem, struc alg->cra_exit(tfm); crypto_exit_ops(tfm); crypto_mod_put(alg); - kzfree(mem); + kfree_sensitive(mem); } EXPORT_SYMBOL_GPL(crypto_destroy_tfm); --- a/crypto/asymmetric_keys/verify_pefile.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/asymmetric_keys/verify_pefile.c @@ -376,7 +376,7 @@ static int pefile_digest_pe(const void * } error: - kzfree(desc); + kfree_sensitive(desc); error_no_desc: crypto_free_shash(tfm); kleave(" = %d", ret); @@ -447,6 +447,6 @@ int verify_pefile_signature(const void * ret = pefile_digest_pe(pebuf, pelen, &ctx); error: - kzfree(ctx.digest); + kfree_sensitive(ctx.digest); return ret; } --- a/crypto/deflate.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/deflate.c @@ -163,7 +163,7 @@ static void __deflate_exit(void *ctx) static void deflate_free_ctx(struct crypto_scomp *tfm, void *ctx) { __deflate_exit(ctx); - kzfree(ctx); + kfree_sensitive(ctx); } static void deflate_exit(struct crypto_tfm *tfm) --- a/crypto/drbg.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/drbg.c @@ -1218,19 +1218,19 @@ static inline void drbg_dealloc_state(st { if (!drbg) return; - kzfree(drbg->Vbuf); + kfree_sensitive(drbg->Vbuf); drbg->Vbuf = NULL; drbg->V = NULL; - kzfree(drbg->Cbuf); + kfree_sensitive(drbg->Cbuf); drbg->Cbuf = NULL; drbg->C = NULL; - kzfree(drbg->scratchpadbuf); + kfree_sensitive(drbg->scratchpadbuf); drbg->scratchpadbuf = NULL; drbg->reseed_ctr = 0; drbg->d_ops = NULL; drbg->core = NULL; if (IS_ENABLED(CONFIG_CRYPTO_FIPS)) { - kzfree(drbg->prev); + kfree_sensitive(drbg->prev); drbg->prev = NULL; drbg->fips_primed = false; } @@ -1701,7 +1701,7 @@ static int drbg_fini_hash_kernel(struct struct sdesc *sdesc = (struct sdesc *)drbg->priv_data; if (sdesc) { crypto_free_shash(sdesc->shash.tfm); - kzfree(sdesc); + kfree_sensitive(sdesc); } drbg->priv_data = NULL; return 0; --- a/crypto/ecc.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/ecc.c @@ -67,7 +67,7 @@ static u64 *ecc_alloc_digits_space(unsig static void ecc_free_digits_space(u64 *space) { - kzfree(space); + kfree_sensitive(space); } static struct ecc_point *ecc_alloc_point(unsigned int ndigits) @@ -101,9 +101,9 @@ static void ecc_free_point(struct ecc_po if (!p) return; - kzfree(p->x); - kzfree(p->y); - kzfree(p); + kfree_sensitive(p->x); + kfree_sensitive(p->y); + kfree_sensitive(p); } static void vli_clear(u64 *vli, unsigned int ndigits) --- a/crypto/ecdh.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/ecdh.c @@ -124,7 +124,7 @@ static int ecdh_compute_value(struct kpp /* fall through */ free_all: - kzfree(shared_secret); + kfree_sensitive(shared_secret); free_pubkey: kfree(public_key); return ret; --- a/crypto/gcm.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/gcm.c @@ -139,7 +139,7 @@ static int crypto_gcm_setkey(struct cryp CRYPTO_TFM_REQ_MASK); err = crypto_ahash_setkey(ghash, (u8 *)&data->hash, sizeof(be128)); out: - kzfree(data); + kfree_sensitive(data); return err; } --- a/crypto/gf128mul.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/gf128mul.c @@ -304,8 +304,8 @@ void gf128mul_free_64k(struct gf128mul_6 int i; for (i = 0; i < 16; i++) - kzfree(t->t[i]); - kzfree(t); + kfree_sensitive(t->t[i]); + kfree_sensitive(t); } EXPORT_SYMBOL(gf128mul_free_64k); --- a/crypto/jitterentropy-kcapi.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/jitterentropy-kcapi.c @@ -57,7 +57,7 @@ void *jent_zalloc(unsigned int len) void jent_zfree(void *ptr) { - kzfree(ptr); + kfree_sensitive(ptr); } int jent_fips_enabled(void) --- a/crypto/rng.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/rng.c @@ -53,7 +53,7 @@ int crypto_rng_reset(struct crypto_rng * err = crypto_rng_alg(tfm)->seed(tfm, seed, slen); crypto_stats_rng_seed(alg, err); out: - kzfree(buf); + kfree_sensitive(buf); return err; } EXPORT_SYMBOL_GPL(crypto_rng_reset); --- a/crypto/rsa-pkcs1pad.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/rsa-pkcs1pad.c @@ -199,7 +199,7 @@ static int pkcs1pad_encrypt_sign_complet sg_copy_from_buffer(req->dst, sg_nents_for_len(req->dst, ctx->key_size), out_buf, ctx->key_size); - kzfree(out_buf); + kfree_sensitive(out_buf); out: req->dst_len = ctx->key_size; @@ -322,7 +322,7 @@ static int pkcs1pad_decrypt_complete(str out_buf + pos, req->dst_len); done: - kzfree(req_ctx->out_buf); + kfree_sensitive(req_ctx->out_buf); return err; } @@ -500,7 +500,7 @@ static int pkcs1pad_verify_complete(stru req->dst_len) != 0) err = -EKEYREJECTED; done: - kzfree(req_ctx->out_buf); + kfree_sensitive(req_ctx->out_buf); return err; } --- a/crypto/seqiv.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/seqiv.c @@ -33,7 +33,7 @@ static void seqiv_aead_encrypt_complete2 memcpy(req->iv, subreq->iv, crypto_aead_ivsize(geniv)); out: - kzfree(subreq->iv); + kfree_sensitive(subreq->iv); } static void seqiv_aead_encrypt_complete(struct crypto_async_request *base, --- a/crypto/shash.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/shash.c @@ -44,7 +44,7 @@ static int shash_setkey_unaligned(struct alignbuffer = (u8 *)ALIGN((unsigned long)buffer, alignmask + 1); memcpy(alignbuffer, key, keylen); err = shash->setkey(tfm, alignbuffer, keylen); - kzfree(buffer); + kfree_sensitive(buffer); return err; } --- a/crypto/skcipher.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/skcipher.c @@ -592,7 +592,7 @@ static int skcipher_setkey_unaligned(str alignbuffer = (u8 *)ALIGN((unsigned long)buffer, alignmask + 1); memcpy(alignbuffer, key, keylen); ret = cipher->setkey(tfm, alignbuffer, keylen); - kzfree(buffer); + kfree_sensitive(buffer); return ret; } --- a/crypto/testmgr.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/testmgr.c @@ -1744,7 +1744,7 @@ out: kfree(vec.plaintext); kfree(vec.digest); crypto_free_shash(generic_tfm); - kzfree(generic_desc); + kfree_sensitive(generic_desc); return err; } #else /* !CONFIG_CRYPTO_MANAGER_EXTRA_TESTS */ @@ -3665,7 +3665,7 @@ static int drbg_cavs_test(const struct d if (IS_ERR(drng)) { printk(KERN_ERR "alg: drbg: could not allocate DRNG handle for " "%s\n", driver); - kzfree(buf); + kfree_sensitive(buf); return -ENOMEM; } @@ -3712,7 +3712,7 @@ static int drbg_cavs_test(const struct d outbuf: crypto_free_rng(drng); - kzfree(buf); + kfree_sensitive(buf); return ret; } --- a/crypto/zstd.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/crypto/zstd.c @@ -137,7 +137,7 @@ static void __zstd_exit(void *ctx) static void zstd_free_ctx(struct crypto_scomp *tfm, void *ctx) { __zstd_exit(ctx); - kzfree(ctx); + kfree_sensitive(ctx); } static void zstd_exit(struct crypto_tfm *tfm) --- a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-cipher.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/allwinner/sun8i-ce/sun8i-ce-cipher.c @@ -254,7 +254,7 @@ theend_iv: offset = areq->cryptlen - ivsize; if (rctx->op_dir & CE_DECRYPTION) { memcpy(areq->iv, backup_iv, ivsize); - kzfree(backup_iv); + kfree_sensitive(backup_iv); } else { scatterwalk_map_and_copy(areq->iv, areq->dst, offset, ivsize, 0); --- a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-cipher.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/allwinner/sun8i-ss/sun8i-ss-cipher.c @@ -249,7 +249,7 @@ theend_iv: if (rctx->op_dir & SS_DECRYPTION) { memcpy(areq->iv, backup_iv, ivsize); memzero_explicit(backup_iv, ivsize); - kzfree(backup_iv); + kfree_sensitive(backup_iv); } else { scatterwalk_map_and_copy(areq->iv, areq->dst, offset, ivsize, 0); --- a/drivers/crypto/amlogic/amlogic-gxl-cipher.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/amlogic/amlogic-gxl-cipher.c @@ -252,8 +252,8 @@ static int meson_cipher(struct skcipher_ } } theend: - kzfree(bkeyiv); - kzfree(backup_iv); + kfree_sensitive(bkeyiv); + kfree_sensitive(backup_iv); return err; } --- a/drivers/crypto/atmel-ecc.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/atmel-ecc.c @@ -69,7 +69,7 @@ static void atmel_ecdh_done(struct atmel /* fall through */ free_work_data: - kzfree(work_data); + kfree_sensitive(work_data); kpp_request_complete(req, status); } --- a/drivers/crypto/caam/caampkc.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/caam/caampkc.c @@ -854,14 +854,14 @@ static int caam_rsa_dec(struct akcipher_ static void caam_rsa_free_key(struct caam_rsa_key *key) { - kzfree(key->d); - kzfree(key->p); - kzfree(key->q); - kzfree(key->dp); - kzfree(key->dq); - kzfree(key->qinv); - kzfree(key->tmp1); - kzfree(key->tmp2); + kfree_sensitive(key->d); + kfree_sensitive(key->p); + kfree_sensitive(key->q); + kfree_sensitive(key->dp); + kfree_sensitive(key->dq); + kfree_sensitive(key->qinv); + kfree_sensitive(key->tmp1); + kfree_sensitive(key->tmp2); kfree(key->e); kfree(key->n); memset(key, 0, sizeof(*key)); @@ -1018,17 +1018,17 @@ static void caam_rsa_set_priv_key_form(s return; free_dq: - kzfree(rsa_key->dq); + kfree_sensitive(rsa_key->dq); free_dp: - kzfree(rsa_key->dp); + kfree_sensitive(rsa_key->dp); free_tmp2: - kzfree(rsa_key->tmp2); + kfree_sensitive(rsa_key->tmp2); free_tmp1: - kzfree(rsa_key->tmp1); + kfree_sensitive(rsa_key->tmp1); free_q: - kzfree(rsa_key->q); + kfree_sensitive(rsa_key->q); free_p: - kzfree(rsa_key->p); + kfree_sensitive(rsa_key->p); } static int caam_rsa_set_priv_key(struct crypto_akcipher *tfm, const void *key, --- a/drivers/crypto/cavium/cpt/cptvf_main.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/cavium/cpt/cptvf_main.c @@ -74,7 +74,7 @@ static void cleanup_worker_threads(struc for (i = 0; i < cptvf->nr_queues; i++) tasklet_kill(&cwqe_info->vq_wqe[i].twork); - kzfree(cwqe_info); + kfree_sensitive(cwqe_info); cptvf->wqe_info = NULL; } @@ -88,7 +88,7 @@ static void free_pending_queues(struct p continue; /* free single queue */ - kzfree((queue->head)); + kfree_sensitive((queue->head)); queue->front = 0; queue->rear = 0; @@ -189,7 +189,7 @@ static void free_command_queues(struct c chunk->head = NULL; chunk->dma_addr = 0; hlist_del(&chunk->nextchunk); - kzfree(chunk); + kfree_sensitive(chunk); } queue->nchunks = 0; --- a/drivers/crypto/cavium/cpt/cptvf_reqmanager.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/cavium/cpt/cptvf_reqmanager.c @@ -305,12 +305,12 @@ static void do_request_cleanup(struct cp } } - kzfree(info->scatter_components); - kzfree(info->gather_components); - kzfree(info->out_buffer); - kzfree(info->in_buffer); - kzfree((void *)info->completion_addr); - kzfree(info); + kfree_sensitive(info->scatter_components); + kfree_sensitive(info->gather_components); + kfree_sensitive(info->out_buffer); + kfree_sensitive(info->in_buffer); + kfree_sensitive((void *)info->completion_addr); + kfree_sensitive(info); } static void do_post_process(struct cpt_vf *cptvf, struct cpt_info_buffer *info) --- a/drivers/crypto/cavium/nitrox/nitrox_lib.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/cavium/nitrox/nitrox_lib.c @@ -90,7 +90,7 @@ static void nitrox_free_aqm_queues(struc for (i = 0; i < ndev->nr_queues; i++) { nitrox_cmdq_cleanup(ndev->aqmq[i]); - kzfree(ndev->aqmq[i]); + kfree_sensitive(ndev->aqmq[i]); ndev->aqmq[i] = NULL; } } @@ -122,7 +122,7 @@ static int nitrox_alloc_aqm_queues(struc err = nitrox_cmdq_init(cmdq, AQM_Q_ALIGN_BYTES); if (err) { - kzfree(cmdq); + kfree_sensitive(cmdq); goto aqmq_fail; } ndev->aqmq[i] = cmdq; --- a/drivers/crypto/cavium/zip/zip_crypto.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/cavium/zip/zip_crypto.c @@ -260,7 +260,7 @@ void *zip_alloc_scomp_ctx_deflate(struct ret = zip_ctx_init(zip_ctx, 0); if (ret) { - kzfree(zip_ctx); + kfree_sensitive(zip_ctx); return ERR_PTR(ret); } @@ -279,7 +279,7 @@ void *zip_alloc_scomp_ctx_lzs(struct cry ret = zip_ctx_init(zip_ctx, 1); if (ret) { - kzfree(zip_ctx); + kfree_sensitive(zip_ctx); return ERR_PTR(ret); } @@ -291,7 +291,7 @@ void zip_free_scomp_ctx(struct crypto_sc struct zip_kernel_ctx *zip_ctx = ctx; zip_ctx_exit(zip_ctx); - kzfree(zip_ctx); + kfree_sensitive(zip_ctx); } int zip_scomp_compress(struct crypto_scomp *tfm, --- a/drivers/crypto/ccp/ccp-crypto-rsa.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/ccp/ccp-crypto-rsa.c @@ -112,13 +112,13 @@ static int ccp_check_key_length(unsigned static void ccp_rsa_free_key_bufs(struct ccp_ctx *ctx) { /* Clean up old key data */ - kzfree(ctx->u.rsa.e_buf); + kfree_sensitive(ctx->u.rsa.e_buf); ctx->u.rsa.e_buf = NULL; ctx->u.rsa.e_len = 0; - kzfree(ctx->u.rsa.n_buf); + kfree_sensitive(ctx->u.rsa.n_buf); ctx->u.rsa.n_buf = NULL; ctx->u.rsa.n_len = 0; - kzfree(ctx->u.rsa.d_buf); + kfree_sensitive(ctx->u.rsa.d_buf); ctx->u.rsa.d_buf = NULL; ctx->u.rsa.d_len = 0; } --- a/drivers/crypto/ccree/cc_aead.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/ccree/cc_aead.c @@ -448,7 +448,7 @@ static int cc_get_plain_hmac_key(struct if (dma_mapping_error(dev, key_dma_addr)) { dev_err(dev, "Mapping key va=0x%p len=%u for DMA failed\n", key, keylen); - kzfree(key); + kfree_sensitive(key); return -ENOMEM; } if (keylen > blocksize) { @@ -533,7 +533,7 @@ static int cc_get_plain_hmac_key(struct if (key_dma_addr) dma_unmap_single(dev, key_dma_addr, keylen, DMA_TO_DEVICE); - kzfree(key); + kfree_sensitive(key); return rc; } --- a/drivers/crypto/ccree/cc_buffer_mgr.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/ccree/cc_buffer_mgr.c @@ -488,7 +488,7 @@ void cc_unmap_aead_request(struct device if (areq_ctx->gen_ctx.iv_dma_addr) { dma_unmap_single(dev, areq_ctx->gen_ctx.iv_dma_addr, hw_iv_size, DMA_BIDIRECTIONAL); - kzfree(areq_ctx->gen_ctx.iv); + kfree_sensitive(areq_ctx->gen_ctx.iv); } /* Release pool */ @@ -559,7 +559,7 @@ static int cc_aead_chain_iv(struct cc_dr if (dma_mapping_error(dev, areq_ctx->gen_ctx.iv_dma_addr)) { dev_err(dev, "Mapping iv %u B at va=%pK for DMA failed\n", hw_iv_size, req->iv); - kzfree(areq_ctx->gen_ctx.iv); + kfree_sensitive(areq_ctx->gen_ctx.iv); areq_ctx->gen_ctx.iv = NULL; rc = -ENOMEM; goto chain_iv_exit; --- a/drivers/crypto/ccree/cc_cipher.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/ccree/cc_cipher.c @@ -257,7 +257,7 @@ static void cc_cipher_exit(struct crypto &ctx_p->user.key_dma_addr); /* Free key buffer in context */ - kzfree(ctx_p->user.key); + kfree_sensitive(ctx_p->user.key); dev_dbg(dev, "Free key buffer in context. key=@%p\n", ctx_p->user.key); } @@ -881,7 +881,7 @@ static void cc_cipher_complete(struct de /* Not a BACKLOG notification */ cc_unmap_cipher_request(dev, req_ctx, ivsize, src, dst); memcpy(req->iv, req_ctx->iv, ivsize); - kzfree(req_ctx->iv); + kfree_sensitive(req_ctx->iv); } skcipher_request_complete(req, err); @@ -994,7 +994,7 @@ static int cc_cipher_process(struct skci exit_process: if (rc != -EINPROGRESS && rc != -EBUSY) { - kzfree(req_ctx->iv); + kfree_sensitive(req_ctx->iv); } return rc; --- a/drivers/crypto/ccree/cc_hash.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/ccree/cc_hash.c @@ -764,7 +764,7 @@ static int cc_hash_setkey(struct crypto_ if (dma_mapping_error(dev, ctx->key_params.key_dma_addr)) { dev_err(dev, "Mapping key va=0x%p len=%u for DMA failed\n", ctx->key_params.key, keylen); - kzfree(ctx->key_params.key); + kfree_sensitive(ctx->key_params.key); return -ENOMEM; } dev_dbg(dev, "mapping key-buffer: key_dma_addr=%pad keylen=%u\n", @@ -913,7 +913,7 @@ out: &ctx->key_params.key_dma_addr, ctx->key_params.keylen); } - kzfree(ctx->key_params.key); + kfree_sensitive(ctx->key_params.key); return rc; } @@ -950,7 +950,7 @@ static int cc_xcbc_setkey(struct crypto_ if (dma_mapping_error(dev, ctx->key_params.key_dma_addr)) { dev_err(dev, "Mapping key va=0x%p len=%u for DMA failed\n", key, keylen); - kzfree(ctx->key_params.key); + kfree_sensitive(ctx->key_params.key); return -ENOMEM; } dev_dbg(dev, "mapping key-buffer: key_dma_addr=%pad keylen=%u\n", @@ -999,7 +999,7 @@ static int cc_xcbc_setkey(struct crypto_ dev_dbg(dev, "Unmapped key-buffer: key_dma_addr=%pad keylen=%u\n", &ctx->key_params.key_dma_addr, ctx->key_params.keylen); - kzfree(ctx->key_params.key); + kfree_sensitive(ctx->key_params.key); return rc; } --- a/drivers/crypto/ccree/cc_request_mgr.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/ccree/cc_request_mgr.c @@ -107,7 +107,7 @@ void cc_req_mgr_fini(struct cc_drvdata * /* Kill tasklet */ tasklet_kill(&req_mgr_h->comptask); #endif - kzfree(req_mgr_h); + kfree_sensitive(req_mgr_h); drvdata->request_mgr_handle = NULL; } --- a/drivers/crypto/marvell/cesa/hash.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/marvell/cesa/hash.c @@ -1157,7 +1157,7 @@ static int mv_cesa_ahmac_pad_init(struct } /* Set the memory region to 0 to avoid any leak. */ - kzfree(keydup); + kfree_sensitive(keydup); if (ret) return ret; --- a/drivers/crypto/marvell/octeontx/otx_cptvf_main.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/marvell/octeontx/otx_cptvf_main.c @@ -68,7 +68,7 @@ static void cleanup_worker_threads(struc for (i = 0; i < cptvf->num_queues; i++) tasklet_kill(&cwqe_info->vq_wqe[i].twork); - kzfree(cwqe_info); + kfree_sensitive(cwqe_info); cptvf->wqe_info = NULL; } @@ -82,7 +82,7 @@ static void free_pending_queues(struct o continue; /* free single queue */ - kzfree((queue->head)); + kfree_sensitive((queue->head)); queue->front = 0; queue->rear = 0; queue->qlen = 0; @@ -176,7 +176,7 @@ static void free_command_queues(struct o chunk->head = NULL; chunk->dma_addr = 0; list_del(&chunk->nextchunk); - kzfree(chunk); + kfree_sensitive(chunk); } queue->num_chunks = 0; queue->idx = 0; --- a/drivers/crypto/marvell/octeontx/otx_cptvf_reqmgr.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/marvell/octeontx/otx_cptvf_reqmgr.h @@ -215,7 +215,7 @@ static inline void do_request_cleanup(st DMA_BIDIRECTIONAL); } } - kzfree(info); + kfree_sensitive(info); } struct otx_cptvf_wqe; --- a/drivers/crypto/nx/nx.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/nx/nx.c @@ -746,7 +746,7 @@ void nx_crypto_ctx_exit(struct crypto_tf { struct nx_crypto_ctx *nx_ctx = crypto_tfm_ctx(tfm); - kzfree(nx_ctx->kmem); + kfree_sensitive(nx_ctx->kmem); nx_ctx->csbcpb = NULL; nx_ctx->csbcpb_aead = NULL; nx_ctx->in_sg = NULL; @@ -762,7 +762,7 @@ void nx_crypto_ctx_aead_exit(struct cryp { struct nx_crypto_ctx *nx_ctx = crypto_aead_ctx(tfm); - kzfree(nx_ctx->kmem); + kfree_sensitive(nx_ctx->kmem); } static int nx_probe(struct vio_dev *viodev, const struct vio_device_id *id) --- a/drivers/crypto/virtio/virtio_crypto_algs.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/virtio/virtio_crypto_algs.c @@ -167,7 +167,7 @@ static int virtio_crypto_alg_skcipher_in num_in, vcrypto, GFP_ATOMIC); if (err < 0) { spin_unlock(&vcrypto->ctrl_lock); - kzfree(cipher_key); + kfree_sensitive(cipher_key); return err; } virtqueue_kick(vcrypto->ctrl_vq); @@ -184,7 +184,7 @@ static int virtio_crypto_alg_skcipher_in spin_unlock(&vcrypto->ctrl_lock); pr_err("virtio_crypto: Create session failed status: %u\n", le32_to_cpu(vcrypto->input.status)); - kzfree(cipher_key); + kfree_sensitive(cipher_key); return -EINVAL; } @@ -197,7 +197,7 @@ static int virtio_crypto_alg_skcipher_in spin_unlock(&vcrypto->ctrl_lock); - kzfree(cipher_key); + kfree_sensitive(cipher_key); return 0; } @@ -472,9 +472,9 @@ __virtio_crypto_skcipher_do_req(struct v return 0; free_iv: - kzfree(iv); + kfree_sensitive(iv); free: - kzfree(req_data); + kfree_sensitive(req_data); kfree(sgs); return err; } @@ -583,7 +583,7 @@ static void virtio_crypto_skcipher_final scatterwalk_map_and_copy(req->iv, req->dst, req->cryptlen - AES_BLOCK_SIZE, AES_BLOCK_SIZE, 0); - kzfree(vc_sym_req->iv); + kfree_sensitive(vc_sym_req->iv); virtcrypto_clear_request(&vc_sym_req->base); crypto_finalize_skcipher_request(vc_sym_req->base.dataq->engine, --- a/drivers/crypto/virtio/virtio_crypto_core.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/crypto/virtio/virtio_crypto_core.c @@ -17,7 +17,7 @@ void virtcrypto_clear_request(struct virtio_crypto_request *vc_req) { if (vc_req) { - kzfree(vc_req->req_data); + kfree_sensitive(vc_req->req_data); kfree(vc_req->sgs); } } --- a/drivers/md/dm-crypt.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/md/dm-crypt.c @@ -407,7 +407,7 @@ static void crypt_iv_lmk_dtr(struct cryp crypto_free_shash(lmk->hash_tfm); lmk->hash_tfm = NULL; - kzfree(lmk->seed); + kfree_sensitive(lmk->seed); lmk->seed = NULL; } @@ -558,9 +558,9 @@ static void crypt_iv_tcw_dtr(struct cryp { struct iv_tcw_private *tcw = &cc->iv_gen_private.tcw; - kzfree(tcw->iv_seed); + kfree_sensitive(tcw->iv_seed); tcw->iv_seed = NULL; - kzfree(tcw->whitening); + kfree_sensitive(tcw->whitening); tcw->whitening = NULL; if (tcw->crc32_tfm && !IS_ERR(tcw->crc32_tfm)) @@ -994,8 +994,8 @@ static int crypt_iv_elephant(struct cryp kunmap_atomic(data); out: - kzfree(ks); - kzfree(es); + kfree_sensitive(ks); + kfree_sensitive(es); skcipher_request_free(req); return r; } @@ -2294,7 +2294,7 @@ static int crypt_set_keyring_key(struct key = request_key(type, key_desc + 1, NULL); if (IS_ERR(key)) { - kzfree(new_key_string); + kfree_sensitive(new_key_string); return PTR_ERR(key); } @@ -2304,7 +2304,7 @@ static int crypt_set_keyring_key(struct if (ret < 0) { up_read(&key->sem); key_put(key); - kzfree(new_key_string); + kfree_sensitive(new_key_string); return ret; } @@ -2318,10 +2318,10 @@ static int crypt_set_keyring_key(struct if (!ret) { set_bit(DM_CRYPT_KEY_VALID, &cc->flags); - kzfree(cc->key_string); + kfree_sensitive(cc->key_string); cc->key_string = new_key_string; } else - kzfree(new_key_string); + kfree_sensitive(new_key_string); return ret; } @@ -2382,7 +2382,7 @@ static int crypt_set_key(struct crypt_co clear_bit(DM_CRYPT_KEY_VALID, &cc->flags); /* wipe references to any kernel keyring key */ - kzfree(cc->key_string); + kfree_sensitive(cc->key_string); cc->key_string = NULL; /* Decode key from its hex representation. */ @@ -2414,7 +2414,7 @@ static int crypt_wipe_key(struct crypt_c return r; } - kzfree(cc->key_string); + kfree_sensitive(cc->key_string); cc->key_string = NULL; r = crypt_setkey(cc); memset(&cc->key, 0, cc->key_size * sizeof(u8)); @@ -2493,15 +2493,15 @@ static void crypt_dtr(struct dm_target * if (cc->dev) dm_put_device(ti, cc->dev); - kzfree(cc->cipher_string); - kzfree(cc->key_string); - kzfree(cc->cipher_auth); - kzfree(cc->authenc_key); + kfree_sensitive(cc->cipher_string); + kfree_sensitive(cc->key_string); + kfree_sensitive(cc->cipher_auth); + kfree_sensitive(cc->authenc_key); mutex_destroy(&cc->bio_alloc_lock); /* Must zero key material before freeing */ - kzfree(cc); + kfree_sensitive(cc); spin_lock(&dm_crypt_clients_lock); WARN_ON(!dm_crypt_clients_n); --- a/drivers/md/dm-integrity.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/md/dm-integrity.c @@ -3405,8 +3405,8 @@ static struct scatterlist **dm_integrity static void free_alg(struct alg_spec *a) { - kzfree(a->alg_string); - kzfree(a->key); + kfree_sensitive(a->alg_string); + kfree_sensitive(a->key); memset(a, 0, sizeof *a); } @@ -4337,7 +4337,7 @@ static void dm_integrity_dtr(struct dm_t for (i = 0; i < ic->journal_sections; i++) { struct skcipher_request *req = ic->sk_requests[i]; if (req) { - kzfree(req->iv); + kfree_sensitive(req->iv); skcipher_request_free(req); } } --- a/drivers/misc/ibmvmc.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/misc/ibmvmc.c @@ -286,7 +286,7 @@ static void *alloc_dma_buffer(struct vio if (dma_mapping_error(&vdev->dev, *dma_handle)) { *dma_handle = 0; - kzfree(buffer); + kfree_sensitive(buffer); return NULL; } @@ -310,7 +310,7 @@ static void free_dma_buffer(struct vio_d dma_unmap_single(&vdev->dev, dma_handle, size, DMA_BIDIRECTIONAL); /* deallocate memory */ - kzfree(vaddr); + kfree_sensitive(vaddr); } /** @@ -883,7 +883,7 @@ static int ibmvmc_close(struct inode *in spin_unlock_irqrestore(&hmc->lock, flags); } - kzfree(session); + kfree_sensitive(session); return rc; } --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mbx.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_mbx.c @@ -137,7 +137,7 @@ static void hclge_free_vector_ring_chain while (chain) { chain_tmp = chain->next; - kzfree(chain); + kfree_sensitive(chain); chain = chain_tmp; } } --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c @@ -960,9 +960,9 @@ int ixgbe_ipsec_vf_add_sa(struct ixgbe_a return 0; err_aead: - kzfree(xs->aead); + kfree_sensitive(xs->aead); err_xs: - kzfree(xs); + kfree_sensitive(xs); err_out: msgbuf[1] = err; return err; @@ -1047,7 +1047,7 @@ int ixgbe_ipsec_vf_del_sa(struct ixgbe_a ixgbe_ipsec_del_sa(xs); /* remove the xs that was made-up in the add request */ - kzfree(xs); + kfree_sensitive(xs); return 0; } --- a/drivers/net/ppp/ppp_mppe.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/net/ppp/ppp_mppe.c @@ -222,7 +222,7 @@ out_free: kfree(state->sha1_digest); if (state->sha1) { crypto_free_shash(state->sha1->tfm); - kzfree(state->sha1); + kfree_sensitive(state->sha1); } kfree(state); out: @@ -238,8 +238,8 @@ static void mppe_free(void *arg) if (state) { kfree(state->sha1_digest); crypto_free_shash(state->sha1->tfm); - kzfree(state->sha1); - kzfree(state); + kfree_sensitive(state->sha1); + kfree_sensitive(state); } } --- a/drivers/net/wireguard/noise.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/net/wireguard/noise.c @@ -114,7 +114,7 @@ static struct noise_keypair *keypair_cre static void keypair_free_rcu(struct rcu_head *rcu) { - kzfree(container_of(rcu, struct noise_keypair, rcu)); + kfree_sensitive(container_of(rcu, struct noise_keypair, rcu)); } static void keypair_free_kref(struct kref *kref) @@ -821,7 +821,7 @@ bool wg_noise_handshake_begin_session(st handshake->entry.peer->device->index_hashtable, &handshake->entry, &new_keypair->entry); } else { - kzfree(new_keypair); + kfree_sensitive(new_keypair); } rcu_read_unlock_bh(); --- a/drivers/net/wireguard/peer.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/net/wireguard/peer.c @@ -203,7 +203,7 @@ static void rcu_release(struct rcu_head /* The final zeroing takes care of clearing any remaining handshake key * material and other potentially sensitive information. */ - kzfree(peer); + kfree_sensitive(peer); } static void kref_release(struct kref *refcount) --- a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c @@ -1369,7 +1369,7 @@ static void iwl_pcie_rx_handle_rb(struct &rxcb, rxq->id); if (reclaim) { - kzfree(txq->entries[cmd_index].free_buf); + kfree_sensitive(txq->entries[cmd_index].free_buf); txq->entries[cmd_index].free_buf = NULL; } --- a/drivers/net/wireless/intel/iwlwifi/pcie/tx.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/net/wireless/intel/iwlwifi/pcie/tx.c @@ -721,8 +721,8 @@ static void iwl_pcie_txq_free(struct iwl /* De-alloc array of command/tx buffers */ if (txq_id == trans->txqs.cmd.q_id) for (i = 0; i < txq->n_window; i++) { - kzfree(txq->entries[i].cmd); - kzfree(txq->entries[i].free_buf); + kfree_sensitive(txq->entries[i].cmd); + kfree_sensitive(txq->entries[i].free_buf); } /* De-alloc circular buffer of TFDs */ @@ -1765,7 +1765,7 @@ static int iwl_pcie_enqueue_hcmd(struct BUILD_BUG_ON(IWL_TFH_NUM_TBS > sizeof(out_meta->tbs) * BITS_PER_BYTE); out_meta->flags = cmd->flags; if (WARN_ON_ONCE(txq->entries[idx].free_buf)) - kzfree(txq->entries[idx].free_buf); + kfree_sensitive(txq->entries[idx].free_buf); txq->entries[idx].free_buf = dup_buf; trace_iwlwifi_dev_hcmd(trans->dev, cmd, cmd_size, &out_cmd->hdr_wide); --- a/drivers/net/wireless/intel/iwlwifi/pcie/tx-gen2.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/net/wireless/intel/iwlwifi/pcie/tx-gen2.c @@ -1026,7 +1026,7 @@ static int iwl_pcie_gen2_enqueue_hcmd(st BUILD_BUG_ON(IWL_TFH_NUM_TBS > sizeof(out_meta->tbs) * BITS_PER_BYTE); out_meta->flags = cmd->flags; if (WARN_ON_ONCE(txq->entries[idx].free_buf)) - kzfree(txq->entries[idx].free_buf); + kfree_sensitive(txq->entries[idx].free_buf); txq->entries[idx].free_buf = dup_buf; trace_iwlwifi_dev_hcmd(trans->dev, cmd, cmd_size, &out_cmd->hdr_wide); @@ -1257,8 +1257,8 @@ static void iwl_pcie_gen2_txq_free(struc /* De-alloc array of command/tx buffers */ if (txq_id == trans->txqs.cmd.q_id) for (i = 0; i < txq->n_window; i++) { - kzfree(txq->entries[i].cmd); - kzfree(txq->entries[i].free_buf); + kfree_sensitive(txq->entries[i].cmd); + kfree_sensitive(txq->entries[i].free_buf); } del_timer_sync(&txq->stuck_timer); --- a/drivers/net/wireless/intersil/orinoco/wext.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/net/wireless/intersil/orinoco/wext.c @@ -31,8 +31,8 @@ static int orinoco_set_key(struct orinoc enum orinoco_alg alg, const u8 *key, int key_len, const u8 *seq, int seq_len) { - kzfree(priv->keys[index].key); - kzfree(priv->keys[index].seq); + kfree_sensitive(priv->keys[index].key); + kfree_sensitive(priv->keys[index].seq); if (key_len) { priv->keys[index].key = kzalloc(key_len, GFP_ATOMIC); --- a/drivers/s390/crypto/ap_bus.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/s390/crypto/ap_bus.h @@ -219,8 +219,8 @@ static inline void ap_init_message(struc */ static inline void ap_release_message(struct ap_message *ap_msg) { - kzfree(ap_msg->msg); - kzfree(ap_msg->private); + kfree_sensitive(ap_msg->msg); + kfree_sensitive(ap_msg->private); } /* --- a/drivers/staging/ks7010/ks_hostif.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/staging/ks7010/ks_hostif.c @@ -245,7 +245,7 @@ michael_mic(u8 *key, u8 *data, unsigned ret = crypto_shash_finup(desc, data + 12, len - 12, result); err_free_desc: - kzfree(desc); + kfree_sensitive(desc); err_free_tfm: crypto_free_shash(tfm); --- a/drivers/staging/rtl8723bs/core/rtw_security.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/staging/rtl8723bs/core/rtw_security.c @@ -2251,7 +2251,7 @@ static void gf_mulx(u8 *pad) static void aes_encrypt_deinit(void *ctx) { - kzfree(ctx); + kfree_sensitive(ctx); } --- a/drivers/staging/wlan-ng/p80211netdev.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/staging/wlan-ng/p80211netdev.c @@ -429,7 +429,7 @@ static netdev_tx_t p80211knetdev_hard_st failed: /* Free up the WEP buffer if it's not the same as the skb */ if ((p80211_wep.data) && (p80211_wep.data != skb->data)) - kzfree(p80211_wep.data); + kfree_sensitive(p80211_wep.data); /* we always free the skb here, never in a lower level. */ if (!result) --- a/drivers/target/iscsi/iscsi_target_auth.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/drivers/target/iscsi/iscsi_target_auth.c @@ -484,7 +484,7 @@ static int chap_server_compute_hash( pr_debug("[server] Sending CHAP_R=0x%s\n", response); auth_ret = 0; out: - kzfree(desc); + kfree_sensitive(desc); if (tfm) crypto_free_shash(tfm); kfree(initiatorchg); --- a/fs/cifs/cifsencrypt.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/fs/cifs/cifsencrypt.c @@ -797,7 +797,7 @@ calc_seckey(struct cifs_ses *ses) ses->auth_key.len = CIFS_SESS_KEY_SIZE; memzero_explicit(sec_key, CIFS_SESS_KEY_SIZE); - kzfree(ctx_arc4); + kfree_sensitive(ctx_arc4); return 0; } --- a/fs/cifs/connect.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/fs/cifs/connect.c @@ -2182,7 +2182,7 @@ cifs_parse_mount_options(const char *mou tmp_end++; if (!(tmp_end < end && tmp_end[1] == delim)) { /* No it is not. Set the password to NULL */ - kzfree(vol->password); + kfree_sensitive(vol->password); vol->password = NULL; break; } @@ -2220,7 +2220,7 @@ cifs_parse_mount_options(const char *mou options = end; } - kzfree(vol->password); + kfree_sensitive(vol->password); /* Now build new password string */ temp_len = strlen(value); vol->password = kzalloc(temp_len+1, GFP_KERNEL); @@ -3198,7 +3198,7 @@ cifs_set_cifscreds(struct smb_vol *vol, rc = -ENOMEM; kfree(vol->username); vol->username = NULL; - kzfree(vol->password); + kfree_sensitive(vol->password); vol->password = NULL; goto out_key_put; } @@ -4219,7 +4219,7 @@ void cifs_cleanup_volume_info_contents(struct smb_vol *volume_info) { kfree(volume_info->username); - kzfree(volume_info->password); + kfree_sensitive(volume_info->password); kfree(volume_info->UNC); kfree(volume_info->domainname); kfree(volume_info->iocharset); @@ -5345,7 +5345,7 @@ cifs_construct_tcon(struct cifs_sb_info out: kfree(vol_info->username); - kzfree(vol_info->password); + kfree_sensitive(vol_info->password); kfree(vol_info); return tcon; --- a/fs/cifs/dfs_cache.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/fs/cifs/dfs_cache.c @@ -1131,7 +1131,7 @@ err_free_domainname: err_free_unc: kfree(new->UNC); err_free_password: - kzfree(new->password); + kfree_sensitive(new->password); err_free_username: kfree(new->username); kfree(new); --- a/fs/cifs/misc.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/fs/cifs/misc.c @@ -103,12 +103,12 @@ sesInfoFree(struct cifs_ses *buf_to_free kfree(buf_to_free->serverOS); kfree(buf_to_free->serverDomain); kfree(buf_to_free->serverNOS); - kzfree(buf_to_free->password); + kfree_sensitive(buf_to_free->password); kfree(buf_to_free->user_name); kfree(buf_to_free->domainName); - kzfree(buf_to_free->auth_key.response); + kfree_sensitive(buf_to_free->auth_key.response); kfree(buf_to_free->iface_list); - kzfree(buf_to_free); + kfree_sensitive(buf_to_free); } struct cifs_tcon * @@ -148,7 +148,7 @@ tconInfoFree(struct cifs_tcon *buf_to_fr } atomic_dec(&tconInfoAllocCount); kfree(buf_to_free->nativeFileSystem); - kzfree(buf_to_free->password); + kfree_sensitive(buf_to_free->password); kfree(buf_to_free->crfid.fid); #ifdef CONFIG_CIFS_DFS_UPCALL kfree(buf_to_free->dfs_path); --- a/fs/crypto/inline_crypt.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/fs/crypto/inline_crypt.c @@ -16,6 +16,7 @@ #include #include #include +#include #include "fscrypt_private.h" @@ -187,7 +188,7 @@ int fscrypt_prepare_inline_crypt_key(str fail: for (i = 0; i < queue_refs; i++) blk_put_queue(blk_key->devs[i]); - kzfree(blk_key); + kfree_sensitive(blk_key); return err; } @@ -201,7 +202,7 @@ void fscrypt_destroy_inline_crypt_key(st blk_crypto_evict_key(blk_key->devs[i], &blk_key->base); blk_put_queue(blk_key->devs[i]); } - kzfree(blk_key); + kfree_sensitive(blk_key); } } --- a/fs/crypto/keyring.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/fs/crypto/keyring.c @@ -51,7 +51,7 @@ static void free_master_key(struct fscry } key_put(mk->mk_users); - kzfree(mk); + kfree_sensitive(mk); } static inline bool valid_key_spec(const struct fscrypt_key_specifier *spec) @@ -531,7 +531,7 @@ static int fscrypt_provisioning_key_prep static void fscrypt_provisioning_key_free_preparse( struct key_preparsed_payload *prep) { - kzfree(prep->payload.data[0]); + kfree_sensitive(prep->payload.data[0]); } static void fscrypt_provisioning_key_describe(const struct key *key, @@ -548,7 +548,7 @@ static void fscrypt_provisioning_key_des static void fscrypt_provisioning_key_destroy(struct key *key) { - kzfree(key->payload.data[0]); + kfree_sensitive(key->payload.data[0]); } static struct key_type key_type_fscrypt_provisioning = { --- a/fs/crypto/keysetup_v1.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/fs/crypto/keysetup_v1.c @@ -155,7 +155,7 @@ static void free_direct_key(struct fscry { if (dk) { fscrypt_destroy_prepared_key(&dk->dk_key); - kzfree(dk); + kfree_sensitive(dk); } } @@ -283,7 +283,7 @@ static int setup_v1_file_key_derived(str err = fscrypt_set_per_file_enc_key(ci, derived_key); out: - kzfree(derived_key); + kfree_sensitive(derived_key); return err; } --- a/fs/ecryptfs/keystore.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/fs/ecryptfs/keystore.c @@ -838,7 +838,7 @@ ecryptfs_write_tag_70_packet(char *dest, out_release_free_unlock: crypto_free_shash(s->hash_tfm); out_free_unlock: - kzfree(s->block_aligned_filename); + kfree_sensitive(s->block_aligned_filename); out_unlock: mutex_unlock(s->tfm_mutex); out: @@ -847,7 +847,7 @@ out: key_put(auth_tok_key); } skcipher_request_free(s->skcipher_req); - kzfree(s->hash_desc); + kfree_sensitive(s->hash_desc); kfree(s); return rc; } --- a/fs/ecryptfs/messaging.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/fs/ecryptfs/messaging.c @@ -175,7 +175,7 @@ int ecryptfs_exorcise_daemon(struct ecry } hlist_del(&daemon->euid_chain); mutex_unlock(&daemon->mux); - kzfree(daemon); + kfree_sensitive(daemon); out: return rc; } --- a/include/crypto/aead.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/include/crypto/aead.h @@ -425,7 +425,7 @@ static inline struct aead_request *aead_ */ static inline void aead_request_free(struct aead_request *req) { - kzfree(req); + kfree_sensitive(req); } /** --- a/include/crypto/akcipher.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/include/crypto/akcipher.h @@ -207,7 +207,7 @@ static inline struct akcipher_request *a */ static inline void akcipher_request_free(struct akcipher_request *req) { - kzfree(req); + kfree_sensitive(req); } /** --- a/include/crypto/gf128mul.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/include/crypto/gf128mul.h @@ -230,7 +230,7 @@ void gf128mul_4k_bbe(be128 *a, const str void gf128mul_x8_ble(le128 *r, const le128 *x); static inline void gf128mul_free_4k(struct gf128mul_4k *t) { - kzfree(t); + kfree_sensitive(t); } --- a/include/crypto/hash.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/include/crypto/hash.h @@ -606,7 +606,7 @@ static inline struct ahash_request *ahas */ static inline void ahash_request_free(struct ahash_request *req) { - kzfree(req); + kfree_sensitive(req); } static inline void ahash_request_zero(struct ahash_request *req) --- a/include/crypto/internal/acompress.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/include/crypto/internal/acompress.h @@ -46,7 +46,7 @@ static inline struct acomp_req *__acomp_ static inline void __acomp_request_free(struct acomp_req *req) { - kzfree(req); + kfree_sensitive(req); } /** --- a/include/crypto/kpp.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/include/crypto/kpp.h @@ -187,7 +187,7 @@ static inline struct kpp_request *kpp_re */ static inline void kpp_request_free(struct kpp_request *req) { - kzfree(req); + kfree_sensitive(req); } /** --- a/include/crypto/skcipher.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/include/crypto/skcipher.h @@ -508,7 +508,7 @@ static inline struct skcipher_request *s */ static inline void skcipher_request_free(struct skcipher_request *req) { - kzfree(req); + kfree_sensitive(req); } static inline void skcipher_request_zero(struct skcipher_request *req) --- a/include/linux/slab.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/include/linux/slab.h @@ -186,10 +186,12 @@ void memcg_deactivate_kmem_caches(struct */ void * __must_check krealloc(const void *, size_t, gfp_t); void kfree(const void *); -void kzfree(const void *); +void kfree_sensitive(const void *); size_t __ksize(const void *); size_t ksize(const void *); +#define kzfree(x) kfree_sensitive(x) /* For backward compatibility */ + #ifdef CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR void __check_heap_object(const void *ptr, unsigned long n, struct page *page, bool to_user); --- a/lib/mpi/mpiutil.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/lib/mpi/mpiutil.c @@ -69,7 +69,7 @@ void mpi_free_limb_space(mpi_ptr_t a) if (!a) return; - kzfree(a); + kfree_sensitive(a); } void mpi_assign_limb_space(MPI a, mpi_ptr_t ap, unsigned nlimbs) @@ -95,7 +95,7 @@ int mpi_resize(MPI a, unsigned nlimbs) if (!p) return -ENOMEM; memcpy(p, a->d, a->alloced * sizeof(mpi_limb_t)); - kzfree(a->d); + kfree_sensitive(a->d); a->d = p; } else { a->d = kcalloc(nlimbs, sizeof(mpi_limb_t), GFP_KERNEL); @@ -112,7 +112,7 @@ void mpi_free(MPI a) return; if (a->flags & 4) - kzfree(a->d); + kfree_sensitive(a->d); else mpi_free_limb_space(a->d); --- a/lib/test_kasan.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/lib/test_kasan.c @@ -766,15 +766,15 @@ static noinline void __init kmalloc_doub char *ptr; size_t size = 16; - pr_info("double-free (kzfree)\n"); + pr_info("double-free (kfree_sensitive)\n"); ptr = kmalloc(size, GFP_KERNEL); if (!ptr) { pr_err("Allocation failed\n"); return; } - kzfree(ptr); - kzfree(ptr); + kfree_sensitive(ptr); + kfree_sensitive(ptr); } #ifdef CONFIG_KASAN_VMALLOC --- a/mm/slab_common.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/mm/slab_common.c @@ -1729,17 +1729,17 @@ void *krealloc(const void *p, size_t new EXPORT_SYMBOL(krealloc); /** - * kzfree - like kfree but zero memory + * kfree_sensitive - Clear sensitive information in memory before freeing * @p: object to free memory of * * The memory of the object @p points to is zeroed before freed. - * If @p is %NULL, kzfree() does nothing. + * If @p is %NULL, kfree_sensitive() does nothing. * * Note: this function zeroes the whole allocated buffer which can be a good * deal bigger than the requested buffer size passed to kmalloc(). So be * careful when using this function in performance sensitive code. */ -void kzfree(const void *p) +void kfree_sensitive(const void *p) { size_t ks; void *mem = (void *)p; @@ -1750,7 +1750,7 @@ void kzfree(const void *p) memzero_explicit(mem, ks); kfree(mem); } -EXPORT_SYMBOL(kzfree); +EXPORT_SYMBOL(kfree_sensitive); /** * ksize - get the actual amount of memory allocated for a given object --- a/net/atm/mpoa_caches.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/atm/mpoa_caches.c @@ -180,7 +180,7 @@ static int cache_hit(in_cache_entry *ent static void in_cache_put(in_cache_entry *entry) { if (refcount_dec_and_test(&entry->use)) { - kzfree(entry); + kfree_sensitive(entry); } } @@ -415,7 +415,7 @@ static eg_cache_entry *eg_cache_get_by_s static void eg_cache_put(eg_cache_entry *entry) { if (refcount_dec_and_test(&entry->use)) { - kzfree(entry); + kfree_sensitive(entry); } } --- a/net/bluetooth/ecdh_helper.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/bluetooth/ecdh_helper.c @@ -104,7 +104,7 @@ int compute_ecdh_secret(struct crypto_kp free_all: kpp_request_free(req); free_tmp: - kzfree(tmp); + kfree_sensitive(tmp); return err; } @@ -151,9 +151,9 @@ int set_ecdh_privkey(struct crypto_kpp * err = crypto_kpp_set_secret(tfm, buf, buf_len); /* fall through */ free_all: - kzfree(buf); + kfree_sensitive(buf); free_tmp: - kzfree(tmp); + kfree_sensitive(tmp); return err; } --- a/net/bluetooth/smp.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/bluetooth/smp.c @@ -753,9 +753,9 @@ static void smp_chan_destroy(struct l2ca complete = test_bit(SMP_FLAG_COMPLETE, &smp->flags); mgmt_smp_complete(hcon, complete); - kzfree(smp->csrk); - kzfree(smp->slave_csrk); - kzfree(smp->link_key); + kfree_sensitive(smp->csrk); + kfree_sensitive(smp->slave_csrk); + kfree_sensitive(smp->link_key); crypto_free_shash(smp->tfm_cmac); crypto_free_kpp(smp->tfm_ecdh); @@ -789,7 +789,7 @@ static void smp_chan_destroy(struct l2ca } chan->data = NULL; - kzfree(smp); + kfree_sensitive(smp); hci_conn_drop(hcon); } @@ -1156,7 +1156,7 @@ static void sc_generate_link_key(struct const u8 salt[16] = { 0x31, 0x70, 0x6d, 0x74 }; if (smp_h7(smp->tfm_cmac, smp->tk, salt, smp->link_key)) { - kzfree(smp->link_key); + kfree_sensitive(smp->link_key); smp->link_key = NULL; return; } @@ -1165,14 +1165,14 @@ static void sc_generate_link_key(struct const u8 tmp1[4] = { 0x31, 0x70, 0x6d, 0x74 }; if (smp_h6(smp->tfm_cmac, smp->tk, tmp1, smp->link_key)) { - kzfree(smp->link_key); + kfree_sensitive(smp->link_key); smp->link_key = NULL; return; } } if (smp_h6(smp->tfm_cmac, smp->link_key, lebr, smp->link_key)) { - kzfree(smp->link_key); + kfree_sensitive(smp->link_key); smp->link_key = NULL; return; } @@ -1407,7 +1407,7 @@ static struct smp_chan *smp_chan_create( free_shash: crypto_free_shash(smp->tfm_cmac); zfree_smp: - kzfree(smp); + kfree_sensitive(smp); return NULL; } @@ -3278,7 +3278,7 @@ static struct l2cap_chan *smp_add_cid(st tfm_cmac = crypto_alloc_shash("cmac(aes)", 0, 0); if (IS_ERR(tfm_cmac)) { BT_ERR("Unable to create CMAC crypto context"); - kzfree(smp); + kfree_sensitive(smp); return ERR_CAST(tfm_cmac); } @@ -3286,7 +3286,7 @@ static struct l2cap_chan *smp_add_cid(st if (IS_ERR(tfm_ecdh)) { BT_ERR("Unable to create ECDH crypto context"); crypto_free_shash(tfm_cmac); - kzfree(smp); + kfree_sensitive(smp); return ERR_CAST(tfm_ecdh); } @@ -3300,7 +3300,7 @@ create_chan: if (smp) { crypto_free_shash(smp->tfm_cmac); crypto_free_kpp(smp->tfm_ecdh); - kzfree(smp); + kfree_sensitive(smp); } return ERR_PTR(-ENOMEM); } @@ -3347,7 +3347,7 @@ static void smp_del_chan(struct l2cap_ch chan->data = NULL; crypto_free_shash(smp->tfm_cmac); crypto_free_kpp(smp->tfm_ecdh); - kzfree(smp); + kfree_sensitive(smp); } l2cap_chan_put(chan); --- a/net/core/sock.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/core/sock.c @@ -2257,7 +2257,7 @@ static inline void __sock_kfree_s(struct if (WARN_ON_ONCE(!mem)) return; if (nullify) - kzfree(mem); + kfree_sensitive(mem); else kfree(mem); atomic_sub(size, &sk->sk_omem_alloc); --- a/net/ipv4/tcp_fastopen.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/ipv4/tcp_fastopen.c @@ -38,7 +38,7 @@ static void tcp_fastopen_ctx_free(struct struct tcp_fastopen_context *ctx = container_of(head, struct tcp_fastopen_context, rcu); - kzfree(ctx); + kfree_sensitive(ctx); } void tcp_fastopen_destroy_cipher(struct sock *sk) --- a/net/mac80211/aead_api.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/mac80211/aead_api.c @@ -41,7 +41,7 @@ int aead_encrypt(struct crypto_aead *tfm aead_request_set_ad(aead_req, sg[0].length); crypto_aead_encrypt(aead_req); - kzfree(aead_req); + kfree_sensitive(aead_req); return 0; } @@ -76,7 +76,7 @@ int aead_decrypt(struct crypto_aead *tfm aead_request_set_ad(aead_req, sg[0].length); err = crypto_aead_decrypt(aead_req); - kzfree(aead_req); + kfree_sensitive(aead_req); return err; } --- a/net/mac80211/aes_gmac.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/mac80211/aes_gmac.c @@ -60,7 +60,7 @@ int ieee80211_aes_gmac(struct crypto_aea aead_request_set_ad(aead_req, GMAC_AAD_LEN + data_len); crypto_aead_encrypt(aead_req); - kzfree(aead_req); + kfree_sensitive(aead_req); return 0; } --- a/net/mac80211/key.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/mac80211/key.c @@ -732,7 +732,7 @@ static void ieee80211_key_free_common(st ieee80211_aes_gcm_key_free(key->u.gcmp.tfm); break; } - kzfree(key); + kfree_sensitive(key); } static void __ieee80211_key_destroy(struct ieee80211_key *key, --- a/net/mac802154/llsec.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/mac802154/llsec.c @@ -49,7 +49,7 @@ void mac802154_llsec_destroy(struct mac8 msl = container_of(sl, struct mac802154_llsec_seclevel, level); list_del(&sl->list); - kzfree(msl); + kfree_sensitive(msl); } list_for_each_entry_safe(dev, dn, &sec->table.devices, list) { @@ -66,7 +66,7 @@ void mac802154_llsec_destroy(struct mac8 mkey = container_of(key->key, struct mac802154_llsec_key, key); list_del(&key->list); llsec_key_put(mkey); - kzfree(key); + kfree_sensitive(key); } } @@ -155,7 +155,7 @@ err_tfm: if (key->tfm[i]) crypto_free_aead(key->tfm[i]); - kzfree(key); + kfree_sensitive(key); return NULL; } @@ -170,7 +170,7 @@ static void llsec_key_release(struct kre crypto_free_aead(key->tfm[i]); crypto_free_sync_skcipher(key->tfm0); - kzfree(key); + kfree_sensitive(key); } static struct mac802154_llsec_key* @@ -261,7 +261,7 @@ int mac802154_llsec_key_add(struct mac80 return 0; fail: - kzfree(new); + kfree_sensitive(new); return -ENOMEM; } @@ -341,10 +341,10 @@ static void llsec_dev_free(struct mac802 devkey); list_del(&pos->list); - kzfree(devkey); + kfree_sensitive(devkey); } - kzfree(dev); + kfree_sensitive(dev); } int mac802154_llsec_dev_add(struct mac802154_llsec *sec, @@ -682,7 +682,7 @@ llsec_do_encrypt_auth(struct sk_buff *sk rc = crypto_aead_encrypt(req); - kzfree(req); + kfree_sensitive(req); return rc; } @@ -886,7 +886,7 @@ llsec_do_decrypt_auth(struct sk_buff *sk rc = crypto_aead_decrypt(req); - kzfree(req); + kfree_sensitive(req); skb_trim(skb, skb->len - authlen); return rc; @@ -926,7 +926,7 @@ llsec_update_devkey_record(struct mac802 if (!devkey) list_add_rcu(&next->devkey.list, &dev->dev.keys); else - kzfree(next); + kfree_sensitive(next); spin_unlock_bh(&dev->lock); } --- a/net/sctp/auth.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/sctp/auth.c @@ -49,7 +49,7 @@ void sctp_auth_key_put(struct sctp_auth_ return; if (refcount_dec_and_test(&key->refcnt)) { - kzfree(key); + kfree_sensitive(key); SCTP_DBG_OBJCNT_DEC(keys); } } --- a/net/sunrpc/auth_gss/gss_krb5_crypto.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/sunrpc/auth_gss/gss_krb5_crypto.c @@ -1003,7 +1003,7 @@ krb5_rc4_setup_seq_key(struct krb5_ctx * err = 0; out_err: - kzfree(desc); + kfree_sensitive(desc); crypto_free_shash(hmac); dprintk("%s: returning %d\n", __func__, err); return err; @@ -1079,7 +1079,7 @@ krb5_rc4_setup_enc_key(struct krb5_ctx * err = 0; out_err: - kzfree(desc); + kfree_sensitive(desc); crypto_free_shash(hmac); dprintk("%s: returning %d\n", __func__, err); return err; --- a/net/sunrpc/auth_gss/gss_krb5_keys.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/sunrpc/auth_gss/gss_krb5_keys.c @@ -228,11 +228,11 @@ u32 krb5_derive_key(const struct gss_krb ret = 0; err_free_raw: - kzfree(rawkey); + kfree_sensitive(rawkey); err_free_out: - kzfree(outblockdata); + kfree_sensitive(outblockdata); err_free_in: - kzfree(inblockdata); + kfree_sensitive(inblockdata); err_free_cipher: crypto_free_sync_skcipher(cipher); err_return: --- a/net/sunrpc/auth_gss/gss_krb5_mech.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/sunrpc/auth_gss/gss_krb5_mech.c @@ -443,7 +443,7 @@ context_derive_keys_rc4(struct krb5_ctx desc->tfm = hmac; err = crypto_shash_digest(desc, sigkeyconstant, slen, ctx->cksum); - kzfree(desc); + kfree_sensitive(desc); if (err) goto out_err_free_hmac; /* --- a/net/tipc/crypto.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/tipc/crypto.c @@ -441,7 +441,7 @@ static int tipc_aead_init(struct tipc_ae /* Allocate per-cpu TFM entry pointer */ tmp->tfm_entry = alloc_percpu(struct tipc_tfm *); if (!tmp->tfm_entry) { - kzfree(tmp); + kfree_sensitive(tmp); return -ENOMEM; } @@ -491,7 +491,7 @@ static int tipc_aead_init(struct tipc_ae /* Not any TFM is allocated? */ if (!tfm_cnt) { free_percpu(tmp->tfm_entry); - kzfree(tmp); + kfree_sensitive(tmp); return err; } @@ -545,7 +545,7 @@ static int tipc_aead_clone(struct tipc_a aead->tfm_entry = alloc_percpu_gfp(struct tipc_tfm *, GFP_ATOMIC); if (unlikely(!aead->tfm_entry)) { - kzfree(aead); + kfree_sensitive(aead); return -ENOMEM; } @@ -1352,7 +1352,7 @@ int tipc_crypto_start(struct tipc_crypto /* Allocate statistic structure */ c->stats = alloc_percpu_gfp(struct tipc_crypto_stats, GFP_ATOMIC); if (!c->stats) { - kzfree(c); + kfree_sensitive(c); return -ENOMEM; } @@ -1408,7 +1408,7 @@ void tipc_crypto_stop(struct tipc_crypto free_percpu(c->stats); *crypto = NULL; - kzfree(c); + kfree_sensitive(c); } void tipc_crypto_timeout(struct tipc_crypto *rx) --- a/net/wireless/core.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/wireless/core.c @@ -1125,7 +1125,7 @@ static void __cfg80211_unregister_wdev(s } #ifdef CONFIG_CFG80211_WEXT - kzfree(wdev->wext.keys); + kfree_sensitive(wdev->wext.keys); wdev->wext.keys = NULL; #endif /* only initialized if we have a netdev */ --- a/net/wireless/ibss.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/wireless/ibss.c @@ -127,7 +127,7 @@ int __cfg80211_join_ibss(struct cfg80211 return -EINVAL; if (WARN_ON(wdev->connect_keys)) - kzfree(wdev->connect_keys); + kfree_sensitive(wdev->connect_keys); wdev->connect_keys = connkeys; wdev->ibss_fixed = params->channel_fixed; @@ -161,7 +161,7 @@ static void __cfg80211_clear_ibss(struct ASSERT_WDEV_LOCK(wdev); - kzfree(wdev->connect_keys); + kfree_sensitive(wdev->connect_keys); wdev->connect_keys = NULL; rdev_set_qos_map(rdev, dev, NULL); --- a/net/wireless/lib80211_crypt_tkip.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/wireless/lib80211_crypt_tkip.c @@ -131,7 +131,7 @@ static void lib80211_tkip_deinit(void *p crypto_free_shash(_priv->tx_tfm_michael); crypto_free_shash(_priv->rx_tfm_michael); } - kzfree(priv); + kfree_sensitive(priv); } static inline u16 RotR1(u16 val) --- a/net/wireless/lib80211_crypt_wep.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/wireless/lib80211_crypt_wep.c @@ -56,7 +56,7 @@ static void *lib80211_wep_init(int keyid static void lib80211_wep_deinit(void *priv) { - kzfree(priv); + kfree_sensitive(priv); } /* Add WEP IV/key info to a frame that has at least 4 bytes of headroom */ --- a/net/wireless/nl80211.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/wireless/nl80211.c @@ -9836,7 +9836,7 @@ static int nl80211_join_ibss(struct sk_b if ((ibss.chandef.width != NL80211_CHAN_WIDTH_20_NOHT) && no_ht) { - kzfree(connkeys); + kfree_sensitive(connkeys); return -EINVAL; } } @@ -9848,7 +9848,7 @@ static int nl80211_join_ibss(struct sk_b int r = validate_pae_over_nl80211(rdev, info); if (r < 0) { - kzfree(connkeys); + kfree_sensitive(connkeys); return r; } @@ -9861,7 +9861,7 @@ static int nl80211_join_ibss(struct sk_b wdev_lock(dev->ieee80211_ptr); err = __cfg80211_join_ibss(rdev, dev, &ibss, connkeys); if (err) - kzfree(connkeys); + kfree_sensitive(connkeys); else if (info->attrs[NL80211_ATTR_SOCKET_OWNER]) dev->ieee80211_ptr->conn_owner_nlportid = info->snd_portid; wdev_unlock(dev->ieee80211_ptr); @@ -10289,7 +10289,7 @@ static int nl80211_connect(struct sk_buf if (info->attrs[NL80211_ATTR_HT_CAPABILITY]) { if (!info->attrs[NL80211_ATTR_HT_CAPABILITY_MASK]) { - kzfree(connkeys); + kfree_sensitive(connkeys); return -EINVAL; } memcpy(&connect.ht_capa, @@ -10307,7 +10307,7 @@ static int nl80211_connect(struct sk_buf if (info->attrs[NL80211_ATTR_VHT_CAPABILITY]) { if (!info->attrs[NL80211_ATTR_VHT_CAPABILITY_MASK]) { - kzfree(connkeys); + kfree_sensitive(connkeys); return -EINVAL; } memcpy(&connect.vht_capa, @@ -10321,7 +10321,7 @@ static int nl80211_connect(struct sk_buf (rdev->wiphy.features & NL80211_FEATURE_QUIET)) && !wiphy_ext_feature_isset(&rdev->wiphy, NL80211_EXT_FEATURE_RRM)) { - kzfree(connkeys); + kfree_sensitive(connkeys); return -EINVAL; } connect.flags |= ASSOC_REQ_USE_RRM; @@ -10329,21 +10329,21 @@ static int nl80211_connect(struct sk_buf connect.pbss = nla_get_flag(info->attrs[NL80211_ATTR_PBSS]); if (connect.pbss && !rdev->wiphy.bands[NL80211_BAND_60GHZ]) { - kzfree(connkeys); + kfree_sensitive(connkeys); return -EOPNOTSUPP; } if (info->attrs[NL80211_ATTR_BSS_SELECT]) { /* bss selection makes no sense if bssid is set */ if (connect.bssid) { - kzfree(connkeys); + kfree_sensitive(connkeys); return -EINVAL; } err = parse_bss_select(info->attrs[NL80211_ATTR_BSS_SELECT], wiphy, &connect.bss_select); if (err) { - kzfree(connkeys); + kfree_sensitive(connkeys); return err; } } @@ -10373,13 +10373,13 @@ static int nl80211_connect(struct sk_buf info->attrs[NL80211_ATTR_FILS_ERP_REALM] || info->attrs[NL80211_ATTR_FILS_ERP_NEXT_SEQ_NUM] || info->attrs[NL80211_ATTR_FILS_ERP_RRK]) { - kzfree(connkeys); + kfree_sensitive(connkeys); return -EINVAL; } if (nla_get_flag(info->attrs[NL80211_ATTR_EXTERNAL_AUTH_SUPPORT])) { if (!info->attrs[NL80211_ATTR_SOCKET_OWNER]) { - kzfree(connkeys); + kfree_sensitive(connkeys); GENL_SET_ERR_MSG(info, "external auth requires connection ownership"); return -EINVAL; @@ -10392,7 +10392,7 @@ static int nl80211_connect(struct sk_buf err = cfg80211_connect(rdev, dev, &connect, connkeys, connect.prev_bssid); if (err) - kzfree(connkeys); + kfree_sensitive(connkeys); if (!err && info->attrs[NL80211_ATTR_SOCKET_OWNER]) { dev->ieee80211_ptr->conn_owner_nlportid = info->snd_portid; --- a/net/wireless/sme.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/wireless/sme.c @@ -742,7 +742,7 @@ void __cfg80211_connect_result(struct ne } if (cr->status != WLAN_STATUS_SUCCESS) { - kzfree(wdev->connect_keys); + kfree_sensitive(wdev->connect_keys); wdev->connect_keys = NULL; wdev->ssid_len = 0; wdev->conn_owner_nlportid = 0; @@ -1098,7 +1098,7 @@ void __cfg80211_disconnected(struct net_ wdev->current_bss = NULL; wdev->ssid_len = 0; wdev->conn_owner_nlportid = 0; - kzfree(wdev->connect_keys); + kfree_sensitive(wdev->connect_keys); wdev->connect_keys = NULL; nl80211_send_disconnected(rdev, dev, reason, ie, ie_len, from_ap); @@ -1281,7 +1281,7 @@ int cfg80211_disconnect(struct cfg80211_ ASSERT_WDEV_LOCK(wdev); - kzfree(wdev->connect_keys); + kfree_sensitive(wdev->connect_keys); wdev->connect_keys = NULL; wdev->conn_owner_nlportid = 0; --- a/net/wireless/util.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/wireless/util.c @@ -871,7 +871,7 @@ void cfg80211_upload_connect_keys(struct } } - kzfree(wdev->connect_keys); + kfree_sensitive(wdev->connect_keys); wdev->connect_keys = NULL; } --- a/net/wireless/wext-sme.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/net/wireless/wext-sme.c @@ -57,7 +57,7 @@ int cfg80211_mgd_wext_connect(struct cfg err = cfg80211_connect(rdev, wdev->netdev, &wdev->wext.connect, ck, prev_bssid); if (err) - kzfree(ck); + kfree_sensitive(ck); return err; } --- a/scripts/coccinelle/free/devm_free.cocci~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/scripts/coccinelle/free/devm_free.cocci @@ -89,7 +89,7 @@ position p; ( kfree@p(x) | - kzfree@p(x) + kfree_sensitive@p(x) | krealloc@p(x, ...) | @@ -112,7 +112,7 @@ position p != safe.p; ( * kfree@p(x) | -* kzfree@p(x) +* kfree_sensitive@p(x) | * krealloc@p(x, ...) | --- a/scripts/coccinelle/free/ifnullfree.cocci~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/scripts/coccinelle/free/ifnullfree.cocci @@ -21,7 +21,7 @@ expression E; ( kfree(E); | - kzfree(E); + kfree_sensitive(E); | debugfs_remove(E); | @@ -42,7 +42,7 @@ position p; @@ * if (E != NULL) -* \(kfree@p\|kzfree@p\|debugfs_remove@p\|debugfs_remove_recursive@p\| +* \(kfree@p\|kfree_sensitive@p\|debugfs_remove@p\|debugfs_remove_recursive@p\| * usb_free_urb@p\|kmem_cache_destroy@p\|mempool_destroy@p\| * dma_pool_destroy@p\)(E); --- a/scripts/coccinelle/free/kfreeaddr.cocci~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/scripts/coccinelle/free/kfreeaddr.cocci @@ -20,7 +20,7 @@ position p; ( * kfree@p(&e->f) | -* kzfree@p(&e->f) +* kfree_sensitive@p(&e->f) ) @script:python depends on org@ --- a/scripts/coccinelle/free/kfree.cocci~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/scripts/coccinelle/free/kfree.cocci @@ -24,7 +24,7 @@ position p1; ( * kfree@p1(E) | -* kzfree@p1(E) +* kfree_sensitive@p1(E) ) @print expression@ @@ -68,7 +68,7 @@ while (1) { ... ( * kfree@ok(E) | -* kzfree@ok(E) +* kfree_sensitive@ok(E) ) ... when != break; when != goto l; @@ -86,7 +86,7 @@ position free.p1!=loop.ok,p2!={print.p,s ( * kfree@p1(E,...) | -* kzfree@p1(E,...) +* kfree_sensitive@p1(E,...) ) ... ( --- a/security/apparmor/domain.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/apparmor/domain.c @@ -40,8 +40,8 @@ void aa_free_domain_entries(struct aa_do return; for (i = 0; i < domain->size; i++) - kzfree(domain->table[i]); - kzfree(domain->table); + kfree_sensitive(domain->table[i]); + kfree_sensitive(domain->table); domain->table = NULL; } } --- a/security/apparmor/include/file.h~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/apparmor/include/file.h @@ -72,7 +72,7 @@ static inline void aa_free_file_ctx(stru { if (ctx) { aa_put_label(rcu_access_pointer(ctx->label)); - kzfree(ctx); + kfree_sensitive(ctx); } } --- a/security/apparmor/policy.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/apparmor/policy.c @@ -187,9 +187,9 @@ static void aa_free_data(void *ptr, void { struct aa_data *data = ptr; - kzfree(data->data); - kzfree(data->key); - kzfree(data); + kfree_sensitive(data->data); + kfree_sensitive(data->key); + kfree_sensitive(data); } /** @@ -217,19 +217,19 @@ void aa_free_profile(struct aa_profile * aa_put_profile(rcu_access_pointer(profile->parent)); aa_put_ns(profile->ns); - kzfree(profile->rename); + kfree_sensitive(profile->rename); aa_free_file_rules(&profile->file); aa_free_cap_rules(&profile->caps); aa_free_rlimit_rules(&profile->rlimits); for (i = 0; i < profile->xattr_count; i++) - kzfree(profile->xattrs[i]); - kzfree(profile->xattrs); + kfree_sensitive(profile->xattrs[i]); + kfree_sensitive(profile->xattrs); for (i = 0; i < profile->secmark_count; i++) - kzfree(profile->secmark[i].label); - kzfree(profile->secmark); - kzfree(profile->dirname); + kfree_sensitive(profile->secmark[i].label); + kfree_sensitive(profile->secmark); + kfree_sensitive(profile->dirname); aa_put_dfa(profile->xmatch); aa_put_dfa(profile->policy.dfa); @@ -237,14 +237,14 @@ void aa_free_profile(struct aa_profile * rht = profile->data; profile->data = NULL; rhashtable_free_and_destroy(rht, aa_free_data, NULL); - kzfree(rht); + kfree_sensitive(rht); } - kzfree(profile->hash); + kfree_sensitive(profile->hash); aa_put_loaddata(profile->rawdata); aa_label_destroy(&profile->label); - kzfree(profile); + kfree_sensitive(profile); } /** --- a/security/apparmor/policy_ns.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/apparmor/policy_ns.c @@ -121,9 +121,9 @@ static struct aa_ns *alloc_ns(const char return ns; fail_unconfined: - kzfree(ns->base.hname); + kfree_sensitive(ns->base.hname); fail_ns: - kzfree(ns); + kfree_sensitive(ns); return NULL; } @@ -145,7 +145,7 @@ void aa_free_ns(struct aa_ns *ns) ns->unconfined->ns = NULL; aa_free_profile(ns->unconfined); - kzfree(ns); + kfree_sensitive(ns); } /** --- a/security/apparmor/policy_unpack.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/apparmor/policy_unpack.c @@ -163,10 +163,10 @@ static void do_loaddata_free(struct work aa_put_ns(ns); } - kzfree(d->hash); - kzfree(d->name); + kfree_sensitive(d->hash); + kfree_sensitive(d->name); kvfree(d->data); - kzfree(d); + kfree_sensitive(d); } void aa_loaddata_kref(struct kref *kref) @@ -894,7 +894,7 @@ static struct aa_profile *unpack_profile while (unpack_strdup(e, &key, NULL)) { data = kzalloc(sizeof(*data), GFP_KERNEL); if (!data) { - kzfree(key); + kfree_sensitive(key); goto fail; } @@ -902,8 +902,8 @@ static struct aa_profile *unpack_profile data->size = unpack_blob(e, &data->data, NULL); data->data = kvmemdup(data->data, data->size); if (data->size && !data->data) { - kzfree(data->key); - kzfree(data); + kfree_sensitive(data->key); + kfree_sensitive(data); goto fail; } @@ -1037,7 +1037,7 @@ void aa_load_ent_free(struct aa_load_ent aa_put_profile(ent->old); aa_put_profile(ent->new); kfree(ent->ns_name); - kzfree(ent); + kfree_sensitive(ent); } } --- a/security/keys/big_key.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/keys/big_key.c @@ -138,7 +138,7 @@ int big_key_preparse(struct key_preparse err_fput: fput(file); err_enckey: - kzfree(enckey); + kfree_sensitive(enckey); error: memzero_explicit(buf, enclen); kvfree(buf); @@ -155,7 +155,7 @@ void big_key_free_preparse(struct key_pr path_put(path); } - kzfree(prep->payload.data[big_key_data]); + kfree_sensitive(prep->payload.data[big_key_data]); } /* @@ -187,7 +187,7 @@ void big_key_destroy(struct key *key) path->mnt = NULL; path->dentry = NULL; } - kzfree(key->payload.data[big_key_data]); + kfree_sensitive(key->payload.data[big_key_data]); key->payload.data[big_key_data] = NULL; } --- a/security/keys/dh.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/keys/dh.c @@ -58,9 +58,9 @@ error: static void dh_free_data(struct dh *dh) { - kzfree(dh->key); - kzfree(dh->p); - kzfree(dh->g); + kfree_sensitive(dh->key); + kfree_sensitive(dh->p); + kfree_sensitive(dh->g); } struct dh_completion { @@ -126,7 +126,7 @@ static void kdf_dealloc(struct kdf_sdesc if (sdesc->shash.tfm) crypto_free_shash(sdesc->shash.tfm); - kzfree(sdesc); + kfree_sensitive(sdesc); } /* @@ -220,7 +220,7 @@ static int keyctl_dh_compute_kdf(struct ret = -EFAULT; err: - kzfree(outbuf); + kfree_sensitive(outbuf); return ret; } @@ -395,11 +395,11 @@ long __keyctl_dh_compute(struct keyctl_d out6: kpp_request_free(req); out5: - kzfree(outbuf); + kfree_sensitive(outbuf); out4: crypto_free_kpp(tfm); out3: - kzfree(secret); + kfree_sensitive(secret); out2: dh_free_data(&dh_inputs); out1: --- a/security/keys/encrypted-keys/encrypted.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/keys/encrypted-keys/encrypted.c @@ -370,7 +370,7 @@ static int get_derived_key(u8 *derived_k master_keylen); ret = crypto_shash_tfm_digest(hash_tfm, derived_buf, derived_buf_len, derived_key); - kzfree(derived_buf); + kfree_sensitive(derived_buf); return ret; } @@ -812,13 +812,13 @@ static int encrypted_instantiate(struct ret = encrypted_init(epayload, key->description, format, master_desc, decrypted_datalen, hex_encoded_iv); if (ret < 0) { - kzfree(epayload); + kfree_sensitive(epayload); goto out; } rcu_assign_keypointer(key, epayload); out: - kzfree(datablob); + kfree_sensitive(datablob); return ret; } @@ -827,7 +827,7 @@ static void encrypted_rcu_free(struct rc struct encrypted_key_payload *epayload; epayload = container_of(rcu, struct encrypted_key_payload, rcu); - kzfree(epayload); + kfree_sensitive(epayload); } /* @@ -885,7 +885,7 @@ static int encrypted_update(struct key * rcu_assign_keypointer(key, new_epayload); call_rcu(&epayload->rcu, encrypted_rcu_free); out: - kzfree(buf); + kfree_sensitive(buf); return ret; } @@ -946,7 +946,7 @@ static long encrypted_read(const struct memzero_explicit(derived_key, sizeof(derived_key)); memcpy(buffer, ascii_buf, asciiblob_len); - kzfree(ascii_buf); + kfree_sensitive(ascii_buf); return asciiblob_len; out: @@ -961,7 +961,7 @@ out: */ static void encrypted_destroy(struct key *key) { - kzfree(key->payload.data[0]); + kfree_sensitive(key->payload.data[0]); } struct key_type key_type_encrypted = { --- a/security/keys/trusted-keys/trusted_tpm1.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/keys/trusted-keys/trusted_tpm1.c @@ -68,7 +68,7 @@ static int TSS_sha1(const unsigned char } ret = crypto_shash_digest(&sdesc->shash, data, datalen, digest); - kzfree(sdesc); + kfree_sensitive(sdesc); return ret; } @@ -112,7 +112,7 @@ static int TSS_rawhmac(unsigned char *di if (!ret) ret = crypto_shash_final(&sdesc->shash, digest); out: - kzfree(sdesc); + kfree_sensitive(sdesc); return ret; } @@ -166,7 +166,7 @@ int TSS_authhmac(unsigned char *digest, paramdigest, TPM_NONCE_SIZE, h1, TPM_NONCE_SIZE, h2, 1, &c, 0, 0); out: - kzfree(sdesc); + kfree_sensitive(sdesc); return ret; } EXPORT_SYMBOL_GPL(TSS_authhmac); @@ -251,7 +251,7 @@ int TSS_checkhmac1(unsigned char *buffer if (memcmp(testhmac, authdata, SHA1_DIGEST_SIZE)) ret = -EINVAL; out: - kzfree(sdesc); + kfree_sensitive(sdesc); return ret; } EXPORT_SYMBOL_GPL(TSS_checkhmac1); @@ -353,7 +353,7 @@ static int TSS_checkhmac2(unsigned char if (memcmp(testhmac2, authdata2, SHA1_DIGEST_SIZE)) ret = -EINVAL; out: - kzfree(sdesc); + kfree_sensitive(sdesc); return ret; } @@ -563,7 +563,7 @@ static int tpm_seal(struct tpm_buf *tb, *bloblen = storedsize; } out: - kzfree(td); + kfree_sensitive(td); return ret; } @@ -1031,12 +1031,12 @@ static int trusted_instantiate(struct ke if (!ret && options->pcrlock) ret = pcrlock(options->pcrlock); out: - kzfree(datablob); - kzfree(options); + kfree_sensitive(datablob); + kfree_sensitive(options); if (!ret) rcu_assign_keypointer(key, payload); else - kzfree(payload); + kfree_sensitive(payload); return ret; } @@ -1045,7 +1045,7 @@ static void trusted_rcu_free(struct rcu_ struct trusted_key_payload *p; p = container_of(rcu, struct trusted_key_payload, rcu); - kzfree(p); + kfree_sensitive(p); } /* @@ -1087,13 +1087,13 @@ static int trusted_update(struct key *ke ret = datablob_parse(datablob, new_p, new_o); if (ret != Opt_update) { ret = -EINVAL; - kzfree(new_p); + kfree_sensitive(new_p); goto out; } if (!new_o->keyhandle) { ret = -EINVAL; - kzfree(new_p); + kfree_sensitive(new_p); goto out; } @@ -1107,22 +1107,22 @@ static int trusted_update(struct key *ke ret = key_seal(new_p, new_o); if (ret < 0) { pr_info("trusted_key: key_seal failed (%d)\n", ret); - kzfree(new_p); + kfree_sensitive(new_p); goto out; } if (new_o->pcrlock) { ret = pcrlock(new_o->pcrlock); if (ret < 0) { pr_info("trusted_key: pcrlock failed (%d)\n", ret); - kzfree(new_p); + kfree_sensitive(new_p); goto out; } } rcu_assign_keypointer(key, new_p); call_rcu(&p->rcu, trusted_rcu_free); out: - kzfree(datablob); - kzfree(new_o); + kfree_sensitive(datablob); + kfree_sensitive(new_o); return ret; } @@ -1154,7 +1154,7 @@ static long trusted_read(const struct ke */ static void trusted_destroy(struct key *key) { - kzfree(key->payload.data[0]); + kfree_sensitive(key->payload.data[0]); } struct key_type key_type_trusted = { --- a/security/keys/user_defined.c~mm-treewide-rename-kzfree-to-kfree_sensitive +++ a/security/keys/user_defined.c @@ -82,7 +82,7 @@ EXPORT_SYMBOL_GPL(user_preparse); */ void user_free_preparse(struct key_preparsed_payload *prep) { - kzfree(prep->payload.data[0]); + kfree_sensitive(prep->payload.data[0]); } EXPORT_SYMBOL_GPL(user_free_preparse); @@ -91,7 +91,7 @@ static void user_free_payload_rcu(struct struct user_key_payload *payload; payload = container_of(head, struct user_key_payload, rcu); - kzfree(payload); + kfree_sensitive(payload); } /* @@ -147,7 +147,7 @@ void user_destroy(struct key *key) { struct user_key_payload *upayload = key->payload.data[0]; - kzfree(upayload); + kfree_sensitive(upayload); } EXPORT_SYMBOL_GPL(user_destroy); From patchwork Fri Aug 7 06:18:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704723 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 63EFF13B6 for ; Fri, 7 Aug 2020 06:18:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 27DCF22C9F for ; Fri, 7 Aug 2020 06:18:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="jdwyvxu7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 27DCF22C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id ED17B8D0018; Fri, 7 Aug 2020 02:18:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E342B8D000F; Fri, 7 Aug 2020 02:18:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFB7C8D0018; Fri, 7 Aug 2020 02:18:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0191.hostedemail.com [216.40.44.191]) by kanga.kvack.org (Postfix) with ESMTP id B64FB8D000F for ; Fri, 7 Aug 2020 02:18:19 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 73B79181AEF10 for ; Fri, 7 Aug 2020 06:18:19 +0000 (UTC) X-FDA: 77122767918.05.home25_3a0c9f826fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 4E6AD18015860 for ; Fri, 7 Aug 2020 06:18:19 +0000 (UTC) X-Spam-Summary: 1,0,0,2aba8f78aed3c02d,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2196:2199:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:5007:6261:6653:6737:7576:7903:8599:8603:9025:9121:9545:10004:10913:11026:11473:11658:11914:12043:12048:12295:12297:12517:12519:12555:12679:12783:12986:13846:14096:14181:14721:14849:21080:21451:21627:21939:21990:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygwp6hrd6iy158at3f1kndu6d17op7h7tfysdaeinzohgw6dp1jc6fk1oqpjq.qcqhi4d1ei43w5a7jcb7satmrwydk6jdrnh9ty8ot5zaxranfnb3tpqsib3pow9.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL :0,DNSBL X-HE-Tag: home25_3a0c9f826fbe X-Filterd-Recvd-Size: 3613 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf09.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:18 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A71342177B; Fri, 7 Aug 2020 06:18:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781098; bh=KtulRCauuG00kNqKhgECuMBgp5+EX5wRaB63o6XwEFo=; h=Date:From:To:Subject:In-Reply-To:From; b=jdwyvxu7Dk2wgnbvWKMVwjJyXfdxghRSKFppexYcnmnJy0dvYj5B96FFgeME8m79g GBt+Y4rJp4hWH9bovddFpfHgT9LuOWAUYJ4urvE+OuljZBa7kke8uPdadQk2J9aBUY 2bNk69MDlFimbiyMd+5Ty03215N0Ce7P1cg2K8bo= Date: Thu, 06 Aug 2020 23:18:17 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, david@redhat.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, william.kucharski@oracle.com, willy@infradead.org Subject: [patch 023/163] mm: ksize() should silently accept a NULL pointer Message-ID: <20200807061817._fTyXX7M0%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 4E6AD18015860 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: William Kucharski Subject: mm: ksize() should silently accept a NULL pointer Other mm routines such as kfree() and kzfree() silently do the right thing if passed a NULL pointer, so ksize() should do the same. Link: http://lkml.kernel.org/r/20200616225409.4670-1-william.kucharski@oracle.com Signed-off-by: William Kucharski Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: David Hildenbrand Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Signed-off-by: Andrew Morton --- mm/slab_common.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) --- a/mm/slab_common.c~mm-ksize-should-silently-accept-a-null-pointer +++ a/mm/slab_common.c @@ -1681,10 +1681,9 @@ static __always_inline void *__do_kreall gfp_t flags) { void *ret; - size_t ks = 0; + size_t ks; - if (p) - ks = ksize(p); + ks = ksize(p); if (ks >= new_size) { p = kasan_krealloc((void *)p, new_size, flags); @@ -1744,10 +1743,9 @@ void kfree_sensitive(const void *p) size_t ks; void *mem = (void *)p; - if (unlikely(ZERO_OR_NULL_PTR(mem))) - return; ks = ksize(mem); - memzero_explicit(mem, ks); + if (ks) + memzero_explicit(mem, ks); kfree(mem); } EXPORT_SYMBOL(kfree_sensitive); @@ -1770,8 +1768,6 @@ size_t ksize(const void *objp) { size_t size; - if (WARN_ON_ONCE(!objp)) - return 0; /* * We need to check that the pointed to object is valid, and only then * unpoison the shadow memory below. We use __kasan_check_read(), to @@ -1785,7 +1781,7 @@ size_t ksize(const void *objp) * We want to perform the check before __ksize(), to avoid potentially * crashing in __ksize() due to accessing invalid metadata. */ - if (unlikely(objp == ZERO_SIZE_PTR) || !__kasan_check_read(objp, 1)) + if (unlikely(ZERO_OR_NULL_PTR(objp)) || !__kasan_check_read(objp, 1)) return 0; size = __ksize(objp); From patchwork Fri Aug 7 06:18:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704725 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D419013B6 for ; Fri, 7 Aug 2020 06:18:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A58722177B for ; Fri, 7 Aug 2020 06:18:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="0V2Hk7nk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A58722177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3371C8D0019; Fri, 7 Aug 2020 02:18:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2E68F8D000F; Fri, 7 Aug 2020 02:18:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FD168D0019; Fri, 7 Aug 2020 02:18:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id 079938D000F for ; Fri, 7 Aug 2020 02:18:24 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B71EE82499B9 for ; Fri, 7 Aug 2020 06:18:23 +0000 (UTC) X-FDA: 77122768086.17.party65_5401c2726fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 90185180D0180 for ; Fri, 7 Aug 2020 06:18:23 +0000 (UTC) X-Spam-Summary: 10,1,0,75e7fa01d452e6bf,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:404:800:960:965:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2565:2682:2685:2731:2859:2894:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4390:4395:5007:6119:6261:6653:6737:7576:7903:8599:8784:8957:9025:9040:9389:9545:10004:10913:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13132:13231:13846:14096:14181:14721:14849:21080:21347:21450:21451:21611:21627:21796:21939:21972:21984:30036:30054:30062:30064:30069,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygjyptqbqo3fz7jugmw8fsumg5fopncq5orbkzdmg418qxinu3rjkq51jchbr.1ie39amss8rq7b4ppygd56bh6ni61twmzjsyyjeh1uasne56xcwmx3a8hsa b83t.1-l X-HE-Tag: party65_5401c2726fbe X-Filterd-Recvd-Size: 5076 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:23 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3F8122177B; Fri, 7 Aug 2020 06:18:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781102; bh=36tB1MKB9BBGhJraUptyrInUB9HTj+GoP5szEqS1CnA=; h=Date:From:To:Subject:In-Reply-To:From; b=0V2Hk7nkniGFMmnvn6ZG6GqCOZV5Y7doZXs3t28u7hsNCTzT56dUbKAxacvVrkUhh 6547mQtofVuor1yi7cKGL7xOB8PKGC6jFEvM6uMjYm1BDHSAe8sRJUiOB0txmg4cOd EdPcn7pwfF6fRr7NRTu4OUcKdjyanIaD/iUcMHSo= Date: Thu, 06 Aug 2020 23:18:20 -0700 From: Andrew Morton To: akpm@linux-foundation.org, alex.popov@linux.com, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mjg59@google.com, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vinmenon@codeaurora.org, vjitta@codeaurora.org Subject: [patch 024/163] mm/slab: expand CONFIG_SLAB_FREELIST_HARDENED to include SLAB Message-ID: <20200807061820.jrCao-wcB%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 90185180D0180 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kees Cook Subject: mm/slab: expand CONFIG_SLAB_FREELIST_HARDENED to include SLAB Patch series "mm: Expand CONFIG_SLAB_FREELIST_HARDENED to include SLAB" In reviewing Vlastimil Babka's latest slub debug series, I realized[1] that several checks under CONFIG_SLAB_FREELIST_HARDENED weren't being applied to SLAB. Fix this by expanding the Kconfig coverage, and adding a simple double-free test for SLAB. This patch (of 2): Include SLAB caches when performing kmem_cache pointer verification. A defense against such corruption[1] should be applied to all the allocators. With this added, the "SLAB_FREE_CROSS" and "SLAB_FREE_PAGE" LKDTM tests now pass on SLAB: lkdtm: Performing direct entry SLAB_FREE_CROSS lkdtm: Attempting cross-cache slab free ... ------------[ cut here ]------------ cache_from_obj: Wrong slab cache. lkdtm-heap-b but object is from lkdtm-heap-a WARNING: CPU: 2 PID: 2195 at mm/slab.h:530 kmem_cache_free+0x8d/0x1d0 ... lkdtm: Performing direct entry SLAB_FREE_PAGE lkdtm: Attempting non-Slab slab free ... ------------[ cut here ]------------ virt_to_cache: Object is not a Slab page! WARNING: CPU: 1 PID: 2202 at mm/slab.h:489 kmem_cache_free+0x196/0x1d0 Additionally clean up neighboring Kconfig entries for clarity, readability, and redundant option removal. [1] https://github.com/ThomasKing2014/slides/raw/master/Building%20universal%20Android%20rooting%20with%20a%20type%20confusion%20vulnerability.pdf Link: http://lkml.kernel.org/r/20200625215548.389774-1-keescook@chromium.org Link: http://lkml.kernel.org/r/20200625215548.389774-2-keescook@chromium.org Fixes: 598a0717a816 ("mm/slab: validate cache membership under freelist hardening") Signed-off-by: Kees Cook Acked-by: Vlastimil Babka Cc: Alexander Popov Cc: Christoph Lameter Cc: David Rientjes Cc: Jann Horn Cc: Joonsoo Kim Cc: Matthew Garrett Cc: Pekka Enberg Cc: Roman Gushchin Cc: Vijayanand Jitta Cc: Vinayak Menon Signed-off-by: Andrew Morton --- init/Kconfig | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) --- a/init/Kconfig~mm-expand-config_slab_freelist_hardened-to-include-slab +++ a/init/Kconfig @@ -1913,9 +1913,8 @@ config SLAB_MERGE_DEFAULT command line. config SLAB_FREELIST_RANDOM - default n + bool "Randomize slab freelist" depends on SLAB || SLUB - bool "SLAB freelist randomization" help Randomizes the freelist order used on creating new pages. This security feature reduces the predictability of the kernel slab @@ -1923,12 +1922,14 @@ config SLAB_FREELIST_RANDOM config SLAB_FREELIST_HARDENED bool "Harden slab freelist metadata" - depends on SLUB + depends on SLAB || SLUB help Many kernel heap attacks try to target slab cache metadata and other infrastructure. This options makes minor performance sacrifices to harden the kernel slab allocator against common - freelist exploit methods. + freelist exploit methods. Some slab implementations have more + sanity-checking than others. This option is most effective with + CONFIG_SLUB. config SHUFFLE_PAGE_ALLOCATOR bool "Page allocator randomization" From patchwork Fri Aug 7 06:18:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704727 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 51BCA13B6 for ; Fri, 7 Aug 2020 06:18:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2240D2177B for ; Fri, 7 Aug 2020 06:18:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="k7/PlVT0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2240D2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C5FAF8D001A; Fri, 7 Aug 2020 02:18:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BE85A8D000F; Fri, 7 Aug 2020 02:18:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFD578D001A; Fri, 7 Aug 2020 02:18:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0157.hostedemail.com [216.40.44.157]) by kanga.kvack.org (Postfix) with ESMTP id 978C38D000F for ; Fri, 7 Aug 2020 02:18:27 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 62EF71E1B for ; Fri, 7 Aug 2020 06:18:27 +0000 (UTC) X-FDA: 77122768254.03.snow66_170908526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 3E8FB28A4E9 for ; Fri, 7 Aug 2020 06:18:27 +0000 (UTC) X-Spam-Summary: 1,0,0,c3e28d68a6016adf,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:965:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2196:2197:2198:2199:2200:2201:2393:2525:2559:2565:2682:2685:2739:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4390:4395:5007:6261:6653:6737:7576:7903:8599:9010:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13846:14181:14721:14849:21080:21451:21627:21939:21990:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yrk6ajk8ohzpmxu4c7f4g66y3amypkejr13hwjsk5dama368aantdw8w6gmkn.wuo7jabax1k1q4d9s74fght7bmdwi4i8qx6u1h4ushjutmb94fjjnp33xzyrmq1.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MS F:not bu X-HE-Tag: snow66_170908526fbe X-Filterd-Recvd-Size: 4166 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:26 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4054D22C9F; Fri, 7 Aug 2020 06:18:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781106; bh=QPd4SUwbzWoRFVG/dAfx5eUxU/5Sx3XlCV1Ji9AfAy8=; h=Date:From:To:Subject:In-Reply-To:From; b=k7/PlVT0hnO+fhkChoUJJwzDwStXExkcMMnCKhOcK9hMI8f+/VpTKpckmkNKneBIB cSo5GHtDowHLEEUs58RzhyZu9OX6OoriJ40FhgUafgZJrsSy/oATz3dOmAWWNnQFAM Es9EhSSpvCfBKx5G0yf/EXtf2742HN9Ku0WWx9hc= Date: Thu, 06 Aug 2020 23:18:24 -0700 From: Andrew Morton To: akpm@linux-foundation.org, alex.popov@linux.com, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mjg59@google.com, mm-commits@vger.kernel.org, penberg@kernel.org, rdunlap@infradead.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vinmenon@codeaurora.org, vjitta@codeaurora.org Subject: [patch 025/163] mm/slab: add naive detection of double free Message-ID: <20200807061824.s-f36OexL%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 3E8FB28A4E9 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kees Cook Subject: mm/slab: add naive detection of double free Similar to commit ce6fa91b9363 ("mm/slub.c: add a naive detection of double free or corruption"), add a very cheap double-free check for SLAB under CONFIG_SLAB_FREELIST_HARDENED. With this added, the "SLAB_FREE_DOUBLE" LKDTM test passes under SLAB: lkdtm: Performing direct entry SLAB_FREE_DOUBLE lkdtm: Attempting double slab free ... ------------[ cut here ]------------ WARNING: CPU: 2 PID: 2193 at mm/slab.c:757 ___cache _free+0x325/0x390 [keescook@chromium.org: fix misplaced __free_one()] Link: http://lkml.kernel.org/r/202006261306.0D82A2B@keescook Link: https://lore.kernel.org/lkml/7ff248c7-d447-340c-a8e2-8c02972aca70@infradead.org Link: http://lkml.kernel.org/r/20200625215548.389774-3-keescook@chromium.org Signed-off-by: Kees Cook Acked-by: Vlastimil Babka Acked-by: Randy Dunlap [build tested] Cc: Roman Gushchin Cc: Christoph Lameter Cc: Alexander Popov Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Vinayak Menon Cc: Matthew Garrett Cc: Jann Horn Cc: Vijayanand Jitta Signed-off-by: Andrew Morton --- mm/slab.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) --- a/mm/slab.c~slab-add-naive-detection-of-double-free +++ a/mm/slab.c @@ -588,6 +588,16 @@ static int transfer_objects(struct array return nr; } +/* &alien->lock must be held by alien callers. */ +static __always_inline void __free_one(struct array_cache *ac, void *objp) +{ + /* Avoid trivial double-free. */ + if (IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) && + WARN_ON_ONCE(ac->avail > 0 && ac->entry[ac->avail - 1] == objp)) + return; + ac->entry[ac->avail++] = objp; +} + #ifndef CONFIG_NUMA #define drain_alien_cache(cachep, alien) do { } while (0) @@ -767,7 +777,7 @@ static int __cache_free_alien(struct kme STATS_INC_ACOVERFLOW(cachep); __drain_alien_cache(cachep, ac, page_node, &list); } - ac->entry[ac->avail++] = objp; + __free_one(ac, objp); spin_unlock(&alien->lock); slabs_destroy(cachep, &list); } else { @@ -3466,7 +3476,7 @@ void ___cache_free(struct kmem_cache *ca } } - ac->entry[ac->avail++] = objp; + __free_one(ac, objp); } /** From patchwork Fri Aug 7 06:18:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704729 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D07414B7 for ; Fri, 7 Aug 2020 06:18:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D759221E5 for ; Fri, 7 Aug 2020 06:18:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="TQ43bzDi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D759221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1FEF88D001B; Fri, 7 Aug 2020 02:18:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1AEB98D000F; Fri, 7 Aug 2020 02:18:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C5F78D001B; Fri, 7 Aug 2020 02:18:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id E6F6A8D000F for ; Fri, 7 Aug 2020 02:18:30 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A2C933628 for ; Fri, 7 Aug 2020 06:18:30 +0000 (UTC) X-FDA: 77122768380.28.hot68_0d1042126fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 7524A6D74 for ; Fri, 7 Aug 2020 06:18:30 +0000 (UTC) X-Spam-Summary: 1,0,0,829efdffe95f9bcc,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:421:560:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1434:1437:1535:1544:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2693:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6119:6261:6653:7514:7576:7903:8599:8603:8660:9025:9040:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12696:12737:12783:12986:13095:13148:13215:13229:13230:13255:14181:14721:14849:21080:21433:21451:21627:21796:21939:21990:30012:30036:30054:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8iduacduncro58i6apb1rs3wj3opia841bhrx5brfhhrw6dd49urpwhpm5oy.rbtdir9uqy3q45d4xyjgfr7oza4hzk1fgg5asx8j6i3pzukx6rh5sa3169d5pno.s-lbl8.mailshell.net-223.238.255.1 00,Cache X-HE-Tag: hot68_0d1042126fbe X-Filterd-Recvd-Size: 5553 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:29 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B40B5221E5; Fri, 7 Aug 2020 06:18:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781109; bh=bLXsSO4PYwiWR+nIydy2OM59iaBT655q2XOxHr6Tvhc=; h=Date:From:To:Subject:In-Reply-To:From; b=TQ43bzDiCoMebEiQWYBbs9j92w3neI36upBlFewSuitT105Z/elXSVK8fTa663D2f QxxyydE15+PgjkIpMqTcBYK4pXJOHBlW1l/Kwq124k0F9jX2nMgTJAl9iswQzLNizE 2ScqvmEWdW+dlCi7JU3upIcdCZAXTa0+aQ/Bl8MY= Date: Thu, 06 Aug 2020 23:18:28 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, lonuxli.64@gmail.com, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 026/163] mm, slab: check GFP_SLAB_BUG_MASK before alloc_pages in kmalloc_order Message-ID: <20200807061828.uPsWNduYb%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 7524A6D74 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Long Li Subject: mm, slab: check GFP_SLAB_BUG_MASK before alloc_pages in kmalloc_order kmalloc cannot allocate memory from HIGHMEM. Allocating large amounts of memory currently bypasses the check and will simply leak the memory when page_address() returns NULL. To fix this, factor the GFP_SLAB_BUG_MASK check out of slab & slub, and call it from kmalloc_order() as well. In order to make the code clear, the warning message is put in one place. Link: http://lkml.kernel.org/r/20200704035027.GA62481@lilong Signed-off-by: Long Li Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: Pekka Enberg Acked-by: David Rientjes Cc: Christoph Lameter Cc: Joonsoo Kim Signed-off-by: Andrew Morton --- mm/slab.c | 10 +++------- mm/slab.h | 1 + mm/slab_common.c | 17 +++++++++++++++++ mm/slub.c | 9 ++------- 4 files changed, 23 insertions(+), 14 deletions(-) --- a/mm/slab.c~mm-slab-check-gfp_slab_bug_mask-before-alloc_pages-in-kmalloc_order +++ a/mm/slab.c @@ -2589,13 +2589,9 @@ static struct page *cache_grow_begin(str * Be lazy and only check for valid flags here, keeping it out of the * critical path in kmem_cache_alloc(). */ - if (unlikely(flags & GFP_SLAB_BUG_MASK)) { - gfp_t invalid_mask = flags & GFP_SLAB_BUG_MASK; - flags &= ~GFP_SLAB_BUG_MASK; - pr_warn("Unexpected gfp: %#x (%pGg). Fixing up to gfp: %#x (%pGg). Fix your code!\n", - invalid_mask, &invalid_mask, flags, &flags); - dump_stack(); - } + if (unlikely(flags & GFP_SLAB_BUG_MASK)) + flags = kmalloc_fix_flags(flags); + WARN_ON_ONCE(cachep->ctor && (flags & __GFP_ZERO)); local_flags = flags & (GFP_CONSTRAINT_MASK|GFP_RECLAIM_MASK); --- a/mm/slab_common.c~mm-slab-check-gfp_slab_bug_mask-before-alloc_pages-in-kmalloc_order +++ a/mm/slab_common.c @@ -26,6 +26,8 @@ #define CREATE_TRACE_POINTS #include +#include "internal.h" + #include "slab.h" enum slab_state slab_state; @@ -1332,6 +1334,18 @@ void __init create_kmalloc_caches(slab_f } #endif /* !CONFIG_SLOB */ +gfp_t kmalloc_fix_flags(gfp_t flags) +{ + gfp_t invalid_mask = flags & GFP_SLAB_BUG_MASK; + + flags &= ~GFP_SLAB_BUG_MASK; + pr_warn("Unexpected gfp: %#x (%pGg). Fixing up to gfp: %#x (%pGg). Fix your code!\n", + invalid_mask, &invalid_mask, flags, &flags); + dump_stack(); + + return flags; +} + /* * To avoid unnecessary overhead, we pass through large allocation requests * directly to the page allocator. We use __GFP_COMP, because we will need to @@ -1342,6 +1356,9 @@ void *kmalloc_order(size_t size, gfp_t f void *ret = NULL; struct page *page; + if (unlikely(flags & GFP_SLAB_BUG_MASK)) + flags = kmalloc_fix_flags(flags); + flags |= __GFP_COMP; page = alloc_pages(flags, order); if (likely(page)) { --- a/mm/slab.h~mm-slab-check-gfp_slab_bug_mask-before-alloc_pages-in-kmalloc_order +++ a/mm/slab.h @@ -152,6 +152,7 @@ void create_kmalloc_caches(slab_flags_t) struct kmem_cache *kmalloc_slab(size_t, gfp_t); #endif +gfp_t kmalloc_fix_flags(gfp_t flags); /* Functions provided by the slab allocators */ int __kmem_cache_create(struct kmem_cache *, slab_flags_t flags); --- a/mm/slub.c~mm-slab-check-gfp_slab_bug_mask-before-alloc_pages-in-kmalloc_order +++ a/mm/slub.c @@ -1745,13 +1745,8 @@ out: static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int node) { - if (unlikely(flags & GFP_SLAB_BUG_MASK)) { - gfp_t invalid_mask = flags & GFP_SLAB_BUG_MASK; - flags &= ~GFP_SLAB_BUG_MASK; - pr_warn("Unexpected gfp: %#x (%pGg). Fixing up to gfp: %#x (%pGg). Fix your code!\n", - invalid_mask, &invalid_mask, flags, &flags); - dump_stack(); - } + if (unlikely(flags & GFP_SLAB_BUG_MASK)) + flags = kmalloc_fix_flags(flags); return allocate_slab(s, flags & (GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK), node); From patchwork Fri Aug 7 06:18:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704731 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D0F9913B6 for ; Fri, 7 Aug 2020 06:18:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A61B32177B for ; Fri, 7 Aug 2020 06:18:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="XHwR3lzs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A61B32177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 819168D001C; Fri, 7 Aug 2020 02:18:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7C9B28D000F; Fri, 7 Aug 2020 02:18:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 691A28D001C; Fri, 7 Aug 2020 02:18:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0037.hostedemail.com [216.40.44.37]) by kanga.kvack.org (Postfix) with ESMTP id 4D5F48D000F for ; Fri, 7 Aug 2020 02:18:34 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 0F3163628 for ; Fri, 7 Aug 2020 06:18:34 +0000 (UTC) X-FDA: 77122768548.24.voice03_1d04a2026fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id CC1D11A4A5 for ; Fri, 7 Aug 2020 06:18:33 +0000 (UTC) X-Spam-Summary: 1,0,0,150ff2608ed39c5c,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:146:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2196:2199:2282:2393:2525:2559:2563:2682:2685:2859:2895:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6261:6653:7576:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12297:12517:12519:12555:12679:12783:12986:13069:13311:13357:14093:14181:14384:14721:14849:21080:21451:21627:21939:30029:30054:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfh14r8uuu95ru745f8kair9bkrop3hcihsqjywdmyh3gmcd55bx5p5in5gao.eapc76g5mcqygee7okjowxukttg118jwn4j734rs8q3iuawb6wukgstn1whoztn.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNS BL:neutr X-HE-Tag: voice03_1d04a2026fbe X-Filterd-Recvd-Size: 2759 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:33 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 416BC22CAE; Fri, 7 Aug 2020 06:18:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781112; bh=yEGq3QmsfsiD7ZQzvEF9Iu4ht28IgB/X7T284JZKEAk=; h=Date:From:To:Subject:In-Reply-To:From; b=XHwR3lzsUlaWXKMS0ay4BbtYJP+9ERo3H2KjPs2HXy8tP4Wowjm/m6NLm18X8RcH0 u/gqBis+tx54K97K/BwcBzRBaq0NdxjE8+OAoluluGCf5pykhhBioExB2qu4RQhGbp zu849xbsDlXCVjNP2ion7Slcjn087SSHUN06vYIQ= Date: Thu, 06 Aug 2020 23:18:31 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, yangx.jy@cn.fujitsu.com Subject: [patch 027/163] mm/slab.c: update outdated kmem_list3 in a comment Message-ID: <20200807061831.mb1JDJlMk%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: CC1D11A4A5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Xiao Yang Subject: mm/slab.c: update outdated kmem_list3 in a comment kmem_list3 has been renamed to kmem_cache_node long long ago so update it. References: 6744f087ba2a ("slab: Common name for the per node structures") ce8eb6c424c7 ("slab: Rename list3/l3 to node") Link: http://lkml.kernel.org/r/20200722033355.26908-1-yangx.jy@cn.fujitsu.com Signed-off-by: Xiao Yang Reviewed-by: Pekka Enberg Cc: Christoph Lameter Cc: David Rientjes Cc: Joonsoo Kim Signed-off-by: Andrew Morton --- mm/slab.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/mm/slab.c~mm-slabc-update-outdated-kmem_list3-in-a-comment +++ a/mm/slab.c @@ -1060,7 +1060,7 @@ int slab_prepare_cpu(unsigned int cpu) * offline. * * Even if all the cpus of a node are down, we don't free the - * kmem_list3 of any cache. This to avoid a race between cpu_down, and + * kmem_cache_node of any cache. This to avoid a race between cpu_down, and * a kmalloc allocation from another cpu for memory from the node of * the cpu going down. The list3 structure is usually allocated from * kmem_cache_create() and gets destroyed at kmem_cache_destroy(). From patchwork Fri Aug 7 06:18:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704733 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 32D6813B6 for ; Fri, 7 Aug 2020 06:18:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EDE8D22C9F for ; Fri, 7 Aug 2020 06:18:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="mC7SlJKv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EDE8D22C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B9CE58D001D; Fri, 7 Aug 2020 02:18:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B280C8D000F; Fri, 7 Aug 2020 02:18:37 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A0478D001D; Fri, 7 Aug 2020 02:18:37 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0031.hostedemail.com [216.40.44.31]) by kanga.kvack.org (Postfix) with ESMTP id 7E6D18D000F for ; Fri, 7 Aug 2020 02:18:37 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 464B782499B9 for ; Fri, 7 Aug 2020 06:18:37 +0000 (UTC) X-FDA: 77122768674.05.bait14_2b02fe526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 24B881801541C for ; Fri, 7 Aug 2020 06:18:37 +0000 (UTC) X-Spam-Summary: 1,0,0,b07a931e2b31a90a,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:2:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1801:2196:2198:2199:2200:2393:2525:2559:2565:2682:2685:2693:2731:2859:2895:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4052:4250:4321:4385:4605:5007:6119:6261:6653:6737:7576:7875:7903:8599:9010:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12219:12296:12297:12438:12517:12519:12555:12679:12783:12986:13146:13161:13229:13230:13846:14096:21080:21451:21627:21795:21939:21966:21990:30029:30030:30051:30054:30062:30064:30070:30074,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8jmqrduzp4ah19rajg3eeqy1fsycc1xh45x14qpi8wy8srdk47h1esggiqxx.r4sqsbyfs6sxupyrj4qyxbw5f6bwgm5n6bupxjb6ydtodhaaeidfxwn63eeuuy8.q-lb l8.mails X-HE-Tag: bait14_2b02fe526fbe X-Filterd-Recvd-Size: 11344 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:36 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 76104221E5; Fri, 7 Aug 2020 06:18:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781116; bh=IrQmuDLnmKO8nJeUqsiRdNTHP8cYpnJR9kJ8og74ckc=; h=Date:From:To:Subject:In-Reply-To:From; b=mC7SlJKviedINVAjcr61oqktSFvF4dnoxWc3twPsq0K5zobIkKeUBSQR9Xps5T+1Y mz7ICWS45PxEIIRKBXkfcWcSCAwyL7ZKsD/KK/8VRqWFfbHLUKSnBIH4f1Z8CBuCmL +UlufmqqBsMpDPmOjvaV5vvhfhvHU9w4W9AAe3/I= Date: Thu, 06 Aug 2020 23:18:35 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vjitta@codeaurora.org Subject: [patch 028/163] mm, slub: extend slub_debug syntax for multiple blocks Message-ID: <20200807061835.tEBeaeVg9%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 24B881801541C X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slub: extend slub_debug syntax for multiple blocks Patch series "slub_debug fixes and improvements". The slub_debug kernel boot parameter can either apply a single set of options to all caches or a list of caches. There is a use case where debugging is applied for all caches and then disabled at runtime for specific caches, for performance and memory consumption reasons [1]. As runtime changes are dangerous, extend the boot parameter syntax so that multiple blocks of either global or slab-specific options can be specified, with blocks delimited by ';'. This will also support the use case of [1] without runtime changes. For details see the updated Documentation/vm/slub.rst [1] https://lore.kernel.org/r/1383cd32-1ddc-4dac-b5f8-9c42282fa81c@codeaurora.org [weiyongjun1@huawei.com: make parse_slub_debug_flags() static] Link: http://lkml.kernel.org/r/20200702150522.4940-1-weiyongjun1@huawei.com Link: http://lkml.kernel.org/r/20200610163135.17364-2-vbabka@suse.cz Signed-off-by: Vlastimil Babka Reviewed-by: Kees Cook Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Jann Horn Cc: Roman Gushchin Cc: Vijayanand Jitta Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Signed-off-by: Andrew Morton --- Documentation/admin-guide/kernel-parameters.txt | 2 Documentation/vm/slub.rst | 18 + mm/slub.c | 179 +++++++++----- 3 files changed, 146 insertions(+), 53 deletions(-) --- a/Documentation/admin-guide/kernel-parameters.txt~mm-slub-extend-slub_debug-syntax-for-multiple-blocks +++ a/Documentation/admin-guide/kernel-parameters.txt @@ -4689,7 +4689,7 @@ fragmentation. Defaults to 1 for systems with more than 32MB of RAM, 0 otherwise. - slub_debug[=options[,slabs]] [MM, SLUB] + slub_debug[=options[,slabs][;[options[,slabs]]...] [MM, SLUB] Enabling slub_debug allows one to determine the culprit if slab objects become corrupted. Enabling slub_debug can create guard zones around objects and --- a/Documentation/vm/slub.rst~mm-slub-extend-slub_debug-syntax-for-multiple-blocks +++ a/Documentation/vm/slub.rst @@ -41,6 +41,11 @@ slub_debug=,, Enable options only for select slabs (no spaces after a comma) +Multiple blocks of options for all slabs or selected slabs can be given, with +blocks of options delimited by ';'. The last of "all slabs" blocks is applied +to all slabs except those that match one of the "select slabs" block. Options +of the first "select slabs" blocks that matches the slab's name are applied. + Possible debug options are:: F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS @@ -83,6 +88,19 @@ switch off debugging for such caches by slub_debug=O +You can apply different options to different list of slab names, using blocks +of options. This will enable red zoning for dentry and user tracking for +kmalloc. All other slabs will not get any debugging enabled:: + + slub_debug=Z,dentry;U,kmalloc-* + +You can also enable options (e.g. sanity checks and poisoning) for all caches +except some that are deemed too performance critical and don't need to be +debugged by specifying global debug options followed by a list of slab names +with "-" as options:: + + slub_debug=FZ;-,zs_handle,zspage + In case you forgot to enable debugging on the kernel command line: It is possible to enable debugging manually when the kernel is up. Look at the contents of:: --- a/mm/slub.c~mm-slub-extend-slub_debug-syntax-for-multiple-blocks +++ a/mm/slub.c @@ -499,7 +499,7 @@ static slab_flags_t slub_debug = DEBUG_D static slab_flags_t slub_debug; #endif -static char *slub_debug_slabs; +static char *slub_debug_string; static int disable_higher_order_debug; /* @@ -1262,68 +1262,132 @@ out: return ret; } -static int __init setup_slub_debug(char *str) +/* + * Parse a block of slub_debug options. Blocks are delimited by ';' + * + * @str: start of block + * @flags: returns parsed flags, or DEBUG_DEFAULT_FLAGS if none specified + * @slabs: return start of list of slabs, or NULL when there's no list + * @init: assume this is initial parsing and not per-kmem-create parsing + * + * returns the start of next block if there's any, or NULL + */ +static char * +parse_slub_debug_flags(char *str, slab_flags_t *flags, char **slabs, bool init) { - slub_debug = DEBUG_DEFAULT_FLAGS; - if (*str++ != '=' || !*str) - /* - * No options specified. Switch on full debugging. - */ - goto out; + bool higher_order_disable = false; - if (*str == ',') + /* Skip any completely empty blocks */ + while (*str && *str == ';') + str++; + + if (*str == ',') { /* * No options but restriction on slabs. This means full * debugging for slabs matching a pattern. */ + *flags = DEBUG_DEFAULT_FLAGS; goto check_slabs; + } + *flags = 0; - slub_debug = 0; - if (*str == '-') - /* - * Switch off all debugging measures. - */ - goto out; - - /* - * Determine which debug features should be switched on - */ - for (; *str && *str != ','; str++) { + /* Determine which debug features should be switched on */ + for (; *str && *str != ',' && *str != ';'; str++) { switch (tolower(*str)) { + case '-': + *flags = 0; + break; case 'f': - slub_debug |= SLAB_CONSISTENCY_CHECKS; + *flags |= SLAB_CONSISTENCY_CHECKS; break; case 'z': - slub_debug |= SLAB_RED_ZONE; + *flags |= SLAB_RED_ZONE; break; case 'p': - slub_debug |= SLAB_POISON; + *flags |= SLAB_POISON; break; case 'u': - slub_debug |= SLAB_STORE_USER; + *flags |= SLAB_STORE_USER; break; case 't': - slub_debug |= SLAB_TRACE; + *flags |= SLAB_TRACE; break; case 'a': - slub_debug |= SLAB_FAILSLAB; + *flags |= SLAB_FAILSLAB; break; case 'o': /* * Avoid enabling debugging on caches if its minimum * order would increase as a result. */ - disable_higher_order_debug = 1; + higher_order_disable = true; break; default: - pr_err("slub_debug option '%c' unknown. skipped\n", - *str); + if (init) + pr_err("slub_debug option '%c' unknown. skipped\n", *str); } } - check_slabs: if (*str == ',') - slub_debug_slabs = str + 1; + *slabs = ++str; + else + *slabs = NULL; + + /* Skip over the slab list */ + while (*str && *str != ';') + str++; + + /* Skip any completely empty blocks */ + while (*str && *str == ';') + str++; + + if (init && higher_order_disable) + disable_higher_order_debug = 1; + + if (*str) + return str; + else + return NULL; +} + +static int __init setup_slub_debug(char *str) +{ + slab_flags_t flags; + char *saved_str; + char *slab_list; + bool global_slub_debug_changed = false; + bool slab_list_specified = false; + + slub_debug = DEBUG_DEFAULT_FLAGS; + if (*str++ != '=' || !*str) + /* + * No options specified. Switch on full debugging. + */ + goto out; + + saved_str = str; + while (str) { + str = parse_slub_debug_flags(str, &flags, &slab_list, true); + + if (!slab_list) { + slub_debug = flags; + global_slub_debug_changed = true; + } else { + slab_list_specified = true; + } + } + + /* + * For backwards compatibility, a single list of flags with list of + * slabs means debugging is only enabled for those slabs, so the global + * slub_debug should be 0. We can extended that to multiple lists as + * long as there is no option specifying flags without a slab list. + */ + if (slab_list_specified) { + if (!global_slub_debug_changed) + slub_debug = 0; + slub_debug_string = saved_str; + } out: if ((static_branch_unlikely(&init_on_alloc) || static_branch_unlikely(&init_on_free)) && @@ -1352,36 +1416,47 @@ slab_flags_t kmem_cache_flags(unsigned i { char *iter; size_t len; + char *next_block; + slab_flags_t block_flags; /* If slub_debug = 0, it folds into the if conditional. */ - if (!slub_debug_slabs) + if (!slub_debug_string) return flags | slub_debug; len = strlen(name); - iter = slub_debug_slabs; - while (*iter) { - char *end, *glob; - size_t cmplen; - - end = strchrnul(iter, ','); - - glob = strnchr(iter, end - iter, '*'); - if (glob) - cmplen = glob - iter; - else - cmplen = max_t(size_t, len, (end - iter)); + next_block = slub_debug_string; + /* Go through all blocks of debug options, see if any matches our slab's name */ + while (next_block) { + next_block = parse_slub_debug_flags(next_block, &block_flags, &iter, false); + if (!iter) + continue; + /* Found a block that has a slab list, search it */ + while (*iter) { + char *end, *glob; + size_t cmplen; + + end = strchrnul(iter, ','); + if (next_block && next_block < end) + end = next_block - 1; + + glob = strnchr(iter, end - iter, '*'); + if (glob) + cmplen = glob - iter; + else + cmplen = max_t(size_t, len, (end - iter)); - if (!strncmp(name, iter, cmplen)) { - flags |= slub_debug; - break; - } + if (!strncmp(name, iter, cmplen)) { + flags |= block_flags; + return flags; + } - if (!*end) - break; - iter = end + 1; + if (!*end || *end == ';') + break; + iter = end + 1; + } } - return flags; + return slub_debug; } #else /* !CONFIG_SLUB_DEBUG */ static inline void setup_object_debug(struct kmem_cache *s, From patchwork Fri Aug 7 06:18:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704735 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9896413B6 for ; Fri, 7 Aug 2020 06:18:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6997F221E5 for ; Fri, 7 Aug 2020 06:18:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ljvfh/Ne" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6997F221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 337DE8D001E; Fri, 7 Aug 2020 02:18:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2C0278D000F; Fri, 7 Aug 2020 02:18:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D8018D001E; Fri, 7 Aug 2020 02:18:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id 0146D8D000F for ; Fri, 7 Aug 2020 02:18:40 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id BCCD7180AD801 for ; Fri, 7 Aug 2020 06:18:40 +0000 (UTC) X-FDA: 77122768800.02.level42_491673626fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 930C110097AA0 for ; Fri, 7 Aug 2020 06:18:40 +0000 (UTC) X-Spam-Summary: 1,0,0,a8022a43be66e7f0,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1606:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2553:2559:2566:2682:2685:2693:2859:2892:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4118:4321:4385:4605:5007:6119:6261:6653:6737:7576:7903:8599:8957:9025:9391:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12294:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13184:13200:13221:13229:13846:21080:21433:21451:21627:21819:21939:21966:21990:30029:30034:30045:30054:30062:30064:30070:30075:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygh65whskbcep7bfwoh5spmwxszoc7qjs14s91traa6kq11nyc9kewujfkjjb.4a3df9arpenskow1fjyh8dcrr5gwmy57qwgjq86dnuctj5efcd 9hqe44w7 X-HE-Tag: level42_491673626fbe X-Filterd-Recvd-Size: 7271 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:40 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id DBD8E2177B; Fri, 7 Aug 2020 06:18:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781119; bh=mCOp824s/TjDQryEW8XBnyyOzo3OO+ME6LhykY5FJCw=; h=Date:From:To:Subject:In-Reply-To:From; b=Ljvfh/Ner5ifuJVt3NcehM93X4xPypGSMCt1fU6bMJ/kFf/2cvMmLR1ZEwmuZWwrp 9fFYnFtn+oKewRKauZWHiO1KGQccwwkirQp8Nz8uAIevAf2BUJ7o3Zvk4gAjd+Tycw 3i+21DgGmeROCfJ0T/jYXdryMU9j7y/OxJuP52tg= Date: Thu, 06 Aug 2020 23:18:38 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vjitta@codeaurora.org Subject: [patch 029/163] mm, slub: make some slub_debug related attributes read-only Message-ID: <20200807061838.DsBROPWcO%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 930C110097AA0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slub: make some slub_debug related attributes read-only SLUB_DEBUG creates several files under /sys/kernel/slab// that can be read to check if the respective debugging options are enabled for given cache. The options can be also toggled at runtime by writing into the files. Some of those, namely red_zone, poison, and store_user can be toggled only when no objects yet exist in the cache. Vijayanand reports [1] that there is a problem with freelist randomization if changing the debugging option's state results in different number of objects per page, and the random sequence cache needs thus needs to be recomputed. However, another problem is that the check for "no objects yet exist in the cache" is racy, as noted by Jann [2] and fixing that would add overhead or otherwise complicate the allocation/freeing paths. Thus it would be much simpler just to remove the runtime toggling support. The documentation describes it's "In case you forgot to enable debugging on the kernel command line", but the neccessity of having no objects limits its usefulness anyway for many caches. Vijayanand describes an use case [3] where debugging is enabled for all but zram caches for memory overhead reasons, and using the runtime toggles was the only way to achieve such configuration. After the previous patch it's now possible to do that directly from the kernel boot option, so we can remove the dangerous runtime toggles by making the /sys attribute files read-only. While updating it, also improve the documentation of the debugging /sys files. [1] https://lkml.kernel.org/r/1580379523-32272-1-git-send-email-vjitta@codeaurora.org [2] https://lore.kernel.org/r/CAG48ez31PP--h6_FzVyfJ4H86QYczAFPdxtJHUEEan+7VJETAQ@mail.gmail.com [3] https://lore.kernel.org/r/1383cd32-1ddc-4dac-b5f8-9c42282fa81c@codeaurora.org Link: http://lkml.kernel.org/r/20200610163135.17364-3-vbabka@suse.cz Signed-off-by: Vlastimil Babka Reported-by: Vijayanand Jitta Reported-by: Jann Horn Reviewed-by: Kees Cook Acked-by: Roman Gushchin Cc: Christoph Lameter Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Signed-off-by: Andrew Morton --- Documentation/vm/slub.rst | 26 ++++++++++++-------- mm/slub.c | 46 ++---------------------------------- 2 files changed, 19 insertions(+), 53 deletions(-) --- a/Documentation/vm/slub.rst~mm-slub-make-some-slub_debug-related-attributes-read-only +++ a/Documentation/vm/slub.rst @@ -101,20 +101,26 @@ with "-" as options:: slub_debug=FZ;-,zs_handle,zspage -In case you forgot to enable debugging on the kernel command line: It is -possible to enable debugging manually when the kernel is up. Look at the -contents of:: +The state of each debug option for a slab can be found in the respective files +under:: /sys/kernel/slab// -Look at the writable files. Writing 1 to them will enable the -corresponding debug option. All options can be set on a slab that does -not contain objects. If the slab already contains objects then sanity checks -and tracing may only be enabled. The other options may cause the realignment -of objects. +If the file contains 1, the option is enabled, 0 means disabled. The debug +options from the ``slub_debug`` parameter translate to the following files:: -Careful with tracing: It may spew out lots of information and never stop if -used on the wrong slab. + F sanity_checks + Z red_zone + P poison + U store_user + T trace + A failslab + +The sanity_checks, trace and failslab files are writable, so writing 1 or 0 +will enable or disable the option at runtime. The writes to trace and failslab +may return -EINVAL if the cache is subject to slab merging. Careful with +tracing: It may spew out lots of information and never stop if used on the +wrong slab. Slab merging ============ --- a/mm/slub.c~mm-slub-make-some-slub_debug-related-attributes-read-only +++ a/mm/slub.c @@ -5335,61 +5335,21 @@ static ssize_t red_zone_show(struct kmem return sprintf(buf, "%d\n", !!(s->flags & SLAB_RED_ZONE)); } -static ssize_t red_zone_store(struct kmem_cache *s, - const char *buf, size_t length) -{ - if (any_slab_objects(s)) - return -EBUSY; - - s->flags &= ~SLAB_RED_ZONE; - if (buf[0] == '1') { - s->flags |= SLAB_RED_ZONE; - } - calculate_sizes(s, -1); - return length; -} -SLAB_ATTR(red_zone); +SLAB_ATTR_RO(red_zone); static ssize_t poison_show(struct kmem_cache *s, char *buf) { return sprintf(buf, "%d\n", !!(s->flags & SLAB_POISON)); } -static ssize_t poison_store(struct kmem_cache *s, - const char *buf, size_t length) -{ - if (any_slab_objects(s)) - return -EBUSY; - - s->flags &= ~SLAB_POISON; - if (buf[0] == '1') { - s->flags |= SLAB_POISON; - } - calculate_sizes(s, -1); - return length; -} -SLAB_ATTR(poison); +SLAB_ATTR_RO(poison); static ssize_t store_user_show(struct kmem_cache *s, char *buf) { return sprintf(buf, "%d\n", !!(s->flags & SLAB_STORE_USER)); } -static ssize_t store_user_store(struct kmem_cache *s, - const char *buf, size_t length) -{ - if (any_slab_objects(s)) - return -EBUSY; - - s->flags &= ~SLAB_STORE_USER; - if (buf[0] == '1') { - s->flags &= ~__CMPXCHG_DOUBLE; - s->flags |= SLAB_STORE_USER; - } - calculate_sizes(s, -1); - return length; -} -SLAB_ATTR(store_user); +SLAB_ATTR_RO(store_user); static ssize_t validate_show(struct kmem_cache *s, char *buf) { From patchwork Fri Aug 7 06:18:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704737 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E78C913B6 for ; Fri, 7 Aug 2020 06:18:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BA8AD2177B for ; Fri, 7 Aug 2020 06:18:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="VJmwIGlO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA8AD2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7F2078D001F; Fri, 7 Aug 2020 02:18:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7A3C98D000F; Fri, 7 Aug 2020 02:18:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61DBF8D001F; Fri, 7 Aug 2020 02:18:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0056.hostedemail.com [216.40.44.56]) by kanga.kvack.org (Postfix) with ESMTP id 45F078D000F for ; Fri, 7 Aug 2020 02:18:44 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0AF53180AD801 for ; Fri, 7 Aug 2020 06:18:44 +0000 (UTC) X-FDA: 77122768968.25.tank91_1b0f20c26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id CF9EA1804E3A0 for ; Fri, 7 Aug 2020 06:18:43 +0000 (UTC) X-Spam-Summary: 1,0,0,6dcb4db0b2651500,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:560:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2564:2682:2685:2731:2859:2895:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3503:3504:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:5007:6261:6653:6737:7576:7903:8599:8957:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13138:13153:13221:13228:13229:13231:13846:14181:14721:14849:21080:21451:21627:21939:21990:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8ihceh5oyywuyb9sc4kcirqihgypcpg39gi3n1f1j846ocgk6k4s1bxkyj3j.uox78emwfz37zuws5uxx3k15raotfj55i3xof7u5yxja5b43new7zmbxitn7u1b.g-lbl8.mailshell.net-223.238.255.100 ,CacheIP X-HE-Tag: tank91_1b0f20c26fbe X-Filterd-Recvd-Size: 4135 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:43 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 24DD322C9F; Fri, 7 Aug 2020 06:18:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781122; bh=qXxWzoMHq2LhcsLbtQJhp5AAWhZEXTOc92gI7jyXIaE=; h=Date:From:To:Subject:In-Reply-To:From; b=VJmwIGlOgz2l/jKEhMCgkSD4OMGSNS6QvQ0Wuy9IVvGkVQ17pNcihRvDcOAqS6OLH ik0cr84mxDj/DWWhfyTpn71llme6NskLi7gQG6TIOh6+jQWyFV0/feTSJIudDZVVmK 95T+kES9SEFqtZXzagpiIOvEPJG6FRt00iUNh8+4= Date: Thu, 06 Aug 2020 23:18:41 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vjitta@codeaurora.org Subject: [patch 030/163] mm, slub: remove runtime allocation order changes Message-ID: <20200807061841.JutInBYGY%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: CF9EA1804E3A0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slub: remove runtime allocation order changes SLUB allows runtime changing of page allocation order by writing into the /sys/kernel/slab//order file. Jann has reported [1] that this interface allows the order to be set too small, leading to crashes. While it's possible to fix the immediate issue, closer inspection reveals potential races. Storing the new order calls calculate_sizes() which non-atomically updates a lot of kmem_cache fields while the cache is still in use. Unexpected behavior might occur even if the fields are set to the same value as they were. This could be fixed by splitting out the part of calculate_sizes() that depends on forced_order, so that we only update kmem_cache.oo field. This could still race with init_cache_random_seq(), shuffle_freelist(), allocate_slab(). Perhaps it's possible to audit and e.g. add some READ_ONCE/WRITE_ONCE accesses, it might be easier just to remove the runtime order changes, which is what this patch does. If there are valid usecases for per-cache order setting, we could e.g. extend the boot parameters to do that. [1] https://lore.kernel.org/r/CAG48ez31PP--h6_FzVyfJ4H86QYczAFPdxtJHUEEan+7VJETAQ@mail.gmail.com Link: http://lkml.kernel.org/r/20200610163135.17364-4-vbabka@suse.cz Signed-off-by: Vlastimil Babka Acked-by: Christoph Lameter Reported-by: Jann Horn Reviewed-by: Kees Cook Acked-by: Roman Gushchin Cc: Vijayanand Jitta Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Signed-off-by: Andrew Morton --- mm/slub.c | 19 +------------------ 1 file changed, 1 insertion(+), 18 deletions(-) --- a/mm/slub.c~mm-slub-remove-runtime-allocation-order-changes +++ a/mm/slub.c @@ -5095,28 +5095,11 @@ static ssize_t objs_per_slab_show(struct } SLAB_ATTR_RO(objs_per_slab); -static ssize_t order_store(struct kmem_cache *s, - const char *buf, size_t length) -{ - unsigned int order; - int err; - - err = kstrtouint(buf, 10, &order); - if (err) - return err; - - if (order > slub_max_order || order < slub_min_order) - return -EINVAL; - - calculate_sizes(s, order); - return length; -} - static ssize_t order_show(struct kmem_cache *s, char *buf) { return sprintf(buf, "%u\n", oo_order(s->oo)); } -SLAB_ATTR(order); +SLAB_ATTR_RO(order); static ssize_t min_partial_show(struct kmem_cache *s, char *buf) { From patchwork Fri Aug 7 06:18:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704739 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 72C5413B6 for ; Fri, 7 Aug 2020 06:18:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 475F922C9F for ; Fri, 7 Aug 2020 06:18:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="mYw5yV9R" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 475F922C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 274868D0020; Fri, 7 Aug 2020 02:18:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 225738D000F; Fri, 7 Aug 2020 02:18:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13AEC8D0020; Fri, 7 Aug 2020 02:18:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0235.hostedemail.com [216.40.44.235]) by kanga.kvack.org (Postfix) with ESMTP id ED00D8D000F for ; Fri, 7 Aug 2020 02:18:47 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B1FCC2C79 for ; Fri, 7 Aug 2020 06:18:47 +0000 (UTC) X-FDA: 77122769094.08.hot68_57166ac26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 789DB1819E76C for ; Fri, 7 Aug 2020 06:18:47 +0000 (UTC) X-Spam-Summary: 1,0,0,1a88b846ed4d35e9,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1544:1605:1711:1730:1747:1777:1792:2198:2199:2393:2525:2553:2559:2563:2682:2685:2693:2859:2892:2895:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4117:4321:4470:4605:5007:6119:6261:6653:6737:7576:7903:7904:8599:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12294:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13138:13153:13161:13228:13229:13231:13255:13846:14181:14721:14849:21080:21433:21451:21627:21939:21966:21990:30054:30062:30064:30070:30075:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yge51w6px8gmnupc31izqqeqkj9ocz4xpjee9g48zs1uebw3d5prjjk84w8r3.rajtiozm4iqa8eg7iqir6uw7rr9kzo3x57fsjw9ehdb7kx17441tfazrxizq 6tw.c-lb X-HE-Tag: hot68_57166ac26fbe X-Filterd-Recvd-Size: 6087 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:46 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 76B4622CAF; Fri, 7 Aug 2020 06:18:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781126; bh=gO2FE7FeoupRREpR0ndTnvKOHLpWsYeQ0jFwu6mPS00=; h=Date:From:To:Subject:In-Reply-To:From; b=mYw5yV9R0NQ0V0lGvfnxnwolXGHhzWPtL+iFyQebwLOVxXENMCzqqFdfWLOwk1p68 lrqSmbwUZ/udr1Aq3W5pXscjEBRFL1eRZxUKcX3fcJo+Ay0xFqwQChY99+qrl540h+ gp95h2tmz0p9Z9tF19aYGzMMhhs7QY7wHZeCYYMg= Date: Thu, 06 Aug 2020 23:18:45 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vjitta@codeaurora.org Subject: [patch 031/163] mm, slub: make remaining slub_debug related attributes read-only Message-ID: <20200807061845.TEH4F98qc%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 789DB1819E76C X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slub: make remaining slub_debug related attributes read-only SLUB_DEBUG creates several files under /sys/kernel/slab// that can be read to check if the respective debugging options are enabled for given cache. Some options, namely sanity_checks, trace, and failslab can be also enabled and disabled at runtime by writing into the files. The runtime toggling is racy. Some options disable __CMPXCHG_DOUBLE when enabled, which means that in case of concurrent allocations, some can still use __CMPXCHG_DOUBLE and some not, leading to potential corruption. The s->flags field is also not updated or checked atomically. The simplest solution is to remove the runtime toggling. The extended slub_debug boot parameter syntax introduced by earlier patch should allow to fine-tune the debugging configuration during boot with same granularity. Link: http://lkml.kernel.org/r/20200610163135.17364-5-vbabka@suse.cz Signed-off-by: Vlastimil Babka Reviewed-by: Kees Cook Acked-by: Roman Gushchin Cc: Christoph Lameter Cc: Jann Horn Cc: Vijayanand Jitta Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Signed-off-by: Andrew Morton --- Documentation/vm/slub.rst | 7 +--- mm/slub.c | 62 +----------------------------------- 2 files changed, 5 insertions(+), 64 deletions(-) --- a/Documentation/vm/slub.rst~mm-slub-make-remaining-slub_debug-related-attributes-read-only +++ a/Documentation/vm/slub.rst @@ -116,11 +116,8 @@ options from the ``slub_debug`` paramete T trace A failslab -The sanity_checks, trace and failslab files are writable, so writing 1 or 0 -will enable or disable the option at runtime. The writes to trace and failslab -may return -EINVAL if the cache is subject to slab merging. Careful with -tracing: It may spew out lots of information and never stop if used on the -wrong slab. +Careful with tracing: It may spew out lots of information and never stop if +used on the wrong slab. Slab merging ============ --- a/mm/slub.c~mm-slub-make-remaining-slub_debug-related-attributes-read-only +++ a/mm/slub.c @@ -5040,20 +5040,6 @@ static ssize_t show_slab_objects(struct return x + sprintf(buf + x, "\n"); } -#ifdef CONFIG_SLUB_DEBUG -static int any_slab_objects(struct kmem_cache *s) -{ - int node; - struct kmem_cache_node *n; - - for_each_kmem_cache_node(s, node, n) - if (atomic_long_read(&n->total_objects)) - return 1; - - return 0; -} -#endif - #define to_slab_attr(n) container_of(n, struct slab_attribute, attr) #define to_slab(n) container_of(n, struct kmem_cache, kobj) @@ -5275,43 +5261,13 @@ static ssize_t sanity_checks_show(struct { return sprintf(buf, "%d\n", !!(s->flags & SLAB_CONSISTENCY_CHECKS)); } - -static ssize_t sanity_checks_store(struct kmem_cache *s, - const char *buf, size_t length) -{ - s->flags &= ~SLAB_CONSISTENCY_CHECKS; - if (buf[0] == '1') { - s->flags &= ~__CMPXCHG_DOUBLE; - s->flags |= SLAB_CONSISTENCY_CHECKS; - } - return length; -} -SLAB_ATTR(sanity_checks); +SLAB_ATTR_RO(sanity_checks); static ssize_t trace_show(struct kmem_cache *s, char *buf) { return sprintf(buf, "%d\n", !!(s->flags & SLAB_TRACE)); } - -static ssize_t trace_store(struct kmem_cache *s, const char *buf, - size_t length) -{ - /* - * Tracing a merged cache is going to give confusing results - * as well as cause other issues like converting a mergeable - * cache into an umergeable one. - */ - if (s->refcount > 1) - return -EINVAL; - - s->flags &= ~SLAB_TRACE; - if (buf[0] == '1') { - s->flags &= ~__CMPXCHG_DOUBLE; - s->flags |= SLAB_TRACE; - } - return length; -} -SLAB_ATTR(trace); +SLAB_ATTR_RO(trace); static ssize_t red_zone_show(struct kmem_cache *s, char *buf) { @@ -5375,19 +5331,7 @@ static ssize_t failslab_show(struct kmem { return sprintf(buf, "%d\n", !!(s->flags & SLAB_FAILSLAB)); } - -static ssize_t failslab_store(struct kmem_cache *s, const char *buf, - size_t length) -{ - if (s->refcount > 1) - return -EINVAL; - - s->flags &= ~SLAB_FAILSLAB; - if (buf[0] == '1') - s->flags |= SLAB_FAILSLAB; - return length; -} -SLAB_ATTR(failslab); +SLAB_ATTR_RO(failslab); #endif static ssize_t shrink_show(struct kmem_cache *s, char *buf) From patchwork Fri Aug 7 06:18:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704741 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7E51214B7 for ; Fri, 7 Aug 2020 06:18:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4FD7122D07 for ; Fri, 7 Aug 2020 06:18:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="csOCmnfY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4FD7122D07 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 13CAC8D0021; Fri, 7 Aug 2020 02:18:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0C54D8D000F; Fri, 7 Aug 2020 02:18:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 001AE8D0021; Fri, 7 Aug 2020 02:18:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0025.hostedemail.com [216.40.44.25]) by kanga.kvack.org (Postfix) with ESMTP id DC7ED8D000F for ; Fri, 7 Aug 2020 02:18:50 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A47F93641 for ; Fri, 7 Aug 2020 06:18:50 +0000 (UTC) X-FDA: 77122769220.18.unit03_46016a526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 823E3100EC676 for ; Fri, 7 Aug 2020 06:18:50 +0000 (UTC) X-Spam-Summary: 1,0,0,93f63a936ffa75f0,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2859:2895:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6119:6261:6653:6737:7576:7903:8599:8784:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13069:13161:13229:13311:13357:13846:14181:14384:14721:14849:21080:21450:21451:21627:21795:21939:21990:30051:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfw893eo1si773y7zrdr9bqb67woc7rq31an3rj9rak6z3zyojm8u5muhnfzo.rtwpih7kbhd5c38ujba5qq4s6tfbj7xi4j6xwnfmn783zscdhzzth8nz1sct1xi.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0 .5,0.5,0 X-HE-Tag: unit03_46016a526fbe X-Filterd-Recvd-Size: 3360 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:50 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1620922CB3; Fri, 7 Aug 2020 06:18:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781129; bh=MfO/fZsVOJp9DxsWMlg6hC+zRDpAKLgREa7mDK4tp+Q=; h=Date:From:To:Subject:In-Reply-To:From; b=csOCmnfY8MqClNgY5H17Q5Lzk/IQ6RD/ss+Jk369U4tzCGN7NYU3NmArV2+f2bGPD NJhwYsA8hHiwrDOBgfaR67LiDzYoIcDXMXG8naHzHzs0p8QRQcaPPjqLL7VcKWey0H uLVzS0J+S9nb9IRJ5TWwTkFoQkLJmVWTlfWuXnrQ= Date: Thu, 06 Aug 2020 23:18:48 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vjitta@codeaurora.org Subject: [patch 032/163] mm, slub: make reclaim_account attribute read-only Message-ID: <20200807061848.vF8DVDKGT%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 823E3100EC676 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slub: make reclaim_account attribute read-only The attribute reflects the SLAB_RECLAIM_ACCOUNT cache flag. It's not clear why this attribute was writable in the first place, as it's tied to how the cache is used by its creator, it's not a user tunable. Furthermore: - it affects slab merging, but that's not being checked while toggled - if affects whether __GFP_RECLAIMABLE flag is used to allocate page, but the runtime toggle doesn't update allocflags - it affects cache_vmstat_idx() so runtime toggling might lead to incosistency of NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE Thus make it read-only. Link: http://lkml.kernel.org/r/20200610163135.17364-6-vbabka@suse.cz Signed-off-by: Vlastimil Babka Reviewed-by: Kees Cook Acked-by: Roman Gushchin Cc: Christoph Lameter Cc: Jann Horn Cc: Vijayanand Jitta Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Signed-off-by: Andrew Morton --- mm/slub.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) --- a/mm/slub.c~mm-slub-make-reclaim_account-attribute-read-only +++ a/mm/slub.c @@ -5207,16 +5207,7 @@ static ssize_t reclaim_account_show(stru { return sprintf(buf, "%d\n", !!(s->flags & SLAB_RECLAIM_ACCOUNT)); } - -static ssize_t reclaim_account_store(struct kmem_cache *s, - const char *buf, size_t length) -{ - s->flags &= ~SLAB_RECLAIM_ACCOUNT; - if (buf[0] == '1') - s->flags |= SLAB_RECLAIM_ACCOUNT; - return length; -} -SLAB_ATTR(reclaim_account); +SLAB_ATTR_RO(reclaim_account); static ssize_t hwcache_align_show(struct kmem_cache *s, char *buf) { From patchwork Fri Aug 7 06:18:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704743 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 241C514B7 for ; Fri, 7 Aug 2020 06:18:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E5608221E5 for ; Fri, 7 Aug 2020 06:18:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="1ow2ptCk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E5608221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B27098D0022; Fri, 7 Aug 2020 02:18:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AAF2C8D000F; Fri, 7 Aug 2020 02:18:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C70C8D0022; Fri, 7 Aug 2020 02:18:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0193.hostedemail.com [216.40.44.193]) by kanga.kvack.org (Postfix) with ESMTP id 81FEC8D000F for ; Fri, 7 Aug 2020 02:18:54 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 43A4B181AEF1A for ; Fri, 7 Aug 2020 06:18:54 +0000 (UTC) X-FDA: 77122769388.25.eye95_30113eb26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 1600E1804E3A1 for ; Fri, 7 Aug 2020 06:18:54 +0000 (UTC) X-Spam-Summary: 1,0,0,3114707b1c84f809,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:1981:2194:2196:2198:2199:2200:2201:2393:2525:2559:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3868:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6261:6653:6737:7576:7808:8599:8660:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13148:13156:13161:13228:13229:13230:13846:14040:14096:14181:14721:14849:21080:21451:21627:21939:21990:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yg4p8m8fccw3hqur5ycwurfz639ypkicm95mjsxiehqimd7n1skdi3f38miqn.er4k1yabuquybybf8eym4ii4y6z9ceyejpf9xmyzbmk1fjth4yihcfase6fh54t.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayes ian:0.5, X-HE-Tag: eye95_30113eb26fbe X-Filterd-Recvd-Size: 3962 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:53 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3EB53221E5; Fri, 7 Aug 2020 06:18:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781132; bh=TzKTOHb/3HVIfRAodnOtDgX1cgsKC/SqYP1Hve5BuWA=; h=Date:From:To:Subject:In-Reply-To:From; b=1ow2ptCk4R/3jGWVF0SjVQFDVMl1lIwCEjUghriOoGqN/U5s/B3B46CHJwG6XcG2H QgSV+tRlLBTfeRNhQCRJscqN2RzPl5vNGtHQu5Ot5kaUGgdmEdfJADNK2lR304nize MwpmiWo0cwRMMG0Xkdqcxc1SSG/e1P17hZSQlZD0= Date: Thu, 06 Aug 2020 23:18:51 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vjitta@codeaurora.org Subject: [patch 033/163] mm, slub: introduce static key for slub_debug() Message-ID: <20200807061851.QI1qSidy8%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 1600E1804E3A1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slub: introduce static key for slub_debug() One advantage of CONFIG_SLUB_DEBUG is that a generic distro kernel can be built with the option enabled, but it's inactive until simply enabled on boot, without rebuilding the kernel. With a static key, we can further eliminate the overhead of checking whether a cache has a particular debug flag enabled if we know that there are no such caches (slub_debug was not enabled during boot). We use the same mechanism also for e.g. page_owner, debug_pagealloc or kmemcg functionality. This patch introduces the static key and makes the general check for per-cache debug flags kmem_cache_debug() use it. This benefits several call sites, including (slow path but still rather frequent) __slab_free(). The next patches will add more uses. Link: http://lkml.kernel.org/r/20200610163135.17364-7-vbabka@suse.cz Signed-off-by: Vlastimil Babka Acked-by: Roman Gushchin Acked-by: Christoph Lameter Reviewed-by: Kees Cook Cc: Jann Horn Cc: Vijayanand Jitta Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Signed-off-by: Andrew Morton --- mm/slub.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) --- a/mm/slub.c~mm-slub-introduce-static-key-for-slub_debug +++ a/mm/slub.c @@ -114,13 +114,21 @@ * the fast path and disables lockless freelists. */ +#ifdef CONFIG_SLUB_DEBUG +#ifdef CONFIG_SLUB_DEBUG_ON +DEFINE_STATIC_KEY_TRUE(slub_debug_enabled); +#else +DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); +#endif +#endif + static inline int kmem_cache_debug(struct kmem_cache *s) { #ifdef CONFIG_SLUB_DEBUG - return unlikely(s->flags & SLAB_DEBUG_FLAGS); -#else - return 0; + if (static_branch_unlikely(&slub_debug_enabled)) + return s->flags & SLAB_DEBUG_FLAGS; #endif + return 0; } void *fixup_red_left(struct kmem_cache *s, void *p) @@ -1389,6 +1397,8 @@ static int __init setup_slub_debug(char slub_debug_string = saved_str; } out: + if (slub_debug != 0 || slub_debug_string) + static_branch_enable(&slub_debug_enabled); if ((static_branch_unlikely(&init_on_alloc) || static_branch_unlikely(&init_on_free)) && (slub_debug & SLAB_POISON)) From patchwork Fri Aug 7 06:18:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704745 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5229913B6 for ; Fri, 7 Aug 2020 06:18:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1AB0722CB3 for ; Fri, 7 Aug 2020 06:18:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="px2oTUnb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1AB0722CB3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B75BB8D0023; Fri, 7 Aug 2020 02:18:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B24F28D000F; Fri, 7 Aug 2020 02:18:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A628F8D0023; Fri, 7 Aug 2020 02:18:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 8CBC98D000F for ; Fri, 7 Aug 2020 02:18:57 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5873882499A8 for ; Fri, 7 Aug 2020 06:18:57 +0000 (UTC) X-FDA: 77122769514.22.ray96_200d40d26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id 2E3F118038E68 for ; Fri, 7 Aug 2020 06:18:57 +0000 (UTC) X-Spam-Summary: 1,0,0,aadeded76ce71728,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2564:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6119:6261:6653:6737:7576:7808:7875:7903:8599:8603:8660:9025:9545:10004:10913:11026:11257:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13148:13161:13229:13230:13846:14096:14181:14721:14849:21080:21220:21451:21627:21740:21939:21990:30012:30029:30054:30064:30070:30091,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yr48crnmb1c5wkg6pherb1ia96mop5mjog8iytwszjmhgef34s1t5js5hyhg5.i6y9osf93ba7phxs7xein9nw3caxqx8atm5imrozphqwtaf8c44ufb39b9kuwsn.s-lbl8.mailshell.net-223.238.2 55.100,C X-HE-Tag: ray96_200d40d26fbe X-Filterd-Recvd-Size: 4306 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:18:56 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8A7FD22CAF; Fri, 7 Aug 2020 06:18:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781136; bh=qQTtSDa6o1vI/6AAveAi9Qdhgiu0qNk/FT0N/qiTuVw=; h=Date:From:To:Subject:In-Reply-To:From; b=px2oTUnbsROli+QWpZEZIMiEfw/nCl+NmAtvrT9IV0woEh3eoXhzgn5igw0hqzV+a GPBANSUtwFnb6xe2P+xuNFQ3lKXEdh1z5OWyR1i+l4SvKn34iz7Oi4fGMnBxOGzMxt mg2cUcs5ycZHP2QaO8PsCFvZtt4ovRAIkYopL6S4= Date: Thu, 06 Aug 2020 23:18:55 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vjitta@codeaurora.org Subject: [patch 034/163] mm, slub: introduce kmem_cache_debug_flags() Message-ID: <20200807061855.wsvIYh1Qy%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 2E3F118038E68 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slub: introduce kmem_cache_debug_flags() There are few places that call kmem_cache_debug(s) (which tests if any of debug flags are enabled for a cache) immediately followed by a test for a specific flag. The compiler can probably eliminate the extra check, but we can make the code nicer by introducing kmem_cache_debug_flags() that works like kmem_cache_debug() (including the static key check) but tests for specific flag(s). The next patches will add more users. [vbabka@suse.cz: change return from int to bool, per Kees. Add VM_WARN_ON_ONCE() for invalid flags, per Roman] Link: http://lkml.kernel.org/r/949b90ed-e0f0-07d7-4d21-e30ec0958a7c@suse.cz Link: http://lkml.kernel.org/r/20200610163135.17364-8-vbabka@suse.cz Signed-off-by: Vlastimil Babka Acked-by: Roman Gushchin Acked-by: Christoph Lameter Acked-by: Kees Cook Cc: Jann Horn Cc: Vijayanand Jitta Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Signed-off-by: Andrew Morton --- mm/slub.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) --- a/mm/slub.c~mm-slub-introduce-kmem_cache_debug_flags +++ a/mm/slub.c @@ -122,18 +122,29 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabl #endif #endif -static inline int kmem_cache_debug(struct kmem_cache *s) +/* + * Returns true if any of the specified slub_debug flags is enabled for the + * cache. Use only for flags parsed by setup_slub_debug() as it also enables + * the static key. + */ +static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t flags) { + VM_WARN_ON_ONCE(!(flags & SLAB_DEBUG_FLAGS)); #ifdef CONFIG_SLUB_DEBUG if (static_branch_unlikely(&slub_debug_enabled)) - return s->flags & SLAB_DEBUG_FLAGS; + return s->flags & flags; #endif - return 0; + return false; +} + +static inline bool kmem_cache_debug(struct kmem_cache *s) +{ + return kmem_cache_debug_flags(s, SLAB_DEBUG_FLAGS); } void *fixup_red_left(struct kmem_cache *s, void *p) { - if (kmem_cache_debug(s) && s->flags & SLAB_RED_ZONE) + if (kmem_cache_debug_flags(s, SLAB_RED_ZONE)) p += s->red_left_pad; return p; @@ -4060,7 +4071,7 @@ void __check_heap_object(const void *ptr offset = (ptr - page_address(page)) % s->size; /* Adjust for redzone and reject if within the redzone. */ - if (kmem_cache_debug(s) && s->flags & SLAB_RED_ZONE) { + if (kmem_cache_debug_flags(s, SLAB_RED_ZONE)) { if (offset < s->red_left_pad) usercopy_abort("SLUB object in left red zone", s->name, to_user, offset, n); From patchwork Fri Aug 7 06:18:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704747 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B40A513B6 for ; Fri, 7 Aug 2020 06:19:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7D0AF22C9F for ; Fri, 7 Aug 2020 06:19:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="s3tUUtDq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7D0AF22C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3865C8D0024; Fri, 7 Aug 2020 02:19:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 30E918D000F; Fri, 7 Aug 2020 02:19:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FD818D0024; Fri, 7 Aug 2020 02:19:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0029.hostedemail.com [216.40.44.29]) by kanga.kvack.org (Postfix) with ESMTP id 042828D000F for ; Fri, 7 Aug 2020 02:19:01 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C3591181AEF1A for ; Fri, 7 Aug 2020 06:19:00 +0000 (UTC) X-FDA: 77122769640.08.yard48_3b02c2f26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 9B9B71819E76C for ; Fri, 7 Aug 2020 06:19:00 +0000 (UTC) X-Spam-Summary: 1,0,0,702663f339033337,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3870:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:5007:6261:6653:6737:7576:7875:7901:7903:8599:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13229:13846:14040:14096:14181:14721:14849:21080:21451:21627:21939:21990:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yr3ndu6esh9tnsfctzqqt48ctwropycw9k8mx5wfhd7hurnux8e98wdmcoc7o.rf8yihs8ykuagafs7yammwq8nhbfxmgge5dzjii9bt668443r87wd1rutmsyxue.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,M SF:not b X-HE-Tag: yard48_3b02c2f26fbe X-Filterd-Recvd-Size: 3529 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:00 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id DCC3F2177B; Fri, 7 Aug 2020 06:18:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781139; bh=bc48BL73VXwV/bxiX0u8KlfYKhcdsULjFa5h3rzwa/g=; h=Date:From:To:Subject:In-Reply-To:From; b=s3tUUtDqyjMcH4mF9j+ZpHeY1PtxjWeQWUyDlmwawgs6w2oyJGpq7teRL7arNLKBY v2FzHTK+3DGWznvJXPDaUJBmRIwKP0KMvtlhesVSUQLvqfjHmgM8p3hU3dSikYrIoK Df52xeRA5alhMfvZziB1ZqQNK1t4rHZgYUsDnrl8= Date: Thu, 06 Aug 2020 23:18:58 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vjitta@codeaurora.org Subject: [patch 035/163] mm, slub: extend checks guarded by slub_debug static key Message-ID: <20200807061858.P7bD7iz08%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 9B9B71819E76C X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slub: extend checks guarded by slub_debug static key There are few more places in SLUB that could benefit from reduced overhead of the static key introduced by a previous patch: - setup_object_debug() called on each object in newly allocated slab page - setup_page_debug() called on newly allocated slab page - __free_slab() called on freed slab page Link: http://lkml.kernel.org/r/20200610163135.17364-9-vbabka@suse.cz Signed-off-by: Vlastimil Babka Acked-by: Roman Gushchin Acked-by: Christoph Lameter Cc: Jann Horn Cc: Kees Cook Cc: Vijayanand Jitta Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Signed-off-by: Andrew Morton --- mm/slub.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --- a/mm/slub.c~mm-slub-extend-checks-guarded-by-slub_debug-static-key +++ a/mm/slub.c @@ -1131,7 +1131,7 @@ static inline void dec_slabs_node(struct static void setup_object_debug(struct kmem_cache *s, struct page *page, void *object) { - if (!(s->flags & (SLAB_STORE_USER|SLAB_RED_ZONE|__OBJECT_POISON))) + if (!kmem_cache_debug_flags(s, SLAB_STORE_USER|SLAB_RED_ZONE|__OBJECT_POISON)) return; init_object(s, object, SLUB_RED_INACTIVE); @@ -1141,7 +1141,7 @@ static void setup_object_debug(struct km static void setup_page_debug(struct kmem_cache *s, struct page *page, void *addr) { - if (!(s->flags & SLAB_POISON)) + if (!kmem_cache_debug_flags(s, SLAB_POISON)) return; metadata_access_enable(); @@ -1853,7 +1853,7 @@ static void __free_slab(struct kmem_cach int order = compound_order(page); int pages = 1 << order; - if (s->flags & SLAB_CONSISTENCY_CHECKS) { + if (kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) { void *p; slab_pad_check(s, page); From patchwork Fri Aug 7 06:19:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704749 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B097C14B7 for ; Fri, 7 Aug 2020 06:19:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7EABB22D08 for ; Fri, 7 Aug 2020 06:19:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Q7irohVy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7EABB22D08 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 421C98D0025; Fri, 7 Aug 2020 02:19:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3D17B8D000F; Fri, 7 Aug 2020 02:19:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C2238D0025; Fri, 7 Aug 2020 02:19:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id 11A9C8D000F for ; Fri, 7 Aug 2020 02:19:05 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CCA963646 for ; Fri, 7 Aug 2020 06:19:04 +0000 (UTC) X-FDA: 77122769808.01.shake20_280ea2526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id A161F1004CB30 for ; Fri, 7 Aug 2020 06:19:04 +0000 (UTC) X-Spam-Summary: 1,0,0,e7c8e4e84f57d1a7,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1606:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2564:2682:2685:2693:2730:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4117:4250:4321:4385:5007:6261:6653:6737:7576:7875:7903:8599:8603:9025:9545:9592:10004:10913:11026:11232:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13221:13229:13846:21080:21324:21450:21451:21627:21740:21939:21972:21990:30029:30054:30069:30070:30075,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yrmtre9874qx3emzsnahcxtrosiop1arr41pwj5fuwn13rcgag7qun7einxgd.1t614of8zt9r7xh91zkmakeu48qfm8myoj83q8cgqmy1y6zy3hiu3zgyhdejcb5.o-lbl8.mailshell.net-223.238.25 5.100,Ca X-HE-Tag: shake20_280ea2526fbe X-Filterd-Recvd-Size: 6561 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:04 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E84952177B; Fri, 7 Aug 2020 06:19:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781143; bh=sM+cKwXInKuDWo/rzdw756i65GUH6Xv44WZJMYlKGX4=; h=Date:From:To:Subject:In-Reply-To:From; b=Q7irohVyW3h5vhzaXQ7fwFM0/EPhg6MGP8h371ys+e31KNrBF1/7sIDN8oMVNCUFu pYnI1bd20e7ArLNI8tIhcl7Dx7XUsemIKjStaV89aoDDrUtRqkhuIOXcT+5QLGji0p 5f7hCfq0Vd0UKnvB3K5eoFD2PAOSvW8mcVlWgAVk= Date: Thu, 06 Aug 2020 23:19:01 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vjitta@codeaurora.org Subject: [patch 036/163] mm, slab/slub: move and improve cache_from_obj() Message-ID: <20200807061901.kDVYQY8r_%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: A161F1004CB30 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slab/slub: move and improve cache_from_obj() The function cache_from_obj() was added by commit b9ce5ef49f00 ("sl[au]b: always get the cache from its page in kmem_cache_free()") to support kmemcg, where per-memcg cache can be different from the root one, so we can't use the kmem_cache pointer given to kmem_cache_free(). Prior to that commit, SLUB already had debugging check+warning that could be enabled to compare the given kmem_cache pointer to one referenced by the slab page where the object-to-be-freed resides. This check was moved to cache_from_obj(). Later the check was also enabled for SLAB_FREELIST_HARDENED configs by commit 598a0717a816 ("mm/slab: validate cache membership under freelist hardening"). These checks and warnings can be useful especially for the debugging, which can be improved. Commit 598a0717a816 changed the pr_err() with WARN_ON_ONCE() to WARN_ONCE() so only the first hit is now reported, others are silent. This patch changes it to WARN() so that all errors are reported. It's also useful to print SLUB allocation/free tracking info for the offending object, if tracking is enabled. We could export the SLUB print_tracking() function and provide an empty one for SLAB, or realize that both the debugging and hardening cases in cache_from_obj() are only supported by SLUB anyway. So this patch moves cache_from_obj() from slab.h to separate instances in slab.c and slub.c, where the SLAB version only does the kmemcg lookup and even could be completely removed once the kmemcg rework [1] is merged. The SLUB version can thus easily use the print_tracking() function. It can also use the kmem_cache_debug_flags() static key check for improved performance in kernels without the hardening and with debugging not enabled on boot. [1] https://lore.kernel.org/r/20200608230654.828134-18-guro@fb.com Link: http://lkml.kernel.org/r/20200610163135.17364-10-vbabka@suse.cz Signed-off-by: Vlastimil Babka Cc: Christoph Lameter Cc: Jann Horn Cc: Kees Cook Cc: Vijayanand Jitta Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Signed-off-by: Andrew Morton --- mm/slab.c | 8 ++++++++ mm/slab.h | 23 ----------------------- mm/slub.c | 21 +++++++++++++++++++++ 3 files changed, 29 insertions(+), 23 deletions(-) --- a/mm/slab.c~mm-slab-slub-move-and-improve-cache_from_obj +++ a/mm/slab.c @@ -3678,6 +3678,14 @@ void *__kmalloc_track_caller(size_t size } EXPORT_SYMBOL(__kmalloc_track_caller); +static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) +{ + if (memcg_kmem_enabled()) + return virt_to_cache(x); + else + return s; +} + /** * kmem_cache_free - Deallocate an object * @cachep: The cache the allocation was from. --- a/mm/slab.h~mm-slab-slub-move-and-improve-cache_from_obj +++ a/mm/slab.h @@ -504,29 +504,6 @@ static __always_inline void uncharge_sla memcg_uncharge_slab(page, order, s); } -static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) -{ - struct kmem_cache *cachep; - - /* - * When kmemcg is not being used, both assignments should return the - * same value. but we don't want to pay the assignment price in that - * case. If it is not compiled in, the compiler should be smart enough - * to not do even the assignment. In that case, slab_equal_or_root - * will also be a constant. - */ - if (!memcg_kmem_enabled() && - !IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) && - !unlikely(s->flags & SLAB_CONSISTENCY_CHECKS)) - return s; - - cachep = virt_to_cache(x); - WARN_ONCE(cachep && !slab_equal_or_root(cachep, s), - "%s: Wrong slab cache. %s but object is from %s\n", - __func__, s->name, cachep->name); - return cachep; -} - static inline size_t slab_ksize(const struct kmem_cache *s) { #ifndef CONFIG_SLUB --- a/mm/slub.c~mm-slab-slub-move-and-improve-cache_from_obj +++ a/mm/slub.c @@ -1525,6 +1525,10 @@ static bool freelist_corrupted(struct km { return false; } + +static void print_tracking(struct kmem_cache *s, void *object) +{ +} #endif /* CONFIG_SLUB_DEBUG */ /* @@ -3171,6 +3175,23 @@ void ___cache_free(struct kmem_cache *ca } #endif +static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) +{ + struct kmem_cache *cachep; + + if (!IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) && + !memcg_kmem_enabled() && + !kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) + return s; + + cachep = virt_to_cache(x); + if (WARN(cachep && !slab_equal_or_root(cachep, s), + "%s: Wrong slab cache. %s but object is from %s\n", + __func__, s->name, cachep->name)) + print_tracking(cachep, x); + return cachep; +} + void kmem_cache_free(struct kmem_cache *s, void *x) { s = cache_from_obj(s, x); From patchwork Fri Aug 7 06:19:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704751 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7452513B6 for ; Fri, 7 Aug 2020 06:19:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3E94422CAF for ; Fri, 7 Aug 2020 06:19:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="KxkwqTzb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E94422CAF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EF7808D000F; Fri, 7 Aug 2020 02:19:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ECF658D0026; Fri, 7 Aug 2020 02:19:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D9EE78D000F; Fri, 7 Aug 2020 02:19:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0066.hostedemail.com [216.40.44.66]) by kanga.kvack.org (Postfix) with ESMTP id BF1F58D000F for ; Fri, 7 Aug 2020 02:19:08 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 81C3B180AD802 for ; Fri, 7 Aug 2020 06:19:08 +0000 (UTC) X-FDA: 77122769976.24.shop86_5e103a626fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 5B75B1A4A5 for ; Fri, 7 Aug 2020 06:19:08 +0000 (UTC) X-Spam-Summary: 1,0,0,dd654e61f2c49409,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2566:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4049:4119:4250:4321:4385:5007:6261:6653:6737:7576:7875:7903:8603:9010:9025:9545:9592:10004:11026:11232:11257:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13221:13229:13255:13846:21080:21450:21451:21627:21939:21972:21990:30029:30054:30064:30069:30070:30075,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yg9p8n3kqbhfwc97nxbbgftx6jzopt4qyhd3zy5ax5wchatwadqwczq8si7eh.jm15piuqq71fjegbtcfudfpnebox3mdmq1dxywzuxzf74f4ugw7qbtwubykwgy4.6-lbl8.mailshell.net-223.238.255.1 00,Cache X-HE-Tag: shop86_5e103a626fbe X-Filterd-Recvd-Size: 8321 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:07 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7138A221E5; Fri, 7 Aug 2020 06:19:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781147; bh=34+m/Kc+Qqv3xOKiIH06kgVQSbItLz0a4ABNLuVaWKY=; h=Date:From:To:Subject:In-Reply-To:From; b=KxkwqTzbntyJd8mONhOarci3XdFlKtT+pTxdAP4vR7HYRvfqmvSJ9zwSsGF4Ntjuz mdJlLAf3WiLBRncgVe1mI9ub1Pv/4VXWA/1yF9T9CLgmHAssFBCj1Jw1nOMd2YzQSh OvpxDs2y7mkVqKpvdlLGlKfaBnOvtteGR9cZTsZc= Date: Thu, 06 Aug 2020 23:19:05 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, iamjoonsoo.kim@lge.com, jannh@google.com, keescook@chromium.org, linux-mm@kvack.org, mjg59@google.com, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org, vbabka@suse.cz, vinmenon@codeaurora.org, vjitta@codeaurora.org Subject: [patch 037/163] mm, slab/slub: improve error reporting and overhead of cache_from_obj() Message-ID: <20200807061905.uWLgaEibF%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 5B75B1A4A5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Vlastimil Babka Subject: mm, slab/slub: improve error reporting and overhead of cache_from_obj() cache_from_obj() was added by commit b9ce5ef49f00 ("sl[au]b: always get the cache from its page in kmem_cache_free()") to support kmemcg, where per-memcg cache can be different from the root one, so we can't use the kmem_cache pointer given to kmem_cache_free(). Prior to that commit, SLUB already had debugging check+warning that could be enabled to compare the given kmem_cache pointer to one referenced by the slab page where the object-to-be-freed resides. This check was moved to cache_from_obj(). Later the check was also enabled for SLAB_FREELIST_HARDENED configs by commit 598a0717a816 ("mm/slab: validate cache membership under freelist hardening"). These checks and warnings can be useful especially for the debugging, which can be improved. Commit 598a0717a816 changed the pr_err() with WARN_ON_ONCE() to WARN_ONCE() so only the first hit is now reported, others are silent. This patch changes it to WARN() so that all errors are reported. It's also useful to print SLUB allocation/free tracking info for the offending object, if tracking is enabled. Thus, export the SLUB print_tracking() function and provide an empty one for SLAB. For SLUB we can also benefit from the static key check in kmem_cache_debug_flags(), but we need to move this function to slab.h and declare the static key there. [1] https://lore.kernel.org/r/20200608230654.828134-18-guro@fb.com [vbabka@suse.cz: avoid bogus WARN()] Link: https://lore.kernel.org/r/20200623090213.GW5535@shao2-debian Link: http://lkml.kernel.org/r/b33e0fa7-cd28-4788-9e54-5927846329ef@suse.cz Link: http://lkml.kernel.org/r/afeda7ac-748b-33d8-a905-56b708148ad5@suse.cz Signed-off-by: Vlastimil Babka Acked-by: Kees Cook Acked-by: Roman Gushchin Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Matthew Garrett Cc: Jann Horn Cc: Vijayanand Jitta Cc: Vinayak Menon Signed-off-by: Andrew Morton --- mm/slab.c | 8 -------- mm/slab.h | 45 +++++++++++++++++++++++++++++++++++++++++++++ mm/slub.c | 38 +------------------------------------- 3 files changed, 46 insertions(+), 45 deletions(-) --- a/mm/slab.c~mm-slab-slub-improve-error-reporting-and-overhead-of-cache_from_obj +++ a/mm/slab.c @@ -3678,14 +3678,6 @@ void *__kmalloc_track_caller(size_t size } EXPORT_SYMBOL(__kmalloc_track_caller); -static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) -{ - if (memcg_kmem_enabled()) - return virt_to_cache(x); - else - return s; -} - /** * kmem_cache_free - Deallocate an object * @cachep: The cache the allocation was from. --- a/mm/slab.h~mm-slab-slub-improve-error-reporting-and-overhead-of-cache_from_obj +++ a/mm/slab.h @@ -276,6 +276,34 @@ static inline int cache_vmstat_idx(struc NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; } +#ifdef CONFIG_SLUB_DEBUG +#ifdef CONFIG_SLUB_DEBUG_ON +DECLARE_STATIC_KEY_TRUE(slub_debug_enabled); +#else +DECLARE_STATIC_KEY_FALSE(slub_debug_enabled); +#endif +extern void print_tracking(struct kmem_cache *s, void *object); +#else +static inline void print_tracking(struct kmem_cache *s, void *object) +{ +} +#endif + +/* + * Returns true if any of the specified slub_debug flags is enabled for the + * cache. Use only for flags parsed by setup_slub_debug() as it also enables + * the static key. + */ +static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t flags) +{ +#ifdef CONFIG_SLUB_DEBUG + VM_WARN_ON_ONCE(!(flags & SLAB_DEBUG_FLAGS)); + if (static_branch_unlikely(&slub_debug_enabled)) + return s->flags & flags; +#endif + return false; +} + #ifdef CONFIG_MEMCG_KMEM /* List of all root caches. */ @@ -504,6 +532,23 @@ static __always_inline void uncharge_sla memcg_uncharge_slab(page, order, s); } +static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) +{ + struct kmem_cache *cachep; + + if (!IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) && + !memcg_kmem_enabled() && + !kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) + return s; + + cachep = virt_to_cache(x); + if (WARN(cachep && !slab_equal_or_root(cachep, s), + "%s: Wrong slab cache. %s but object is from %s\n", + __func__, s->name, cachep->name)) + print_tracking(cachep, x); + return cachep; +} + static inline size_t slab_ksize(const struct kmem_cache *s) { #ifndef CONFIG_SLUB --- a/mm/slub.c~mm-slab-slub-improve-error-reporting-and-overhead-of-cache_from_obj +++ a/mm/slub.c @@ -122,21 +122,6 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabl #endif #endif -/* - * Returns true if any of the specified slub_debug flags is enabled for the - * cache. Use only for flags parsed by setup_slub_debug() as it also enables - * the static key. - */ -static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t flags) -{ - VM_WARN_ON_ONCE(!(flags & SLAB_DEBUG_FLAGS)); -#ifdef CONFIG_SLUB_DEBUG - if (static_branch_unlikely(&slub_debug_enabled)) - return s->flags & flags; -#endif - return false; -} - static inline bool kmem_cache_debug(struct kmem_cache *s) { return kmem_cache_debug_flags(s, SLAB_DEBUG_FLAGS); @@ -653,7 +638,7 @@ static void print_track(const char *s, s #endif } -static void print_tracking(struct kmem_cache *s, void *object) +void print_tracking(struct kmem_cache *s, void *object) { unsigned long pr_time = jiffies; if (!(s->flags & SLAB_STORE_USER)) @@ -1525,10 +1510,6 @@ static bool freelist_corrupted(struct km { return false; } - -static void print_tracking(struct kmem_cache *s, void *object) -{ -} #endif /* CONFIG_SLUB_DEBUG */ /* @@ -3175,23 +3156,6 @@ void ___cache_free(struct kmem_cache *ca } #endif -static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) -{ - struct kmem_cache *cachep; - - if (!IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) && - !memcg_kmem_enabled() && - !kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) - return s; - - cachep = virt_to_cache(x); - if (WARN(cachep && !slab_equal_or_root(cachep, s), - "%s: Wrong slab cache. %s but object is from %s\n", - __func__, s->name, cachep->name)) - print_tracking(cachep, x); - return cachep; -} - void kmem_cache_free(struct kmem_cache *s, void *x) { s = cache_from_obj(s, x); From patchwork Fri Aug 7 06:19:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704753 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CFF2B14B7 for ; Fri, 7 Aug 2020 06:19:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A12502177B for ; Fri, 7 Aug 2020 06:19:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="k8WeoTUV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A12502177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4217C8D0027; Fri, 7 Aug 2020 02:19:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3F82F8D0026; Fri, 7 Aug 2020 02:19:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30F068D0027; Fri, 7 Aug 2020 02:19:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0189.hostedemail.com [216.40.44.189]) by kanga.kvack.org (Postfix) with ESMTP id 14CD08D0026 for ; Fri, 7 Aug 2020 02:19:12 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id CECD582499A8 for ; Fri, 7 Aug 2020 06:19:11 +0000 (UTC) X-FDA: 77122770102.17.loss59_4c0f8b826fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 8429F180D0181 for ; Fri, 7 Aug 2020 06:19:11 +0000 (UTC) X-Spam-Summary: 1,0,0,66e6438602e2ad97,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4419:5007:6261:6653:6737:7576:8599:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12114:12297:12517:12519:12555:12679:12783:12986:13069:13161:13229:13311:13357:13846:14181:14384:14721:14849:21080:21451:21627:21740:21939:21987:30054:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8h9igitpt7fhjg6ahfyrwmq9pwoprbjejckrq9tyoi4haq41ouq89s9mjtgk.3fprfryhznzj97isgjyt4ssec514bbxiasngroq66wnb5htoppg9effounqqrdh.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF: not bulk X-HE-Tag: loss59_4c0f8b826fbe X-Filterd-Recvd-Size: 2668 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:11 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E26932177B; Fri, 7 Aug 2020 06:19:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781150; bh=t5PucQZuKToNKwVENT0iXxoiOqnhQXXF6xvuCDnnKj4=; h=Date:From:To:Subject:In-Reply-To:From; b=k8WeoTUVc4zSsbLPRavwgmy7nAG3F0u2TnPjKmiphcQazY5jfvPQtArkezWLsR09U wQ1ITzXzSDOyNwp2XUYGukaGVs2nAA22Sk9JP78K4B6LiUYAplDA15cCGmT86gi4A+ 7fN6M3xsuHL5iPyskU7dsM0Kdn03SpG+lCcpn8EU= Date: Thu, 06 Aug 2020 23:19:09 -0700 From: Andrew Morton To: akpm@linux-foundation.org, bigeasy@linutronix.de, cl@linux.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, tglx@linutronix.de, torvalds@linux-foundation.org, yuzhao@google.com Subject: [patch 038/163] mm/slub.c: drop lockdep_assert_held() from put_map() Message-ID: <20200807061909.ouNDTlYfw%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 8429F180D0181 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Sebastian Andrzej Siewior Subject: mm/slub.c: drop lockdep_assert_held() from put_map() There is no point in using lockdep_assert_held() unlock that is about to be unlocked. It works only with lockdep and lockdep will complain if spin_unlock() is used on a lock that has not been locked. Remove superfluous lockdep_assert_held(). Link: http://lkml.kernel.org/r/20200618201234.795692-2-bigeasy@linutronix.de Signed-off-by: Sebastian Andrzej Siewior Cc: Yu Zhao Cc: Christopher Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Thomas Gleixner Signed-off-by: Andrew Morton --- mm/slub.c | 2 -- 1 file changed, 2 deletions(-) --- a/mm/slub.c~slub-drop-lockdep_assert_held-from-put_map +++ a/mm/slub.c @@ -473,8 +473,6 @@ static unsigned long *get_map(struct kme static void put_map(unsigned long *map) __releases(&object_map_lock) { VM_BUG_ON(map != object_map); - lockdep_assert_held(&object_map_lock); - spin_unlock(&object_map_lock); } From patchwork Fri Aug 7 06:19:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704755 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1B36414B7 for ; Fri, 7 Aug 2020 06:19:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E234C22D07 for ; Fri, 7 Aug 2020 06:19:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="0K1uw6Cg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E234C22D07 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 757188D0028; Fri, 7 Aug 2020 02:19:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 72CA58D0026; Fri, 7 Aug 2020 02:19:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 642CF8D0028; Fri, 7 Aug 2020 02:19:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0215.hostedemail.com [216.40.44.215]) by kanga.kvack.org (Postfix) with ESMTP id 4BDEF8D0026 for ; Fri, 7 Aug 2020 02:19:15 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 126363637 for ; Fri, 7 Aug 2020 06:19:15 +0000 (UTC) X-FDA: 77122770270.24.berry34_5009a6e26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id DBDAA1A4A5 for ; Fri, 7 Aug 2020 06:19:14 +0000 (UTC) X-Spam-Summary: 1,0,0,50269d6eeb93c641,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:965:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2194:2196:2199:2200:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4390:4395:5007:6261:6653:6737:7576:7904:8599:8603:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12114:12297:12438:12517:12519:12555:12679:12783:12986:13180:13229:13255:13846:14181:14721:14849:21080:21451:21627:21796:21939:21990:30036:30054,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yf7nt699fxzkgwuwyhbc4e4hi54ocd66u76ueq4seuyih1omyomj8xzjc9bem.atn8r3c6ufwqmun3ugu5fri81xky4rpbprcd5669wrh5gqwr3bamwcyd8r8utt8.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,D omainCac X-HE-Tag: berry34_5009a6e26fbe X-Filterd-Recvd-Size: 3675 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:14 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 36B6722CAF; Fri, 7 Aug 2020 06:19:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781153; bh=2UE3dQE/YR8k4zIoUAYLHNZ1AerjcU14FqZxzHA1uEE=; h=Date:From:To:Subject:In-Reply-To:From; b=0K1uw6CgAKdDBpkkaQoF6X23Dm+oxYxzfkfrGGQtXSoaJ2Uw73n8Mc/h02cDUVzxe mo0z78fhfArfol1fxuiUmhUWQ/KhObAofJk40A2qdnF92MQ8FH4HGwbUq+hclZTjCk v8D/rnmKx0hH9QwECG4miQNd5UAONdC2jvzyzoQM= Date: Thu, 06 Aug 2020 23:19:12 -0700 From: Andrew Morton To: akpm@linux-foundation.org, andreyknvl@google.com, cl@linux.com, elver@google.com, glider@google.com, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, torvalds@linux-foundation.org Subject: [patch 039/163] mm, kcsan: instrument SLAB/SLUB free with "ASSERT_EXCLUSIVE_ACCESS" Message-ID: <20200807061912.r0alqJmL-%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: DBDAA1A4A5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Marco Elver Subject: mm, kcsan: instrument SLAB/SLUB free with "ASSERT_EXCLUSIVE_ACCESS" Provide the necessary KCSAN checks to assist with debugging racy use-after-frees. While KASAN is more reliable at generally catching such use-after-frees (due to its use of a quarantine), it can be difficult to debug racy use-after-frees. If a reliable reproducer exists, KCSAN can assist in debugging such issues. Note: ASSERT_EXCLUSIVE_ACCESS is a convenience wrapper if the size is simply sizeof(var). Instead, here we just use __kcsan_check_access() explicitly to pass the correct size. Link: http://lkml.kernel.org/r/20200623072653.114563-1-elver@google.com Signed-off-by: Marco Elver Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Signed-off-by: Andrew Morton --- mm/slab.c | 5 +++++ mm/slub.c | 5 +++++ 2 files changed, 10 insertions(+) --- a/mm/slab.c~mm-kcsan-instrument-slab-slub-free-with-assert_exclusive_access +++ a/mm/slab.c @@ -3432,6 +3432,11 @@ static __always_inline void __cache_free if (kasan_slab_free(cachep, objp, _RET_IP_)) return; + /* Use KCSAN to help debug racy use-after-free. */ + if (!(cachep->flags & SLAB_TYPESAFE_BY_RCU)) + __kcsan_check_access(objp, cachep->object_size, + KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT); + ___cache_free(cachep, objp, caller); } --- a/mm/slub.c~mm-kcsan-instrument-slab-slub-free-with-assert_exclusive_access +++ a/mm/slub.c @@ -1549,6 +1549,11 @@ static __always_inline bool slab_free_ho if (!(s->flags & SLAB_DEBUG_OBJECTS)) debug_check_no_obj_freed(x, s->object_size); + /* Use KCSAN to help debug racy use-after-free. */ + if (!(s->flags & SLAB_TYPESAFE_BY_RCU)) + __kcsan_check_access(x, s->object_size, + KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT); + /* KASAN might put x into memory quarantine, delaying its reuse */ return kasan_slab_free(s, x, _RET_IP_); } From patchwork Fri Aug 7 06:19:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704757 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7710114B7 for ; Fri, 7 Aug 2020 06:19:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2AC4122CB3 for ; Fri, 7 Aug 2020 06:19:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="DskOLY6a" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2AC4122CB3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D70838D0029; Fri, 7 Aug 2020 02:19:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D47378D0026; Fri, 7 Aug 2020 02:19:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5DFA8D0029; Fri, 7 Aug 2020 02:19:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0047.hostedemail.com [216.40.44.47]) by kanga.kvack.org (Postfix) with ESMTP id AD41A8D0026 for ; Fri, 7 Aug 2020 02:19:19 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 71CA0180AD804 for ; Fri, 7 Aug 2020 06:19:19 +0000 (UTC) X-FDA: 77122770438.29.yard87_3d169a326fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 47E05180868E6 for ; Fri, 7 Aug 2020 06:19:19 +0000 (UTC) X-Spam-Summary: 50,0,0,eff572daf6e80d5a,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:4:41:334:355:368:369:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1963:2194:2196:2199:2200:2393:2525:2560:2566:2639:2682:2685:2689:2693:2730:2859:2899:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3167:3865:3866:3867:3868:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:5007:6117:6119:6261:6653:6737:6738:7576:7901:7903:8957:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12895:12986:13053:13141:13161:13200:13229:13230:13255:13548:13846:14096:21063:21080:21433:21451:21611:21627:21788:21819:21939:21990:30054:30064:30069:30089,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygqdhrp9m4uzcxwwsdk116jpbx5ypekcz7tkjnnupz4jeatn7sfsdymboz7bi.4jshq4fmu85h794kcm3e69k955rr1xq75qzf1 x4pd4szh X-HE-Tag: yard87_3d169a326fbe X-Filterd-Recvd-Size: 15524 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:18 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C73A12177B; Fri, 7 Aug 2020 06:19:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781157; bh=HBFAslhuKXaOPh5f+h7wPd2tS96nNlCng1cIn0lEfEw=; h=Date:From:To:Subject:In-Reply-To:From; b=DskOLY6aRqmY3TTIn+Into42VoLe6KSqDgwjrRso2WHsgNVJnBnXqm7QHMzoFzRQ6 rHyNqLduW16V2W883FUQYw5Evrgl8yRAUL+PnddH84wKKii2jmGyS7f29BdLTqScXR SF0yAoIk+shdNFm/86k+zPqC7AvV+8Q0r1F6S1lY= Date: Thu, 06 Aug 2020 23:19:16 -0700 From: Andrew Morton To: akpm@linux-foundation.org, anshuman.khandual@arm.com, benh@kernel.crashing.org, borntraeger@de.ibm.com, bp@alien8.de, catalin.marinas@arm.com, corbet@lwn.net, gor@linux.ibm.com, heiko.carstens@de.ibm.com, hpa@zytor.com, kirill@shutemov.name, linux-mm@kvack.org, mingo@redhat.com, mm-commits@vger.kernel.org, mpe@ellerman.id.au, palmer@dabbelt.com, paul.walmsley@sifive.com, paulus@samba.org, rppt@kernel.org, rppt@linux.ibm.com, steven.price@arm.com, tglx@linutronix.de, torvalds@linux-foundation.org, vgupta@synopsys.com, will@kernel.org, ziy@nvidia.com Subject: [patch 040/163] mm/debug_vm_pgtable: add tests validating arch helpers for core MM features Message-ID: <20200807061916.zBaC-yU7x%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 47E05180868E6 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Anshuman Khandual Subject: mm/debug_vm_pgtable: add tests validating arch helpers for core MM features Patch series "mm/debug_vm_pgtable: Add some more tests", v5. This series adds some more arch page table helper validation tests which are related to core and advanced memory functions. This also creates a documentation, enlisting expected semantics for all page table helpers as suggested by Mike Rapoport previously (https://lkml.org/lkml/2020/1/30/40). There are many TRANSPARENT_HUGEPAGE and ARCH_HAS_TRANSPARENT_HUGEPAGE_PUD ifdefs scattered across the test. But consolidating all the fallback stubs is not very straight forward because ARCH_HAS_TRANSPARENT_HUGEPAGE_PUD is not explicitly dependent on ARCH_HAS_TRANSPARENT_HUGEPAGE. Tested on arm64, x86 platforms but only build tested on all other enabled platforms through ARCH_HAS_DEBUG_VM_PGTABLE i.e powerpc, arc, s390. The following failure on arm64 still exists which was mentioned previously. It will be fixed with the upcoming THP migration on arm64 enablement series. WARNING .... mm/debug_vm_pgtable.c:860 debug_vm_pgtable+0x940/0xa54 WARN_ON(!pmd_present(pmd_mkinvalid(pmd_mkhuge(pmd)))) This patch (of 4): This adds new tests validating arch page table helpers for these following core memory features. These tests create and test specific mapping types at various page table levels. 1. SPECIAL mapping 2. PROTNONE mapping 3. DEVMAP mapping 4. SOFTDIRTY mapping 5. SWAP mapping 6. MIGRATION mapping 7. HUGETLB mapping 8. THP mapping Link: http://lkml.kernel.org/r/1594610587-4172-1-git-send-email-anshuman.khandual@arm.com Link: http://lkml.kernel.org/r/1593996516-7186-1-git-send-email-anshuman.khandual@arm.com Link: http://lkml.kernel.org/r/1593996516-7186-2-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual Reviewed-by: Zi Yan Tested-by: Vineet Gupta [arc] Suggested-by: Catalin Marinas Cc: Mike Rapoport Cc: Vineet Gupta Cc: Catalin Marinas Cc: Will Deacon Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: Michael Ellerman Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Christian Borntraeger Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: "H. Peter Anvin" Cc: Kirill A. Shutemov Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Jonathan Corbet Cc: Mike Rapoport Cc: Steven Price Signed-off-by: Andrew Morton --- mm/debug_vm_pgtable.c | 302 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 301 insertions(+), 1 deletion(-) --- a/mm/debug_vm_pgtable.c~mm-debug_vm_pgtable-add-tests-validating-arch-helpers-for-core-mm-features +++ a/mm/debug_vm_pgtable.c @@ -282,6 +282,278 @@ static void __init pmd_populate_tests(st WARN_ON(pmd_bad(pmd)); } +static void __init pte_special_tests(unsigned long pfn, pgprot_t prot) +{ + pte_t pte = pfn_pte(pfn, prot); + + if (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL)) + return; + + WARN_ON(!pte_special(pte_mkspecial(pte))); +} + +static void __init pte_protnone_tests(unsigned long pfn, pgprot_t prot) +{ + pte_t pte = pfn_pte(pfn, prot); + + if (!IS_ENABLED(CONFIG_NUMA_BALANCING)) + return; + + WARN_ON(!pte_protnone(pte)); + WARN_ON(!pte_present(pte)); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void __init pmd_protnone_tests(unsigned long pfn, pgprot_t prot) +{ + pmd_t pmd = pmd_mkhuge(pfn_pmd(pfn, prot)); + + if (!IS_ENABLED(CONFIG_NUMA_BALANCING)) + return; + + WARN_ON(!pmd_protnone(pmd)); + WARN_ON(!pmd_present(pmd)); +} +#else /* !CONFIG_TRANSPARENT_HUGEPAGE */ +static void __init pmd_protnone_tests(unsigned long pfn, pgprot_t prot) { } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +#ifdef CONFIG_ARCH_HAS_PTE_DEVMAP +static void __init pte_devmap_tests(unsigned long pfn, pgprot_t prot) +{ + pte_t pte = pfn_pte(pfn, prot); + + WARN_ON(!pte_devmap(pte_mkdevmap(pte))); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void __init pmd_devmap_tests(unsigned long pfn, pgprot_t prot) +{ + pmd_t pmd = pfn_pmd(pfn, prot); + + WARN_ON(!pmd_devmap(pmd_mkdevmap(pmd))); +} + +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +static void __init pud_devmap_tests(unsigned long pfn, pgprot_t prot) +{ + pud_t pud = pfn_pud(pfn, prot); + + WARN_ON(!pud_devmap(pud_mkdevmap(pud))); +} +#else /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +static void __init pud_devmap_tests(unsigned long pfn, pgprot_t prot) { } +#endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +#else /* CONFIG_TRANSPARENT_HUGEPAGE */ +static void __init pmd_devmap_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pud_devmap_tests(unsigned long pfn, pgprot_t prot) { } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#else +static void __init pte_devmap_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pmd_devmap_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pud_devmap_tests(unsigned long pfn, pgprot_t prot) { } +#endif /* CONFIG_ARCH_HAS_PTE_DEVMAP */ + +static void __init pte_soft_dirty_tests(unsigned long pfn, pgprot_t prot) +{ + pte_t pte = pfn_pte(pfn, prot); + + if (!IS_ENABLED(CONFIG_MEM_SOFT_DIRTY)) + return; + + WARN_ON(!pte_soft_dirty(pte_mksoft_dirty(pte))); + WARN_ON(pte_soft_dirty(pte_clear_soft_dirty(pte))); +} + +static void __init pte_swap_soft_dirty_tests(unsigned long pfn, pgprot_t prot) +{ + pte_t pte = pfn_pte(pfn, prot); + + if (!IS_ENABLED(CONFIG_MEM_SOFT_DIRTY)) + return; + + WARN_ON(!pte_swp_soft_dirty(pte_swp_mksoft_dirty(pte))); + WARN_ON(pte_swp_soft_dirty(pte_swp_clear_soft_dirty(pte))); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void __init pmd_soft_dirty_tests(unsigned long pfn, pgprot_t prot) +{ + pmd_t pmd = pfn_pmd(pfn, prot); + + if (!IS_ENABLED(CONFIG_MEM_SOFT_DIRTY)) + return; + + WARN_ON(!pmd_soft_dirty(pmd_mksoft_dirty(pmd))); + WARN_ON(pmd_soft_dirty(pmd_clear_soft_dirty(pmd))); +} + +static void __init pmd_swap_soft_dirty_tests(unsigned long pfn, pgprot_t prot) +{ + pmd_t pmd = pfn_pmd(pfn, prot); + + if (!IS_ENABLED(CONFIG_MEM_SOFT_DIRTY) || + !IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION)) + return; + + WARN_ON(!pmd_swp_soft_dirty(pmd_swp_mksoft_dirty(pmd))); + WARN_ON(pmd_swp_soft_dirty(pmd_swp_clear_soft_dirty(pmd))); +} +#else /* !CONFIG_ARCH_HAS_PTE_DEVMAP */ +static void __init pmd_soft_dirty_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pmd_swap_soft_dirty_tests(unsigned long pfn, pgprot_t prot) +{ +} +#endif /* CONFIG_ARCH_HAS_PTE_DEVMAP */ + +static void __init pte_swap_tests(unsigned long pfn, pgprot_t prot) +{ + swp_entry_t swp; + pte_t pte; + + pte = pfn_pte(pfn, prot); + swp = __pte_to_swp_entry(pte); + pte = __swp_entry_to_pte(swp); + WARN_ON(pfn != pte_pfn(pte)); +} + +#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION +static void __init pmd_swap_tests(unsigned long pfn, pgprot_t prot) +{ + swp_entry_t swp; + pmd_t pmd; + + pmd = pfn_pmd(pfn, prot); + swp = __pmd_to_swp_entry(pmd); + pmd = __swp_entry_to_pmd(swp); + WARN_ON(pfn != pmd_pfn(pmd)); +} +#else /* !CONFIG_ARCH_ENABLE_THP_MIGRATION */ +static void __init pmd_swap_tests(unsigned long pfn, pgprot_t prot) { } +#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ + +static void __init swap_migration_tests(void) +{ + struct page *page; + swp_entry_t swp; + + if (!IS_ENABLED(CONFIG_MIGRATION)) + return; + /* + * swap_migration_tests() requires a dedicated page as it needs to + * be locked before creating a migration entry from it. Locking the + * page that actually maps kernel text ('start_kernel') can be real + * problematic. Lets allocate a dedicated page explicitly for this + * purpose that will be freed subsequently. + */ + page = alloc_page(GFP_KERNEL); + if (!page) { + pr_err("page allocation failed\n"); + return; + } + + /* + * make_migration_entry() expects given page to be + * locked, otherwise it stumbles upon a BUG_ON(). + */ + __SetPageLocked(page); + swp = make_migration_entry(page, 1); + WARN_ON(!is_migration_entry(swp)); + WARN_ON(!is_write_migration_entry(swp)); + + make_migration_entry_read(&swp); + WARN_ON(!is_migration_entry(swp)); + WARN_ON(is_write_migration_entry(swp)); + + swp = make_migration_entry(page, 0); + WARN_ON(!is_migration_entry(swp)); + WARN_ON(is_write_migration_entry(swp)); + __ClearPageLocked(page); + __free_page(page); +} + +#ifdef CONFIG_HUGETLB_PAGE +static void __init hugetlb_basic_tests(unsigned long pfn, pgprot_t prot) +{ + struct page *page; + pte_t pte; + + /* + * Accessing the page associated with the pfn is safe here, + * as it was previously derived from a real kernel symbol. + */ + page = pfn_to_page(pfn); + pte = mk_huge_pte(page, prot); + + WARN_ON(!huge_pte_dirty(huge_pte_mkdirty(pte))); + WARN_ON(!huge_pte_write(huge_pte_mkwrite(huge_pte_wrprotect(pte)))); + WARN_ON(huge_pte_write(huge_pte_wrprotect(huge_pte_mkwrite(pte)))); + +#ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB + pte = pfn_pte(pfn, prot); + + WARN_ON(!pte_huge(pte_mkhuge(pte))); +#endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ +} +#else /* !CONFIG_HUGETLB_PAGE */ +static void __init hugetlb_basic_tests(unsigned long pfn, pgprot_t prot) { } +#endif /* CONFIG_HUGETLB_PAGE */ + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void __init pmd_thp_tests(unsigned long pfn, pgprot_t prot) +{ + pmd_t pmd; + + if (!has_transparent_hugepage()) + return; + + /* + * pmd_trans_huge() and pmd_present() must return positive after + * MMU invalidation with pmd_mkinvalid(). This behavior is an + * optimization for transparent huge page. pmd_trans_huge() must + * be true if pmd_page() returns a valid THP to avoid taking the + * pmd_lock when others walk over non transhuge pmds (i.e. there + * are no THP allocated). Especially when splitting a THP and + * removing the present bit from the pmd, pmd_trans_huge() still + * needs to return true. pmd_present() should be true whenever + * pmd_trans_huge() returns true. + */ + pmd = pfn_pmd(pfn, prot); + WARN_ON(!pmd_trans_huge(pmd_mkhuge(pmd))); + +#ifndef __HAVE_ARCH_PMDP_INVALIDATE + WARN_ON(!pmd_trans_huge(pmd_mkinvalid(pmd_mkhuge(pmd)))); + WARN_ON(!pmd_present(pmd_mkinvalid(pmd_mkhuge(pmd)))); +#endif /* __HAVE_ARCH_PMDP_INVALIDATE */ +} + +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +static void __init pud_thp_tests(unsigned long pfn, pgprot_t prot) +{ + pud_t pud; + + if (!has_transparent_hugepage()) + return; + + pud = pfn_pud(pfn, prot); + WARN_ON(!pud_trans_huge(pud_mkhuge(pud))); + + /* + * pud_mkinvalid() has been dropped for now. Enable back + * these tests when it comes back with a modified pud_present(). + * + * WARN_ON(!pud_trans_huge(pud_mkinvalid(pud_mkhuge(pud)))); + * WARN_ON(!pud_present(pud_mkinvalid(pud_mkhuge(pud)))); + */ +} +#else /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +static void __init pud_thp_tests(unsigned long pfn, pgprot_t prot) { } +#endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +#else /* !CONFIG_TRANSPARENT_HUGEPAGE */ +static void __init pmd_thp_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pud_thp_tests(unsigned long pfn, pgprot_t prot) { } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + static unsigned long __init get_random_vaddr(void) { unsigned long random_vaddr, random_pages, total_user_pages; @@ -303,7 +575,7 @@ static int __init debug_vm_pgtable(void) pmd_t *pmdp, *saved_pmdp, pmd; pte_t *ptep; pgtable_t saved_ptep; - pgprot_t prot; + pgprot_t prot, protnone; phys_addr_t paddr; unsigned long vaddr, pte_aligned, pmd_aligned; unsigned long pud_aligned, p4d_aligned, pgd_aligned; @@ -319,6 +591,12 @@ static int __init debug_vm_pgtable(void) } /* + * __P000 (or even __S000) will help create page table entries with + * PROT_NONE permission as required for pxx_protnone_tests(). + */ + protnone = __P000; + + /* * PFN for mapping at PTE level is determined from a standard kernel * text symbol. But pfns for higher page table levels are derived by * masking lower bits of this real pfn. These derived pfns might not @@ -373,6 +651,28 @@ static int __init debug_vm_pgtable(void) p4d_populate_tests(mm, p4dp, saved_pudp); pgd_populate_tests(mm, pgdp, saved_p4dp); + pte_special_tests(pte_aligned, prot); + pte_protnone_tests(pte_aligned, protnone); + pmd_protnone_tests(pmd_aligned, protnone); + + pte_devmap_tests(pte_aligned, prot); + pmd_devmap_tests(pmd_aligned, prot); + pud_devmap_tests(pud_aligned, prot); + + pte_soft_dirty_tests(pte_aligned, prot); + pmd_soft_dirty_tests(pmd_aligned, prot); + pte_swap_soft_dirty_tests(pte_aligned, prot); + pmd_swap_soft_dirty_tests(pmd_aligned, prot); + + pte_swap_tests(pte_aligned, prot); + pmd_swap_tests(pmd_aligned, prot); + + swap_migration_tests(); + hugetlb_basic_tests(pte_aligned, prot); + + pmd_thp_tests(pmd_aligned, prot); + pud_thp_tests(pud_aligned, prot); + p4d_free(mm, saved_p4dp); pud_free(mm, saved_pudp); pmd_free(mm, saved_pmdp); From patchwork Fri Aug 7 06:19:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704759 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1023D14B7 for ; Fri, 7 Aug 2020 06:19:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BAA0022C9F for ; Fri, 7 Aug 2020 06:19:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="neZJvq2p" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BAA0022C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 951148D002A; Fri, 7 Aug 2020 02:19:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8B3B98D0026; Fri, 7 Aug 2020 02:19:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A2EE8D002A; Fri, 7 Aug 2020 02:19:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0192.hostedemail.com [216.40.44.192]) by kanga.kvack.org (Postfix) with ESMTP id 60B5E8D0026 for ; Fri, 7 Aug 2020 02:19:24 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 20700249D for ; Fri, 7 Aug 2020 06:19:24 +0000 (UTC) X-FDA: 77122770648.28.soda30_240056026fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id E9F486D74 for ; Fri, 7 Aug 2020 06:19:23 +0000 (UTC) X-Spam-Summary: 1,0,0,c9c145afc6165978,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:4:41:334:355:368:369:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1963:2196:2199:2393:2525:2559:2564:2640:2682:2685:2693:2730:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3167:3865:3868:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4605:5007:6117:6119:6261:6653:6737:6738:7576:7903:9025:9545:10004:10913:11026:11233:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13255:13548:13846:14096:21080:21433:21451:21627:21819:21939:21990:30054:30064:30089,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygo1k1e9nho9qkyhpjh956947b9ochnfz1h6wngun36yh1igsxqts7k1zya7m.fbfuxuegrosgepxnzfjp5ujkwqop9jcxzhtrrnkccpnoo3sibqis1wkcwipxapy.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Net check:no X-HE-Tag: soda30_240056026fbe X-Filterd-Recvd-Size: 15974 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:23 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E0E94221E5; Fri, 7 Aug 2020 06:19:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781162; bh=mVY7YChuZxKX/8PbAqTOlsIycxWKRxUwAIeUP2ZjWqM=; h=Date:From:To:Subject:In-Reply-To:From; b=neZJvq2pi2rxnKfrSBLZ9PfnZ8bHkj4fX+j2rQPy9ary9Zrf5WBmUEh/SMDK/6fua KeE5MZt/QTHDzWCvzPfsJdSCNv5QmpYirAl1rEgE+rFBMxIA7Bynxq6bqvWGrF/s27 QMJAV77YVA5t7b6zTqTFPyOp9Be3LE8vHkLYw/6s= Date: Thu, 06 Aug 2020 23:19:20 -0700 From: Andrew Morton To: akpm@linux-foundation.org, anshuman.khandual@arm.com, benh@kernel.crashing.org, borntraeger@de.ibm.com, bp@alien8.de, catalin.marinas@arm.com, corbet@lwn.net, gor@linux.ibm.com, heiko.carstens@de.ibm.com, hpa@zytor.com, kirill@shutemov.name, linux-mm@kvack.org, mingo@redhat.com, mm-commits@vger.kernel.org, mpe@ellerman.id.au, palmer@dabbelt.com, paul.walmsley@sifive.com, paulus@samba.org, rppt@kernel.org, rppt@linux.ibm.com, steven.price@arm.com, tglx@linutronix.de, torvalds@linux-foundation.org, vgupta@synopsys.com, will@kernel.org, ziy@nvidia.com Subject: [patch 041/163] mm/debug_vm_pgtable: add tests validating advanced arch page table helpers Message-ID: <20200807061920.70HUcXwC9%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E9F486D74 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Anshuman Khandual Subject: mm/debug_vm_pgtable: add tests validating advanced arch page table helpers This adds new tests validating for these following arch advanced page table helpers. These tests create and test specific mapping types at various page table levels. 1. pxxp_set_wrprotect() 2. pxxp_get_and_clear() 3. pxxp_set_access_flags() 4. pxxp_get_and_clear_full() 5. pxxp_test_and_clear_young() 6. pxx_leaf() 7. pxx_set_huge() 8. pxx_(clear|mk)_savedwrite() 9. huge_pxxp_xxx() [anshuman.khandual@arm.com: drop RANDOM_ORVALUE from hugetlb_advanced_tests()] Link: http://lkml.kernel.org/r/1594610587-4172-3-git-send-email-anshuman.khandual@arm.com Link: http://lkml.kernel.org/r/1593996516-7186-3-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual Reviewed-by: Zi Yan Tested-by: Vineet Gupta [arc] Suggested-by: Catalin Marinas Cc: Mike Rapoport Cc: Vineet Gupta Cc: Catalin Marinas Cc: Will Deacon Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: Michael Ellerman Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Christian Borntraeger Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: "H. Peter Anvin" Cc: Kirill A. Shutemov Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Jonathan Corbet Cc: Mike Rapoport Cc: Steven Price Signed-off-by: Andrew Morton --- mm/debug_vm_pgtable.c | 312 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 312 insertions(+) --- a/mm/debug_vm_pgtable.c~mm-debug_vm_pgtable-add-tests-validating-advanced-arch-page-table-helpers +++ a/mm/debug_vm_pgtable.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -28,6 +29,7 @@ #include #include #include +#include #define VMFLAGS (VM_READ|VM_WRITE|VM_EXEC) @@ -55,6 +57,55 @@ static void __init pte_basic_tests(unsig WARN_ON(pte_write(pte_wrprotect(pte_mkwrite(pte)))); } +static void __init pte_advanced_tests(struct mm_struct *mm, + struct vm_area_struct *vma, pte_t *ptep, + unsigned long pfn, unsigned long vaddr, + pgprot_t prot) +{ + pte_t pte = pfn_pte(pfn, prot); + + pte = pfn_pte(pfn, prot); + set_pte_at(mm, vaddr, ptep, pte); + ptep_set_wrprotect(mm, vaddr, ptep); + pte = ptep_get(ptep); + WARN_ON(pte_write(pte)); + + pte = pfn_pte(pfn, prot); + set_pte_at(mm, vaddr, ptep, pte); + ptep_get_and_clear(mm, vaddr, ptep); + pte = ptep_get(ptep); + WARN_ON(!pte_none(pte)); + + pte = pfn_pte(pfn, prot); + pte = pte_wrprotect(pte); + pte = pte_mkclean(pte); + set_pte_at(mm, vaddr, ptep, pte); + pte = pte_mkwrite(pte); + pte = pte_mkdirty(pte); + ptep_set_access_flags(vma, vaddr, ptep, pte, 1); + pte = ptep_get(ptep); + WARN_ON(!(pte_write(pte) && pte_dirty(pte))); + + pte = pfn_pte(pfn, prot); + set_pte_at(mm, vaddr, ptep, pte); + ptep_get_and_clear_full(mm, vaddr, ptep, 1); + pte = ptep_get(ptep); + WARN_ON(!pte_none(pte)); + + pte = pte_mkyoung(pte); + set_pte_at(mm, vaddr, ptep, pte); + ptep_test_and_clear_young(vma, vaddr, ptep); + pte = ptep_get(ptep); + WARN_ON(pte_young(pte)); +} + +static void __init pte_savedwrite_tests(unsigned long pfn, pgprot_t prot) +{ + pte_t pte = pfn_pte(pfn, prot); + + WARN_ON(!pte_savedwrite(pte_mk_savedwrite(pte_clear_savedwrite(pte)))); + WARN_ON(pte_savedwrite(pte_clear_savedwrite(pte_mk_savedwrite(pte)))); +} #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void __init pmd_basic_tests(unsigned long pfn, pgprot_t prot) { @@ -77,6 +128,90 @@ static void __init pmd_basic_tests(unsig WARN_ON(!pmd_bad(pmd_mkhuge(pmd))); } +static void __init pmd_advanced_tests(struct mm_struct *mm, + struct vm_area_struct *vma, pmd_t *pmdp, + unsigned long pfn, unsigned long vaddr, + pgprot_t prot) +{ + pmd_t pmd = pfn_pmd(pfn, prot); + + if (!has_transparent_hugepage()) + return; + + /* Align the address wrt HPAGE_PMD_SIZE */ + vaddr = (vaddr & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE; + + pmd = pfn_pmd(pfn, prot); + set_pmd_at(mm, vaddr, pmdp, pmd); + pmdp_set_wrprotect(mm, vaddr, pmdp); + pmd = READ_ONCE(*pmdp); + WARN_ON(pmd_write(pmd)); + + pmd = pfn_pmd(pfn, prot); + set_pmd_at(mm, vaddr, pmdp, pmd); + pmdp_huge_get_and_clear(mm, vaddr, pmdp); + pmd = READ_ONCE(*pmdp); + WARN_ON(!pmd_none(pmd)); + + pmd = pfn_pmd(pfn, prot); + pmd = pmd_wrprotect(pmd); + pmd = pmd_mkclean(pmd); + set_pmd_at(mm, vaddr, pmdp, pmd); + pmd = pmd_mkwrite(pmd); + pmd = pmd_mkdirty(pmd); + pmdp_set_access_flags(vma, vaddr, pmdp, pmd, 1); + pmd = READ_ONCE(*pmdp); + WARN_ON(!(pmd_write(pmd) && pmd_dirty(pmd))); + + pmd = pmd_mkhuge(pfn_pmd(pfn, prot)); + set_pmd_at(mm, vaddr, pmdp, pmd); + pmdp_huge_get_and_clear_full(vma, vaddr, pmdp, 1); + pmd = READ_ONCE(*pmdp); + WARN_ON(!pmd_none(pmd)); + + pmd = pmd_mkyoung(pmd); + set_pmd_at(mm, vaddr, pmdp, pmd); + pmdp_test_and_clear_young(vma, vaddr, pmdp); + pmd = READ_ONCE(*pmdp); + WARN_ON(pmd_young(pmd)); +} + +static void __init pmd_leaf_tests(unsigned long pfn, pgprot_t prot) +{ + pmd_t pmd = pfn_pmd(pfn, prot); + + /* + * PMD based THP is a leaf entry. + */ + pmd = pmd_mkhuge(pmd); + WARN_ON(!pmd_leaf(pmd)); +} + +static void __init pmd_huge_tests(pmd_t *pmdp, unsigned long pfn, pgprot_t prot) +{ + pmd_t pmd; + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) + return; + /* + * X86 defined pmd_set_huge() verifies that the given + * PMD is not a populated non-leaf entry. + */ + WRITE_ONCE(*pmdp, __pmd(0)); + WARN_ON(!pmd_set_huge(pmdp, __pfn_to_phys(pfn), prot)); + WARN_ON(!pmd_clear_huge(pmdp)); + pmd = READ_ONCE(*pmdp); + WARN_ON(!pmd_none(pmd)); +} + +static void __init pmd_savedwrite_tests(unsigned long pfn, pgprot_t prot) +{ + pmd_t pmd = pfn_pmd(pfn, prot); + + WARN_ON(!pmd_savedwrite(pmd_mk_savedwrite(pmd_clear_savedwrite(pmd)))); + WARN_ON(pmd_savedwrite(pmd_clear_savedwrite(pmd_mk_savedwrite(pmd)))); +} + #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD static void __init pud_basic_tests(unsigned long pfn, pgprot_t prot) { @@ -100,12 +235,119 @@ static void __init pud_basic_tests(unsig */ WARN_ON(!pud_bad(pud_mkhuge(pud))); } + +static void __init pud_advanced_tests(struct mm_struct *mm, + struct vm_area_struct *vma, pud_t *pudp, + unsigned long pfn, unsigned long vaddr, + pgprot_t prot) +{ + pud_t pud = pfn_pud(pfn, prot); + + if (!has_transparent_hugepage()) + return; + + /* Align the address wrt HPAGE_PUD_SIZE */ + vaddr = (vaddr & HPAGE_PUD_MASK) + HPAGE_PUD_SIZE; + + set_pud_at(mm, vaddr, pudp, pud); + pudp_set_wrprotect(mm, vaddr, pudp); + pud = READ_ONCE(*pudp); + WARN_ON(pud_write(pud)); + +#ifndef __PAGETABLE_PMD_FOLDED + pud = pfn_pud(pfn, prot); + set_pud_at(mm, vaddr, pudp, pud); + pudp_huge_get_and_clear(mm, vaddr, pudp); + pud = READ_ONCE(*pudp); + WARN_ON(!pud_none(pud)); + + pud = pfn_pud(pfn, prot); + set_pud_at(mm, vaddr, pudp, pud); + pudp_huge_get_and_clear_full(mm, vaddr, pudp, 1); + pud = READ_ONCE(*pudp); + WARN_ON(!pud_none(pud)); +#endif /* __PAGETABLE_PMD_FOLDED */ + pud = pfn_pud(pfn, prot); + pud = pud_wrprotect(pud); + pud = pud_mkclean(pud); + set_pud_at(mm, vaddr, pudp, pud); + pud = pud_mkwrite(pud); + pud = pud_mkdirty(pud); + pudp_set_access_flags(vma, vaddr, pudp, pud, 1); + pud = READ_ONCE(*pudp); + WARN_ON(!(pud_write(pud) && pud_dirty(pud))); + + pud = pud_mkyoung(pud); + set_pud_at(mm, vaddr, pudp, pud); + pudp_test_and_clear_young(vma, vaddr, pudp); + pud = READ_ONCE(*pudp); + WARN_ON(pud_young(pud)); +} + +static void __init pud_leaf_tests(unsigned long pfn, pgprot_t prot) +{ + pud_t pud = pfn_pud(pfn, prot); + + /* + * PUD based THP is a leaf entry. + */ + pud = pud_mkhuge(pud); + WARN_ON(!pud_leaf(pud)); +} + +static void __init pud_huge_tests(pud_t *pudp, unsigned long pfn, pgprot_t prot) +{ + pud_t pud; + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) + return; + /* + * X86 defined pud_set_huge() verifies that the given + * PUD is not a populated non-leaf entry. + */ + WRITE_ONCE(*pudp, __pud(0)); + WARN_ON(!pud_set_huge(pudp, __pfn_to_phys(pfn), prot)); + WARN_ON(!pud_clear_huge(pudp)); + pud = READ_ONCE(*pudp); + WARN_ON(!pud_none(pud)); +} #else /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ static void __init pud_basic_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pud_advanced_tests(struct mm_struct *mm, + struct vm_area_struct *vma, pud_t *pudp, + unsigned long pfn, unsigned long vaddr, + pgprot_t prot) +{ +} +static void __init pud_leaf_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pud_huge_tests(pud_t *pudp, unsigned long pfn, pgprot_t prot) +{ +} #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ #else /* !CONFIG_TRANSPARENT_HUGEPAGE */ static void __init pmd_basic_tests(unsigned long pfn, pgprot_t prot) { } static void __init pud_basic_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pmd_advanced_tests(struct mm_struct *mm, + struct vm_area_struct *vma, pmd_t *pmdp, + unsigned long pfn, unsigned long vaddr, + pgprot_t prot) +{ +} +static void __init pud_advanced_tests(struct mm_struct *mm, + struct vm_area_struct *vma, pud_t *pudp, + unsigned long pfn, unsigned long vaddr, + pgprot_t prot) +{ +} +static void __init pmd_leaf_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pud_leaf_tests(unsigned long pfn, pgprot_t prot) { } +static void __init pmd_huge_tests(pmd_t *pmdp, unsigned long pfn, pgprot_t prot) +{ +} +static void __init pud_huge_tests(pud_t *pudp, unsigned long pfn, pgprot_t prot) +{ +} +static void __init pmd_savedwrite_tests(unsigned long pfn, pgprot_t prot) { } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static void __init p4d_basic_tests(unsigned long pfn, pgprot_t prot) @@ -495,8 +737,56 @@ static void __init hugetlb_basic_tests(u WARN_ON(!pte_huge(pte_mkhuge(pte))); #endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ } + +static void __init hugetlb_advanced_tests(struct mm_struct *mm, + struct vm_area_struct *vma, + pte_t *ptep, unsigned long pfn, + unsigned long vaddr, pgprot_t prot) +{ + struct page *page = pfn_to_page(pfn); + pte_t pte = ptep_get(ptep); + unsigned long paddr = __pfn_to_phys(pfn) & PMD_MASK; + + pte = pte_mkhuge(mk_pte(pfn_to_page(PHYS_PFN(paddr)), prot)); + set_huge_pte_at(mm, vaddr, ptep, pte); + barrier(); + WARN_ON(!pte_same(pte, huge_ptep_get(ptep))); + huge_pte_clear(mm, vaddr, ptep, PMD_SIZE); + pte = huge_ptep_get(ptep); + WARN_ON(!huge_pte_none(pte)); + + pte = mk_huge_pte(page, prot); + set_huge_pte_at(mm, vaddr, ptep, pte); + barrier(); + huge_ptep_set_wrprotect(mm, vaddr, ptep); + pte = huge_ptep_get(ptep); + WARN_ON(huge_pte_write(pte)); + + pte = mk_huge_pte(page, prot); + set_huge_pte_at(mm, vaddr, ptep, pte); + barrier(); + huge_ptep_get_and_clear(mm, vaddr, ptep); + pte = huge_ptep_get(ptep); + WARN_ON(!huge_pte_none(pte)); + + pte = mk_huge_pte(page, prot); + pte = huge_pte_wrprotect(pte); + set_huge_pte_at(mm, vaddr, ptep, pte); + barrier(); + pte = huge_pte_mkwrite(pte); + pte = huge_pte_mkdirty(pte); + huge_ptep_set_access_flags(vma, vaddr, ptep, pte, 1); + pte = huge_ptep_get(ptep); + WARN_ON(!(huge_pte_write(pte) && huge_pte_dirty(pte))); +} #else /* !CONFIG_HUGETLB_PAGE */ static void __init hugetlb_basic_tests(unsigned long pfn, pgprot_t prot) { } +static void __init hugetlb_advanced_tests(struct mm_struct *mm, + struct vm_area_struct *vma, + pte_t *ptep, unsigned long pfn, + unsigned long vaddr, pgprot_t prot) +{ +} #endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -568,6 +858,7 @@ static unsigned long __init get_random_v static int __init debug_vm_pgtable(void) { + struct vm_area_struct *vma; struct mm_struct *mm; pgd_t *pgdp; p4d_t *p4dp, *saved_p4dp; @@ -596,6 +887,12 @@ static int __init debug_vm_pgtable(void) */ protnone = __P000; + vma = vm_area_alloc(mm); + if (!vma) { + pr_err("vma allocation failed\n"); + return 1; + } + /* * PFN for mapping at PTE level is determined from a standard kernel * text symbol. But pfns for higher page table levels are derived by @@ -644,6 +941,20 @@ static int __init debug_vm_pgtable(void) p4d_clear_tests(mm, p4dp); pgd_clear_tests(mm, pgdp); + pte_advanced_tests(mm, vma, ptep, pte_aligned, vaddr, prot); + pmd_advanced_tests(mm, vma, pmdp, pmd_aligned, vaddr, prot); + pud_advanced_tests(mm, vma, pudp, pud_aligned, vaddr, prot); + hugetlb_advanced_tests(mm, vma, ptep, pte_aligned, vaddr, prot); + + pmd_leaf_tests(pmd_aligned, prot); + pud_leaf_tests(pud_aligned, prot); + + pmd_huge_tests(pmdp, pmd_aligned, prot); + pud_huge_tests(pudp, pud_aligned, prot); + + pte_savedwrite_tests(pte_aligned, prot); + pmd_savedwrite_tests(pmd_aligned, prot); + pte_unmap_unlock(ptep, ptl); pmd_populate_tests(mm, pmdp, saved_ptep); @@ -678,6 +989,7 @@ static int __init debug_vm_pgtable(void) pmd_free(mm, saved_pmdp); pte_free(mm, saved_ptep); + vm_area_free(vma); mm_dec_nr_puds(mm); mm_dec_nr_pmds(mm); mm_dec_nr_ptes(mm); From patchwork Fri Aug 7 06:19:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704761 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E30EC13B6 for ; Fri, 7 Aug 2020 06:19:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9DE1122CB3 for ; Fri, 7 Aug 2020 06:19:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="YmWcac4d" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9DE1122CB3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 55A0D8D002B; Fri, 7 Aug 2020 02:19:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4E3F78D0026; Fri, 7 Aug 2020 02:19:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AA4A8D002B; Fri, 7 Aug 2020 02:19:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0241.hostedemail.com [216.40.44.241]) by kanga.kvack.org (Postfix) with ESMTP id 20E608D0026 for ; Fri, 7 Aug 2020 02:19:28 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id DC422181AEF10 for ; Fri, 7 Aug 2020 06:19:27 +0000 (UTC) X-FDA: 77122770774.18.self90_5b06b3a26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id B1E7E100EC676 for ; Fri, 7 Aug 2020 06:19:27 +0000 (UTC) X-Spam-Summary: 1,0,0,d7963c4080cfca30,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:4:41:334:355:368:369:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1963:2194:2199:2393:2525:2559:2563:2639:2682:2685:2693:2859:2892:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3167:3865:3867:3868:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4605:5007:6119:6261:6653:6737:6738:7576:7903:8784:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12291:12297:12438:12517:12519:12555:12679:12683:12783:12986:13191:13192:13229:13846:21080:21433:21451:21611:21627:21819:21939:21990:30054:30064:30075:30089,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygwbfj1bgdsd5b9pucwmt61k33wycoopp567kyb4zq4qf71fai835opucfbrq.bm9u6r4i9mq6nrigbxbo4bka5dd56d577fzrauj1dzbfgqsb9736qmmx6enq54n.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Ne tcheck:n X-HE-Tag: self90_5b06b3a26fbe X-Filterd-Recvd-Size: 15155 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:26 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7D33922CAF; Fri, 7 Aug 2020 06:19:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781166; bh=K1MsK7olPmfJwYDth7hRqN939txt5TNcQL1GYmdxar4=; h=Date:From:To:Subject:In-Reply-To:From; b=YmWcac4denZeHkclRnR6hGMbgLeg7Yc5sXYNh7NOpViOfn0Wx1L4+CEFYhQC0drq4 J1OvSf7wBBWN7KeUlYcRsYiXxNsIr6cp4MYonnui0bvBOh/EGD0Q5KV/OwR2n1tUpx 7437B7KTYkBbXVGPSHDCjp8ex4AGPSJEudx/4Fag= Date: Thu, 06 Aug 2020 23:19:25 -0700 From: Andrew Morton To: akpm@linux-foundation.org, anshuman.khandual@arm.com, benh@kernel.crashing.org, borntraeger@de.ibm.com, bp@alien8.de, catalin.marinas@arm.com, corbet@lwn.net, gor@linux.ibm.com, heiko.carstens@de.ibm.com, hpa@zytor.com, kirill@shutemov.name, linux-mm@kvack.org, mingo@redhat.com, mm-commits@vger.kernel.org, mpe@ellerman.id.au, palmer@dabbelt.com, paul.walmsley@sifive.com, paulus@samba.org, rppt@kernel.org, rppt@linux.ibm.com, tglx@linutronix.de, torvalds@linux-foundation.org, vgupta@synopsys.com, will@kernel.org, ziy@nvidia.com Subject: [patch 042/163] mm/debug_vm_pgtable: add debug prints for individual tests Message-ID: <20200807061925.0PZTbfDIA%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: B1E7E100EC676 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Anshuman Khandual Subject: mm/debug_vm_pgtable: add debug prints for individual tests This adds debug print information that enlists all tests getting executed on a given platform. With dynamic debug enabled, the following information will be splashed during boot. For compactness purpose, dropped both time stamp and prefix (i.e debug_vm_pgtable) from this sample output. [debug_vm_pgtable ]: Validating architecture page table helpers [pte_basic_tests ]: Validating PTE basic [pmd_basic_tests ]: Validating PMD basic [p4d_basic_tests ]: Validating P4D basic [pgd_basic_tests ]: Validating PGD basic [pte_clear_tests ]: Validating PTE clear [pmd_clear_tests ]: Validating PMD clear [pte_advanced_tests ]: Validating PTE advanced [pmd_advanced_tests ]: Validating PMD advanced [hugetlb_advanced_tests]: Validating HugeTLB advanced [pmd_leaf_tests ]: Validating PMD leaf [pmd_huge_tests ]: Validating PMD huge [pte_savedwrite_tests ]: Validating PTE saved write [pmd_savedwrite_tests ]: Validating PMD saved write [pmd_populate_tests ]: Validating PMD populate [pte_special_tests ]: Validating PTE special [pte_protnone_tests ]: Validating PTE protnone [pmd_protnone_tests ]: Validating PMD protnone [pte_devmap_tests ]: Validating PTE devmap [pmd_devmap_tests ]: Validating PMD devmap [pte_swap_tests ]: Validating PTE swap [swap_migration_tests ]: Validating swap migration [hugetlb_basic_tests ]: Validating HugeTLB basic [pmd_thp_tests ]: Validating PMD based THP Link: http://lkml.kernel.org/r/1593996516-7186-4-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual Reviewed-by: Zi Yan Tested-by: Vineet Gupta [arc] Cc: Catalin Marinas Cc: Mike Rapoport Cc: Vineet Gupta Cc: Will Deacon Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: Michael Ellerman Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Christian Borntraeger Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: "H. Peter Anvin" Cc: Kirill A. Shutemov Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Jonathan Corbet Cc: Mike Rapoport Signed-off-by: Andrew Morton --- mm/debug_vm_pgtable.c | 46 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) --- a/mm/debug_vm_pgtable.c~mm-debug_vm_pgtable-add-debug-prints-for-individual-tests +++ a/mm/debug_vm_pgtable.c @@ -8,7 +8,7 @@ * * Author: Anshuman Khandual */ -#define pr_fmt(fmt) "debug_vm_pgtable: %s: " fmt, __func__ +#define pr_fmt(fmt) "debug_vm_pgtable: [%-25s]: " fmt, __func__ #include #include @@ -48,6 +48,7 @@ static void __init pte_basic_tests(unsig { pte_t pte = pfn_pte(pfn, prot); + pr_debug("Validating PTE basic\n"); WARN_ON(!pte_same(pte, pte)); WARN_ON(!pte_young(pte_mkyoung(pte_mkold(pte)))); WARN_ON(!pte_dirty(pte_mkdirty(pte_mkclean(pte)))); @@ -64,6 +65,7 @@ static void __init pte_advanced_tests(st { pte_t pte = pfn_pte(pfn, prot); + pr_debug("Validating PTE advanced\n"); pte = pfn_pte(pfn, prot); set_pte_at(mm, vaddr, ptep, pte); ptep_set_wrprotect(mm, vaddr, ptep); @@ -103,6 +105,7 @@ static void __init pte_savedwrite_tests( { pte_t pte = pfn_pte(pfn, prot); + pr_debug("Validating PTE saved write\n"); WARN_ON(!pte_savedwrite(pte_mk_savedwrite(pte_clear_savedwrite(pte)))); WARN_ON(pte_savedwrite(pte_clear_savedwrite(pte_mk_savedwrite(pte)))); } @@ -114,6 +117,7 @@ static void __init pmd_basic_tests(unsig if (!has_transparent_hugepage()) return; + pr_debug("Validating PMD basic\n"); WARN_ON(!pmd_same(pmd, pmd)); WARN_ON(!pmd_young(pmd_mkyoung(pmd_mkold(pmd)))); WARN_ON(!pmd_dirty(pmd_mkdirty(pmd_mkclean(pmd)))); @@ -138,6 +142,7 @@ static void __init pmd_advanced_tests(st if (!has_transparent_hugepage()) return; + pr_debug("Validating PMD advanced\n"); /* Align the address wrt HPAGE_PMD_SIZE */ vaddr = (vaddr & HPAGE_PMD_MASK) + HPAGE_PMD_SIZE; @@ -180,6 +185,7 @@ static void __init pmd_leaf_tests(unsign { pmd_t pmd = pfn_pmd(pfn, prot); + pr_debug("Validating PMD leaf\n"); /* * PMD based THP is a leaf entry. */ @@ -193,6 +199,8 @@ static void __init pmd_huge_tests(pmd_t if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) return; + + pr_debug("Validating PMD huge\n"); /* * X86 defined pmd_set_huge() verifies that the given * PMD is not a populated non-leaf entry. @@ -208,6 +216,7 @@ static void __init pmd_savedwrite_tests( { pmd_t pmd = pfn_pmd(pfn, prot); + pr_debug("Validating PMD saved write\n"); WARN_ON(!pmd_savedwrite(pmd_mk_savedwrite(pmd_clear_savedwrite(pmd)))); WARN_ON(pmd_savedwrite(pmd_clear_savedwrite(pmd_mk_savedwrite(pmd)))); } @@ -220,6 +229,7 @@ static void __init pud_basic_tests(unsig if (!has_transparent_hugepage()) return; + pr_debug("Validating PUD basic\n"); WARN_ON(!pud_same(pud, pud)); WARN_ON(!pud_young(pud_mkyoung(pud_mkold(pud)))); WARN_ON(!pud_write(pud_mkwrite(pud_wrprotect(pud)))); @@ -246,6 +256,7 @@ static void __init pud_advanced_tests(st if (!has_transparent_hugepage()) return; + pr_debug("Validating PUD advanced\n"); /* Align the address wrt HPAGE_PUD_SIZE */ vaddr = (vaddr & HPAGE_PUD_MASK) + HPAGE_PUD_SIZE; @@ -288,6 +299,7 @@ static void __init pud_leaf_tests(unsign { pud_t pud = pfn_pud(pfn, prot); + pr_debug("Validating PUD leaf\n"); /* * PUD based THP is a leaf entry. */ @@ -301,6 +313,8 @@ static void __init pud_huge_tests(pud_t if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) return; + + pr_debug("Validating PUD huge\n"); /* * X86 defined pud_set_huge() verifies that the given * PUD is not a populated non-leaf entry. @@ -354,6 +368,7 @@ static void __init p4d_basic_tests(unsig { p4d_t p4d; + pr_debug("Validating P4D basic\n"); memset(&p4d, RANDOM_NZVALUE, sizeof(p4d_t)); WARN_ON(!p4d_same(p4d, p4d)); } @@ -362,6 +377,7 @@ static void __init pgd_basic_tests(unsig { pgd_t pgd; + pr_debug("Validating PGD basic\n"); memset(&pgd, RANDOM_NZVALUE, sizeof(pgd_t)); WARN_ON(!pgd_same(pgd, pgd)); } @@ -374,6 +390,7 @@ static void __init pud_clear_tests(struc if (mm_pmd_folded(mm)) return; + pr_debug("Validating PUD clear\n"); pud = __pud(pud_val(pud) | RANDOM_ORVALUE); WRITE_ONCE(*pudp, pud); pud_clear(pudp); @@ -388,6 +405,8 @@ static void __init pud_populate_tests(st if (mm_pmd_folded(mm)) return; + + pr_debug("Validating PUD populate\n"); /* * This entry points to next level page table page. * Hence this must not qualify as pud_bad(). @@ -414,6 +433,7 @@ static void __init p4d_clear_tests(struc if (mm_pud_folded(mm)) return; + pr_debug("Validating P4D clear\n"); p4d = __p4d(p4d_val(p4d) | RANDOM_ORVALUE); WRITE_ONCE(*p4dp, p4d); p4d_clear(p4dp); @@ -429,6 +449,7 @@ static void __init p4d_populate_tests(st if (mm_pud_folded(mm)) return; + pr_debug("Validating P4D populate\n"); /* * This entry points to next level page table page. * Hence this must not qualify as p4d_bad(). @@ -447,6 +468,7 @@ static void __init pgd_clear_tests(struc if (mm_p4d_folded(mm)) return; + pr_debug("Validating PGD clear\n"); pgd = __pgd(pgd_val(pgd) | RANDOM_ORVALUE); WRITE_ONCE(*pgdp, pgd); pgd_clear(pgdp); @@ -462,6 +484,7 @@ static void __init pgd_populate_tests(st if (mm_p4d_folded(mm)) return; + pr_debug("Validating PGD populate\n"); /* * This entry points to next level page table page. * Hence this must not qualify as pgd_bad(). @@ -490,6 +513,7 @@ static void __init pte_clear_tests(struc { pte_t pte = ptep_get(ptep); + pr_debug("Validating PTE clear\n"); pte = __pte(pte_val(pte) | RANDOM_ORVALUE); set_pte_at(mm, vaddr, ptep, pte); barrier(); @@ -502,6 +526,7 @@ static void __init pmd_clear_tests(struc { pmd_t pmd = READ_ONCE(*pmdp); + pr_debug("Validating PMD clear\n"); pmd = __pmd(pmd_val(pmd) | RANDOM_ORVALUE); WRITE_ONCE(*pmdp, pmd); pmd_clear(pmdp); @@ -514,6 +539,7 @@ static void __init pmd_populate_tests(st { pmd_t pmd; + pr_debug("Validating PMD populate\n"); /* * This entry points to next level page table page. * Hence this must not qualify as pmd_bad(). @@ -531,6 +557,7 @@ static void __init pte_special_tests(uns if (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_SPECIAL)) return; + pr_debug("Validating PTE special\n"); WARN_ON(!pte_special(pte_mkspecial(pte))); } @@ -541,6 +568,7 @@ static void __init pte_protnone_tests(un if (!IS_ENABLED(CONFIG_NUMA_BALANCING)) return; + pr_debug("Validating PTE protnone\n"); WARN_ON(!pte_protnone(pte)); WARN_ON(!pte_present(pte)); } @@ -553,6 +581,7 @@ static void __init pmd_protnone_tests(un if (!IS_ENABLED(CONFIG_NUMA_BALANCING)) return; + pr_debug("Validating PMD protnone\n"); WARN_ON(!pmd_protnone(pmd)); WARN_ON(!pmd_present(pmd)); } @@ -565,6 +594,7 @@ static void __init pte_devmap_tests(unsi { pte_t pte = pfn_pte(pfn, prot); + pr_debug("Validating PTE devmap\n"); WARN_ON(!pte_devmap(pte_mkdevmap(pte))); } @@ -573,6 +603,7 @@ static void __init pmd_devmap_tests(unsi { pmd_t pmd = pfn_pmd(pfn, prot); + pr_debug("Validating PMD devmap\n"); WARN_ON(!pmd_devmap(pmd_mkdevmap(pmd))); } @@ -581,6 +612,7 @@ static void __init pud_devmap_tests(unsi { pud_t pud = pfn_pud(pfn, prot); + pr_debug("Validating PUD devmap\n"); WARN_ON(!pud_devmap(pud_mkdevmap(pud))); } #else /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ @@ -603,6 +635,7 @@ static void __init pte_soft_dirty_tests( if (!IS_ENABLED(CONFIG_MEM_SOFT_DIRTY)) return; + pr_debug("Validating PTE soft dirty\n"); WARN_ON(!pte_soft_dirty(pte_mksoft_dirty(pte))); WARN_ON(pte_soft_dirty(pte_clear_soft_dirty(pte))); } @@ -614,6 +647,7 @@ static void __init pte_swap_soft_dirty_t if (!IS_ENABLED(CONFIG_MEM_SOFT_DIRTY)) return; + pr_debug("Validating PTE swap soft dirty\n"); WARN_ON(!pte_swp_soft_dirty(pte_swp_mksoft_dirty(pte))); WARN_ON(pte_swp_soft_dirty(pte_swp_clear_soft_dirty(pte))); } @@ -626,6 +660,7 @@ static void __init pmd_soft_dirty_tests( if (!IS_ENABLED(CONFIG_MEM_SOFT_DIRTY)) return; + pr_debug("Validating PMD soft dirty\n"); WARN_ON(!pmd_soft_dirty(pmd_mksoft_dirty(pmd))); WARN_ON(pmd_soft_dirty(pmd_clear_soft_dirty(pmd))); } @@ -638,6 +673,7 @@ static void __init pmd_swap_soft_dirty_t !IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION)) return; + pr_debug("Validating PMD swap soft dirty\n"); WARN_ON(!pmd_swp_soft_dirty(pmd_swp_mksoft_dirty(pmd))); WARN_ON(pmd_swp_soft_dirty(pmd_swp_clear_soft_dirty(pmd))); } @@ -653,6 +689,7 @@ static void __init pte_swap_tests(unsign swp_entry_t swp; pte_t pte; + pr_debug("Validating PTE swap\n"); pte = pfn_pte(pfn, prot); swp = __pte_to_swp_entry(pte); pte = __swp_entry_to_pte(swp); @@ -665,6 +702,7 @@ static void __init pmd_swap_tests(unsign swp_entry_t swp; pmd_t pmd; + pr_debug("Validating PMD swap\n"); pmd = pfn_pmd(pfn, prot); swp = __pmd_to_swp_entry(pmd); pmd = __swp_entry_to_pmd(swp); @@ -681,6 +719,8 @@ static void __init swap_migration_tests( if (!IS_ENABLED(CONFIG_MIGRATION)) return; + + pr_debug("Validating swap migration\n"); /* * swap_migration_tests() requires a dedicated page as it needs to * be locked before creating a migration entry from it. Locking the @@ -720,6 +760,7 @@ static void __init hugetlb_basic_tests(u struct page *page; pte_t pte; + pr_debug("Validating HugeTLB basic\n"); /* * Accessing the page associated with the pfn is safe here, * as it was previously derived from a real kernel symbol. @@ -747,6 +788,7 @@ static void __init hugetlb_advanced_test pte_t pte = ptep_get(ptep); unsigned long paddr = __pfn_to_phys(pfn) & PMD_MASK; + pr_debug("Validating HugeTLB advanced\n"); pte = pte_mkhuge(mk_pte(pfn_to_page(PHYS_PFN(paddr)), prot)); set_huge_pte_at(mm, vaddr, ptep, pte); barrier(); @@ -797,6 +839,7 @@ static void __init pmd_thp_tests(unsigne if (!has_transparent_hugepage()) return; + pr_debug("Validating PMD based THP\n"); /* * pmd_trans_huge() and pmd_present() must return positive after * MMU invalidation with pmd_mkinvalid(). This behavior is an @@ -825,6 +868,7 @@ static void __init pud_thp_tests(unsigne if (!has_transparent_hugepage()) return; + pr_debug("Validating PUD based THP\n"); pud = pfn_pud(pfn, prot); WARN_ON(!pud_trans_huge(pud_mkhuge(pud))); From patchwork Fri Aug 7 06:19:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704763 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DB96A13B6 for ; Fri, 7 Aug 2020 06:19:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 94DD32177B for ; Fri, 7 Aug 2020 06:19:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="uEsNrWY8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 94DD32177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 336B18D002C; Fri, 7 Aug 2020 02:19:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2E6DC8D0026; Fri, 7 Aug 2020 02:19:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FE948D002C; Fri, 7 Aug 2020 02:19:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id 014288D0026 for ; Fri, 7 Aug 2020 02:19:31 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id CA215180AD801 for ; Fri, 7 Aug 2020 06:19:31 +0000 (UTC) X-FDA: 77122770942.13.sink13_3d02b6126fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 9CB9A18140B67 for ; Fri, 7 Aug 2020 06:19:31 +0000 (UTC) X-Spam-Summary: 1,0,0,39a093451aaf90a1,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:327:355:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1801:1963:1981:2194:2199:2393:2525:2559:2565:2682:2685:2689:2693:2859:2901:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3167:3865:3867:3868:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4605:5007:6117:6119:6261:6653:6737:6738:7576:7903:9025:9545:10004:10226:10913:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13149:13230:13846:21080:21433:21451:21627:21749:21811:21819:21939:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrcapif4tfnk5mcz7641154wwmwyc187nz4cb53emsnbsay5jbtdsormgweip.w74wcs1dnyd8rmu7iy7z4m1xw67igzxq9otkdprushqgy356bxtf555yeogt6wx.6-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,N etcheck: X-HE-Tag: sink13_3d02b6126fbe X-Filterd-Recvd-Size: 23287 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:30 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5347122C9F; Fri, 7 Aug 2020 06:19:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781170; bh=t1meuJS9gVKSggsdND/htzTiHPaD/GvbXNGvZ8IM2ig=; h=Date:From:To:Subject:In-Reply-To:From; b=uEsNrWY83cj8CgQTqxOQn6+VGBSFsD+FC79kVfgJ+RWQN2o9/xB6s54/z7bsyb4At L+NuCvHsApu8qn524AVmNXyGaX7zc/HhIauHkpYkLtABCuAqOIfmOEnVdjKcGppuIk j2bv98MuOpFi5JqvAH/Mw5I1ILTDFS6y2ADBj3Yo= Date: Thu, 06 Aug 2020 23:19:28 -0700 From: Andrew Morton To: akpm@linux-foundation.org, anshuman.khandual@arm.com, benh@kernel.crashing.org, borntraeger@de.ibm.com, bp@alien8.de, catalin.marinas@arm.com, corbet@lwn.net, gor@linux.ibm.com, heiko.carstens@de.ibm.com, hpa@zytor.com, kirill@shutemov.name, linux-mm@kvack.org, mingo@redhat.com, mm-commits@vger.kernel.org, mpe@ellerman.id.au, palmer@dabbelt.com, paul.walmsley@sifive.com, paulus@samba.org, rppt@kernel.org, rppt@linux.ibm.com, tglx@linutronix.de, torvalds@linux-foundation.org, vgupta@synopsys.com, will@kernel.org, ziy@nvidia.com Subject: [patch 043/163] Documentation/mm: add descriptions for arch page table helpers Message-ID: <20200807061928.owCi0L_UF%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 9CB9A18140B67 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Anshuman Khandual Subject: Documentation/mm: add descriptions for arch page table helpers This adds a specific description file for all arch page table helpers which is in sync with the semantics being tested via CONFIG_DEBUG_VM_PGTABLE. All future changes either to these descriptions here or the debug test should always remain in sync. [anshuman.khandual@arm.com: fold in Mike's patch for the rst document, fix typos in the rst document] Link: http://lkml.kernel.org/r/1594610587-4172-5-git-send-email-anshuman.khandual@arm.com Link: http://lkml.kernel.org/r/1593996516-7186-5-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual Suggested-by: Mike Rapoport Acked-by: Mike Rapoport Cc: Jonathan Corbet Cc: Mike Rapoport Cc: Vineet Gupta Cc: Catalin Marinas Cc: Will Deacon Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: Michael Ellerman Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Christian Borntraeger Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: "H. Peter Anvin" Cc: Kirill A. Shutemov Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Zi Yan Signed-off-by: Andrew Morton --- Documentation/vm/arch_pgtable_helpers.rst | 258 ++++++++++++++++++++ mm/debug_vm_pgtable.c | 6 2 files changed, 264 insertions(+) --- /dev/null +++ a/Documentation/vm/arch_pgtable_helpers.rst @@ -0,0 +1,258 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _arch_page_table_helpers: + +=============================== +Architecture Page Table Helpers +=============================== + +Generic MM expects architectures (with MMU) to provide helpers to create, access +and modify page table entries at various level for different memory functions. +These page table helpers need to conform to a common semantics across platforms. +Following tables describe the expected semantics which can also be tested during +boot via CONFIG_DEBUG_VM_PGTABLE option. All future changes in here or the debug +test need to be in sync. + +====================== +PTE Page Table Helpers +====================== + ++---------------------------+--------------------------------------------------+ +| pte_same | Tests whether both PTE entries are the same | ++---------------------------+--------------------------------------------------+ +| pte_bad | Tests a non-table mapped PTE | ++---------------------------+--------------------------------------------------+ +| pte_present | Tests a valid mapped PTE | ++---------------------------+--------------------------------------------------+ +| pte_young | Tests a young PTE | ++---------------------------+--------------------------------------------------+ +| pte_dirty | Tests a dirty PTE | ++---------------------------+--------------------------------------------------+ +| pte_write | Tests a writable PTE | ++---------------------------+--------------------------------------------------+ +| pte_special | Tests a special PTE | ++---------------------------+--------------------------------------------------+ +| pte_protnone | Tests a PROT_NONE PTE | ++---------------------------+--------------------------------------------------+ +| pte_devmap | Tests a ZONE_DEVICE mapped PTE | ++---------------------------+--------------------------------------------------+ +| pte_soft_dirty | Tests a soft dirty PTE | ++---------------------------+--------------------------------------------------+ +| pte_swp_soft_dirty | Tests a soft dirty swapped PTE | ++---------------------------+--------------------------------------------------+ +| pte_mkyoung | Creates a young PTE | ++---------------------------+--------------------------------------------------+ +| pte_mkold | Creates an old PTE | ++---------------------------+--------------------------------------------------+ +| pte_mkdirty | Creates a dirty PTE | ++---------------------------+--------------------------------------------------+ +| pte_mkclean | Creates a clean PTE | ++---------------------------+--------------------------------------------------+ +| pte_mkwrite | Creates a writable PTE | ++---------------------------+--------------------------------------------------+ +| pte_mkwrprotect | Creates a write protected PTE | ++---------------------------+--------------------------------------------------+ +| pte_mkspecial | Creates a special PTE | ++---------------------------+--------------------------------------------------+ +| pte_mkdevmap | Creates a ZONE_DEVICE mapped PTE | ++---------------------------+--------------------------------------------------+ +| pte_mksoft_dirty | Creates a soft dirty PTE | ++---------------------------+--------------------------------------------------+ +| pte_clear_soft_dirty | Clears a soft dirty PTE | ++---------------------------+--------------------------------------------------+ +| pte_swp_mksoft_dirty | Creates a soft dirty swapped PTE | ++---------------------------+--------------------------------------------------+ +| pte_swp_clear_soft_dirty | Clears a soft dirty swapped PTE | ++---------------------------+--------------------------------------------------+ +| pte_mknotpresent | Invalidates a mapped PTE | ++---------------------------+--------------------------------------------------+ +| ptep_get_and_clear | Clears a PTE | ++---------------------------+--------------------------------------------------+ +| ptep_get_and_clear_full | Clears a PTE | ++---------------------------+--------------------------------------------------+ +| ptep_test_and_clear_young | Clears young from a PTE | ++---------------------------+--------------------------------------------------+ +| ptep_set_wrprotect | Converts into a write protected PTE | ++---------------------------+--------------------------------------------------+ +| ptep_set_access_flags | Converts into a more permissive PTE | ++---------------------------+--------------------------------------------------+ + +====================== +PMD Page Table Helpers +====================== + ++---------------------------+--------------------------------------------------+ +| pmd_same | Tests whether both PMD entries are the same | ++---------------------------+--------------------------------------------------+ +| pmd_bad | Tests a non-table mapped PMD | ++---------------------------+--------------------------------------------------+ +| pmd_leaf | Tests a leaf mapped PMD | ++---------------------------+--------------------------------------------------+ +| pmd_huge | Tests a HugeTLB mapped PMD | ++---------------------------+--------------------------------------------------+ +| pmd_trans_huge | Tests a Transparent Huge Page (THP) at PMD | ++---------------------------+--------------------------------------------------+ +| pmd_present | Tests a valid mapped PMD | ++---------------------------+--------------------------------------------------+ +| pmd_young | Tests a young PMD | ++---------------------------+--------------------------------------------------+ +| pmd_dirty | Tests a dirty PMD | ++---------------------------+--------------------------------------------------+ +| pmd_write | Tests a writable PMD | ++---------------------------+--------------------------------------------------+ +| pmd_special | Tests a special PMD | ++---------------------------+--------------------------------------------------+ +| pmd_protnone | Tests a PROT_NONE PMD | ++---------------------------+--------------------------------------------------+ +| pmd_devmap | Tests a ZONE_DEVICE mapped PMD | ++---------------------------+--------------------------------------------------+ +| pmd_soft_dirty | Tests a soft dirty PMD | ++---------------------------+--------------------------------------------------+ +| pmd_swp_soft_dirty | Tests a soft dirty swapped PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mkyoung | Creates a young PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mkold | Creates an old PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mkdirty | Creates a dirty PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mkclean | Creates a clean PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mkwrite | Creates a writable PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mkwrprotect | Creates a write protected PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mkspecial | Creates a special PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mkdevmap | Creates a ZONE_DEVICE mapped PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mksoft_dirty | Creates a soft dirty PMD | ++---------------------------+--------------------------------------------------+ +| pmd_clear_soft_dirty | Clears a soft dirty PMD | ++---------------------------+--------------------------------------------------+ +| pmd_swp_mksoft_dirty | Creates a soft dirty swapped PMD | ++---------------------------+--------------------------------------------------+ +| pmd_swp_clear_soft_dirty | Clears a soft dirty swapped PMD | ++---------------------------+--------------------------------------------------+ +| pmd_mkinvalid | Invalidates a mapped PMD [1] | ++---------------------------+--------------------------------------------------+ +| pmd_set_huge | Creates a PMD huge mapping | ++---------------------------+--------------------------------------------------+ +| pmd_clear_huge | Clears a PMD huge mapping | ++---------------------------+--------------------------------------------------+ +| pmdp_get_and_clear | Clears a PMD | ++---------------------------+--------------------------------------------------+ +| pmdp_get_and_clear_full | Clears a PMD | ++---------------------------+--------------------------------------------------+ +| pmdp_test_and_clear_young | Clears young from a PMD | ++---------------------------+--------------------------------------------------+ +| pmdp_set_wrprotect | Converts into a write protected PMD | ++---------------------------+--------------------------------------------------+ +| pmdp_set_access_flags | Converts into a more permissive PMD | ++---------------------------+--------------------------------------------------+ + +====================== +PUD Page Table Helpers +====================== + ++---------------------------+--------------------------------------------------+ +| pud_same | Tests whether both PUD entries are the same | ++---------------------------+--------------------------------------------------+ +| pud_bad | Tests a non-table mapped PUD | ++---------------------------+--------------------------------------------------+ +| pud_leaf | Tests a leaf mapped PUD | ++---------------------------+--------------------------------------------------+ +| pud_huge | Tests a HugeTLB mapped PUD | ++---------------------------+--------------------------------------------------+ +| pud_trans_huge | Tests a Transparent Huge Page (THP) at PUD | ++---------------------------+--------------------------------------------------+ +| pud_present | Tests a valid mapped PUD | ++---------------------------+--------------------------------------------------+ +| pud_young | Tests a young PUD | ++---------------------------+--------------------------------------------------+ +| pud_dirty | Tests a dirty PUD | ++---------------------------+--------------------------------------------------+ +| pud_write | Tests a writable PUD | ++---------------------------+--------------------------------------------------+ +| pud_devmap | Tests a ZONE_DEVICE mapped PUD | ++---------------------------+--------------------------------------------------+ +| pud_mkyoung | Creates a young PUD | ++---------------------------+--------------------------------------------------+ +| pud_mkold | Creates an old PUD | ++---------------------------+--------------------------------------------------+ +| pud_mkdirty | Creates a dirty PUD | ++---------------------------+--------------------------------------------------+ +| pud_mkclean | Creates a clean PUD | ++---------------------------+--------------------------------------------------+ +| pud_mkwrite | Creates a writable PUD | ++---------------------------+--------------------------------------------------+ +| pud_mkwrprotect | Creates a write protected PUD | ++---------------------------+--------------------------------------------------+ +| pud_mkdevmap | Creates a ZONE_DEVICE mapped PUD | ++---------------------------+--------------------------------------------------+ +| pud_mkinvalid | Invalidates a mapped PUD [1] | ++---------------------------+--------------------------------------------------+ +| pud_set_huge | Creates a PUD huge mapping | ++---------------------------+--------------------------------------------------+ +| pud_clear_huge | Clears a PUD huge mapping | ++---------------------------+--------------------------------------------------+ +| pudp_get_and_clear | Clears a PUD | ++---------------------------+--------------------------------------------------+ +| pudp_get_and_clear_full | Clears a PUD | ++---------------------------+--------------------------------------------------+ +| pudp_test_and_clear_young | Clears young from a PUD | ++---------------------------+--------------------------------------------------+ +| pudp_set_wrprotect | Converts into a write protected PUD | ++---------------------------+--------------------------------------------------+ +| pudp_set_access_flags | Converts into a more permissive PUD | ++---------------------------+--------------------------------------------------+ + +========================== +HugeTLB Page Table Helpers +========================== + ++---------------------------+--------------------------------------------------+ +| pte_huge | Tests a HugeTLB | ++---------------------------+--------------------------------------------------+ +| pte_mkhuge | Creates a HugeTLB | ++---------------------------+--------------------------------------------------+ +| huge_pte_dirty | Tests a dirty HugeTLB | ++---------------------------+--------------------------------------------------+ +| huge_pte_write | Tests a writable HugeTLB | ++---------------------------+--------------------------------------------------+ +| huge_pte_mkdirty | Creates a dirty HugeTLB | ++---------------------------+--------------------------------------------------+ +| huge_pte_mkwrite | Creates a writable HugeTLB | ++---------------------------+--------------------------------------------------+ +| huge_pte_mkwrprotect | Creates a write protected HugeTLB | ++---------------------------+--------------------------------------------------+ +| huge_ptep_get_and_clear | Clears a HugeTLB | ++---------------------------+--------------------------------------------------+ +| huge_ptep_set_wrprotect | Converts into a write protected HugeTLB | ++---------------------------+--------------------------------------------------+ +| huge_ptep_set_access_flags | Converts into a more permissive HugeTLB | ++---------------------------+--------------------------------------------------+ + +======================== +SWAP Page Table Helpers +======================== + ++---------------------------+--------------------------------------------------+ +| __pte_to_swp_entry | Creates a swapped entry (arch) from a mapped PTE | ++---------------------------+--------------------------------------------------+ +| __swp_to_pte_entry | Creates a mapped PTE from a swapped entry (arch) | ++---------------------------+--------------------------------------------------+ +| __pmd_to_swp_entry | Creates a swapped entry (arch) from a mapped PMD | ++---------------------------+--------------------------------------------------+ +| __swp_to_pmd_entry | Creates a mapped PMD from a swapped entry (arch) | ++---------------------------+--------------------------------------------------+ +| is_migration_entry | Tests a migration (read or write) swapped entry | ++---------------------------+--------------------------------------------------+ +| is_write_migration_entry | Tests a write migration swapped entry | ++---------------------------+--------------------------------------------------+ +| make_migration_entry_read | Converts into read migration swapped entry | ++---------------------------+--------------------------------------------------+ +| make_migration_entry | Creates a migration swapped entry (read or write)| ++---------------------------+--------------------------------------------------+ + +[1] https://lore.kernel.org/linux-mm/20181017020930.GN30832@redhat.com/ --- a/mm/debug_vm_pgtable.c~documentation-mm-add-descriptions-for-arch-page-table-helpers +++ a/mm/debug_vm_pgtable.c @@ -31,6 +31,12 @@ #include #include +/* + * Please refer Documentation/vm/arch_pgtable_helpers.rst for the semantics + * expectations that are being validated here. All future changes in here + * or the documentation need to be in sync. + */ + #define VMFLAGS (VM_READ|VM_WRITE|VM_EXEC) /* From patchwork Fri Aug 7 06:19:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704765 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E496113B6 for ; Fri, 7 Aug 2020 06:19:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AD2C72177B for ; Fri, 7 Aug 2020 06:19:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="zaH+Z2Vf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AD2C72177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5342E8D002D; Fri, 7 Aug 2020 02:19:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4BADD8D0026; Fri, 7 Aug 2020 02:19:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AC118D002D; Fri, 7 Aug 2020 02:19:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0135.hostedemail.com [216.40.44.135]) by kanga.kvack.org (Postfix) with ESMTP id 1B2B78D0026 for ; Fri, 7 Aug 2020 02:19:35 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E34011730840 for ; Fri, 7 Aug 2020 06:19:34 +0000 (UTC) X-FDA: 77122771068.18.rings45_2b1053026fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id B9768100EC679 for ; Fri, 7 Aug 2020 06:19:34 +0000 (UTC) X-Spam-Summary: 1,0,0,485e27f7bc81695e,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2194:2198:2199:2200:2393:2525:2559:2565:2682:2685:2693:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6119:6261:6653:7576:7903:8599:8660:8957:9025:9163:9545:10004:10913:11026:11257:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12663:12679:12783:12986:13148:13230:13846:13870:14181:14721:14849:21080:21451:21611:21627:21939:30012:30029:30054:30056:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8nxne58s8f5cd6sd3kdpdx3cyhop5fee87pyyua6aydjai9344433js8cg4w.ymaw7a934788ftinmm6ygbbswq8uooqnat859j5n6b8qm53ecsmctpm8yhaefbd.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5 ,Netchec X-HE-Tag: rings45_2b1053026fbe X-Filterd-Recvd-Size: 4030 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:34 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 20F74221E5; Fri, 7 Aug 2020 06:19:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781173; bh=n/ea/zQi/au7IEKqAk5RxZjCnmqcoq9RcR6QlNuyd88=; h=Date:From:To:Subject:In-Reply-To:From; b=zaH+Z2Vfle6tRskem2213ncqk3fjlyON7ywHvIYd2WMSxyYJQ2p2U5//qitCajcAI 8xM5P3ytEXo/v/s87PbXnK4kYsDiU1gMQd4RQv6B+dVJvofJS8SHAN4ZZ9KeOQexsz +yYeR/bd4e0kYsRXdhpetQgrmfu3lJnX2BjeG+PM= Date: Thu, 06 Aug 2020 23:19:32 -0700 From: Andrew Morton To: akpm@linux-foundation.org, jhubbard@nvidia.com, kirill@shutemov.name, linux-mm@kvack.org, mm-commits@vger.kernel.org, rppt@linux.ibm.com, torvalds@linux-foundation.org, vbabka@suse.cz, william.kucharski@oracle.com, willy@infradead.org Subject: [patch 044/163] mm/debug: handle page->mapping better in dump_page Message-ID: <20200807061932.6mICkpGjI%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: B9768100EC679 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000028, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Subject: mm/debug: handle page->mapping better in dump_page Patch series "Improvements for dump_page()", v2. Here's a sample dump of a pagecache tail page with all of the patches applied: page:000000006d1c49ca refcount:6 mapcount:0 mapping:00000000136b8d90 index:0x109 pfn:0x6c645 head:000000008bd38076 order:2 compound_mapcount:0 compound_pincount:0 aops:xfs_address_space_operations ino:800042 dentry name:"fd" flags: 0x4000000000012014(uptodate|lru|private|head) raw: 4000000000000000 ffffd46ac1b19101 ffffffff00000202 dead000000000004 raw: 0000000000000001 0000000000000000 00000000ffffffff 0000000000000000 head: 4000000000012014 ffffd46ac1b1bbc8 ffffd46ac1b1bc08 ffff91976f659560 head: 0000000000000108 ffff919773220680 00000006ffffffff 0000000000000000 page dumped because: testing This patch (of 6): If we can't call page_mapping() to get the page mapping, handle the anon/ksm/movable bits correctly. [akpm@linux-foundation.org: augmented code comment from John] Link: http://lkml.kernel.org/r/15cff11a-6762-8a6a-3f0e-dd227280cd6f@nvidia.com Link: http://lkml.kernel.org/r/20200709202117.7216-1-willy@infradead.org Link: http://lkml.kernel.org/r/20200709202117.7216-2-willy@infradead.org Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Mike Rapoport Reviewed-by: John Hubbard Acked-by: Vlastimil Babka Cc: William Kucharski Cc: "Kirill A. Shutemov" Signed-off-by: Andrew Morton --- mm/debug.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) --- a/mm/debug.c~mm-handle-page-mapping-better-in-dump_page +++ a/mm/debug.c @@ -69,8 +69,19 @@ void __dump_page(struct page *page, cons } if (page < head || (page >= head + MAX_ORDER_NR_PAGES)) { - /* Corrupt page, cannot call page_mapping */ - mapping = page->mapping; + /* + * Corrupt page, so we cannot call page_mapping. Instead, do a + * safe subset of the steps that page_mapping() does. Caution: + * this will be misleading for tail pages, PageSwapCache pages, + * and potentially other situations. (See the page_mapping() + * implementation for what's missing here.) + */ + unsigned long tmp = (unsigned long)page->mapping; + + if (tmp & PAGE_MAPPING_ANON) + mapping = NULL; + else + mapping = (void *)(tmp & ~PAGE_MAPPING_FLAGS); head = page; compound = false; } else { From patchwork Fri Aug 7 06:19:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704767 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3067013B6 for ; Fri, 7 Aug 2020 06:19:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EF3302177B for ; Fri, 7 Aug 2020 06:19:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="gjWBkB0Y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EF3302177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 99FC28D002E; Fri, 7 Aug 2020 02:19:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 94FDB8D0026; Fri, 7 Aug 2020 02:19:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88CCB8D002E; Fri, 7 Aug 2020 02:19:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0133.hostedemail.com [216.40.44.133]) by kanga.kvack.org (Postfix) with ESMTP id 6DADD8D0026 for ; Fri, 7 Aug 2020 02:19:38 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2165B1730840 for ; Fri, 7 Aug 2020 06:19:38 +0000 (UTC) X-FDA: 77122771236.29.owner38_5b0351d26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id E68E2180868E6 for ; Fri, 7 Aug 2020 06:19:37 +0000 (UTC) X-Spam-Summary: 1,0,0,8d6024d5a5ae7482,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2731:2859:2892:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3868:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6119:6261:6653:7576:7903:8599:8957:9025:9545:10004:10913:11026:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12683:12783:12986:13221:13229:13846:14181:14721:14849:21080:21451:21627:21939:30054:30064:30075,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygc7nht8szxidxhrcbeneweg4gxyp5fy87woyr5d9efampz6iud1rebhmoxjs.bp3qx79n3a13nxau9az1s6749rkpgt9fwnn63epotrbxghejyomd83jfefa9sik.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_ru les:0:0: X-HE-Tag: owner38_5b0351d26fbe X-Filterd-Recvd-Size: 3930 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:37 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6B57B221E5; Fri, 7 Aug 2020 06:19:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781176; bh=86nAWFLKlXEPp/TPLCAlfDHKQibf1P+rJgdFUDeUgF0=; h=Date:From:To:Subject:In-Reply-To:From; b=gjWBkB0Y3oWxX7FUGfGBpCcQInt5FCtFdKuL/p4jxe6gUWwsKoWvPgLFH5kehKDkY P48dy1hepKqEz4pRfRZKK3LXkJ+TC0vTG+esgWN/UXMJAEVKwY7emZDqnrV5ywq2gN IojXfoFcwTxnE/Jm2c8cqVyztR097S6sah4pCm/4= Date: Thu, 06 Aug 2020 23:19:35 -0700 From: Andrew Morton To: akpm@linux-foundation.org, jhubbard@nvidia.com, kirill@shutemov.name, linux-mm@kvack.org, mm-commits@vger.kernel.org, rppt@linux.ibm.com, torvalds@linux-foundation.org, vbabka@suse.cz, william.kucharski@oracle.com, willy@infradead.org Subject: [patch 045/163] mm/debug: dump compound page information on a second line Message-ID: <20200807061935.l0Lk7LmQ5%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E68E2180868E6 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Subject: mm/debug: dump compound page information on a second line Simplify both the implementation and the output by splitting all the compound page information onto a second line. Link: http://lkml.kernel.org/r/20200709202117.7216-3-willy@infradead.org Signed-off-by: Matthew Wilcox (Oracle) Reported-by: John Hubbard Reviewed-by: John Hubbard Tested-by: John Hubbard Acked-by: Mike Rapoport Acked-by: Vlastimil Babka Cc: "Kirill A. Shutemov" Cc: William Kucharski Signed-off-by: Andrew Morton --- mm/debug.c | 30 ++++++++++++------------------ 1 file changed, 12 insertions(+), 18 deletions(-) --- a/mm/debug.c~mm-dump-compound-page-information-on-a-second-line +++ a/mm/debug.c @@ -95,27 +95,21 @@ void __dump_page(struct page *page, cons */ mapcount = PageSlab(head) ? 0 : page_mapcount(page); - if (compound) + pr_warn("page:%px refcount:%d mapcount:%d mapping:%p index:%#lx\n", + page, page_ref_count(head), mapcount, mapping, + page_to_pgoff(page)); + if (compound) { if (hpage_pincount_available(page)) { - pr_warn("page:%px refcount:%d mapcount:%d mapping:%p " - "index:%#lx head:%px order:%u " - "compound_mapcount:%d compound_pincount:%d\n", - page, page_ref_count(head), mapcount, - mapping, page_to_pgoff(page), head, - compound_order(head), compound_mapcount(page), - compound_pincount(page)); + pr_warn("head:%px order:%u compound_mapcount:%d compound_pincount:%d\n", + head, compound_order(head), + compound_mapcount(head), + compound_pincount(head)); } else { - pr_warn("page:%px refcount:%d mapcount:%d mapping:%p " - "index:%#lx head:%px order:%u " - "compound_mapcount:%d\n", - page, page_ref_count(head), mapcount, - mapping, page_to_pgoff(page), head, - compound_order(head), compound_mapcount(page)); + pr_warn("head:%px order:%u compound_mapcount:%d\n", + head, compound_order(head), + compound_mapcount(head)); } - else - pr_warn("page:%px refcount:%d mapcount:%d mapping:%p index:%#lx\n", - page, page_ref_count(page), mapcount, - mapping, page_to_pgoff(page)); + } if (PageKsm(page)) type = "ksm "; else if (PageAnon(page)) From patchwork Fri Aug 7 06:19:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704769 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 34FC714B7 for ; Fri, 7 Aug 2020 06:19:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 060A72177B for ; Fri, 7 Aug 2020 06:19:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="AiqF741c" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 060A72177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AFE348D002F; Fri, 7 Aug 2020 02:19:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AAECF8D0026; Fri, 7 Aug 2020 02:19:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EC208D002F; Fri, 7 Aug 2020 02:19:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0092.hostedemail.com [216.40.44.92]) by kanga.kvack.org (Postfix) with ESMTP id 878BC8D0026 for ; Fri, 7 Aug 2020 02:19:41 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3FAC1181AEF10 for ; Fri, 7 Aug 2020 06:19:41 +0000 (UTC) X-FDA: 77122771362.27.dad38_2c0868126fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id 1B1943D663 for ; Fri, 7 Aug 2020 06:19:41 +0000 (UTC) X-Spam-Summary: 1,0,0,391e294e23c93b54,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2693:2859:2892:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3867:3868:3870:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:8599:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12517:12519:12555:12679:12783:12986:13069:13311:13357:13846:14181:14384:14721:14849:21080:21451:21627:21939:30054:30064:30075,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygqcn94jpwsb3qheg53m9ntzmujocjtnnhuesppgw593qbzrqxww1uf986s4g.uyr5hfwnzr1k9hhd8eaew7h3mzhpj6utwxbu8amrpt9e5nxk3ghyswuyeheh8rb.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custo m_rules: X-HE-Tag: dad38_2c0868126fbe X-Filterd-Recvd-Size: 2764 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:40 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 96BE1221E5; Fri, 7 Aug 2020 06:19:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781180; bh=hcjiEN1mC7jp49mnfMnVEO2vmtArkqbgEDjs37HuOMQ=; h=Date:From:To:Subject:In-Reply-To:From; b=AiqF741cpSznTP/wzTjW3LjLDPCIoUE5GUe6vK6sOj8twiO/TBmXR/EBcjT99Rari lnvx35MOGfrm5VqMglEJkAxGXRPP8PvzKu4YbhdimYYvCev5sTRVAWRYTZ7uOXaJHE WykvXC+LYU3HFRiKb4tUyNlQzNpH4X6puNf8+4W0= Date: Thu, 06 Aug 2020 23:19:39 -0700 From: Andrew Morton To: akpm@linux-foundation.org, jhubbard@nvidia.com, kirill@shutemov.name, linux-mm@kvack.org, mm-commits@vger.kernel.org, rppt@linux.ibm.com, torvalds@linux-foundation.org, vbabka@suse.cz, william.kucharski@oracle.com, willy@infradead.org Subject: [patch 046/163] mm/debug: print head flags in dump_page Message-ID: <20200807061939.k2nP4sOpR%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 1B1943D663 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000289, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Subject: mm/debug: print head flags in dump_page Tail page flags contain very little useful information. Print the head page's flags instead. While the flags will contain "head" for tail pages, this should not be too confusing as the previous line starts with the word "head:" and so the flags should be interpreted as belonging to the head page. Link: http://lkml.kernel.org/r/20200709202117.7216-4-willy@infradead.org Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Mike Rapoport Reviewed-by: John Hubbard Acked-by: Vlastimil Babka Cc: "Kirill A. Shutemov" Cc: William Kucharski Signed-off-by: Andrew Morton --- mm/debug.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/mm/debug.c~mm-print-head-flags-in-dump_page +++ a/mm/debug.c @@ -168,7 +168,7 @@ void __dump_page(struct page *page, cons out_mapping: BUILD_BUG_ON(ARRAY_SIZE(pageflag_names) != __NR_PAGEFLAGS + 1); - pr_warn("%sflags: %#lx(%pGp)%s\n", type, page->flags, &page->flags, + pr_warn("%sflags: %#lx(%pGp)%s\n", type, head->flags, &head->flags, page_cma ? " CMA" : ""); hex_only: From patchwork Fri Aug 7 06:19:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704771 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53EDA13B6 for ; Fri, 7 Aug 2020 06:19:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 286EF2177B for ; Fri, 7 Aug 2020 06:19:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="OV2wbO+3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 286EF2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BC3578D0030; Fri, 7 Aug 2020 02:19:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B725A8D0026; Fri, 7 Aug 2020 02:19:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB13F8D0030; Fri, 7 Aug 2020 02:19:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id 8ECE38D0026 for ; Fri, 7 Aug 2020 02:19:44 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 56F05181AEF10 for ; Fri, 7 Aug 2020 06:19:44 +0000 (UTC) X-FDA: 77122771488.13.self56_1600cdb26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 2864118140B67 for ; Fri, 7 Aug 2020 06:19:44 +0000 (UTC) X-Spam-Summary: 1,0,0,3f155e3b7763e016,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1543:1711:1730:1747:1777:1792:2198:2199:2393:2525:2553:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:8599:8957:9025:9163:9545:10004:10913:11026:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13229:13846:14181:14721:14849:21080:21433:21451:21627:21939:30003:30029:30054:30064:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yf1w5pqne1qodpm85c7ygfufjqwop9x4w16e3xud3iptgncayjy7pt88eq7ie.h3g8bygff91oetnc1z5fqybey76x1nagqrxm43z575nwnfwtn9fs685ws63kya9.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bul k,SPF:fp X-HE-Tag: self56_1600cdb26fbe X-Filterd-Recvd-Size: 4784 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:43 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AB518221E5; Fri, 7 Aug 2020 06:19:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781183; bh=7QtTJ3FPkWRiiCZlJHLjLumqsLmAtsFK+JAWD/dAsXE=; h=Date:From:To:Subject:In-Reply-To:From; b=OV2wbO+3Xctd4TWvlscu5Vg0/Ysevl1jQb1nI5V/FNontVotj5j5tNhcLm8MUWzaQ zjTRJs7aL0De4V1KL62xN/+PS9+1Xkm+6iJ6NIoFUHxakEN6KyJLUX9HxDeRThzySn 0bCuuUfHZWyW4P6Rej4tbCfiFAn/unSErQHYOPlo= Date: Thu, 06 Aug 2020 23:19:42 -0700 From: Andrew Morton To: akpm@linux-foundation.org, jhubbard@nvidia.com, kirill@shutemov.name, linux-mm@kvack.org, mm-commits@vger.kernel.org, rppt@linux.ibm.com, torvalds@linux-foundation.org, vbabka@suse.cz, william.kucharski@oracle.com, willy@infradead.org Subject: [patch 047/163] mm/debug: switch dump_page to get_kernel_nofault Message-ID: <20200807061942.vp2ZT_bvM%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 2864118140B67 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Subject: mm/debug: switch dump_page to get_kernel_nofault This is simpler to use than copy_from_kernel_nofault(). Also make some of the related error messages less verbose. Link: http://lkml.kernel.org/r/20200709202117.7216-5-willy@infradead.org Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Mike Rapoport Acked-by: Vlastimil Babka Cc: John Hubbard Cc: "Kirill A. Shutemov" Cc: William Kucharski Signed-off-by: Andrew Morton --- mm/debug.c | 36 ++++++++++++++++-------------------- 1 file changed, 16 insertions(+), 20 deletions(-) --- a/mm/debug.c~mm-switch-dump_page-to-get_kernel_nofault +++ a/mm/debug.c @@ -115,54 +115,50 @@ void __dump_page(struct page *page, cons else if (PageAnon(page)) type = "anon "; else if (mapping) { - const struct inode *host; + struct inode *host; const struct address_space_operations *a_ops; - const struct hlist_node *dentry_first; - const struct dentry *dentry_ptr; + struct hlist_node *dentry_first; + struct dentry *dentry_ptr; struct dentry dentry; /* * mapping can be invalid pointer and we don't want to crash * accessing it, so probe everything depending on it carefully */ - if (copy_from_kernel_nofault(&host, &mapping->host, - sizeof(struct inode *)) || - copy_from_kernel_nofault(&a_ops, &mapping->a_ops, - sizeof(struct address_space_operations *))) { - pr_warn("failed to read mapping->host or a_ops, mapping not a valid kernel address?\n"); + if (get_kernel_nofault(host, &mapping->host) || + get_kernel_nofault(a_ops, &mapping->a_ops)) { + pr_warn("failed to read mapping contents, not a valid kernel address?\n"); goto out_mapping; } if (!host) { - pr_warn("mapping->a_ops:%ps\n", a_ops); + pr_warn("aops:%ps\n", a_ops); goto out_mapping; } - if (copy_from_kernel_nofault(&dentry_first, - &host->i_dentry.first, sizeof(struct hlist_node *))) { - pr_warn("mapping->a_ops:%ps with invalid mapping->host inode address %px\n", - a_ops, host); + if (get_kernel_nofault(dentry_first, &host->i_dentry.first)) { + pr_warn("aops:%ps with invalid host inode %px\n", + a_ops, host); goto out_mapping; } if (!dentry_first) { - pr_warn("mapping->a_ops:%ps\n", a_ops); + pr_warn("aops:%ps\n", a_ops); goto out_mapping; } dentry_ptr = container_of(dentry_first, struct dentry, d_u.d_alias); - if (copy_from_kernel_nofault(&dentry, dentry_ptr, - sizeof(struct dentry))) { - pr_warn("mapping->aops:%ps with invalid mapping->host->i_dentry.first %px\n", - a_ops, dentry_ptr); + if (get_kernel_nofault(dentry, dentry_ptr)) { + pr_warn("aops:%ps with invalid dentry %px\n", a_ops, + dentry_ptr); } else { /* * if dentry is corrupted, the %pd handler may still * crash, but it's unlikely that we reach here with a * corrupted struct page */ - pr_warn("mapping->aops:%ps dentry name:\"%pd\"\n", - a_ops, &dentry); + pr_warn("aops:%ps dentry name:\"%pd\"\n", a_ops, + &dentry); } } out_mapping: From patchwork Fri Aug 7 06:19:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704773 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 76E3813B6 for ; Fri, 7 Aug 2020 06:19:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4A5CA2177B for ; Fri, 7 Aug 2020 06:19:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="R5qu9uOw" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4A5CA2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 160B98D0031; Fri, 7 Aug 2020 02:19:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 113ED8D0026; Fri, 7 Aug 2020 02:19:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02AD88D0031; Fri, 7 Aug 2020 02:19:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0128.hostedemail.com [216.40.44.128]) by kanga.kvack.org (Postfix) with ESMTP id DE34E8D0026 for ; Fri, 7 Aug 2020 02:19:47 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A7F7082499A8 for ; Fri, 7 Aug 2020 06:19:47 +0000 (UTC) X-FDA: 77122771614.22.beast08_0604bdb26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id 5514918038E67 for ; Fri, 7 Aug 2020 06:19:47 +0000 (UTC) X-Spam-Summary: 1,0,0,8f3cae57dc7bab73,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:560:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3870:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:8599:9025:9163:9545:10004:10913:11658:11914:12043:12048:12296:12297:12517:12519:12555:12679:12783:12986:13069:13161:13229:13311:13357:13846:14181:14384:14721:14849:21080:21433:21451:21627:21939:30029:30034:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygstupumkpwf1z74wgf3j1gs8kaopfsj8ago963993ebqwys1r44dirjxznbh.m6syakkozai1fignagb4rfu116znexc17nok946hk5774zi6c991dq7sp5s743z.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:ne utral,Cu X-HE-Tag: beast08_0604bdb26fbe X-Filterd-Recvd-Size: 2804 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:46 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D0166221E5; Fri, 7 Aug 2020 06:19:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781186; bh=t93+l4YlV+uP0W6/wlwhRLXpXRg9wB62OX2UFHk/pog=; h=Date:From:To:Subject:In-Reply-To:From; b=R5qu9uOwka0Z/XaxmbPb+6V2RutsVCkgDJPSNdcVhnZa3H+nx5DJwhhnqztL/KJcN y4xYXHJEzYZKmnm3SCkuYgSvbK4Wvhn9CZdF47uWxXPoyApvAAWJDCdh91mCNI01zp EGhkKw0Er6QfbqydVQIr4JEOkXdLdKGCUo/q/doE= Date: Thu, 06 Aug 2020 23:19:45 -0700 From: Andrew Morton To: akpm@linux-foundation.org, jhubbard@nvidia.com, kirill@shutemov.name, linux-mm@kvack.org, mm-commits@vger.kernel.org, rppt@linux.ibm.com, torvalds@linux-foundation.org, vbabka@suse.cz, william.kucharski@oracle.com, willy@infradead.org Subject: [patch 048/163] mm/debug: print the inode number in dump_page Message-ID: <20200807061945.pbfgaDBnc%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 5514918038E67 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Subject: mm/debug: print the inode number in dump_page The inode number helps correlate this page with debug messages elsewhere in the kernel. Link: http://lkml.kernel.org/r/20200709202117.7216-6-willy@infradead.org Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Mike Rapoport Reviewed-by: John Hubbard Cc: "Kirill A. Shutemov" Cc: Vlastimil Babka Cc: William Kucharski Signed-off-by: Andrew Morton --- mm/debug.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --- a/mm/debug.c~mm-print-the-inode-number-in-dump_page +++ a/mm/debug.c @@ -143,7 +143,7 @@ void __dump_page(struct page *page, cons } if (!dentry_first) { - pr_warn("aops:%ps\n", a_ops); + pr_warn("aops:%ps ino:%lx\n", a_ops, host->i_ino); goto out_mapping; } @@ -157,8 +157,8 @@ void __dump_page(struct page *page, cons * crash, but it's unlikely that we reach here with a * corrupted struct page */ - pr_warn("aops:%ps dentry name:\"%pd\"\n", a_ops, - &dentry); + pr_warn("aops:%ps ino:%lx dentry name:\"%pd\"\n", + a_ops, host->i_ino, &dentry); } } out_mapping: From patchwork Fri Aug 7 06:19:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704777 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C22614B7 for ; Fri, 7 Aug 2020 06:19:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 537F2221E5 for ; Fri, 7 Aug 2020 06:19:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Se3yqsVe" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 537F2221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3D38F8D0033; Fri, 7 Aug 2020 02:19:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3AB5F8D0026; Fri, 7 Aug 2020 02:19:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C28E8D0033; Fri, 7 Aug 2020 02:19:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0060.hostedemail.com [216.40.44.60]) by kanga.kvack.org (Postfix) with ESMTP id 10F098D0026 for ; Fri, 7 Aug 2020 02:19:57 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id CACF582499B9 for ; Fri, 7 Aug 2020 06:19:56 +0000 (UTC) X-FDA: 77122771992.08.horse23_470d26026fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 83A871819E785 for ; Fri, 7 Aug 2020 06:19:50 +0000 (UTC) X-Spam-Summary: 1,0,0,4104334550c3fb2e,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:5007:6119:6261:6653:7576:8599:8957:9025:9545:10004:10913:11026:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12783:12986:13069:13161:13229:13311:13357:13846:14181:14384:14721:14849:21080:21433:21451:21627:21939:30003:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8eu9iag6zuuhyw6gx56of6x9yeoc7ottursgj1hx53835kchzcbmn4hrbunf.f3eqyksf4nyx37bpoqgo8rruu4n6myxzgz3z7f58x5833aj49udxm1tfgsmdwxf.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,S PF:fp,MS X-HE-Tag: horse23_470d26026fbe X-Filterd-Recvd-Size: 3331 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:50 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0D9EB221E5; Fri, 7 Aug 2020 06:19:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781189; bh=ItaENuLzbga2ZYUsDhfJatfHSMpzLwPFCFJTP1wRIYs=; h=Date:From:To:Subject:In-Reply-To:From; b=Se3yqsVe+qRWcqEugDyTGzYFzMlGA79IK0HrXfxZdk8eTN03tpned3oCWLfKz238F fsdYKWuydeAa0QqNRbHqyVKLtd+/OLHGvh8GVVzdkLV0yL+S9PUm6FyfU0atwMvaON LF+pe8fgWPsz+IcuhSlNp3LPahlAjCzanL/WtPfo= Date: Thu, 06 Aug 2020 23:19:48 -0700 From: Andrew Morton To: akpm@linux-foundation.org, jhubbard@nvidia.com, kirill@shutemov.name, linux-mm@kvack.org, mm-commits@vger.kernel.org, rppt@linux.ibm.com, torvalds@linux-foundation.org, vbabka@suse.cz, william.kucharski@oracle.com, willy@infradead.org Subject: [patch 049/163] mm/debug: print hashed address of struct page Message-ID: <20200807061948.8LG6yOu9l%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 83A871819E785 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000020, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Subject: mm/debug: print hashed address of struct page The actual address of the struct page isn't particularly helpful, while the hashed address helps match with other messages elsewhere. Add the PFN that the page refers to in order to help diagnose problems where the page is improperly aligned for the purpose. Link: http://lkml.kernel.org/r/20200709202117.7216-7-willy@infradead.org Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Mike Rapoport Reviewed-by: John Hubbard Cc: "Kirill A. Shutemov" Cc: Vlastimil Babka Cc: William Kucharski Signed-off-by: Andrew Morton --- mm/debug.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) --- a/mm/debug.c~mm-print-hashed-address-of-struct-page +++ a/mm/debug.c @@ -95,17 +95,17 @@ void __dump_page(struct page *page, cons */ mapcount = PageSlab(head) ? 0 : page_mapcount(page); - pr_warn("page:%px refcount:%d mapcount:%d mapping:%p index:%#lx\n", + pr_warn("page:%p refcount:%d mapcount:%d mapping:%p index:%#lx pfn:%#lx\n", page, page_ref_count(head), mapcount, mapping, - page_to_pgoff(page)); + page_to_pgoff(page), page_to_pfn(page)); if (compound) { if (hpage_pincount_available(page)) { - pr_warn("head:%px order:%u compound_mapcount:%d compound_pincount:%d\n", + pr_warn("head:%p order:%u compound_mapcount:%d compound_pincount:%d\n", head, compound_order(head), compound_mapcount(head), compound_pincount(head)); } else { - pr_warn("head:%px order:%u compound_mapcount:%d\n", + pr_warn("head:%p order:%u compound_mapcount:%d\n", head, compound_order(head), compound_mapcount(head)); } From patchwork Fri Aug 7 06:19:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704775 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAEA313B6 for ; Fri, 7 Aug 2020 06:19:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7F92522C9F for ; Fri, 7 Aug 2020 06:19:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="pVNbSfkP" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7F92522C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6920B8D0032; Fri, 7 Aug 2020 02:19:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6432C8D0026; Fri, 7 Aug 2020 02:19:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52FB98D0032; Fri, 7 Aug 2020 02:19:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id 361798D0026 for ; Fri, 7 Aug 2020 02:19:54 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id F3EDA181AEF10 for ; Fri, 7 Aug 2020 06:19:53 +0000 (UTC) X-FDA: 77122771908.18.shirt92_560699026fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id CCD72100EC676 for ; Fri, 7 Aug 2020 06:19:53 +0000 (UTC) X-Spam-Summary: 1,0,0,ff92fde3a116059c,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:4605:5007:6261:6653:6737:7576:7903:8599:8957:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12783:12986:13138:13221:13229:13231:13846:14181:14721:14849:21080:21451:21611:21627:21939:21990:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y89zphkwrewkkeew99wjypfoyoiopu3n5wkb3cfh3isnimsgz5i1sygdoscpd.n4gm1kti6w7tijnujj1q5mbadumzah9f81ngnbtyg1ff45rq46ma63n65bksc5w.r-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none, DomainCa X-HE-Tag: shirt92_560699026fbe X-Filterd-Recvd-Size: 4929 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:53 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 49D312177B; Fri, 7 Aug 2020 06:19:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781192; bh=5rfT/i4s+593qml7Ut21PXhKcASWpR843y+S52K+/I8=; h=Date:From:To:Subject:In-Reply-To:From; b=pVNbSfkP/mhAK07lrNMLSJy8xE86Zpy2cAWWscTHHAymt0BtbS+L0P8ZAlFQDsPn1 XzJwBlNdVcC9jHPukZtHnUZSt6/i7a0KnnbZz9PoJQGCOCanfqzLrSFD+KF9h91U4r CMchpkui1VL9J/mSpwGL7owpZgZV7fXjbp0uKsCY= Date: Thu, 06 Aug 2020 23:19:51 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cai@lca.pw, jhubbard@nvidia.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, rppt@linux.ibm.com, torvalds@linux-foundation.org, vbabka@suse.cz, william.kucharski@oracle.com, willy@infradead.org Subject: [patch 050/163] mm, dump_page: do not crash with bad compound_mapcount() Message-ID: <20200807061951.-wx0cOeLH%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: CCD72100EC676 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: John Hubbard Subject: mm, dump_page: do not crash with bad compound_mapcount() If a compound page is being split while dump_page() is being run on that page, we can end up calling compound_mapcount() on a page that is no longer compound. This leads to a crash (already seen at least once in the field), due to the VM_BUG_ON_PAGE() assertion inside compound_mapcount(). (The above is from Matthew Wilcox's analysis of Qian Cai's bug report.) A similar problem is possible, via compound_pincount() instead of compound_mapcount(). In order to avoid this kind of crash, make dump_page() slightly more robust, by providing a pair of simpler routines that don't contain assertions: head_mapcount() and head_pincount(). For debug tools, we don't want to go *too* far in this direction, but this is a simple small fix, and the crash has already been seen, so it's a good trade-off. Link: http://lkml.kernel.org/r/20200804214807.169256-1-jhubbard@nvidia.com Signed-off-by: John Hubbard Reported-by: Qian Cai Suggested-by: Matthew Wilcox Acked-by: Vlastimil Babka Cc: Kirill A. Shutemov Cc: Mike Rapoport Cc: William Kucharski Signed-off-by: Andrew Morton --- include/linux/mm.h | 14 ++++++++++++-- mm/debug.c | 6 +++--- 2 files changed, 15 insertions(+), 5 deletions(-) --- a/include/linux/mm.h~mm-dump_page-do-not-crash-with-bad-compound_mapcount +++ a/include/linux/mm.h @@ -779,6 +779,11 @@ static inline void *kvcalloc(size_t n, s extern void kvfree(const void *addr); extern void kvfree_sensitive(const void *addr, size_t len); +static inline int head_mapcount(struct page *head) +{ + return atomic_read(compound_mapcount_ptr(head)) + 1; +} + /* * Mapcount of compound page as a whole, does not include mapped sub-pages. * @@ -788,7 +793,7 @@ static inline int compound_mapcount(stru { VM_BUG_ON_PAGE(!PageCompound(page), page); page = compound_head(page); - return atomic_read(compound_mapcount_ptr(page)) + 1; + return head_mapcount(page); } /* @@ -901,11 +906,16 @@ static inline bool hpage_pincount_availa return PageCompound(page) && compound_order(page) > 1; } +static inline int head_pincount(struct page *head) +{ + return atomic_read(compound_pincount_ptr(head)); +} + static inline int compound_pincount(struct page *page) { VM_BUG_ON_PAGE(!hpage_pincount_available(page), page); page = compound_head(page); - return atomic_read(compound_pincount_ptr(page)); + return head_pincount(page); } static inline void set_compound_order(struct page *page, unsigned int order) --- a/mm/debug.c~mm-dump_page-do-not-crash-with-bad-compound_mapcount +++ a/mm/debug.c @@ -102,12 +102,12 @@ void __dump_page(struct page *page, cons if (hpage_pincount_available(page)) { pr_warn("head:%p order:%u compound_mapcount:%d compound_pincount:%d\n", head, compound_order(head), - compound_mapcount(head), - compound_pincount(head)); + head_mapcount(head), + head_pincount(head)); } else { pr_warn("head:%p order:%u compound_mapcount:%d\n", head, compound_order(head), - compound_mapcount(head)); + head_mapcount(head)); } } if (PageKsm(page)) From patchwork Fri Aug 7 06:19:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704779 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0CF5E138C for ; Fri, 7 Aug 2020 06:20:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C6151221E5 for ; Fri, 7 Aug 2020 06:20:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="R8F3PuCY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C6151221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 61CDD8D0034; Fri, 7 Aug 2020 02:19:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5F61F8D0026; Fri, 7 Aug 2020 02:19:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 535C78D0034; Fri, 7 Aug 2020 02:19:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id 331968D0026 for ; Fri, 7 Aug 2020 02:19:59 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id F0C0582499B9 for ; Fri, 7 Aug 2020 06:19:58 +0000 (UTC) X-FDA: 77122772076.01.run76_19122ef26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 06BE110051B20 for ; Fri, 7 Aug 2020 06:19:57 +0000 (UTC) X-Spam-Summary: 1,0,0,be4fe7a3afbf2f39,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2690:2859:2901:2902:2918:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:7903:8603:9025:9391:9545:10004:10226:10913:11026:11473:11658:11914:12043:12048:12297:12517:12519:12555:12679:12783:12986:13069:13161:13221:13229:13311:13357:14093:14181:14384:14721:14777:14849:21060:21067:21080:21433:21451:21627:21740:21819:21939:30054,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8xwj3tfmejs8g1nnc9sza6dbc1opzuygo6jij1zz71jax95nm773tg8xsbj5.mzikjfumboiaqgnn74u9fnbi7g56au4ykd1kdkd45i8op367pidny39ihgst16m.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCa che:0,MS X-HE-Tag: run76_19122ef26fbe X-Filterd-Recvd-Size: 3115 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:56 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 925D122D03; Fri, 7 Aug 2020 06:19:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781195; bh=2ygTrxq33L8OrQfhcyX0bwMi+QHMXvPJRyccVxwO+Yw=; h=Date:From:To:Subject:In-Reply-To:From; b=R8F3PuCYNZ4dM/XJ9J5DjA79RtMlYGlplQDmiXXNt6BR6QIM/iLzFqtnYjvMs9ABt wjb+IOWCz8zeSdopKiMydNEc+Z7mF7rfm5KT7qY4uFYlk4u0pZq/2Xs+v0d4/rr0C6 SEU7Ij8CRpvs1StWOvYEyiy4bNaNg/5HlLL+C2N8= Date: Thu, 06 Aug 2020 23:19:55 -0700 From: Andrew Morton To: akpm@linux-foundation.org, gavin.dg@linux.alibaba.com, hannes@cmpxchg.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, riel@surriel.com, shakeelb@google.com, torvalds@linux-foundation.org, yang.shi@linux.alibaba.com Subject: [patch 051/163] mm: filemap: clear idle flag for writes Message-ID: <20200807061955.ErOcFZbpk%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 06BE110051B20 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yang Shi Subject: mm: filemap: clear idle flag for writes Since commit bbddabe2e436aa ("mm: filemap: only do access activations on reads"), mark_page_accessed() is called for reads only. But the idle flag is cleared by mark_page_accessed() so the idle flag won't get cleared if the page is write accessed only. Basically idle page tracking is used to estimate workingset size of workload, noticeable size of workingset might be missed if the idle flag is not maintained correctly. It seems good enough to just clear idle flag for write operations. Link: http://lkml.kernel.org/r/1593020612-13051-1-git-send-email-yang.shi@linux.alibaba.com Fixes: bbddabe2e436 ("mm: filemap: only do access activations on reads") Signed-off-by: Yang Shi Reported-by: Gang Deng Reviewed-by: Shakeel Butt Cc: Johannes Weiner Cc: Rik van Riel Signed-off-by: Andrew Morton --- mm/filemap.c | 6 ++++++ 1 file changed, 6 insertions(+) --- a/mm/filemap.c~mm-filemap-clear-idle-flag-for-writes +++ a/mm/filemap.c @@ -41,6 +41,7 @@ #include #include #include +#include #include "internal.h" #define CREATE_TRACE_POINTS @@ -1689,6 +1690,11 @@ repeat: if (fgp_flags & FGP_ACCESSED) mark_page_accessed(page); + else if (fgp_flags & FGP_WRITE) { + /* Clear idle flag for buffer write */ + if (page_is_idle(page)) + clear_page_idle(page); + } no_page: if (!page && (fgp_flags & FGP_CREAT)) { From patchwork Fri Aug 7 06:19:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704781 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 52D3E14B7 for ; Fri, 7 Aug 2020 06:20:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1B8DD22CAF for ; Fri, 7 Aug 2020 06:20:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="q/Hy8+5/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1B8DD22CAF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 273058D0035; Fri, 7 Aug 2020 02:20:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 24B238D0026; Fri, 7 Aug 2020 02:20:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 113DA8D0035; Fri, 7 Aug 2020 02:20:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0195.hostedemail.com [216.40.44.195]) by kanga.kvack.org (Postfix) with ESMTP id EEDED8D0026 for ; Fri, 7 Aug 2020 02:20:00 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BB6553628 for ; Fri, 7 Aug 2020 06:20:00 +0000 (UTC) X-FDA: 77122772160.08.gold66_6008b8a26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 5288D1819E793 for ; Fri, 7 Aug 2020 06:20:00 +0000 (UTC) X-Spam-Summary: 1,0,0,b573e9186fff939e,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2553:2559:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3867:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4398:5007:6119:6261:6653:6691:7576:7903:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12517:12519:12555:12679:12783:12986:13069:13255:13311:13357:14181:14384:14721:14777:14849:21060:21080:21324:21433:21451:21627:21819:21939:21983:30054:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrq81nduj9j461b8umxn377s553ypu3poqkewhywd86cqgs81rrxpsitosheo.6yt4k3at5jyqb6umhimdc1wk1o7x7pektz4rasfz6dkh34tc1bgaetxybwmgahs.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0, MSF:not X-HE-Tag: gold66_6008b8a26fbe X-Filterd-Recvd-Size: 2793 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:19:59 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id BDE7D2177B; Fri, 7 Aug 2020 06:19:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781199; bh=WPCsUZVSwftMcswc4eKyUaSUjWe8J64jW4G6JO1yuLE=; h=Date:From:To:Subject:In-Reply-To:From; b=q/Hy8+5/h6lTZ0yGQu6PyxaQyq4QbhFVexO72F9zajODoCPIhmajOomaNvS66VAzT DS3QMRTIOEo2zVEbkcwzh1a1tsWUO/ZGIHbjGH1igUlFydDHxh69AAv155AfLpNlJM CHl7at2r9Da+ZSI0rdQh04vh62Siw4SVt75FzMYw= Date: Thu, 06 Aug 2020 23:19:58 -0700 From: Andrew Morton To: akpm@linux-foundation.org, gavin.dg@linux.alibaba.com, hannes@cmpxchg.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, riel@surriel.com, shakeelb@google.com, torvalds@linux-foundation.org, yang.shi@linux.alibaba.com Subject: [patch 052/163] mm: filemap: add missing FGP_ flags in kerneldoc comment for pagecache_get_page Message-ID: <20200807061958.JLQGcG_41%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 5288D1819E793 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000015, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yang Shi Subject: mm: filemap: add missing FGP_ flags in kerneldoc comment for pagecache_get_page FGP_{WRITE|NOFS|NOWAIT} were missed in pagecache_get_page's kerneldoc comment. Link: http://lkml.kernel.org/r/1593031747-4249-1-git-send-email-yang.shi@linux.alibaba.com Signed-off-by: Yang Shi Cc: Gang Deng Cc: Shakeel Butt Cc: Johannes Weiner Cc: Rik van Riel Signed-off-by: Andrew Morton --- mm/filemap.c | 3 +++ 1 file changed, 3 insertions(+) --- a/mm/filemap.c~mm-filemap-add-missing-fgp_-flags-in-kerneldoc-comment-for-pagecache_get_page +++ a/mm/filemap.c @@ -1649,6 +1649,9 @@ EXPORT_SYMBOL(find_lock_entry); * * %FGP_FOR_MMAP - The caller wants to do its own locking dance if the * page is already in cache. If the page was allocated, unlock it before * returning so the caller can do the same dance. + * * %FGP_WRITE - The page will be written + * * %FGP_NOFS - __GFP_FS will get cleared in gfp mask + * * %FGP_NOWAIT - Don't get blocked by page lock * * If %FGP_LOCK or %FGP_CREAT are specified then the function may sleep even * if the %GFP flags specified for %FGP_CREAT are atomic. From patchwork Fri Aug 7 06:20:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704783 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 353A3138C for ; Fri, 7 Aug 2020 06:20:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EF3022177B for ; Fri, 7 Aug 2020 06:20:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="FRaI3u1p" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EF3022177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 83F8A8D0036; Fri, 7 Aug 2020 02:20:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8169B8D0026; Fri, 7 Aug 2020 02:20:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7552C8D0036; Fri, 7 Aug 2020 02:20:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0056.hostedemail.com [216.40.44.56]) by kanga.kvack.org (Postfix) with ESMTP id 5C2EB8D0026 for ; Fri, 7 Aug 2020 02:20:04 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2FA7B3628 for ; Fri, 7 Aug 2020 06:20:04 +0000 (UTC) X-FDA: 77122772328.03.lace59_040f3c826fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 003D228A4E9 for ; Fri, 7 Aug 2020 06:20:03 +0000 (UTC) X-Spam-Summary: 1,0,0,424e840884edf45d,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3868:3870:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13069:13161:13229:13255:13311:13357:13846:14181:14384:14721:14849:21080:21451:21627:21939:21990:30012:30034:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yruwq4hoazyci68k46t87zxrm9cycroee9prweshm85op3abfc7wrzihoebb8.zpgiszzsyrggn3cjdmhao1hju5myfb1sgjfo6ih7oejcoaer5qxhiuxzs74cuu8.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL: 0,DNSBL: X-HE-Tag: lace59_040f3c826fbe X-Filterd-Recvd-Size: 2420 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:03 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 763E322CB3; Fri, 7 Aug 2020 06:20:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781202; bh=UyXjNW6AyOGJvcaoAVH9fd3CR2zERdJZqy7geiMs9as=; h=Date:From:To:Subject:In-Reply-To:From; b=FRaI3u1pllaSXMO1cZthcjvH2XtwUaLbLwyq2ByScW0tguWP2TSiEhZzWBKRIh5tW zhmjQcJTl1hhaDTlRjaxZW2DDuGbziy5hV6uc03iugiOixKMVyZoLjWC3Qs/wi+lrI btTRmF7JzUItbP1UgvPAk6b6EFFEwoutKmPOMsEE= Date: Thu, 06 Aug 2020 23:20:01 -0700 From: Andrew Morton To: akpm@linux-foundation.org, ira.weiny@intel.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, tangyizhou@huawei.com, torvalds@linux-foundation.org Subject: [patch 053/163] mm/gup.c: fix the comment of return value for populate_vma_page_range() Message-ID: <20200807062001.1NvKva6WQ%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 003D228A4E9 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Tang Yizhou Subject: mm/gup.c: fix the comment of return value for populate_vma_page_range() The return value of populate_vma_page_range() is consistent with __get_user_pages(), and so is the function comment of return value. Link: http://lkml.kernel.org/r/20200720034303.29920-1-tangyizhou@huawei.com Signed-off-by: Tang Yizhou Reviewed-by: Andrew Morton Reviewed-by: Ira Weiny Signed-off-by: Andrew Morton --- mm/gup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --- a/mm/gup.c~mm-gupc-fix-the-comment-of-return-value-for-populate_vma_page_range +++ a/mm/gup.c @@ -1404,7 +1404,8 @@ retry: * * This takes care of mlocking the pages too if VM_LOCKED is set. * - * return 0 on success, negative error code on error. + * Return either number of pages pinned in the vma, or a negative error + * code on error. * * vma->vm_mm->mmap_lock must be held. * From patchwork Fri Aug 7 06:20:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704785 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5841814B7 for ; Fri, 7 Aug 2020 06:20:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 284FA2177B for ; Fri, 7 Aug 2020 06:20:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="xgxm6Rb+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 284FA2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B74088D0037; Fri, 7 Aug 2020 02:20:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B24318D0026; Fri, 7 Aug 2020 02:20:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A61D58D0037; Fri, 7 Aug 2020 02:20:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id 8DCA88D0026 for ; Fri, 7 Aug 2020 02:20:07 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5C4213637 for ; Fri, 7 Aug 2020 06:20:07 +0000 (UTC) X-FDA: 77122772454.05.crime17_54103a326fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 391921801D2F2 for ; Fri, 7 Aug 2020 06:20:07 +0000 (UTC) X-Spam-Summary: 1,0,0,417295ad7c9dd734,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2525:2559:2564:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6261:6653:7558:7576:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12114:12297:12438:12517:12519:12555:12679:12783:12986:13069:13311:13357:13846:14096:14181:14384:14721:14849:21080:21451:21627:21939:21990:30012:30054:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrtiaappbid4ck7eq81pfu5qfuzocyr5k6qhm5g7pajxbc71c9sypd45bcnxy.8ji5rp4bohigxra4368ed8fx8zo17npkqkfd43wfs8h59pyarzjc1whj4k4h3wp.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL: 0,DNSBL: X-HE-Tag: crime17_54103a326fbe X-Filterd-Recvd-Size: 3122 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:06 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AD061221E5; Fri, 7 Aug 2020 06:20:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781206; bh=lQafCK3Id4C0kM1ht3TIXdfhV0GFqWjodpCMHx6vd+Q=; h=Date:From:To:Subject:In-Reply-To:From; b=xgxm6Rb+fJdEBodRlEfynP8zaweNw+PEFCMPmClZ8XvOkFqSa3z+Ts38lsb0WEHiZ GJMI92mnLmaSbtyEbSVNfX7FEH6kG+wRjnpYHzCBCja5R6v7+iQM7HjZY0VleaG6pM KaHC7J5I72f9bTLpRhfXT5nh6/VOlLRCdSDfSss8= Date: Thu, 06 Aug 2020 23:20:05 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, thunder.leizhen@huawei.com, tim.c.chen@linux.intel.com, torvalds@linux-foundation.org Subject: [patch 054/163] mm/swap_slots.c: simplify alloc_swap_slot_cache() Message-ID: <20200807062005.O6oLXIaBo%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 391921801D2F2 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zhen Lei Subject: mm/swap_slots.c: simplify alloc_swap_slot_cache() Patch series "clean up some functions in mm/swap_slots.c". When I studied the code of mm/swap_slots.c, I found some places can be improved. This patch (of 3): Both "slots" and "slots_ret" are only need to be freed when cache already allocated. Make them closer, seems more clear. No functional change. Link: http://lkml.kernel.org/r/20200430061143.450-1-thunder.leizhen@huawei.com Link: http://lkml.kernel.org/r/20200430061143.450-2-thunder.leizhen@huawei.com Signed-off-by: Zhen Lei Acked-by: Tim Chen Signed-off-by: Andrew Morton --- mm/swap_slots.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) --- a/mm/swap_slots.c~mm-swap-simplify-alloc_swap_slot_cache +++ a/mm/swap_slots.c @@ -136,9 +136,16 @@ static int alloc_swap_slot_cache(unsigne mutex_lock(&swap_slots_cache_mutex); cache = &per_cpu(swp_slots, cpu); - if (cache->slots || cache->slots_ret) + if (cache->slots || cache->slots_ret) { /* cache already allocated */ - goto out; + mutex_unlock(&swap_slots_cache_mutex); + + kvfree(slots); + kvfree(slots_ret); + + return 0; + } + if (!cache->lock_initialized) { mutex_init(&cache->alloc_lock); spin_lock_init(&cache->free_lock); @@ -155,15 +162,8 @@ static int alloc_swap_slot_cache(unsigne */ mb(); cache->slots = slots; - slots = NULL; cache->slots_ret = slots_ret; - slots_ret = NULL; -out: mutex_unlock(&swap_slots_cache_mutex); - if (slots) - kvfree(slots); - if (slots_ret) - kvfree(slots_ret); return 0; } From patchwork Fri Aug 7 06:20:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704787 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 480EB138C for ; Fri, 7 Aug 2020 06:20:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1709522D00 for ; Fri, 7 Aug 2020 06:20:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="y7m20ZAw" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1709522D00 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A65248D0038; Fri, 7 Aug 2020 02:20:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A162A8D0026; Fri, 7 Aug 2020 02:20:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 905428D0038; Fri, 7 Aug 2020 02:20:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0164.hostedemail.com [216.40.44.164]) by kanga.kvack.org (Postfix) with ESMTP id 7B34A8D0026 for ; Fri, 7 Aug 2020 02:20:10 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 45991180AD801 for ; Fri, 7 Aug 2020 06:20:10 +0000 (UTC) X-FDA: 77122772580.23.horse95_070d3e926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 19D5537608 for ; Fri, 7 Aug 2020 06:20:10 +0000 (UTC) X-Spam-Summary: 1,0,0,84e33bc43e060516,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2194:2196:2199:2200:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3867:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:4605:5007:6261:6653:7558:7576:8599:8660:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12114:12297:12438:12517:12519:12555:12679:12783:12986:13069:13148:13230:13311:13357:13846:14181:14384:14721:14849:21080:21451:21627:21939:30054:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygrohhyigdimuwxzrhbg5ohmd4zyco5uhxu9c5a5rzu7mcfza8n66yet3jirg.qm1sbkbifjc4jtk6e8gri5iissp54z659e78dgw3z65f3jxo89z6xszyoqcqmtt.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SP X-HE-Tag: horse95_070d3e926fbe X-Filterd-Recvd-Size: 3264 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:09 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C19E722C9F; Fri, 7 Aug 2020 06:20:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781209; bh=A5XTRG/WG6FSRjHL8HCFSEvfNsiBkY/8EkP+IYiv6Kk=; h=Date:From:To:Subject:In-Reply-To:From; b=y7m20ZAwRlQQUjSMR3WjhBLCXkE0dt3X+NwF7czXSEcH2UGgyp3Fdr92itU79Ocjj OCaY9gIzK86PgtcUDpcrar70ff7mrPlDTqEebKZV8EFRi9jJuFWRkO0dY2jB0Au6DQ WYBqRpxHsKyNjm3mL7oOZQsv2oFz6/SZ9nN0OQAU= Date: Thu, 06 Aug 2020 23:20:08 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, thunder.leizhen@huawei.com, tim.c.chen@linux.intel.com, torvalds@linux-foundation.org Subject: [patch 055/163] mm/swap_slots.c: simplify enable_swap_slots_cache() Message-ID: <20200807062008.XfHKlbvxo%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 19D5537608 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zhen Lei Subject: mm/swap_slots.c: simplify enable_swap_slots_cache() Whether swap_slot_cache_initialized is true or false, __reenable_swap_slots_cache() is always called. To make this meaning clear, leave only one call to __reenable_swap_slots_cache(). This also make it clearer what extra needs be done when swap_slot_cache_initialized is false. No functional change. Link: http://lkml.kernel.org/r/20200430061143.450-3-thunder.leizhen@huawei.com Signed-off-by: Zhen Lei Acked-by: Tim Chen Signed-off-by: Andrew Morton --- mm/swap_slots.c | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) --- a/mm/swap_slots.c~mm-swap-simplify-enable_swap_slots_cache +++ a/mm/swap_slots.c @@ -240,21 +240,19 @@ static int free_slot_cache(unsigned int int enable_swap_slots_cache(void) { - int ret = 0; - mutex_lock(&swap_slots_cache_enable_mutex); - if (swap_slot_cache_initialized) { - __reenable_swap_slots_cache(); - goto out_unlock; - } + if (!swap_slot_cache_initialized) { + int ret; - ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "swap_slots_cache", - alloc_swap_slot_cache, free_slot_cache); - if (WARN_ONCE(ret < 0, "Cache allocation failed (%s), operating " - "without swap slots cache.\n", __func__)) - goto out_unlock; + ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "swap_slots_cache", + alloc_swap_slot_cache, free_slot_cache); + if (WARN_ONCE(ret < 0, "Cache allocation failed (%s), operating " + "without swap slots cache.\n", __func__)) + goto out_unlock; + + swap_slot_cache_initialized = true; + } - swap_slot_cache_initialized = true; __reenable_swap_slots_cache(); out_unlock: mutex_unlock(&swap_slots_cache_enable_mutex); From patchwork Fri Aug 7 06:20:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704789 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6A52114B7 for ; Fri, 7 Aug 2020 06:20:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3D401221E5 for ; Fri, 7 Aug 2020 06:20:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="z6qwmgWI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3D401221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 002FF8D003B; Fri, 7 Aug 2020 02:20:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F1B388D0026; Fri, 7 Aug 2020 02:20:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E589B8D003B; Fri, 7 Aug 2020 02:20:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id CBA258D0026 for ; Fri, 7 Aug 2020 02:20:13 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 98B0D180AD801 for ; Fri, 7 Aug 2020 06:20:13 +0000 (UTC) X-FDA: 77122772706.14.guide62_140cac226fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 6317A18229818 for ; Fri, 7 Aug 2020 06:20:13 +0000 (UTC) X-Spam-Summary: 1,0,0,e391931aded8667d,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3867:3868:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:8599:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13069:13255:13311:13357:13846:14093:14181:14384:14721:14849:21080:21451:21627:21939:21990:30054:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygcussa69xfie7supejba64ubeuyckmy5tio4qhid4roktptp79nrigythafo.kpahrr9gckdpw97xjnwh883i1a9n6ap4e98owpdy83rjriecsber4e7zdfidco4.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,D NSBL:neu X-HE-Tag: guide62_140cac226fbe X-Filterd-Recvd-Size: 3158 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:12 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EA6D42177B; Fri, 7 Aug 2020 06:20:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781212; bh=/i2Y/Aazfea7Zv50E/etBC9vl3UoPf9bBDuPixAtoDw=; h=Date:From:To:Subject:In-Reply-To:From; b=z6qwmgWIYnx228fShuZVxw7Zc0/tFJac8S1xQO34ZVNm5AwtrufE6koG5UumHSC05 FZUE4TtCQrbcmVvE6F82OIbO/JxFDQg3QATwMb8WjILtnxMRTQOxE2lmnj//5X2ytp LKA23tv9lQZJTXlt3LKgb355YZlL2/gcttF4XCak= Date: Thu, 06 Aug 2020 23:20:11 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, thunder.leizhen@huawei.com, tim.c.chen@linux.intel.com, torvalds@linux-foundation.org Subject: [patch 056/163] mm/swap_slots.c: remove redundant check for swap_slot_cache_initialized Message-ID: <20200807062011.jX4N4N-JP%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 6317A18229818 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zhen Lei Subject: mm/swap_slots.c: remove redundant check for swap_slot_cache_initialized Because enable_swap_slots_cache can only become true in enable_swap_slots_cache(), and depends on swap_slot_cache_initialized is true before. That means, when enable_swap_slots_cache is true, swap_slot_cache_initialized is true also. So the condition: "swap_slot_cache_enabled && swap_slot_cache_initialized" can be reduced to "swap_slot_cache_enabled" And in mathematics: "!swap_slot_cache_enabled || !swap_slot_cache_initialized" is equal to "!(swap_slot_cache_enabled && swap_slot_cache_initialized)" So no functional change. Link: http://lkml.kernel.org/r/20200430061143.450-4-thunder.leizhen@huawei.com Signed-off-by: Zhen Lei Acked-by: Tim Chen Signed-off-by: Andrew Morton --- mm/swap_slots.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) --- a/mm/swap_slots.c~mm-swap-remove-redundant-check-for-swap_slot_cache_initialized +++ a/mm/swap_slots.c @@ -46,8 +46,7 @@ static void __drain_swap_slots_cache(uns static void deactivate_swap_slots_cache(void); static void reactivate_swap_slots_cache(void); -#define use_swap_slot_cache (swap_slot_cache_active && \ - swap_slot_cache_enabled && swap_slot_cache_initialized) +#define use_swap_slot_cache (swap_slot_cache_active && swap_slot_cache_enabled) #define SLOTS_CACHE 0x1 #define SLOTS_CACHE_RET 0x2 @@ -94,7 +93,7 @@ static bool check_cache_active(void) { long pages; - if (!swap_slot_cache_enabled || !swap_slot_cache_initialized) + if (!swap_slot_cache_enabled) return false; pages = get_nr_swap_pages(); From patchwork Fri Aug 7 06:20:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704791 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 643EA138C for ; Fri, 7 Aug 2020 06:20:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 30EBD22CAF for ; Fri, 7 Aug 2020 06:20:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="uGvyHdZT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 30EBD22CAF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D49118D0040; Fri, 7 Aug 2020 02:20:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CF8EB8D0026; Fri, 7 Aug 2020 02:20:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C39398D0040; Fri, 7 Aug 2020 02:20:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0068.hostedemail.com [216.40.44.68]) by kanga.kvack.org (Postfix) with ESMTP id A78C08D0026 for ; Fri, 7 Aug 2020 02:20:16 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 6A857180AD801 for ; Fri, 7 Aug 2020 06:20:16 +0000 (UTC) X-FDA: 77122772832.04.suit90_0017a9226fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 44F56800E2D1 for ; Fri, 7 Aug 2020 06:20:16 +0000 (UTC) X-Spam-Summary: 1,0,0,07ac0554329ff004,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:982:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2892:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3866:3870:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:7875:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12114:12296:12297:12517:12519:12555:12679:12783:12986:13069:13311:13357:13846:14094:14181:14384:14721:14849:21080:21451:21627:21939:21972:30054:30075,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yrnxqnhiatgdocq84beexywemu9ypeew5zt46o8ey53nfdhea99ru5fwn1ift.bmj78i971supinxay7zpru8acwbg9hqsmz6jz3uft8mi3igragtyk5tmsuh6uox.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0: 0:0,LFti X-HE-Tag: suit90_0017a9226fbe X-Filterd-Recvd-Size: 2401 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:15 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0B90A22C9F; Fri, 7 Aug 2020 06:20:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781215; bh=tFm7AY9ZlSHtPVwC4YNYViMLjySw34F+CPflENjU16c=; h=Date:From:To:Subject:In-Reply-To:From; b=uGvyHdZT1T6NHISEE+Dyw1PBdVqBCEr9lQ0rdPDxzJWvYYisT7D15dSE6KXHac6MF f/KWsxVCNfDwPhz45DPUC6DrpSl2yVxkpDt4SvdyyXNS1Go/G+UQSy4Y5zxlxyWgC9 +Z+PTqMCAplx3d/OJKIv29bQ6s7HDECW7yvCxKGA= Date: Thu, 06 Aug 2020 23:20:14 -0700 From: Andrew Morton To: akpm@linux-foundation.org, krzk@kernel.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 057/163] mm: swap: fix kerneldoc of swap_vma_readahead() Message-ID: <20200807062014.N_3885n7O%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 44F56800E2D1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Krzysztof Kozlowski Subject: mm: swap: fix kerneldoc of swap_vma_readahead() Fix W=1 compile warnings (invalid kerneldoc): mm/swap_state.c:742: warning: Function parameter or member 'fentry' not described in 'swap_vma_readahead' mm/swap_state.c:742: warning: Excess function parameter 'entry' description in 'swap_vma_readahead' Link: http://lkml.kernel.org/r/20200728171109.28687-2-krzk@kernel.org Signed-off-by: Krzysztof Kozlowski Reviewed-by: Andrew Morton Signed-off-by: Andrew Morton --- mm/swap_state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/mm/swap_state.c~mm-swap-fix-kerneldoc-of-swap_vma_readahead +++ a/mm/swap_state.c @@ -725,7 +725,7 @@ static void swap_ra_info(struct vm_fault /** * swap_vma_readahead - swap in pages in hope we need them soon - * @entry: swap entry of this memory + * @fentry: swap entry of this memory * @gfp_mask: memory allocation flags * @vmf: fault information * From patchwork Fri Aug 7 06:20:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704793 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D3EAB14B7 for ; Fri, 7 Aug 2020 06:20:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 992FB221E5 for ; Fri, 7 Aug 2020 06:20:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="f2AXjbDy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 992FB221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 533118D0041; Fri, 7 Aug 2020 02:20:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4E2D68D0026; Fri, 7 Aug 2020 02:20:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F8FC8D0041; Fri, 7 Aug 2020 02:20:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0182.hostedemail.com [216.40.44.182]) by kanga.kvack.org (Postfix) with ESMTP id 260D08D0026 for ; Fri, 7 Aug 2020 02:20:20 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E1A171730840 for ; Fri, 7 Aug 2020 06:20:19 +0000 (UTC) X-FDA: 77122772958.11.soup44_010372f26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id B327E180F8B80 for ; Fri, 7 Aug 2020 06:20:19 +0000 (UTC) X-Spam-Summary: 1,0,0,cc7776fff2d1fcbc,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2553:2559:2563:2682:2685:2859:2899:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3151:3352:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4384:5007:6117:6261:6653:7576:7901:7903:9025:9391:9545:10004:10913:11026:11658:11914:12043:12048:12297:12517:12519:12555:12679:12783:12986:13069:13255:13311:13357:13846:14181:14384:14721:14777:14849:21080:21324:21433:21451:21600:21627:21819:21939:30054:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yfox91bhzuujy6ji3kh5qaiyiwkopzswoejnrh38sidb6ndz5546api11y5q4.8a4n4ptnozcwyqfimnqtzy9eci5ux91xzif3fzy81c587i5ckot4p6rrrgqzq99.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache: 0,MSF:no X-HE-Tag: soup44_010372f26fbe X-Filterd-Recvd-Size: 2888 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:19 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 24BEC2177B; Fri, 7 Aug 2020 06:20:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781218; bh=5Ljwl9+hAOQKO5kGK9vO0ejknpmWlSa6fzSTlSSNcoE=; h=Date:From:To:Subject:In-Reply-To:From; b=f2AXjbDyMWjyzmrSNonI+3C05GJklJI5EUBWwivLVQlqf1M8NSe/OBfKT6UR9EFGk T7CHVyBz2v/hvn/K50HRZFGMpEnBJQgSokSHiv5A4I7hb424IWwPzhsvCX3kajYRKk 2wCf2F2PO3S9zOsVF/LEBASRVsaIyFz3TApLv33w= Date: Thu, 06 Aug 2020 23:20:17 -0700 From: Andrew Morton To: akpm@linux-foundation.org, axboe@kernel.dk, bvanassche@acm.org, hare@suse.de, hughd@google.com, linux-mm@kvack.org, ming.lei@redhat.com, mm-commits@vger.kernel.org, torvalds@linux-foundation.org, xianting_tian@126.com Subject: [patch 058/163] mm/page_io.c: use blk_io_schedule() for avoiding task hung in sync io Message-ID: <20200807062017.4V6PayTV1%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: B327E180F8B80 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Xianting Tian Subject: mm/page_io.c: use blk_io_schedule() for avoiding task hung in sync io swap_readpage() does the sync io for one page, the io is not big, normally, the io can be finished quickly, but it may take long time or wait forever in case of io failure or discard. This patch uses blk_io_schedule() instead of io_schedule() to avoid task hung and crash (when set /proc/sys/kernel/hung_task_panic) when the above exception occurs. This is similar to the hung task avoidance in submit_bio_wait(), blk_execute_rq() and __blkdev_direct_IO(). Link: http://lkml.kernel.org/r/1596461807-21087-1-git-send-email-xianting_tian@126.com Signed-off-by: Xianting Tian Reviewed-by: Andrew Morton Cc: Ming Lei Cc: Bart Van Assche Cc: Hannes Reinecke Cc: Jens Axboe Cc: Hugh Dickins Signed-off-by: Andrew Morton --- mm/page_io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/mm/page_io.c~mm-use-blk_io_schedule-for-avoiding-task-hung-in-sync-io +++ a/mm/page_io.c @@ -441,7 +441,7 @@ int swap_readpage(struct page *page, boo break; if (!blk_poll(disk->queue, qc, true)) - io_schedule(); + blk_io_schedule(); } __set_current_state(TASK_RUNNING); bio_put(bio); From patchwork Fri Aug 7 06:20:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704795 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7D0CC14B7 for ; Fri, 7 Aug 2020 06:20:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 425AA22C9F for ; Fri, 7 Aug 2020 06:20:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="vC4BnVm2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 425AA22C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1D6878D0047; Fri, 7 Aug 2020 02:20:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 187CD8D0026; Fri, 7 Aug 2020 02:20:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09D588D0047; Fri, 7 Aug 2020 02:20:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0111.hostedemail.com [216.40.44.111]) by kanga.kvack.org (Postfix) with ESMTP id E66BA8D0026 for ; Fri, 7 Aug 2020 02:20:24 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id AE7F1249D for ; Fri, 7 Aug 2020 06:20:24 +0000 (UTC) X-FDA: 77122773168.25.tree18_3b0a38626fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 6DA771804E3A0 for ; Fri, 7 Aug 2020 06:20:24 +0000 (UTC) X-Spam-Summary: 10,1,0,a6e9d9968d01073b,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:2:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2525:2553:2559:2566:2682:2685:2689:2731:2859:2898:2902:2910:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4052:4250:4321:4384:4385:4395:4605:5007:6119:6261:6653:6737:7514:7576:7875:7903:8599:8603:9010:9025:9545:10004:11026:11232:11257:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13053:13221:13229:13846:14096:21080:21433:21450:21451:21627:21740:21889:21939:21990:22013:30034:30054:30066:30070:30075:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y88k3tyxpgxocaeg93yozwr8m7goc9pdoe3qwxit4zmdpu66jziswfotbn4qf.sb4zt9j3xo6tmibaxdptzbxwfk4naw dmomgk9y X-HE-Tag: tree18_3b0a38626fbe X-Filterd-Recvd-Size: 10531 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:23 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5EBE522CF6; Fri, 7 Aug 2020 06:20:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781223; bh=VqiFsLhUvzWDsnbBeIlUs9wGC6URW0MJz9rKg1JG/Vg=; h=Date:From:To:Subject:In-Reply-To:From; b=vC4BnVm2WuvkBAlBbGJlH4iDpddWSa/CSMGPRPmFYD9S43++kuUPqqTsI7gLiLDGI Qw8Z1GvvvYFAI9YdyipuYWYUGQfIEjjn7GqBb8qQUvlqXxKL40B+qDZ4JFBNJlJ8oM g0xROGPlncDr4KQtBCerCjfU2qV9ZhD7R67s9H9I= Date: Thu, 06 Aug 2020 23:20:20 -0700 From: Andrew Morton To: akpm@linux-foundation.org, amir73il@gmail.com, chris@chrisdown.name, hannes@cmpxchg.org, hughd@google.com, jlayton@kernel.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, tj@kernel.org, torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, willy@infradead.org Subject: [patch 059/163] tmpfs: per-superblock i_ino support Message-ID: <20200807062020.G315bvcEC%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 6DA771804E3A0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chris Down Subject: tmpfs: per-superblock i_ino support Patch series "tmpfs: inode: Reduce risk of inum overflow", v7. In Facebook production we are seeing heavy i_ino wraparounds on tmpfs. On affected tiers, in excess of 10% of hosts show multiple files with different content and the same inode number, with some servers even having as many as 150 duplicated inode numbers with differing file content. This causes actual, tangible problems in production. For example, we have complaints from those working on remote caches that their application is reporting cache corruptions because it uses (device, inodenum) to establish the identity of a particular cache object, but because it's not unique any more, the application refuses to continue and reports cache corruption. Even worse, sometimes applications may not even detect the corruption but may continue anyway, causing phantom and hard to debug behaviour. In general, userspace applications expect that (device, inodenum) should be enough to be uniquely point to one inode, which seems fair enough. One might also need to check the generation, but in this case: 1. That's not currently exposed to userspace (ioctl(...FS_IOC_GETVERSION...) returns ENOTTY on tmpfs); 2. Even with generation, there shouldn't be two live inodes with the same inode number on one device. In order to mitigate this, we take a two-pronged approach: 1. Moving inum generation from being global to per-sb for tmpfs. This itself allows some reduction in i_ino churn. This works on both 64- and 32- bit machines. 2. Adding inode{64,32} for tmpfs. This fix is supported on machines with 64-bit ino_t only: we allow users to mount tmpfs with a new inode64 option that uses the full width of ino_t, or CONFIG_TMPFS_INODE64. You can see how this compares to previous related patches which didn't implement this per-superblock: - https://patchwork.kernel.org/patch/11254001/ - https://patchwork.kernel.org/patch/11023915/ This patch (of 2): get_next_ino has a number of problems: - It uses and returns a uint, which is susceptible to become overflowed if a lot of volatile inodes that use get_next_ino are created. - It's global, with no specificity per-sb or even per-filesystem. This means it's not that difficult to cause inode number wraparounds on a single device, which can result in having multiple distinct inodes with the same inode number. This patch adds a per-superblock counter that mitigates the second case. This design also allows us to later have a specific i_ino size per-device, for example, allowing users to choose whether to use 32- or 64-bit inodes for each tmpfs mount. This is implemented in the next commit. For internal shmem mounts which may be less tolerant to spinlock delays, we implement a percpu batching scheme which only takes the stat_lock at each batch boundary. Link: http://lkml.kernel.org/r/cover.1594661218.git.chris@chrisdown.name Link: http://lkml.kernel.org/r/1986b9d63b986f08ec07a4aa4b2275e718e47d8a.1594661218.git.chris@chrisdown.name Signed-off-by: Chris Down Acked-by: Hugh Dickins Cc: Amir Goldstein Cc: Al Viro Cc: Matthew Wilcox Cc: Jeff Layton Cc: Johannes Weiner Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/fs.h | 15 ++++++++ include/linux/shmem_fs.h | 2 + mm/shmem.c | 66 ++++++++++++++++++++++++++++++++++--- 3 files changed, 78 insertions(+), 5 deletions(-) --- a/include/linux/fs.h~tmpfs-per-superblock-i_ino-support +++ a/include/linux/fs.h @@ -2946,6 +2946,21 @@ extern void discard_new_inode(struct ino extern unsigned int get_next_ino(void); extern void evict_inodes(struct super_block *sb); +/* + * Userspace may rely on the the inode number being non-zero. For example, glibc + * simply ignores files with zero i_ino in unlink() and other places. + * + * As an additional complication, if userspace was compiled with + * _FILE_OFFSET_BITS=32 on a 64-bit kernel we'll only end up reading out the + * lower 32 bits, so we need to check that those aren't zero explicitly. With + * _FILE_OFFSET_BITS=64, this may cause some harmless false-negatives, but + * better safe than sorry. + */ +static inline bool is_zero_ino(ino_t ino) +{ + return (u32)ino == 0; +} + extern void __iget(struct inode * inode); extern void iget_failed(struct inode *); extern void clear_inode(struct inode *); --- a/include/linux/shmem_fs.h~tmpfs-per-superblock-i_ino-support +++ a/include/linux/shmem_fs.h @@ -36,6 +36,8 @@ struct shmem_sb_info { unsigned char huge; /* Whether to try for hugepages */ kuid_t uid; /* Mount uid for root directory */ kgid_t gid; /* Mount gid for root directory */ + ino_t next_ino; /* The next per-sb inode number to use */ + ino_t __percpu *ino_batch; /* The next per-cpu inode number to use */ struct mempolicy *mpol; /* default memory policy for mappings */ spinlock_t shrinklist_lock; /* Protects shrinklist */ struct list_head shrinklist; /* List of shinkable inodes */ --- a/mm/shmem.c~tmpfs-per-superblock-i_ino-support +++ a/mm/shmem.c @@ -260,18 +260,67 @@ bool vma_is_shmem(struct vm_area_struct static LIST_HEAD(shmem_swaplist); static DEFINE_MUTEX(shmem_swaplist_mutex); -static int shmem_reserve_inode(struct super_block *sb) +/* + * shmem_reserve_inode() performs bookkeeping to reserve a shmem inode, and + * produces a novel ino for the newly allocated inode. + * + * It may also be called when making a hard link to permit the space needed by + * each dentry. However, in that case, no new inode number is needed since that + * internally draws from another pool of inode numbers (currently global + * get_next_ino()). This case is indicated by passing NULL as inop. + */ +#define SHMEM_INO_BATCH 1024 +static int shmem_reserve_inode(struct super_block *sb, ino_t *inop) { struct shmem_sb_info *sbinfo = SHMEM_SB(sb); - if (sbinfo->max_inodes) { + ino_t ino; + + if (!(sb->s_flags & SB_KERNMOUNT)) { spin_lock(&sbinfo->stat_lock); if (!sbinfo->free_inodes) { spin_unlock(&sbinfo->stat_lock); return -ENOSPC; } sbinfo->free_inodes--; + if (inop) { + ino = sbinfo->next_ino++; + if (unlikely(is_zero_ino(ino))) + ino = sbinfo->next_ino++; + if (unlikely(ino > UINT_MAX)) { + /* + * Emulate get_next_ino uint wraparound for + * compatibility + */ + ino = 1; + } + *inop = ino; + } spin_unlock(&sbinfo->stat_lock); + } else if (inop) { + /* + * __shmem_file_setup, one of our callers, is lock-free: it + * doesn't hold stat_lock in shmem_reserve_inode since + * max_inodes is always 0, and is called from potentially + * unknown contexts. As such, use a per-cpu batched allocator + * which doesn't require the per-sb stat_lock unless we are at + * the batch boundary. + */ + ino_t *next_ino; + next_ino = per_cpu_ptr(sbinfo->ino_batch, get_cpu()); + ino = *next_ino; + if (unlikely(ino % SHMEM_INO_BATCH == 0)) { + spin_lock(&sbinfo->stat_lock); + ino = sbinfo->next_ino; + sbinfo->next_ino += SHMEM_INO_BATCH; + spin_unlock(&sbinfo->stat_lock); + if (unlikely(is_zero_ino(ino))) + ino++; + } + *inop = ino; + *next_ino = ++ino; + put_cpu(); } + return 0; } @@ -2222,13 +2271,14 @@ static struct inode *shmem_get_inode(str struct inode *inode; struct shmem_inode_info *info; struct shmem_sb_info *sbinfo = SHMEM_SB(sb); + ino_t ino; - if (shmem_reserve_inode(sb)) + if (shmem_reserve_inode(sb, &ino)) return NULL; inode = new_inode(sb); if (inode) { - inode->i_ino = get_next_ino(); + inode->i_ino = ino; inode_init_owner(inode, dir, mode); inode->i_blocks = 0; inode->i_atime = inode->i_mtime = inode->i_ctime = current_time(inode); @@ -2932,7 +2982,7 @@ static int shmem_link(struct dentry *old * first link must skip that, to get the accounting right. */ if (inode->i_nlink) { - ret = shmem_reserve_inode(inode->i_sb); + ret = shmem_reserve_inode(inode->i_sb, NULL); if (ret) goto out; } @@ -3584,6 +3634,7 @@ static void shmem_put_super(struct super { struct shmem_sb_info *sbinfo = SHMEM_SB(sb); + free_percpu(sbinfo->ino_batch); percpu_counter_destroy(&sbinfo->used_blocks); mpol_put(sbinfo->mpol); kfree(sbinfo); @@ -3626,6 +3677,11 @@ static int shmem_fill_super(struct super #endif sbinfo->max_blocks = ctx->blocks; sbinfo->free_inodes = sbinfo->max_inodes = ctx->inodes; + if (sb->s_flags & SB_KERNMOUNT) { + sbinfo->ino_batch = alloc_percpu(ino_t); + if (!sbinfo->ino_batch) + goto failed; + } sbinfo->uid = ctx->uid; sbinfo->gid = ctx->gid; sbinfo->mode = ctx->mode; From patchwork Fri Aug 7 06:20:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704797 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E7A0D138C for ; Fri, 7 Aug 2020 06:20:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AC6FB2177B for ; Fri, 7 Aug 2020 06:20:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="2hoZGHkm" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AC6FB2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 494698D0048; Fri, 7 Aug 2020 02:20:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 445538D0026; Fri, 7 Aug 2020 02:20:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35D348D0048; Fri, 7 Aug 2020 02:20:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0186.hostedemail.com [216.40.44.186]) by kanga.kvack.org (Postfix) with ESMTP id 1FDC88D0026 for ; Fri, 7 Aug 2020 02:20:28 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C9A6B180AD801 for ; Fri, 7 Aug 2020 06:20:27 +0000 (UTC) X-FDA: 77122773294.11.camp50_230a99526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 9DC62180F8B80 for ; Fri, 7 Aug 2020 06:20:27 +0000 (UTC) X-Spam-Summary: 1,0,0,fcc1dd2c276e7e88,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:2:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1801:2198:2199:2393:2525:2553:2559:2564:2682:2685:2693:2859:2895:2902:2909:2910:2918:2924:2925:2926:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4052:4250:4321:4605:5007:6119:6261:6630:6653:6737:7514:7576:7903:8599:8603:8784:9025:9164:9545:10004:10946:11026:11257:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12700:12737:12783:12895:12986:13053:13161:13229:13846:14096:21080:21433:21451:21611:21627:21939:21990:30016:30030:30034:30054:30062:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8kj75ccdp8s6ma6n76fxe1h4aaycb586se6ta4btyatpppfc3asb4kx4kg3s.65g8tiqj68d4ckdjmbp6fbhidz3fmep yq5wxiri X-HE-Tag: camp50_230a99526fbe X-Filterd-Recvd-Size: 10827 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:27 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EFB7222CB3; Fri, 7 Aug 2020 06:20:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781226; bh=vC8PJadMiLum6Wk3wBw3my5YygU2rTn7SeI+0FcBL28=; h=Date:From:To:Subject:In-Reply-To:From; b=2hoZGHkm9+AaTTpw4QMpDE/D+C0oL52agXmQ0vCp95wFYJWFXy5AUhqtDcQ5qV6Z5 OA3UGOUDIBHTmHaSZAX15bYvsa55M8Gt0xn/BZ6I0l1u4CInSEgI+Shf5T5F7d5VXU DiRkL0xhX6B9NRSEU7Zl1wLven25IPQMrxsNLMGA= Date: Thu, 06 Aug 2020 23:20:25 -0700 From: Andrew Morton To: akpm@linux-foundation.org, amir73il@gmail.com, chris@chrisdown.name, hannes@cmpxchg.org, hughd@google.com, jlayton@kernel.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, tj@kernel.org, torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, willy@infradead.org Subject: [patch 060/163] tmpfs: support 64-bit inums per-sb Message-ID: <20200807062025.I1TSnFj3V%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 9DC62180F8B80 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chris Down Subject: tmpfs: support 64-bit inums per-sb The default is still set to inode32 for backwards compatibility, but system administrators can opt in to the new 64-bit inode numbers by either: 1. Passing inode64 on the command line when mounting, or 2. Configuring the kernel with CONFIG_TMPFS_INODE64=y The inode64 and inode32 names are used based on existing precedent from XFS. [hughd@google.com: Kconfig fixes] Link: http://lkml.kernel.org/r/alpine.LSU.2.11.2008011928010.13320@eggly.anvils Link: http://lkml.kernel.org/r/8b23758d0c66b5e2263e08baf9c4b6a7565cbd8f.1594661218.git.chris@chrisdown.name Signed-off-by: Chris Down Signed-off-by: Hugh Dickins Reviewed-by: Amir Goldstein Acked-by: Hugh Dickins Cc: Al Viro Cc: Matthew Wilcox Cc: Jeff Layton Cc: Johannes Weiner Cc: Tejun Heo Signed-off-by: Andrew Morton --- Documentation/filesystems/tmpfs.rst | 18 +++++++ fs/Kconfig | 21 ++++++++ include/linux/shmem_fs.h | 1 mm/shmem.c | 65 +++++++++++++++++++++++++- 4 files changed, 103 insertions(+), 2 deletions(-) --- a/Documentation/filesystems/tmpfs.rst~tmpfs-support-64-bit-inums-per-sb +++ a/Documentation/filesystems/tmpfs.rst @@ -150,6 +150,22 @@ These options do not have any effect on parameters with chmod(1), chown(1) and chgrp(1) on a mounted filesystem. +tmpfs has a mount option to select whether it will wrap at 32- or 64-bit inode +numbers: + +======= ======================== +inode64 Use 64-bit inode numbers +inode32 Use 32-bit inode numbers +======= ======================== + +On a 32-bit kernel, inode32 is implicit, and inode64 is refused at mount time. +On a 64-bit kernel, CONFIG_TMPFS_INODE64 sets the default. inode64 avoids the +possibility of multiple files with the same inode number on a single device; +but risks glibc failing with EOVERFLOW once 33-bit inode numbers are reached - +if a long-lived tmpfs is accessed by 32-bit applications so ancient that +opening a file larger than 2GiB fails with EINVAL. + + So 'mount -t tmpfs -o size=10G,nr_inodes=10k,mode=700 tmpfs /mytmpfs' will give you tmpfs instance on /mytmpfs which can allocate 10GB RAM/SWAP in 10240 inodes and it is only accessible by root. @@ -161,3 +177,5 @@ RAM/SWAP in 10240 inodes and it is only Hugh Dickins, 4 June 2007 :Updated: KOSAKI Motohiro, 16 Mar 2010 +:Updated: + Chris Down, 13 July 2020 --- a/fs/Kconfig~tmpfs-support-64-bit-inums-per-sb +++ a/fs/Kconfig @@ -201,6 +201,27 @@ config TMPFS_XATTR If unsure, say N. +config TMPFS_INODE64 + bool "Use 64-bit ino_t by default in tmpfs" + depends on TMPFS && 64BIT + default n + help + tmpfs has historically used only inode numbers as wide as an unsigned + int. In some cases this can cause wraparound, potentially resulting + in multiple files with the same inode number on a single device. This + option makes tmpfs use the full width of ino_t by default, without + needing to specify the inode64 option when mounting. + + But if a long-lived tmpfs is to be accessed by 32-bit applications so + ancient that opening a file larger than 2GiB fails with EINVAL, then + the INODE64 config option and inode64 mount option risk operations + failing with EOVERFLOW once 33-bit inode numbers are reached. + + To override this configured default, use the inode32 or inode64 + option when mounting. + + If unsure, say N. + config HUGETLBFS bool "HugeTLB file system support" depends on X86 || IA64 || SPARC64 || (S390 && 64BIT) || \ --- a/include/linux/shmem_fs.h~tmpfs-support-64-bit-inums-per-sb +++ a/include/linux/shmem_fs.h @@ -36,6 +36,7 @@ struct shmem_sb_info { unsigned char huge; /* Whether to try for hugepages */ kuid_t uid; /* Mount uid for root directory */ kgid_t gid; /* Mount gid for root directory */ + bool full_inums; /* If i_ino should be uint or ino_t */ ino_t next_ino; /* The next per-sb inode number to use */ ino_t __percpu *ino_batch; /* The next per-cpu inode number to use */ struct mempolicy *mpol; /* default memory policy for mappings */ --- a/mm/shmem.c~tmpfs-support-64-bit-inums-per-sb +++ a/mm/shmem.c @@ -114,11 +114,13 @@ struct shmem_options { kuid_t uid; kgid_t gid; umode_t mode; + bool full_inums; int huge; int seen; #define SHMEM_SEEN_BLOCKS 1 #define SHMEM_SEEN_INODES 2 #define SHMEM_SEEN_HUGE 4 +#define SHMEM_SEEN_INUMS 8 }; #ifdef CONFIG_TMPFS @@ -286,12 +288,17 @@ static int shmem_reserve_inode(struct su ino = sbinfo->next_ino++; if (unlikely(is_zero_ino(ino))) ino = sbinfo->next_ino++; - if (unlikely(ino > UINT_MAX)) { + if (unlikely(!sbinfo->full_inums && + ino > UINT_MAX)) { /* * Emulate get_next_ino uint wraparound for * compatibility */ - ino = 1; + if (IS_ENABLED(CONFIG_64BIT)) + pr_warn("%s: inode number overflow on device %d, consider using inode64 mount option\n", + __func__, MINOR(sb->s_dev)); + sbinfo->next_ino = 1; + ino = sbinfo->next_ino++; } *inop = ino; } @@ -304,6 +311,10 @@ static int shmem_reserve_inode(struct su * unknown contexts. As such, use a per-cpu batched allocator * which doesn't require the per-sb stat_lock unless we are at * the batch boundary. + * + * We don't need to worry about inode{32,64} since SB_KERNMOUNT + * shmem mounts are not exposed to userspace, so we don't need + * to worry about things like glibc compatibility. */ ino_t *next_ino; next_ino = per_cpu_ptr(sbinfo->ino_batch, get_cpu()); @@ -3397,6 +3408,8 @@ enum shmem_param { Opt_nr_inodes, Opt_size, Opt_uid, + Opt_inode32, + Opt_inode64, }; static const struct constant_table shmem_param_enums_huge[] = { @@ -3416,6 +3429,8 @@ const struct fs_parameter_spec shmem_fs_ fsparam_string("nr_inodes", Opt_nr_inodes), fsparam_string("size", Opt_size), fsparam_u32 ("uid", Opt_uid), + fsparam_flag ("inode32", Opt_inode32), + fsparam_flag ("inode64", Opt_inode64), {} }; @@ -3487,6 +3502,18 @@ static int shmem_parse_one(struct fs_con break; } goto unsupported_parameter; + case Opt_inode32: + ctx->full_inums = false; + ctx->seen |= SHMEM_SEEN_INUMS; + break; + case Opt_inode64: + if (sizeof(ino_t) < 8) { + return invalfc(fc, + "Cannot use inode64 with <64bit inums in kernel\n"); + } + ctx->full_inums = true; + ctx->seen |= SHMEM_SEEN_INUMS; + break; } return 0; @@ -3578,8 +3605,16 @@ static int shmem_reconfigure(struct fs_c } } + if ((ctx->seen & SHMEM_SEEN_INUMS) && !ctx->full_inums && + sbinfo->next_ino > UINT_MAX) { + err = "Current inum too high to switch to 32-bit inums"; + goto out; + } + if (ctx->seen & SHMEM_SEEN_HUGE) sbinfo->huge = ctx->huge; + if (ctx->seen & SHMEM_SEEN_INUMS) + sbinfo->full_inums = ctx->full_inums; if (ctx->seen & SHMEM_SEEN_BLOCKS) sbinfo->max_blocks = ctx->blocks; if (ctx->seen & SHMEM_SEEN_INODES) { @@ -3619,6 +3654,29 @@ static int shmem_show_options(struct seq if (!gid_eq(sbinfo->gid, GLOBAL_ROOT_GID)) seq_printf(seq, ",gid=%u", from_kgid_munged(&init_user_ns, sbinfo->gid)); + + /* + * Showing inode{64,32} might be useful even if it's the system default, + * since then people don't have to resort to checking both here and + * /proc/config.gz to confirm 64-bit inums were successfully applied + * (which may not even exist if IKCONFIG_PROC isn't enabled). + * + * We hide it when inode64 isn't the default and we are using 32-bit + * inodes, since that probably just means the feature isn't even under + * consideration. + * + * As such: + * + * +-----------------+-----------------+ + * | TMPFS_INODE64=y | TMPFS_INODE64=n | + * +------------------+-----------------+-----------------+ + * | full_inums=true | show | show | + * | full_inums=false | show | hide | + * +------------------+-----------------+-----------------+ + * + */ + if (IS_ENABLED(CONFIG_TMPFS_INODE64) || sbinfo->full_inums) + seq_printf(seq, ",inode%d", (sbinfo->full_inums ? 64 : 32)); #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* Rightly or wrongly, show huge mount option unmasked by shmem_huge */ if (sbinfo->huge) @@ -3667,6 +3725,8 @@ static int shmem_fill_super(struct super ctx->blocks = shmem_default_max_blocks(); if (!(ctx->seen & SHMEM_SEEN_INODES)) ctx->inodes = shmem_default_max_inodes(); + if (!(ctx->seen & SHMEM_SEEN_INUMS)) + ctx->full_inums = IS_ENABLED(CONFIG_TMPFS_INODE64); } else { sb->s_flags |= SB_NOUSER; } @@ -3684,6 +3744,7 @@ static int shmem_fill_super(struct super } sbinfo->uid = ctx->uid; sbinfo->gid = ctx->gid; + sbinfo->full_inums = ctx->full_inums; sbinfo->mode = ctx->mode; sbinfo->huge = ctx->huge; sbinfo->mpol = ctx->mpol; From patchwork Fri Aug 7 06:20:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704799 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 05DEA138C for ; Fri, 7 Aug 2020 06:20:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CA853221E5 for ; Fri, 7 Aug 2020 06:20:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="ICF0VlBE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA853221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4E5C78D0049; Fri, 7 Aug 2020 02:20:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4934B8D0026; Fri, 7 Aug 2020 02:20:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3ADF98D0049; Fri, 7 Aug 2020 02:20:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id 23E398D0026 for ; Fri, 7 Aug 2020 02:20:31 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E9348181AEF10 for ; Fri, 7 Aug 2020 06:20:30 +0000 (UTC) X-FDA: 77122773420.26.root59_3103b4d26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id B9D331804B667 for ; Fri, 7 Aug 2020 06:20:30 +0000 (UTC) X-Spam-Summary: 1,0,0,006c5fb0816ea1c8,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2525:2559:2563:2682:2685:2693:2741:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6261:6653:7576:7903:8599:9025:9545:9592:10004:10913:11026:11232:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13221:13227:13229:14096:14181:14721:14849:21080:21450:21451:21627:21795:21939:21966:21987:21990:30012:30051:30054:30064:30070:30080,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yfxnr4iduttdibc4xkwobsrno97opwn8e4awapm7rs7ay8ak951zrpqwn5y56.eoza37cwkmyxk3qweknur6qryyn8cirtozof4q963n388hy484a8jqo8y46rzwz.4-lbl8.mailshell.net-223.238.255.100,Cache IP:none, X-HE-Tag: root59_3103b4d26fbe X-Filterd-Recvd-Size: 3819 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:30 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 34DEC221E5; Fri, 7 Aug 2020 06:20:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781229; bh=/+GjuyI/einNEPcWggkqcOYCxaQ8Rpt+T+rqa38Zqwo=; h=Date:From:To:Subject:In-Reply-To:From; b=ICF0VlBESrUh5vw8C0dxRLkpAmZRmhBBGP/WSdcbcNNMaSO5z7NK4DPnq8I0aB9SL pD8d8LETxQdWF9bIsy0FxzXXYLDlsGL2HjKhNDFK9ihKMxavLHsed88cprvDATEyIu unc2z0X1jtSCv8TQfT0Snj42+OZ1SoFgftHh3P2U= Date: Thu, 06 Aug 2020 23:20:28 -0700 From: Andrew Morton To: akpm@linux-foundation.org, guro@fb.com, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, naresh.kamboju@linaro.org, shakeelb@google.com, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 061/163] mm: kmem: make memcg_kmem_enabled() irreversible Message-ID: <20200807062028.9dTsF8Lvx%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: B9D331804B667 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000158, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: kmem: make memcg_kmem_enabled() irreversible Historically the kernel memory accounting was an opt-in feature, which could be enabled for individual cgroups. But now it's not true, and it's on by default both on cgroup v1 and cgroup v2. And as long as a user has at least one non-root memory cgroup, the kernel memory accounting is on. So in most setups it's either always on (if memory cgroups are in use and kmem accounting is not disabled), either always off (otherwise). memcg_kmem_enabled() is used in many places to guard the kernel memory accounting code. If memcg_kmem_enabled() can reverse from returning true to returning false (as now), we can't rely on it on release paths and have to check if it was on before. If we'll make memcg_kmem_enabled() irreversible (always returning true after returning it for the first time), it'll make the general logic more simple and robust. It also will allow to guard some checks which otherwise would stay unguarded. Link: http://lkml.kernel.org/r/20200702180926.1330769-1-guro@fb.com Signed-off-by: Roman Gushchin Reported-by: Naresh Kamboju Tested-by: Naresh Kamboju Reviewed-by: Shakeel Butt Acked-by: Vlastimil Babka Acked-by: Michal Hocko Signed-off-by: Andrew Morton --- mm/memcontrol.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) --- a/mm/memcontrol.c~mm-kmem-make-memcg_kmem_enabled-irreversible +++ a/mm/memcontrol.c @@ -3416,7 +3416,8 @@ static int memcg_online_kmem(struct mem_ if (memcg_id < 0) return memcg_id; - static_branch_inc(&memcg_kmem_enabled_key); + static_branch_enable(&memcg_kmem_enabled_key); + /* * A memory cgroup is considered kmem-online as soon as it gets * kmemcg_id. Setting the id after enabling static branching will @@ -3486,11 +3487,6 @@ static void memcg_free_kmem(struct mem_c /* css_alloc() failed, offlining didn't happen */ if (unlikely(memcg->kmem_state == KMEM_ONLINE)) memcg_offline_kmem(memcg); - - if (memcg->kmem_state == KMEM_ALLOCATED) { - WARN_ON(!list_empty(&memcg->kmem_caches)); - static_branch_dec(&memcg_kmem_enabled_key); - } } #else static int memcg_online_kmem(struct mem_cgroup *memcg) From patchwork Fri Aug 7 06:20:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704801 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E396C138C for ; Fri, 7 Aug 2020 06:20:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A03A322C9F for ; Fri, 7 Aug 2020 06:20:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="aAbSVwcR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A03A322C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1AE028D004A; Fri, 7 Aug 2020 02:20:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 15F828D0026; Fri, 7 Aug 2020 02:20:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 026C48D004A; Fri, 7 Aug 2020 02:20:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0024.hostedemail.com [216.40.44.24]) by kanga.kvack.org (Postfix) with ESMTP id DFD558D0026 for ; Fri, 7 Aug 2020 02:20:34 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A3D453637 for ; Fri, 7 Aug 2020 06:20:34 +0000 (UTC) X-FDA: 77122773588.25.grip16_620cfc126fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 794721804E3A8 for ; Fri, 7 Aug 2020 06:20:34 +0000 (UTC) X-Spam-Summary: 1,0,0,f6d3a741b4220273,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:2:41:69:355:379:421:467:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:2198:2199:2393:2525:2559:2565:2682:2685:2731:2859:2895:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4051:4250:4321:4423:4605:5007:6261:6630:6653:6737:7576:7875:7903:7974:8599:8603:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12050:12295:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13153:13161:13208:13228:13229:13255:14096:21067:21080:21222:21325:21433:21450:21451:21627:21740:21939:21966:21990:22013:30012:30034:30045:30054:30064:30066:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygsqwxsdao1mmczbk5i8gz1f64topmt44owb1qy7ttnwgz8u6r3nye54zxrbk.puub5zz65t5n4hpuiqchnpjex7h1dc5fqmoyjj hb4gkjh3 X-HE-Tag: grip16_620cfc126fbe X-Filterd-Recvd-Size: 10220 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:33 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A71D02177B; Fri, 7 Aug 2020 06:20:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781233; bh=/MQJhpgxRti/bb9C8yvPEHdXg3jHNWOqyZUdWhzLx+U=; h=Date:From:To:Subject:In-Reply-To:From; b=aAbSVwcRRZ4OSEAzGSIXsqL4SegCinZqmfo2RCAV8AyrR2j/Wzp1uQ5ZifeDLsDBa 4q/gMflTIBxc6d94KAkQ65Qz98zu51cbOY4VkwVzCAj2DuYcjo3aRmoZIM8PBbauU+ He9C1u2nZnEUIDiS4SGcHuya/t/xNXWH58Ww/0hc= Date: Thu, 06 Aug 2020 23:20:32 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 062/163] mm: memcg: factor out memcg- and lruvec-level changes out of __mod_lruvec_state() Message-ID: <20200807062032.W074XjPoi%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 794721804E3A8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg: factor out memcg- and lruvec-level changes out of __mod_lruvec_state() Patch series "The new cgroup slab memory controller", v7. The patchset moves the accounting from the page level to the object level. It allows to share slab pages between memory cgroups. This leads to a significant win in the slab utilization (up to 45%) and the corresponding drop in the total kernel memory footprint. The reduced number of unmovable slab pages should also have a positive effect on the memory fragmentation. The patchset makes the slab accounting code simpler: there is no more need in the complicated dynamic creation and destruction of per-cgroup slab caches, all memory cgroups use a global set of shared slab caches. The lifetime of slab caches is not more connected to the lifetime of memory cgroups. The more precise accounting does require more CPU, however in practice the difference seems to be negligible. We've been using the new slab controller in Facebook production for several months with different workloads and haven't seen any noticeable regressions. What we've seen were memory savings in order of 1 GB per host (it varied heavily depending on the actual workload, size of RAM, number of CPUs, memory pressure, etc). The third version of the patchset added yet another step towards the simplification of the code: sharing of slab caches between accounted and non-accounted allocations. It comes with significant upsides (most noticeable, a complete elimination of dynamic slab caches creation) but not without some regression risks, so this change sits on top of the patchset and is not completely merged in. So in the unlikely event of a noticeable performance regression it can be reverted separately. The slab memory accounting works in exactly the same way for SLAB and SLUB. With both allocators the new controller shows significant memory savings, with SLUB the difference is bigger. On my 16-core desktop machine running Fedora 32 the size of the slab memory measured after the start of the system was lower by 58% and 38% with SLUB and SLAB correspondingly. As an estimation of a potential CPU overhead, below are results of slab_bulk_test01 test, kindly provided by Jesper D. Brouer. He also helped with the evaluation of results. The test can be found here: https://github.com/netoptimizer/prototype-kernel/ The smallest number in each row should be picked for a comparison. SLUB-patched - bulk-API - SLUB-patched : bulk_quick_reuse objects=1 : 187 - 90 - 224 cycles(tsc) - SLUB-patched : bulk_quick_reuse objects=2 : 110 - 53 - 133 cycles(tsc) - SLUB-patched : bulk_quick_reuse objects=3 : 88 - 95 - 42 cycles(tsc) - SLUB-patched : bulk_quick_reuse objects=4 : 91 - 85 - 36 cycles(tsc) - SLUB-patched : bulk_quick_reuse objects=8 : 32 - 66 - 32 cycles(tsc) SLUB-original - bulk-API - SLUB-original: bulk_quick_reuse objects=1 : 87 - 87 - 142 cycles(tsc) - SLUB-original: bulk_quick_reuse objects=2 : 52 - 53 - 53 cycles(tsc) - SLUB-original: bulk_quick_reuse objects=3 : 42 - 42 - 91 cycles(tsc) - SLUB-original: bulk_quick_reuse objects=4 : 91 - 37 - 37 cycles(tsc) - SLUB-original: bulk_quick_reuse objects=8 : 31 - 79 - 76 cycles(tsc) SLAB-patched - bulk-API - SLAB-patched : bulk_quick_reuse objects=1 : 67 - 67 - 140 cycles(tsc) - SLAB-patched : bulk_quick_reuse objects=2 : 55 - 46 - 46 cycles(tsc) - SLAB-patched : bulk_quick_reuse objects=3 : 93 - 94 - 39 cycles(tsc) - SLAB-patched : bulk_quick_reuse objects=4 : 35 - 88 - 85 cycles(tsc) - SLAB-patched : bulk_quick_reuse objects=8 : 30 - 30 - 30 cycles(tsc) SLAB-original- bulk-API - SLAB-original: bulk_quick_reuse objects=1 : 143 - 136 - 67 cycles(tsc) - SLAB-original: bulk_quick_reuse objects=2 : 45 - 46 - 46 cycles(tsc) - SLAB-original: bulk_quick_reuse objects=3 : 38 - 39 - 39 cycles(tsc) - SLAB-original: bulk_quick_reuse objects=4 : 35 - 87 - 87 cycles(tsc) - SLAB-original: bulk_quick_reuse objects=8 : 29 - 66 - 30 cycles(tsc) This patch (of 19): To convert memcg and lruvec slab counters to bytes there must be a way to change these counters without touching node counters. Factor out __mod_memcg_lruvec_state() out of __mod_lruvec_state(). Link: http://lkml.kernel.org/r/20200623174037.3951353-1-guro@fb.com Link: http://lkml.kernel.org/r/20200623174037.3951353-2-guro@fb.com Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 17 +++++++++++++ mm/memcontrol.c | 43 +++++++++++++++++++---------------- 2 files changed, 41 insertions(+), 19 deletions(-) --- a/include/linux/memcontrol.h~mm-memcg-factor-out-memcg-and-lruvec-level-changes-out-of-__mod_lruvec_state +++ a/include/linux/memcontrol.h @@ -679,11 +679,23 @@ static inline unsigned long lruvec_page_ return x; } +void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, + int val); void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); void __mod_lruvec_slab_state(void *p, enum node_stat_item idx, int val); void mod_memcg_obj_state(void *p, int idx, int val); +static inline void mod_memcg_lruvec_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) +{ + unsigned long flags; + + local_irq_save(flags); + __mod_memcg_lruvec_state(lruvec, idx, val); + local_irq_restore(flags); +} + static inline void mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val) { @@ -1057,6 +1069,11 @@ static inline unsigned long lruvec_page_ return node_page_state(lruvec_pgdat(lruvec), idx); } +static inline void __mod_memcg_lruvec_state(struct lruvec *lruvec, + enum node_stat_item idx, int val) +{ +} + static inline void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val) { --- a/mm/memcontrol.c~mm-memcg-factor-out-memcg-and-lruvec-level-changes-out-of-__mod_lruvec_state +++ a/mm/memcontrol.c @@ -713,30 +713,13 @@ parent_nodeinfo(struct mem_cgroup_per_no return mem_cgroup_nodeinfo(parent, nid); } -/** - * __mod_lruvec_state - update lruvec memory statistics - * @lruvec: the lruvec - * @idx: the stat item - * @val: delta to add to the counter, can be negative - * - * The lruvec is the intersection of the NUMA node and a cgroup. This - * function updates the all three counters that are affected by a - * change of state at this level: per-node, per-cgroup, per-lruvec. - */ -void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, - int val) +void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, + int val) { - pg_data_t *pgdat = lruvec_pgdat(lruvec); struct mem_cgroup_per_node *pn; struct mem_cgroup *memcg; long x; - /* Update node */ - __mod_node_page_state(pgdat, idx, val); - - if (mem_cgroup_disabled()) - return; - pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); memcg = pn->memcg; @@ -748,6 +731,7 @@ void __mod_lruvec_state(struct lruvec *l x = val + __this_cpu_read(pn->lruvec_stat_cpu->count[idx]); if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + pg_data_t *pgdat = lruvec_pgdat(lruvec); struct mem_cgroup_per_node *pi; for (pi = pn; pi; pi = parent_nodeinfo(pi, pgdat->node_id)) @@ -757,6 +741,27 @@ void __mod_lruvec_state(struct lruvec *l __this_cpu_write(pn->lruvec_stat_cpu->count[idx], x); } +/** + * __mod_lruvec_state - update lruvec memory statistics + * @lruvec: the lruvec + * @idx: the stat item + * @val: delta to add to the counter, can be negative + * + * The lruvec is the intersection of the NUMA node and a cgroup. This + * function updates the all three counters that are affected by a + * change of state at this level: per-node, per-cgroup, per-lruvec. + */ +void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, + int val) +{ + /* Update node */ + __mod_node_page_state(lruvec_pgdat(lruvec), idx, val); + + /* Update memcg and lruvec */ + if (!mem_cgroup_disabled()) + __mod_memcg_lruvec_state(lruvec, idx, val); +} + void __mod_lruvec_slab_state(void *p, enum node_stat_item idx, int val) { pg_data_t *pgdat = page_pgdat(virt_to_page(p)); From patchwork Fri Aug 7 06:20:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704803 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7742714B7 for ; Fri, 7 Aug 2020 06:20:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 396E42177B for ; Fri, 7 Aug 2020 06:20:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="erEuyib1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 396E42177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A60DB8D004B; Fri, 7 Aug 2020 02:20:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A116B8D0026; Fri, 7 Aug 2020 02:20:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9278C8D004B; Fri, 7 Aug 2020 02:20:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0139.hostedemail.com [216.40.44.139]) by kanga.kvack.org (Postfix) with ESMTP id 7CE808D0026 for ; Fri, 7 Aug 2020 02:20:38 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 43BC31E1B for ; Fri, 7 Aug 2020 06:20:38 +0000 (UTC) X-FDA: 77122773756.24.boat12_4d06b7926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 1E4191A4A5 for ; Fri, 7 Aug 2020 06:20:38 +0000 (UTC) X-Spam-Summary: 1,0,0,1e08fee699412d6a,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:2:41:355:379:560:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2693:2859:2895:2901:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3369:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4051:4250:4321:4362:4605:5007:6119:6261:6653:6737:7576:7903:7904:8599:9025:9036:9545:10004:10913:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13229:13869:14877:21080:21433:21450:21451:21611:21627:21939:21966:21990:30005:30012:30034:30054:30056:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yf478m6t7ratdspsf3wc5dxi9z1ycpwk5bfwx4g74f4qqiosderyy9osqha8i.9dbncauai8pratqhnq3dwdqihcumtit74kafc355e6wi9zp6eettmob1i8sytqu.1-lbl8.mailshell.net-223.238.255.1 00,Cache X-HE-Tag: boat12_4d06b7926fbe X-Filterd-Recvd-Size: 10030 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:37 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3162C22CAF; Fri, 7 Aug 2020 06:20:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781236; bh=O75JKJbWs2woGqPh0xCflNLWM40HPNgNPbf+KdfLXno=; h=Date:From:To:Subject:In-Reply-To:From; b=erEuyib1S8ceatUpecdI01ShsO1PynZAjOpqynrBUmFiSyy0jm8WV5QjJetNg8HEh S1BfAfkpIMNxNbYLD94clDO8asB+H3ZIpJ0puUoK/eyYvG6unI07N/A8W5Wwewme8g 5+gn2hOjQYyaheYOlAT8JD+IKxCBaDyVrXCixRwE= Date: Thu, 06 Aug 2020 23:20:35 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 063/163] mm: memcg: prepare for byte-sized vmstat items Message-ID: <20200807062035.okJt5Kevx%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 1E4191A4A5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg: prepare for byte-sized vmstat items To implement per-object slab memory accounting, we need to convert slab vmstat counters to bytes. Actually, out of 4 levels of counters: global, per-node, per-memcg and per-lruvec only two last levels will require byte-sized counters. It's because global and per-node counters will be counting the number of slab pages, and per-memcg and per-lruvec will be counting the amount of memory taken by charged slab objects. Converting all vmstat counters to bytes or even all slab counters to bytes would introduce an additional overhead. So instead let's store global and per-node counters in pages, and memcg and lruvec counters in bytes. To make the API clean all access helpers (both on the read and write sides) are dealing with bytes. To avoid back-and-forth conversions a new flavor of read-side helpers is introduced, which always returns values in pages: node_page_state_pages() and global_node_page_state_pages(). Actually new helpers are just reading raw values. Old helpers are simple wrappers, which will complain on an attempt to read byte value, because at the moment no one actually needs bytes. Thanks to Johannes Weiner for the idea of having the byte-sized API on top of the page-sized internal storage. Link: http://lkml.kernel.org/r/20200623174037.3951353-3-guro@fb.com Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- drivers/base/node.c | 2 +- include/linux/mmzone.h | 10 ++++++++++ include/linux/vmstat.h | 14 +++++++++++++- mm/memcontrol.c | 14 ++++++++++---- mm/vmstat.c | 30 ++++++++++++++++++++++++++---- 5 files changed, 60 insertions(+), 10 deletions(-) --- a/drivers/base/node.c~mm-memcg-prepare-for-byte-sized-vmstat-items +++ a/drivers/base/node.c @@ -513,7 +513,7 @@ static ssize_t node_read_vmstat(struct d for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) n += sprintf(buf+n, "%s %lu\n", node_stat_name(i), - node_page_state(pgdat, i)); + node_page_state_pages(pgdat, i)); return n; } --- a/include/linux/mmzone.h~mm-memcg-prepare-for-byte-sized-vmstat-items +++ a/include/linux/mmzone.h @@ -207,6 +207,16 @@ enum node_stat_item { }; /* + * Returns true if the value is measured in bytes (most vmstat values are + * measured in pages). This defines the API part, the internal representation + * might be different. + */ +static __always_inline bool vmstat_item_in_bytes(int idx) +{ + return false; +} + +/* * We do arithmetic on the LRU lists in various places in the code, * so it is important to keep the active lists LRU_ACTIVE higher in * the array than the corresponding inactive lists, and to keep --- a/include/linux/vmstat.h~mm-memcg-prepare-for-byte-sized-vmstat-items +++ a/include/linux/vmstat.h @@ -8,6 +8,7 @@ #include #include #include +#include extern int sysctl_stat_interval; @@ -192,7 +193,8 @@ static inline unsigned long global_zone_ return x; } -static inline unsigned long global_node_page_state(enum node_stat_item item) +static inline +unsigned long global_node_page_state_pages(enum node_stat_item item) { long x = atomic_long_read(&vm_node_stat[item]); #ifdef CONFIG_SMP @@ -202,6 +204,13 @@ static inline unsigned long global_node_ return x; } +static inline unsigned long global_node_page_state(enum node_stat_item item) +{ + VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); + + return global_node_page_state_pages(item); +} + static inline unsigned long zone_page_state(struct zone *zone, enum zone_stat_item item) { @@ -242,9 +251,12 @@ extern unsigned long sum_zone_node_page_ extern unsigned long sum_zone_numa_state(int node, enum numa_stat_item item); extern unsigned long node_page_state(struct pglist_data *pgdat, enum node_stat_item item); +extern unsigned long node_page_state_pages(struct pglist_data *pgdat, + enum node_stat_item item); #else #define sum_zone_node_page_state(node, item) global_zone_page_state(item) #define node_page_state(node, item) global_node_page_state(item) +#define node_page_state_pages(node, item) global_node_page_state_pages(item) #endif /* CONFIG_NUMA */ #ifdef CONFIG_SMP --- a/mm/memcontrol.c~mm-memcg-prepare-for-byte-sized-vmstat-items +++ a/mm/memcontrol.c @@ -681,13 +681,16 @@ mem_cgroup_largest_soft_limit_node(struc */ void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) { - long x; + long x, threshold = MEMCG_CHARGE_BATCH; if (mem_cgroup_disabled()) return; + if (vmstat_item_in_bytes(idx)) + threshold <<= PAGE_SHIFT; + x = val + __this_cpu_read(memcg->vmstats_percpu->stat[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + if (unlikely(abs(x) > threshold)) { struct mem_cgroup *mi; /* @@ -718,7 +721,7 @@ void __mod_memcg_lruvec_state(struct lru { struct mem_cgroup_per_node *pn; struct mem_cgroup *memcg; - long x; + long x, threshold = MEMCG_CHARGE_BATCH; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); memcg = pn->memcg; @@ -729,8 +732,11 @@ void __mod_memcg_lruvec_state(struct lru /* Update lruvec */ __this_cpu_add(pn->lruvec_stat_local->count[idx], val); + if (vmstat_item_in_bytes(idx)) + threshold <<= PAGE_SHIFT; + x = val + __this_cpu_read(pn->lruvec_stat_cpu->count[idx]); - if (unlikely(abs(x) > MEMCG_CHARGE_BATCH)) { + if (unlikely(abs(x) > threshold)) { pg_data_t *pgdat = lruvec_pgdat(lruvec); struct mem_cgroup_per_node *pi; --- a/mm/vmstat.c~mm-memcg-prepare-for-byte-sized-vmstat-items +++ a/mm/vmstat.c @@ -341,6 +341,11 @@ void __mod_node_page_state(struct pglist long x; long t; + if (vmstat_item_in_bytes(item)) { + VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); + delta >>= PAGE_SHIFT; + } + x = delta + __this_cpu_read(*p); t = __this_cpu_read(pcp->stat_threshold); @@ -398,6 +403,8 @@ void __inc_node_state(struct pglist_data s8 __percpu *p = pcp->vm_node_stat_diff + item; s8 v, t; + VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); + v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { @@ -442,6 +449,8 @@ void __dec_node_state(struct pglist_data s8 __percpu *p = pcp->vm_node_stat_diff + item; s8 v, t; + VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); + v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { @@ -541,6 +550,11 @@ static inline void mod_node_state(struct s8 __percpu *p = pcp->vm_node_stat_diff + item; long o, n, t, z; + if (vmstat_item_in_bytes(item)) { + VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); + delta >>= PAGE_SHIFT; + } + do { z = 0; /* overflow to node counters */ @@ -989,8 +1003,8 @@ unsigned long sum_zone_numa_state(int no /* * Determine the per node value of a stat item. */ -unsigned long node_page_state(struct pglist_data *pgdat, - enum node_stat_item item) +unsigned long node_page_state_pages(struct pglist_data *pgdat, + enum node_stat_item item) { long x = atomic_long_read(&pgdat->vm_stat[item]); #ifdef CONFIG_SMP @@ -999,6 +1013,14 @@ unsigned long node_page_state(struct pgl #endif return x; } + +unsigned long node_page_state(struct pglist_data *pgdat, + enum node_stat_item item) +{ + VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); + + return node_page_state_pages(pgdat, item); +} #endif #ifdef CONFIG_COMPACTION @@ -1577,7 +1599,7 @@ static void zoneinfo_show_print(struct s seq_printf(m, "\n per-node stats"); for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) { seq_printf(m, "\n %-12s %lu", node_stat_name(i), - node_page_state(pgdat, i)); + node_page_state_pages(pgdat, i)); } } seq_printf(m, @@ -1698,7 +1720,7 @@ static void *vmstat_start(struct seq_fil #endif for (i = 0; i < NR_VM_NODE_STAT_ITEMS; i++) - v[i] = global_node_page_state(i); + v[i] = global_node_page_state_pages(i); v += NR_VM_NODE_STAT_ITEMS; global_dirty_limits(v + NR_DIRTY_BG_THRESHOLD, From patchwork Fri Aug 7 06:20:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704805 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 45783138C for ; Fri, 7 Aug 2020 06:20:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 00BE6221E5 for ; Fri, 7 Aug 2020 06:20:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Iv7M7jIh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 00BE6221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3842D8D004C; Fri, 7 Aug 2020 02:20:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 35B128D0026; Fri, 7 Aug 2020 02:20:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2717D8D004C; Fri, 7 Aug 2020 02:20:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0250.hostedemail.com [216.40.44.250]) by kanga.kvack.org (Postfix) with ESMTP id 0CABC8D0026 for ; Fri, 7 Aug 2020 02:20:42 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C7EBE181AEF10 for ; Fri, 7 Aug 2020 06:20:41 +0000 (UTC) X-FDA: 77122773882.07.song13_3c1145226fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 9F39A1803F9AA for ; Fri, 7 Aug 2020 06:20:41 +0000 (UTC) X-Spam-Summary: 1,0,0,c889880c29468ef5,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2553:2559:2563:2639:2682:2685:2690:2693:2731:2859:2890:2902:2914:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4041:4042:4321:4385:4605:5007:6119:6261:6653:6737:7576:7903:8599:8603:8660:8957:9025:9036:9121:9545:10004:10913:11026:11232:11233:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13148:13161:13229:13230:21080:21325:21433:21450:21451:21611:21627:21740:21939:21990:30001:30034:30054:30055:30064:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfsauc9ay3uu1jmxz7it7xaqsfqyc3kmcn3nmc9ri7fjnowkcyeq5m4g5fxe9.ac4bfwnb6cjbq5dwbohxueqe8e3dpw5dy5fe9wf4fz5g6u7ay144d3 43bjqo4m X-HE-Tag: song13_3c1145226fbe X-Filterd-Recvd-Size: 14817 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:40 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B3ECF221E5; Fri, 7 Aug 2020 06:20:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781240; bh=Wr6Ao82alTcvrS8s+OMnF9azCleNhu8TDRkyZlYwk/c=; h=Date:From:To:Subject:In-Reply-To:From; b=Iv7M7jIhfUL3QIDmdiIySY5KcsLEzxJM/a8dtjf2651C/nVMpje0+w/7Vcu9UOmpT n/SStelds3r7giNvtBR/pcDEg48Uj7fGJl3iowhWspfERQ/oIChhAg8t+0OTAwrF/u ZW6j1icvBevTo8Z/xTuTAnHynUcxvVmB7AnCqPDE= Date: Thu, 06 Aug 2020 23:20:39 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 064/163] mm: memcg: convert vmstat slab counters to bytes Message-ID: <20200807062039.9Bzmjuqpw%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 9F39A1803F9AA X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg: convert vmstat slab counters to bytes In order to prepare for per-object slab memory accounting, convert NR_SLAB_RECLAIMABLE and NR_SLAB_UNRECLAIMABLE vmstat items to bytes. To make it obvious, rename them to NR_SLAB_RECLAIMABLE_B and NR_SLAB_UNRECLAIMABLE_B (similar to NR_KERNEL_STACK_KB). Internally global and per-node counters are stored in pages, however memcg and lruvec counters are stored in bytes. This scheme may look weird, but only for now. As soon as slab pages will be shared between multiple cgroups, global and node counters will reflect the total number of slab pages. However memcg and lruvec counters will be used for per-memcg slab memory tracking, which will take separate kernel objects in the account. Keeping global and node counters in pages helps to avoid additional overhead. The size of slab memory shouldn't exceed 4Gb on 32-bit machines, so it will fit into atomic_long_t we use for vmstats. Link: http://lkml.kernel.org/r/20200623174037.3951353-4-guro@fb.com Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner Acked-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- drivers/base/node.c | 4 ++-- fs/proc/meminfo.c | 4 ++-- include/linux/mmzone.h | 16 +++++++++++++--- kernel/power/snapshot.c | 2 +- mm/memcontrol.c | 11 ++++------- mm/oom_kill.c | 2 +- mm/page_alloc.c | 8 ++++---- mm/slab.h | 15 ++++++++------- mm/slab_common.c | 4 ++-- mm/slob.c | 12 ++++++------ mm/slub.c | 8 ++++---- mm/vmscan.c | 3 ++- mm/workingset.c | 6 ++++-- 13 files changed, 53 insertions(+), 42 deletions(-) --- a/drivers/base/node.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/drivers/base/node.c @@ -368,8 +368,8 @@ static ssize_t node_read_meminfo(struct unsigned long sreclaimable, sunreclaimable; si_meminfo_node(&i, nid); - sreclaimable = node_page_state(pgdat, NR_SLAB_RECLAIMABLE); - sunreclaimable = node_page_state(pgdat, NR_SLAB_UNRECLAIMABLE); + sreclaimable = node_page_state_pages(pgdat, NR_SLAB_RECLAIMABLE_B); + sunreclaimable = node_page_state_pages(pgdat, NR_SLAB_UNRECLAIMABLE_B); n = sprintf(buf, "Node %d MemTotal: %8lu kB\n" "Node %d MemFree: %8lu kB\n" --- a/fs/proc/meminfo.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/fs/proc/meminfo.c @@ -52,8 +52,8 @@ static int meminfo_proc_show(struct seq_ pages[lru] = global_node_page_state(NR_LRU_BASE + lru); available = si_mem_available(); - sreclaimable = global_node_page_state(NR_SLAB_RECLAIMABLE); - sunreclaim = global_node_page_state(NR_SLAB_UNRECLAIMABLE); + sreclaimable = global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B); + sunreclaim = global_node_page_state_pages(NR_SLAB_UNRECLAIMABLE_B); show_val_kb(m, "MemTotal: ", i.totalram); show_val_kb(m, "MemFree: ", i.freeram); --- a/include/linux/mmzone.h~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/include/linux/mmzone.h @@ -174,8 +174,8 @@ enum node_stat_item { NR_INACTIVE_FILE, /* " " " " " */ NR_ACTIVE_FILE, /* " " " " " */ NR_UNEVICTABLE, /* " " " " " */ - NR_SLAB_RECLAIMABLE, - NR_SLAB_UNRECLAIMABLE, + NR_SLAB_RECLAIMABLE_B, + NR_SLAB_UNRECLAIMABLE_B, NR_ISOLATED_ANON, /* Temporary isolated pages from anon lru */ NR_ISOLATED_FILE, /* Temporary isolated pages from file lru */ WORKINGSET_NODES, @@ -213,7 +213,17 @@ enum node_stat_item { */ static __always_inline bool vmstat_item_in_bytes(int idx) { - return false; + /* + * Global and per-node slab counters track slab pages. + * It's expected that changes are multiples of PAGE_SIZE. + * Internally values are stored in pages. + * + * Per-memcg and per-lruvec counters track memory, consumed + * by individual slab objects. These counters are actually + * byte-precise. + */ + return (idx == NR_SLAB_RECLAIMABLE_B || + idx == NR_SLAB_UNRECLAIMABLE_B); } /* --- a/kernel/power/snapshot.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/kernel/power/snapshot.c @@ -1663,7 +1663,7 @@ static unsigned long minimum_image_size( { unsigned long size; - size = global_node_page_state(NR_SLAB_RECLAIMABLE) + size = global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B) + global_node_page_state(NR_ACTIVE_ANON) + global_node_page_state(NR_INACTIVE_ANON) + global_node_page_state(NR_ACTIVE_FILE) --- a/mm/memcontrol.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/mm/memcontrol.c @@ -1391,9 +1391,8 @@ static char *memory_stat_format(struct m (u64)memcg_page_state(memcg, MEMCG_KERNEL_STACK_KB) * 1024); seq_buf_printf(&s, "slab %llu\n", - (u64)(memcg_page_state(memcg, NR_SLAB_RECLAIMABLE) + - memcg_page_state(memcg, NR_SLAB_UNRECLAIMABLE)) * - PAGE_SIZE); + (u64)(memcg_page_state(memcg, NR_SLAB_RECLAIMABLE_B) + + memcg_page_state(memcg, NR_SLAB_UNRECLAIMABLE_B))); seq_buf_printf(&s, "sock %llu\n", (u64)memcg_page_state(memcg, MEMCG_SOCK) * PAGE_SIZE); @@ -1423,11 +1422,9 @@ static char *memory_stat_format(struct m PAGE_SIZE); seq_buf_printf(&s, "slab_reclaimable %llu\n", - (u64)memcg_page_state(memcg, NR_SLAB_RECLAIMABLE) * - PAGE_SIZE); + (u64)memcg_page_state(memcg, NR_SLAB_RECLAIMABLE_B)); seq_buf_printf(&s, "slab_unreclaimable %llu\n", - (u64)memcg_page_state(memcg, NR_SLAB_UNRECLAIMABLE) * - PAGE_SIZE); + (u64)memcg_page_state(memcg, NR_SLAB_UNRECLAIMABLE_B)); /* Accumulated memory events */ --- a/mm/oom_kill.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/mm/oom_kill.c @@ -184,7 +184,7 @@ static bool is_dump_unreclaim_slabs(void global_node_page_state(NR_ISOLATED_FILE) + global_node_page_state(NR_UNEVICTABLE); - return (global_node_page_state(NR_SLAB_UNRECLAIMABLE) > nr_lru); + return (global_node_page_state_pages(NR_SLAB_UNRECLAIMABLE_B) > nr_lru); } /** --- a/mm/page_alloc.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/mm/page_alloc.c @@ -5220,8 +5220,8 @@ long si_mem_available(void) * items that are in use, and cannot be freed. Cap this estimate at the * low watermark. */ - reclaimable = global_node_page_state(NR_SLAB_RECLAIMABLE) + - global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE); + reclaimable = global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B) + + global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE); available += reclaimable - min(reclaimable / 2, wmark_low); if (available < 0) @@ -5364,8 +5364,8 @@ void show_free_areas(unsigned int filter global_node_page_state(NR_UNEVICTABLE), global_node_page_state(NR_FILE_DIRTY), global_node_page_state(NR_WRITEBACK), - global_node_page_state(NR_SLAB_RECLAIMABLE), - global_node_page_state(NR_SLAB_UNRECLAIMABLE), + global_node_page_state_pages(NR_SLAB_RECLAIMABLE_B), + global_node_page_state_pages(NR_SLAB_UNRECLAIMABLE_B), global_node_page_state(NR_FILE_MAPPED), global_node_page_state(NR_SHMEM), global_zone_page_state(NR_PAGETABLE), --- a/mm/slab_common.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/mm/slab_common.c @@ -1363,8 +1363,8 @@ void *kmalloc_order(size_t size, gfp_t f page = alloc_pages(flags, order); if (likely(page)) { ret = page_address(page); - mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, - 1 << order); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE_B, + PAGE_SIZE << order); } ret = kasan_kmalloc_large(ret, size, flags); /* As ret might get tagged, call kmemleak hook after KASAN. */ --- a/mm/slab.h~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/mm/slab.h @@ -273,7 +273,7 @@ int __kmem_cache_alloc_bulk(struct kmem_ static inline int cache_vmstat_idx(struct kmem_cache *s) { return (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE; + NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; } #ifdef CONFIG_SLUB_DEBUG @@ -390,7 +390,7 @@ static __always_inline int memcg_charge_ if (unlikely(!memcg || mem_cgroup_is_root(memcg))) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), - nr_pages); + nr_pages << PAGE_SHIFT); percpu_ref_get_many(&s->memcg_params.refcnt, nr_pages); return 0; } @@ -400,7 +400,7 @@ static __always_inline int memcg_charge_ goto out; lruvec = mem_cgroup_lruvec(memcg, page_pgdat(page)); - mod_lruvec_state(lruvec, cache_vmstat_idx(s), nr_pages); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), nr_pages << PAGE_SHIFT); /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, nr_pages); @@ -425,11 +425,12 @@ static __always_inline void memcg_unchar memcg = READ_ONCE(s->memcg_params.memcg); if (likely(!mem_cgroup_is_root(memcg))) { lruvec = mem_cgroup_lruvec(memcg, page_pgdat(page)); - mod_lruvec_state(lruvec, cache_vmstat_idx(s), -nr_pages); + mod_lruvec_state(lruvec, cache_vmstat_idx(s), + -(nr_pages << PAGE_SHIFT)); memcg_kmem_uncharge(memcg, nr_pages); } else { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), - -nr_pages); + -(nr_pages << PAGE_SHIFT)); } rcu_read_unlock(); @@ -513,7 +514,7 @@ static __always_inline int charge_slab_p { if (is_root_cache(s)) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), - 1 << order); + PAGE_SIZE << order); return 0; } @@ -525,7 +526,7 @@ static __always_inline void uncharge_sla { if (is_root_cache(s)) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), - -(1 << order)); + -(PAGE_SIZE << order)); return; } --- a/mm/slob.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/mm/slob.c @@ -202,8 +202,8 @@ static void *slob_new_pages(gfp_t gfp, i if (!page) return NULL; - mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, - 1 << order); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE_B, + PAGE_SIZE << order); return page_address(page); } @@ -214,8 +214,8 @@ static void slob_free_pages(void *b, int if (current->reclaim_state) current->reclaim_state->reclaimed_slab += 1 << order; - mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE, - -(1 << order)); + mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); __free_pages(sp, order); } @@ -552,8 +552,8 @@ void kfree(const void *block) slob_free(m, *m + align); } else { unsigned int order = compound_order(sp); - mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE, - -(1 << order)); + mod_node_page_state(page_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); __free_pages(sp, order); } --- a/mm/slub.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/mm/slub.c @@ -3991,8 +3991,8 @@ static void *kmalloc_large_node(size_t s page = alloc_pages_node(node, flags, order); if (page) { ptr = page_address(page); - mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, - 1 << order); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE_B, + PAGE_SIZE << order); } return kmalloc_large_node_hook(ptr, size, flags); @@ -4123,8 +4123,8 @@ void kfree(const void *x) BUG_ON(!PageCompound(page)); kfree_hook(object); - mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE, - -(1 << order)); + mod_node_page_state(page_pgdat(page), NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); __free_pages(page, order); return; } --- a/mm/vmscan.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/mm/vmscan.c @@ -4222,7 +4222,8 @@ int node_reclaim(struct pglist_data *pgd * unmapped file backed pages. */ if (node_pagecache_reclaimable(pgdat) <= pgdat->min_unmapped_pages && - node_page_state(pgdat, NR_SLAB_RECLAIMABLE) <= pgdat->min_slab_pages) + node_page_state_pages(pgdat, NR_SLAB_RECLAIMABLE_B) <= + pgdat->min_slab_pages) return NODE_RECLAIM_FULL; /* --- a/mm/workingset.c~mm-memcg-convert-vmstat-slab-counters-to-bytes +++ a/mm/workingset.c @@ -486,8 +486,10 @@ static unsigned long count_shadow_nodes( for (pages = 0, i = 0; i < NR_LRU_LISTS; i++) pages += lruvec_page_state_local(lruvec, NR_LRU_BASE + i); - pages += lruvec_page_state_local(lruvec, NR_SLAB_RECLAIMABLE); - pages += lruvec_page_state_local(lruvec, NR_SLAB_UNRECLAIMABLE); + pages += lruvec_page_state_local( + lruvec, NR_SLAB_RECLAIMABLE_B) >> PAGE_SHIFT; + pages += lruvec_page_state_local( + lruvec, NR_SLAB_UNRECLAIMABLE_B) >> PAGE_SHIFT; } else #endif pages = node_present_pages(sc->nid); From patchwork Fri Aug 7 06:20:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704807 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 434DA14B7 for ; Fri, 7 Aug 2020 06:20:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0E10A2177B for ; Fri, 7 Aug 2020 06:20:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="cYcxa1sv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E10A2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 55D748D004D; Fri, 7 Aug 2020 02:20:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3F8418D0026; Fri, 7 Aug 2020 02:20:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C2858D004D; Fri, 7 Aug 2020 02:20:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0E08A8D0026 for ; Fri, 7 Aug 2020 02:20:45 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C588982499A8 for ; Fri, 7 Aug 2020 06:20:44 +0000 (UTC) X-FDA: 77122774008.01.seed92_361007626fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 98A1610050780 for ; Fri, 7 Aug 2020 06:20:44 +0000 (UTC) X-Spam-Summary: 1,0,0,0ce6df26936897aa,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1544:1605:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2538:2559:2563:2682:2685:2693:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4117:4250:4321:4385:4605:5007:6261:6653:6737:7576:7875:7903:7904:8599:8603:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13141:13230:14096:14181:14721:14849:21067:21080:21450:21451:21627:21740:21939:21990:30003:30034:30054:30064:30075,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yr6k9nwbb6yxgiffcr69eugby9zyc5f8a4rfbpowex63f1n4szewkx78zah54.qh7yxshgaa1rjgsdyq9urawfcsm696mgkm56a8z3ezrwg4h9im7epban6r97i97.r-lbl8.mailshell.ne t-223.23 X-HE-Tag: seed92_361007626fbe X-Filterd-Recvd-Size: 6389 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:44 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0EB852177B; Fri, 7 Aug 2020 06:20:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781243; bh=QEkddzHsPzcFJl6bK129y1bGxKDyUlj86tDV9kStRPo=; h=Date:From:To:Subject:In-Reply-To:From; b=cYcxa1svXcaYbEBpeQRFduBoWUh0q0ANE0f4l+8UMZTfvqSdHsLPviRkf2M2iVqWr oz+Ul3Ds9SHmWbO6ETgsvwaa0TS7Cpjf6s+LQLVQBG+oPQAKlSgMWApwTzC/WNgWBu qtLqUokD4bW3eEnnUtWmNupe5mFk5JHEnB14ZekA= Date: Thu, 06 Aug 2020 23:20:42 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 065/163] mm: slub: implement SLUB version of obj_to_index() Message-ID: <20200807062042.okGJP8Ttd%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 98A1610050780 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: slub: implement SLUB version of obj_to_index() This commit implements SLUB version of the obj_to_index() function, which will be required to calculate the offset of obj_cgroup in the obj_cgroups vector to store/obtain the objcg ownership data. To make it faster, let's repeat the SLAB's trick introduced by commit 6a2d7a955d8d ("SLAB: use a multiply instead of a divide in obj_to_index()") and avoid an expensive division. Vlastimil Babka noticed, that SLUB does have already a similar function called slab_index(), which is defined only if SLUB_DEBUG is enabled. The function does a similar math, but with a division, and it also takes a page address instead of a page pointer. Let's remove slab_index() and replace it with the new helper __obj_to_index(), which takes a page address. obj_to_index() will be a simple wrapper taking a page pointer and passing page_address(page) into __obj_to_index(). Link: http://lkml.kernel.org/r/20200623174037.3951353-5-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/slub_def.h | 16 ++++++++++++++++ mm/slub.c | 15 +++++---------- 2 files changed, 21 insertions(+), 10 deletions(-) --- a/include/linux/slub_def.h~mm-slub-implement-slub-version-of-obj_to_index +++ a/include/linux/slub_def.h @@ -8,6 +8,7 @@ * (C) 2007 SGI, Christoph Lameter */ #include +#include enum stat_item { ALLOC_FASTPATH, /* Allocation from cpu slab */ @@ -86,6 +87,7 @@ struct kmem_cache { unsigned long min_partial; unsigned int size; /* The size of an object including metadata */ unsigned int object_size;/* The size of an object without metadata */ + struct reciprocal_value reciprocal_size; unsigned int offset; /* Free pointer offset */ #ifdef CONFIG_SLUB_CPU_PARTIAL /* Number of per cpu partial objects to keep around */ @@ -182,4 +184,18 @@ static inline void *nearest_obj(struct k return result; } +/* Determine object index from a given position */ +static inline unsigned int __obj_to_index(const struct kmem_cache *cache, + void *addr, void *obj) +{ + return reciprocal_divide(kasan_reset_tag(obj) - addr, + cache->reciprocal_size); +} + +static inline unsigned int obj_to_index(const struct kmem_cache *cache, + const struct page *page, void *obj) +{ + return __obj_to_index(cache, page_address(page), obj); +} + #endif /* _LINUX_SLUB_DEF_H */ --- a/mm/slub.c~mm-slub-implement-slub-version-of-obj_to_index +++ a/mm/slub.c @@ -317,12 +317,6 @@ static inline void set_freepointer(struc __p < (__addr) + (__objects) * (__s)->size; \ __p += (__s)->size) -/* Determine object index from a given position */ -static inline unsigned int slab_index(void *p, struct kmem_cache *s, void *addr) -{ - return (kasan_reset_tag(p) - addr) / s->size; -} - static inline unsigned int order_objects(unsigned int order, unsigned int size) { return ((unsigned int)PAGE_SIZE << order) / size; @@ -465,7 +459,7 @@ static unsigned long *get_map(struct kme bitmap_zero(object_map, page->objects); for (p = page->freelist; p; p = get_freepointer(s, p)) - set_bit(slab_index(p, s, addr), object_map); + set_bit(__obj_to_index(s, addr, p), object_map); return object_map; } @@ -3754,6 +3748,7 @@ static int calculate_sizes(struct kmem_c */ size = ALIGN(size, s->align); s->size = size; + s->reciprocal_size = reciprocal_value(size); if (forced_order >= 0) order = forced_order; else @@ -3858,7 +3853,7 @@ static void list_slab_objects(struct kme map = get_map(s, page); for_each_object(p, s, addr, page->objects) { - if (!test_bit(slab_index(p, s, addr), map)) { + if (!test_bit(__obj_to_index(s, addr, p), map)) { pr_err("INFO: Object 0x%p @offset=%tu\n", p, p - addr); print_tracking(s, p); } @@ -4574,7 +4569,7 @@ static void validate_slab(struct kmem_ca /* Now we know that a valid freelist exists */ map = get_map(s, page); for_each_object(p, s, addr, page->objects) { - u8 val = test_bit(slab_index(p, s, addr), map) ? + u8 val = test_bit(__obj_to_index(s, addr, p), map) ? SLUB_RED_INACTIVE : SLUB_RED_ACTIVE; if (!check_object(s, page, p, val)) @@ -4765,7 +4760,7 @@ static void process_slab(struct loc_trac map = get_map(s, page); for_each_object(p, s, addr, page->objects) - if (!test_bit(slab_index(p, s, addr), map)) + if (!test_bit(__obj_to_index(s, addr, p), map)) add_location(t, s, get_track(s, p, alloc)); put_map(map); } From patchwork Fri Aug 7 06:20:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704809 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95DBA14B7 for ; Fri, 7 Aug 2020 06:20:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4F64E22CAF for ; Fri, 7 Aug 2020 06:20:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="xcHNkCb0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F64E22CAF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 94F8F8D004E; Fri, 7 Aug 2020 02:20:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 900258D0026; Fri, 7 Aug 2020 02:20:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83C888D004E; Fri, 7 Aug 2020 02:20:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id 6EB6D8D0026 for ; Fri, 7 Aug 2020 02:20:48 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 3CF13180AD801 for ; Fri, 7 Aug 2020 06:20:48 +0000 (UTC) X-FDA: 77122774176.29.rub61_39168e726fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 16BB2180868E6 for ; Fri, 7 Aug 2020 06:20:48 +0000 (UTC) X-Spam-Summary: 1,0,0,a7b1195ae758f57f,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:472:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:2195:2196:2198:2199:2200:2201:2393:2525:2559:2564:2682:2685:2693:2859:2898:2902:2919:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4050:4120:4321:4385:4605:5007:6119:6261:6653:6737:7576:7875:7903:8599:8957:9025:9168:9545:9592:10004:10913:11026:11232:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12895:12986:13161:13229:13255:21080:21433:21450:21451:21611:21627:21740:21795:21939:21966:21987:21990:30051:30054:30064:30070:30089,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygzthu1ca3ru9dync1uzgiyny8moc8t8nb5aspum7cyqtd36j7ph9z7f6do4t.zijowiojjd5k9gk9kjx55j5jnku7xsdguojipr4dt4 m8pgcnfh X-HE-Tag: rub61_39168e726fbe X-Filterd-Recvd-Size: 9351 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:47 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7D9D9221E5; Fri, 7 Aug 2020 06:20:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781247; bh=TWdGH2iD3gJ1mjv8ERqfAFtuk1gCjJcVcJxU+ZVQQQg=; h=Date:From:To:Subject:In-Reply-To:From; b=xcHNkCb0WZg+8ob1gUfThBIUT9bnqUtxe1f99Q9v+9U2K7cG4vSgWvbihl6+q/LTp c/LAwDUAiZ2nO1u388Sedo14OO08UZMBjvOq1FljHu8V0MURW5bWIc/PRF3VqUaleO 7Q2nTxyboYcfBY3YS0lnkaMp6fWsAAGIPSg3JnfI= Date: Thu, 06 Aug 2020 23:20:45 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, hughd@google.com, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 066/163] mm: memcontrol: decouple reference counting from page accounting Message-ID: <20200807062045.cn7WdkUEx%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 16BB2180868E6 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Johannes Weiner Subject: mm: memcontrol: decouple reference counting from page accounting The reference counting of a memcg is currently coupled directly to how many 4k pages are charged to it. This doesn't work well with Roman's new slab controller, which maintains pools of objects and doesn't want to keep an extra balance sheet for the pages backing those objects. This unusual refcounting design (reference counts usually track pointers to an object) is only for historical reasons: memcg used to not take any css references and simply stalled offlining until all charges had been reparented and the page counters had dropped to zero. When we got rid of the reparenting requirement, the simple mechanical translation was to take a reference for every charge. More historical context can be found in commit e8ea14cc6ead ("mm: memcontrol: take a css reference for each charged page"), commit 64f219938941 ("mm: memcontrol: remove obsolete kmemcg pinning tricks") and commit b2052564e66d ("mm: memcontrol: continue cache reclaim from offlined groups"). The new slab controller exposes the limitations in this scheme, so let's switch it to a more idiomatic reference counting model based on actual kernel pointers to the memcg: - The per-cpu stock holds a reference to the memcg its caching - User pages hold a reference for their page->mem_cgroup. Transparent huge pages will no longer acquire tail references in advance, we'll get them if needed during the split. - Kernel pages hold a reference for their page->mem_cgroup - Pages allocated in the root cgroup will acquire and release css references for simplicity. css_get() and css_put() optimize that. - The current memcg_charge_slab() already hacked around the per-charge references; this change gets rid of that as well. - tcp accounting will handle reference in mem_cgroup_sk_{alloc,free} Roman: 1) Rebased on top of the current mm tree: added css_get() in mem_cgroup_charge(), dropped mem_cgroup_try_charge() part 2) I've reformatted commit references in the commit log to make checkpatch.pl happy. [hughd@google.com: remove css_put_many() from __mem_cgroup_clear_mc()] Link: http://lkml.kernel.org/r/alpine.LSU.2.11.2007302011450.2347@eggly.anvils Link: http://lkml.kernel.org/r/20200623174037.3951353-6-guro@fb.com Signed-off-by: Johannes Weiner Signed-off-by: Roman Gushchin Signed-off-by: Hugh Dickins Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Michal Hocko Cc: Christoph Lameter Cc: Tejun Heo Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/memcontrol.c | 39 +++++++++++++++++++++------------------ mm/slab.h | 2 -- 2 files changed, 21 insertions(+), 20 deletions(-) --- a/mm/memcontrol.c~mm-memcontrol-decouple-reference-counting-from-page-accounting +++ a/mm/memcontrol.c @@ -2094,13 +2094,17 @@ static void drain_stock(struct memcg_sto { struct mem_cgroup *old = stock->cached; + if (!old) + return; + if (stock->nr_pages) { page_counter_uncharge(&old->memory, stock->nr_pages); if (do_memsw_account()) page_counter_uncharge(&old->memsw, stock->nr_pages); - css_put_many(&old->css, stock->nr_pages); stock->nr_pages = 0; } + + css_put(&old->css); stock->cached = NULL; } @@ -2136,6 +2140,7 @@ static void refill_stock(struct mem_cgro stock = this_cpu_ptr(&memcg_stock); if (stock->cached != memcg) { /* reset if necessary */ drain_stock(stock); + css_get(&memcg->css); stock->cached = memcg; } stock->nr_pages += nr_pages; @@ -2594,12 +2599,10 @@ force: page_counter_charge(&memcg->memory, nr_pages); if (do_memsw_account()) page_counter_charge(&memcg->memsw, nr_pages); - css_get_many(&memcg->css, nr_pages); return 0; done_restock: - css_get_many(&memcg->css, batch); if (batch > nr_pages) refill_stock(memcg, batch - nr_pages); @@ -2657,8 +2660,6 @@ static void cancel_charge(struct mem_cgr page_counter_uncharge(&memcg->memory, nr_pages); if (do_memsw_account()) page_counter_uncharge(&memcg->memsw, nr_pages); - - css_put_many(&memcg->css, nr_pages); } #endif @@ -2966,6 +2967,7 @@ int __memcg_kmem_charge_page(struct page if (!ret) { page->mem_cgroup = memcg; __SetPageKmemcg(page); + return 0; } } css_put(&memcg->css); @@ -2988,12 +2990,11 @@ void __memcg_kmem_uncharge_page(struct p VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); __memcg_kmem_uncharge(memcg, nr_pages); page->mem_cgroup = NULL; + css_put(&memcg->css); /* slab pages do not have PageKmemcg flag set */ if (PageKmemcg(page)) __ClearPageKmemcg(page); - - css_put_many(&memcg->css, nr_pages); } #endif /* CONFIG_MEMCG_KMEM */ @@ -3005,13 +3006,16 @@ void __memcg_kmem_uncharge_page(struct p */ void mem_cgroup_split_huge_fixup(struct page *head) { + struct mem_cgroup *memcg = head->mem_cgroup; int i; if (mem_cgroup_disabled()) return; - for (i = 1; i < HPAGE_PMD_NR; i++) - head[i].mem_cgroup = head->mem_cgroup; + for (i = 1; i < HPAGE_PMD_NR; i++) { + css_get(&memcg->css); + head[i].mem_cgroup = memcg; + } } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -5452,7 +5456,10 @@ static int mem_cgroup_move_account(struc */ smp_mb(); - page->mem_cgroup = to; /* caller should have done css_get */ + css_get(&to->css); + css_put(&from->css); + + page->mem_cgroup = to; __unlock_page_memcg(from); @@ -5673,8 +5680,6 @@ static void __mem_cgroup_clear_mc(void) if (!mem_cgroup_is_root(mc.to)) page_counter_uncharge(&mc.to->memory, mc.moved_swap); - css_put_many(&mc.to->css, mc.moved_swap); - mc.moved_swap = 0; } memcg_oom_recover(from); @@ -6502,6 +6507,7 @@ int mem_cgroup_charge(struct page *page, if (ret) goto out_put; + css_get(&memcg->css); commit_charge(page, memcg); local_irq_disable(); @@ -6556,9 +6562,6 @@ static void uncharge_batch(const struct __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, ug->nr_pages); memcg_check_events(ug->memcg, ug->dummy_page); local_irq_restore(flags); - - if (!mem_cgroup_is_root(ug->memcg)) - css_put_many(&ug->memcg->css, ug->nr_pages); } static void uncharge_page(struct page *page, struct uncharge_gather *ug) @@ -6596,6 +6599,7 @@ static void uncharge_page(struct page *p ug->dummy_page = page; page->mem_cgroup = NULL; + css_put(&ug->memcg->css); } static void uncharge_list(struct list_head *page_list) @@ -6701,8 +6705,8 @@ void mem_cgroup_migrate(struct page *old page_counter_charge(&memcg->memory, nr_pages); if (do_memsw_account()) page_counter_charge(&memcg->memsw, nr_pages); - css_get_many(&memcg->css, nr_pages); + css_get(&memcg->css); commit_charge(newpage, memcg); local_irq_save(flags); @@ -6939,8 +6943,7 @@ void mem_cgroup_swapout(struct page *pag mem_cgroup_charge_statistics(memcg, page, -nr_entries); memcg_check_events(memcg, page); - if (!mem_cgroup_is_root(memcg)) - css_put_many(&memcg->css, nr_entries); + css_put(&memcg->css); } /** --- a/mm/slab.h~mm-memcontrol-decouple-reference-counting-from-page-accounting +++ a/mm/slab.h @@ -402,9 +402,7 @@ static __always_inline int memcg_charge_ lruvec = mem_cgroup_lruvec(memcg, page_pgdat(page)); mod_lruvec_state(lruvec, cache_vmstat_idx(s), nr_pages << PAGE_SHIFT); - /* transer try_charge() page references to kmem_cache */ percpu_ref_get_many(&s->memcg_params.refcnt, nr_pages); - css_put_many(&memcg->css, nr_pages); out: css_put(&memcg->css); return ret; From patchwork Fri Aug 7 06:20:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704811 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65CC0138C for ; Fri, 7 Aug 2020 06:20:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1A35122D07 for ; Fri, 7 Aug 2020 06:20:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="pc961Etr" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1A35122D07 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7A37C8D004F; Fri, 7 Aug 2020 02:20:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 779828D0026; Fri, 7 Aug 2020 02:20:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68F5F8D004F; Fri, 7 Aug 2020 02:20:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0159.hostedemail.com [216.40.44.159]) by kanga.kvack.org (Postfix) with ESMTP id 52F678D0026 for ; Fri, 7 Aug 2020 02:20:52 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 23F1C1F0A for ; Fri, 7 Aug 2020 06:20:52 +0000 (UTC) X-FDA: 77122774344.27.drum17_0a106f026fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id EC0913D663 for ; Fri, 7 Aug 2020 06:20:51 +0000 (UTC) X-Spam-Summary: 1,0,0,4af51602347bd3d5,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:4:41:355:379:472:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:2194:2195:2196:2198:2199:2200:2201:2202:2393:2525:2559:2563:2682:2685:2693:2731:2732:2741:2859:2890:2902:2903:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3608:3743:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4042:4250:4321:4385:4605:5007:6119:6261:6653:6737:7576:7903:8599:8603:8784:8957:9010:9025:9036:9545:10004:10394:10913:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13138:13149:13153:13161:13228:13229:13230:13231:13869:14096:21080:21433:21450:21451:21627:21740:21789:21795:21939:21987:21990:30001:30005:30051:30054:30056:30064:30070:30080:30089,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygxd3e51tz5crqhb7u94xh mmxf1ocj X-HE-Tag: drum17_0a106f026fbe X-Filterd-Recvd-Size: 17028 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:51 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E34A82177B; Fri, 7 Aug 2020 06:20:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781250; bh=OPxdpRXEIH+KV/wH+3KvvKntRC9YyhUO3JKCi+oDdcI=; h=Date:From:To:Subject:In-Reply-To:From; b=pc961Etr47QVNkq2fjV3x+P44iSMQcuJI02S8g74/sHhAVghGkuaqS/m3etUBw2AP KXercMwWCj9+gBP/x8kgpo50PZCrF52L1DWNn19PwFZHUUqP9rWxloA0HrwuGajWSD w1TqIEwpqcEyEtox4xHTvj0Nxbq5WhPhepkiu0HU= Date: Thu, 06 Aug 2020 23:20:49 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 067/163] mm: memcg/slab: obj_cgroup API Message-ID: <20200807062049.id3xOVXTn%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: EC0913D663 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: obj_cgroup API Obj_cgroup API provides an ability to account sub-page sized kernel objects, which potentially outlive the original memory cgroup. The top-level API consists of the following functions: bool obj_cgroup_tryget(struct obj_cgroup *objcg); void obj_cgroup_get(struct obj_cgroup *objcg); void obj_cgroup_put(struct obj_cgroup *objcg); int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size); void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg); struct obj_cgroup *get_obj_cgroup_from_current(void); Object cgroup is basically a pointer to a memory cgroup with a per-cpu reference counter. It substitutes a memory cgroup in places where it's necessary to charge a custom amount of bytes instead of pages. All charged memory rounded down to pages is charged to the corresponding memory cgroup using __memcg_kmem_charge(). It implements reparenting: on memcg offlining it's getting reattached to the parent memory cgroup. Each online memory cgroup has an associated active object cgroup to handle new allocations and the list of all attached object cgroups. On offlining of a cgroup this list is reparented and for each object cgroup in the list the memcg pointer is swapped to the parent memory cgroup. It prevents long-living objects from pinning the original memory cgroup in the memory. The implementation is based on byte-sized per-cpu stocks. A sub-page sized leftover is stored in an atomic field, which is a part of obj_cgroup object. So on cgroup offlining the leftover is automatically reparented. memcg->objcg is rcu protected. objcg->memcg is a raw pointer, which is always pointing at a memory cgroup, but can be atomically swapped to the parent memory cgroup. So a user must ensure the lifetime of the cgroup, e.g. grab rcu_read_lock or css_set_lock. Link: http://lkml.kernel.org/r/20200623174037.3951353-7-guro@fb.com Suggested-by: Johannes Weiner Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Michal Hocko Cc: Tejun Heo Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 51 ++++++ mm/memcontrol.c | 288 ++++++++++++++++++++++++++++++++++- 2 files changed, 338 insertions(+), 1 deletion(-) --- a/include/linux/memcontrol.h~mm-memcg-slab-obj_cgroup-api +++ a/include/linux/memcontrol.h @@ -23,6 +23,7 @@ #include struct mem_cgroup; +struct obj_cgroup; struct page; struct mm_struct; struct kmem_cache; @@ -193,6 +194,22 @@ struct memcg_cgwb_frn { }; /* + * Bucket for arbitrarily byte-sized objects charged to a memory + * cgroup. The bucket can be reparented in one piece when the cgroup + * is destroyed, without having to round up the individual references + * of all live memory objects in the wild. + */ +struct obj_cgroup { + struct percpu_ref refcnt; + struct mem_cgroup *memcg; + atomic_t nr_charged_bytes; + union { + struct list_head list; + struct rcu_head rcu; + }; +}; + +/* * The memory controller data structure. The memory controller controls both * page cache and RSS per cgroup. We would eventually like to provide * statistics based on the statistics developed by Rik Van Riel for clock-pro, @@ -301,6 +318,8 @@ struct mem_cgroup { int kmemcg_id; enum memcg_kmem_state kmem_state; struct list_head kmem_caches; + struct obj_cgroup __rcu *objcg; + struct list_head objcg_list; /* list of inherited objcgs */ #endif #ifdef CONFIG_CGROUP_WRITEBACK @@ -416,6 +435,33 @@ struct mem_cgroup *mem_cgroup_from_css(s return css ? container_of(css, struct mem_cgroup, css) : NULL; } +static inline bool obj_cgroup_tryget(struct obj_cgroup *objcg) +{ + return percpu_ref_tryget(&objcg->refcnt); +} + +static inline void obj_cgroup_get(struct obj_cgroup *objcg) +{ + percpu_ref_get(&objcg->refcnt); +} + +static inline void obj_cgroup_put(struct obj_cgroup *objcg) +{ + percpu_ref_put(&objcg->refcnt); +} + +/* + * After the initialization objcg->memcg is always pointing at + * a valid memcg, but can be atomically swapped to the parent memcg. + * + * The caller must ensure that the returned memcg won't be released: + * e.g. acquire the rcu_read_lock or css_set_lock. + */ +static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) +{ + return READ_ONCE(objcg->memcg); +} + static inline void mem_cgroup_put(struct mem_cgroup *memcg) { if (memcg) @@ -1368,6 +1414,11 @@ void __memcg_kmem_uncharge(struct mem_cg int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); +struct obj_cgroup *get_obj_cgroup_from_current(void); + +int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size); +void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); + extern struct static_key_false memcg_kmem_enabled_key; extern struct workqueue_struct *memcg_kmem_cache_wq; --- a/mm/memcontrol.c~mm-memcg-slab-obj_cgroup-api +++ a/mm/memcontrol.c @@ -257,6 +257,98 @@ struct cgroup_subsys_state *vmpressure_t } #ifdef CONFIG_MEMCG_KMEM +extern spinlock_t css_set_lock; + +static void obj_cgroup_release(struct percpu_ref *ref) +{ + struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt); + struct mem_cgroup *memcg; + unsigned int nr_bytes; + unsigned int nr_pages; + unsigned long flags; + + /* + * At this point all allocated objects are freed, and + * objcg->nr_charged_bytes can't have an arbitrary byte value. + * However, it can be PAGE_SIZE or (x * PAGE_SIZE). + * + * The following sequence can lead to it: + * 1) CPU0: objcg == stock->cached_objcg + * 2) CPU1: we do a small allocation (e.g. 92 bytes), + * PAGE_SIZE bytes are charged + * 3) CPU1: a process from another memcg is allocating something, + * the stock if flushed, + * objcg->nr_charged_bytes = PAGE_SIZE - 92 + * 5) CPU0: we do release this object, + * 92 bytes are added to stock->nr_bytes + * 6) CPU0: stock is flushed, + * 92 bytes are added to objcg->nr_charged_bytes + * + * In the result, nr_charged_bytes == PAGE_SIZE. + * This page will be uncharged in obj_cgroup_release(). + */ + nr_bytes = atomic_read(&objcg->nr_charged_bytes); + WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1)); + nr_pages = nr_bytes >> PAGE_SHIFT; + + spin_lock_irqsave(&css_set_lock, flags); + memcg = obj_cgroup_memcg(objcg); + if (nr_pages) + __memcg_kmem_uncharge(memcg, nr_pages); + list_del(&objcg->list); + mem_cgroup_put(memcg); + spin_unlock_irqrestore(&css_set_lock, flags); + + percpu_ref_exit(ref); + kfree_rcu(objcg, rcu); +} + +static struct obj_cgroup *obj_cgroup_alloc(void) +{ + struct obj_cgroup *objcg; + int ret; + + objcg = kzalloc(sizeof(struct obj_cgroup), GFP_KERNEL); + if (!objcg) + return NULL; + + ret = percpu_ref_init(&objcg->refcnt, obj_cgroup_release, 0, + GFP_KERNEL); + if (ret) { + kfree(objcg); + return NULL; + } + INIT_LIST_HEAD(&objcg->list); + return objcg; +} + +static void memcg_reparent_objcgs(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct obj_cgroup *objcg, *iter; + + objcg = rcu_replace_pointer(memcg->objcg, NULL, true); + + spin_lock_irq(&css_set_lock); + + /* Move active objcg to the parent's list */ + xchg(&objcg->memcg, parent); + css_get(&parent->css); + list_add(&objcg->list, &parent->objcg_list); + + /* Move already reparented objcgs to the parent's list */ + list_for_each_entry(iter, &memcg->objcg_list, list) { + css_get(&parent->css); + xchg(&iter->memcg, parent); + css_put(&memcg->css); + } + list_splice(&memcg->objcg_list, &parent->objcg_list); + + spin_unlock_irq(&css_set_lock); + + percpu_ref_kill(&objcg->refcnt); +} + /* * This will be the memcg's index in each cache's ->memcg_params.memcg_caches. * The main reason for not using cgroup id for this: @@ -2047,6 +2139,12 @@ EXPORT_SYMBOL(unlock_page_memcg); struct memcg_stock_pcp { struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; + +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *cached_objcg; + unsigned int nr_bytes; +#endif + struct work_struct work; unsigned long flags; #define FLUSHING_CACHED_CHARGE 0 @@ -2054,6 +2152,22 @@ struct memcg_stock_pcp { static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock); static DEFINE_MUTEX(percpu_charge_mutex); +#ifdef CONFIG_MEMCG_KMEM +static void drain_obj_stock(struct memcg_stock_pcp *stock); +static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, + struct mem_cgroup *root_memcg); + +#else +static inline void drain_obj_stock(struct memcg_stock_pcp *stock) +{ +} +static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, + struct mem_cgroup *root_memcg) +{ + return false; +} +#endif + /** * consume_stock: Try to consume stocked charge on this cpu. * @memcg: memcg to consume from. @@ -2120,6 +2234,7 @@ static void drain_local_stock(struct wor local_irq_save(flags); stock = this_cpu_ptr(&memcg_stock); + drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); @@ -2179,6 +2294,8 @@ static void drain_all_stock(struct mem_c if (memcg && stock->nr_pages && mem_cgroup_is_descendant(memcg, root_memcg)) flush = true; + if (obj_stock_flush_required(stock, root_memcg)) + flush = true; rcu_read_unlock(); if (flush && @@ -2705,6 +2822,30 @@ struct mem_cgroup *mem_cgroup_from_obj(v return page->mem_cgroup; } +__always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) +{ + struct obj_cgroup *objcg = NULL; + struct mem_cgroup *memcg; + + if (unlikely(!current->mm && !current->active_memcg)) + return NULL; + + rcu_read_lock(); + if (unlikely(current->active_memcg)) + memcg = rcu_dereference(current->active_memcg); + else + memcg = mem_cgroup_from_task(current); + + for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) { + objcg = rcu_dereference(memcg->objcg); + if (objcg && obj_cgroup_tryget(objcg)) + break; + } + rcu_read_unlock(); + + return objcg; +} + static int memcg_alloc_cache_id(void) { int id, size; @@ -2996,6 +3137,140 @@ void __memcg_kmem_uncharge_page(struct p if (PageKmemcg(page)) __ClearPageKmemcg(page); } + +static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) +{ + struct memcg_stock_pcp *stock; + unsigned long flags; + bool ret = false; + + local_irq_save(flags); + + stock = this_cpu_ptr(&memcg_stock); + if (objcg == stock->cached_objcg && stock->nr_bytes >= nr_bytes) { + stock->nr_bytes -= nr_bytes; + ret = true; + } + + local_irq_restore(flags); + + return ret; +} + +static void drain_obj_stock(struct memcg_stock_pcp *stock) +{ + struct obj_cgroup *old = stock->cached_objcg; + + if (!old) + return; + + if (stock->nr_bytes) { + unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT; + unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1); + + if (nr_pages) { + rcu_read_lock(); + __memcg_kmem_uncharge(obj_cgroup_memcg(old), nr_pages); + rcu_read_unlock(); + } + + /* + * The leftover is flushed to the centralized per-memcg value. + * On the next attempt to refill obj stock it will be moved + * to a per-cpu stock (probably, on an other CPU), see + * refill_obj_stock(). + * + * How often it's flushed is a trade-off between the memory + * limit enforcement accuracy and potential CPU contention, + * so it might be changed in the future. + */ + atomic_add(nr_bytes, &old->nr_charged_bytes); + stock->nr_bytes = 0; + } + + obj_cgroup_put(old); + stock->cached_objcg = NULL; +} + +static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, + struct mem_cgroup *root_memcg) +{ + struct mem_cgroup *memcg; + + if (stock->cached_objcg) { + memcg = obj_cgroup_memcg(stock->cached_objcg); + if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) + return true; + } + + return false; +} + +static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) +{ + struct memcg_stock_pcp *stock; + unsigned long flags; + + local_irq_save(flags); + + stock = this_cpu_ptr(&memcg_stock); + if (stock->cached_objcg != objcg) { /* reset if necessary */ + drain_obj_stock(stock); + obj_cgroup_get(objcg); + stock->cached_objcg = objcg; + stock->nr_bytes = atomic_xchg(&objcg->nr_charged_bytes, 0); + } + stock->nr_bytes += nr_bytes; + + if (stock->nr_bytes > PAGE_SIZE) + drain_obj_stock(stock); + + local_irq_restore(flags); +} + +int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) +{ + struct mem_cgroup *memcg; + unsigned int nr_pages, nr_bytes; + int ret; + + if (consume_obj_stock(objcg, size)) + return 0; + + /* + * In theory, memcg->nr_charged_bytes can have enough + * pre-charged bytes to satisfy the allocation. However, + * flushing memcg->nr_charged_bytes requires two atomic + * operations, and memcg->nr_charged_bytes can't be big, + * so it's better to ignore it and try grab some new pages. + * memcg->nr_charged_bytes will be flushed in + * refill_obj_stock(), called from this function or + * independently later. + */ + rcu_read_lock(); + memcg = obj_cgroup_memcg(objcg); + css_get(&memcg->css); + rcu_read_unlock(); + + nr_pages = size >> PAGE_SHIFT; + nr_bytes = size & (PAGE_SIZE - 1); + + if (nr_bytes) + nr_pages += 1; + + ret = __memcg_kmem_charge(memcg, gfp, nr_pages); + if (!ret && nr_bytes) + refill_obj_stock(objcg, PAGE_SIZE - nr_bytes); + + css_put(&memcg->css); + return ret; +} + +void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) +{ + refill_obj_stock(objcg, size); +} + #endif /* CONFIG_MEMCG_KMEM */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -3416,6 +3691,7 @@ static void memcg_flush_percpu_vmevents( #ifdef CONFIG_MEMCG_KMEM static int memcg_online_kmem(struct mem_cgroup *memcg) { + struct obj_cgroup *objcg; int memcg_id; if (cgroup_memory_nokmem) @@ -3428,6 +3704,14 @@ static int memcg_online_kmem(struct mem_ if (memcg_id < 0) return memcg_id; + objcg = obj_cgroup_alloc(); + if (!objcg) { + memcg_free_cache_id(memcg_id); + return -ENOMEM; + } + objcg->memcg = memcg; + rcu_assign_pointer(memcg->objcg, objcg); + static_branch_enable(&memcg_kmem_enabled_key); /* @@ -3464,9 +3748,10 @@ static void memcg_offline_kmem(struct me parent = root_mem_cgroup; /* - * Deactivate and reparent kmem_caches. + * Deactivate and reparent kmem_caches and objcgs. */ memcg_deactivate_kmem_caches(memcg, parent); + memcg_reparent_objcgs(memcg, parent); kmemcg_id = memcg->kmemcg_id; BUG_ON(kmemcg_id < 0); @@ -5030,6 +5315,7 @@ static struct mem_cgroup *mem_cgroup_all memcg->socket_pressure = jiffies; #ifdef CONFIG_MEMCG_KMEM memcg->kmemcg_id = -1; + INIT_LIST_HEAD(&memcg->objcg_list); #endif #ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD(&memcg->cgwb_list); From patchwork Fri Aug 7 06:20:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704813 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B8FBF138C for ; Fri, 7 Aug 2020 06:20:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7E33A22D03 for ; Fri, 7 Aug 2020 06:20:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="fNXDWE8Z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7E33A22D03 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B9B718D0050; Fri, 7 Aug 2020 02:20:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B49C98D0026; Fri, 7 Aug 2020 02:20:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5F858D0050; Fri, 7 Aug 2020 02:20:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8D9F08D0026 for ; Fri, 7 Aug 2020 02:20:55 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 56AE73625 for ; Fri, 7 Aug 2020 06:20:55 +0000 (UTC) X-FDA: 77122774470.03.shoes22_07122ff26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 2C44A28A4E8 for ; Fri, 7 Aug 2020 06:20:55 +0000 (UTC) X-Spam-Summary: 1,0,0,c1a03acae5b57e1c,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2525:2559:2563:2682:2685:2689:2693:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4049:4119:4321:4385:4605:5007:6261:6653:6737:7576:7903:8599:8957:9025:9121:9545:10004:10913:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13146:13161:13221:13227:13229:13230:14096:21080:21450:21451:21627:21939:21987:21990:30045:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfbq698gfnm6zei8scc55h94yduocfptzc1oc8kfqopiih8t4ot8uttckzz1r.pn94pkcf67e31zcsuqkansxhkifcdg3a1yt1wy3wpmhpmjzpmykefh99y97hj4d.o-lbl8.mailshell.net-223.238 .255.100 X-HE-Tag: shoes22_07122ff26fbe X-Filterd-Recvd-Size: 8041 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:54 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6E5D22177B; Fri, 7 Aug 2020 06:20:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781253; bh=DStW/HX07G4oUvaIN6kcs8n/BmYvQOTIjqp5tncLiqc=; h=Date:From:To:Subject:In-Reply-To:From; b=fNXDWE8Zk2ghqKc6gvoqIexDyamBoFBc5ZAICLOATkem4mIX3CkCUHk2jqr1QQEp2 uUQHN/r2fbKoExves+NzDz+DgCDYK0PKIvnPqTq7lxQ1PeHy8Xgv+PHc2JjQhJnIi1 G9anaXbmavF2s4Kc1X50EJiTXVspueuoMb/tOUxc= Date: Thu, 06 Aug 2020 23:20:52 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 068/163] mm: memcg/slab: allocate obj_cgroups for non-root slab pages Message-ID: <20200807062052.tZ84SDxmn%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 2C44A28A4E8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.017976, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: allocate obj_cgroups for non-root slab pages Allocate and release memory to store obj_cgroup pointers for each non-root slab page. Reuse page->mem_cgroup pointer to store a pointer to the allocated space. This commit temporarily increases the memory footprint of the kernel memory accounting. To store obj_cgroup pointers we'll need a place for an objcg_pointer for each allocated object. However, the following patches in the series will enable sharing of slab pages between memory cgroups, which will dramatically increase the total slab utilization. And the final memory footprint will be significantly smaller than before. To distinguish between obj_cgroups and memcg pointers in case when it's not obvious which one is used (as in page_cgroup_ino()), let's always set the lowest bit in the obj_cgroup case. The original obj_cgroups pointer is marked to be ignored by kmemleak, which otherwise would report a memory leak for each allocated vector. Link: http://lkml.kernel.org/r/20200623174037.3951353-8-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Johannes Weiner Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/mm_types.h | 5 ++- include/linux/slab_def.h | 6 ++++ include/linux/slub_def.h | 5 +++ mm/memcontrol.c | 17 +++++++++--- mm/slab.h | 52 +++++++++++++++++++++++++++++++++++++ 5 files changed, 81 insertions(+), 4 deletions(-) --- a/include/linux/mm_types.h~mm-memcg-slab-allocate-obj_cgroups-for-non-root-slab-pages +++ a/include/linux/mm_types.h @@ -198,7 +198,10 @@ struct page { atomic_t _refcount; #ifdef CONFIG_MEMCG - struct mem_cgroup *mem_cgroup; + union { + struct mem_cgroup *mem_cgroup; + struct obj_cgroup **obj_cgroups; + }; #endif /* --- a/include/linux/slab_def.h~mm-memcg-slab-allocate-obj_cgroups-for-non-root-slab-pages +++ a/include/linux/slab_def.h @@ -114,4 +114,10 @@ static inline unsigned int obj_to_index( return reciprocal_divide(offset, cache->reciprocal_buffer_size); } +static inline int objs_per_slab_page(const struct kmem_cache *cache, + const struct page *page) +{ + return cache->num; +} + #endif /* _LINUX_SLAB_DEF_H */ --- a/include/linux/slub_def.h~mm-memcg-slab-allocate-obj_cgroups-for-non-root-slab-pages +++ a/include/linux/slub_def.h @@ -198,4 +198,9 @@ static inline unsigned int obj_to_index( return __obj_to_index(cache, page_address(page), obj); } +static inline int objs_per_slab_page(const struct kmem_cache *cache, + const struct page *page) +{ + return page->objects; +} #endif /* _LINUX_SLUB_DEF_H */ --- a/mm/memcontrol.c~mm-memcg-slab-allocate-obj_cgroups-for-non-root-slab-pages +++ a/mm/memcontrol.c @@ -569,10 +569,21 @@ ino_t page_cgroup_ino(struct page *page) unsigned long ino = 0; rcu_read_lock(); - if (PageSlab(page) && !PageTail(page)) + if (PageSlab(page) && !PageTail(page)) { memcg = memcg_from_slab_page(page); - else - memcg = READ_ONCE(page->mem_cgroup); + } else { + memcg = page->mem_cgroup; + + /* + * The lowest bit set means that memcg isn't a valid + * memcg pointer, but a obj_cgroups pointer. + * In this case the page is shared and doesn't belong + * to any specific memory cgroup. + */ + if ((unsigned long) memcg & 0x1UL) + memcg = NULL; + } + while (memcg && !(memcg->css.flags & CSS_ONLINE)) memcg = parent_mem_cgroup(memcg); if (memcg) --- a/mm/slab.h~mm-memcg-slab-allocate-obj_cgroups-for-non-root-slab-pages +++ a/mm/slab.h @@ -109,6 +109,7 @@ struct memcg_cache_params { #include #include #include +#include /* * State of the slab allocator. @@ -348,6 +349,18 @@ static inline struct kmem_cache *memcg_r return s->memcg_params.root_cache; } +static inline struct obj_cgroup **page_obj_cgroups(struct page *page) +{ + /* + * page->mem_cgroup and page->obj_cgroups are sharing the same + * space. To distinguish between them in case we don't know for sure + * that the page is a slab page (e.g. page_cgroup_ino()), let's + * always set the lowest bit of obj_cgroups. + */ + return (struct obj_cgroup **) + ((unsigned long)page->obj_cgroups & ~0x1UL); +} + /* * Expects a pointer to a slab page. Please note, that PageSlab() check * isn't sufficient, as it returns true also for tail compound slab pages, @@ -435,6 +448,28 @@ static __always_inline void memcg_unchar percpu_ref_put_many(&s->memcg_params.refcnt, nr_pages); } +static inline int memcg_alloc_page_obj_cgroups(struct page *page, + struct kmem_cache *s, gfp_t gfp) +{ + unsigned int objects = objs_per_slab_page(s, page); + void *vec; + + vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, + page_to_nid(page)); + if (!vec) + return -ENOMEM; + + kmemleak_not_leak(vec); + page->obj_cgroups = (struct obj_cgroup **) ((unsigned long)vec | 0x1UL); + return 0; +} + +static inline void memcg_free_page_obj_cgroups(struct page *page) +{ + kfree(page_obj_cgroups(page)); + page->obj_cgroups = NULL; +} + extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); @@ -484,6 +519,16 @@ static inline void memcg_uncharge_slab(s { } +static inline int memcg_alloc_page_obj_cgroups(struct page *page, + struct kmem_cache *s, gfp_t gfp) +{ + return 0; +} + +static inline void memcg_free_page_obj_cgroups(struct page *page) +{ +} + static inline void slab_init_memcg_params(struct kmem_cache *s) { } @@ -510,12 +555,18 @@ static __always_inline int charge_slab_p gfp_t gfp, int order, struct kmem_cache *s) { + int ret; + if (is_root_cache(s)) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), PAGE_SIZE << order); return 0; } + ret = memcg_alloc_page_obj_cgroups(page, s, gfp); + if (ret) + return ret; + return memcg_charge_slab(page, gfp, order, s); } @@ -528,6 +579,7 @@ static __always_inline void uncharge_sla return; } + memcg_free_page_obj_cgroups(page); memcg_uncharge_slab(page, order, s); } From patchwork Fri Aug 7 06:20:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704815 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3C45D14B7 for ; Fri, 7 Aug 2020 06:21:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EB5FB22D07 for ; Fri, 7 Aug 2020 06:21:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="PbJO9203" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EB5FB22D07 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 30E438D0051; Fri, 7 Aug 2020 02:20:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2BD108D0026; Fri, 7 Aug 2020 02:20:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 160D68D0051; Fri, 7 Aug 2020 02:20:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0205.hostedemail.com [216.40.44.205]) by kanga.kvack.org (Postfix) with ESMTP id F1E388D0026 for ; Fri, 7 Aug 2020 02:20:58 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B731B180AD804 for ; Fri, 7 Aug 2020 06:20:58 +0000 (UTC) X-FDA: 77122774596.05.hole82_130eb5126fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 94A9B18023903 for ; Fri, 7 Aug 2020 06:20:58 +0000 (UTC) X-Spam-Summary: 1,0,0,5010566a57cb5abc,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:2:41:355:379:800:960:966:967:973:981:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4052:4321:4385:4605:5007:6261:6653:6737:7576:8599:8603:8660:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13148:13161:13229:13230:14096:21080:21433:21450:21451:21627:21939:21987:21990:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8jzch3jmrqkcu5916srmfeh6f7yc3smz3cqyrotfe3f69kxu3itegj1ydjbw.mcpydqzn5xj5o7eadyft36etcnyak8refrdngu6pisbzjm4pxrnjasdtim83q55.e-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netc heck:non X-HE-Tag: hole82_130eb5126fbe X-Filterd-Recvd-Size: 11284 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:20:57 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id BB0EF221E5; Fri, 7 Aug 2020 06:20:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781257; bh=tRpWt0fuoKmDnpvqr7gt2u6u36gOZ6DN2arw5tvMfW8=; h=Date:From:To:Subject:In-Reply-To:From; b=PbJO9203JW/rEuNg/0NTqdqVhExIvTKJvGY0LZizonWO2MM9HUsgXRRsaE6ara5N/ /QSf6JbGRhPoXT9qjzyDLbZLAwFZQfK/IIr9WdlQe9AJ+d4caKU8dbiDMove/7tMLX pSv0ax7gK9mer2a0SWNcTO/v38fxNjo2l4QzZmPM= Date: Thu, 06 Aug 2020 23:20:56 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 069/163] mm: memcg/slab: save obj_cgroup for non-root slab objects Message-ID: <20200807062056.GMH1InlNM%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 94A9B18023903 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.412271, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: save obj_cgroup for non-root slab objects Store the obj_cgroup pointer in the corresponding place of page->obj_cgroups for each allocated non-root slab object. Make sure that each allocated object holds a reference to obj_cgroup. Objcg pointer is obtained from the memcg->objcg dereferencing in memcg_kmem_get_cache() and passed from pre_alloc_hook to post_alloc_hook. Then in case of successful allocation(s) it's getting stored in the page->obj_cgroups vector. The objcg obtaining part look a bit bulky now, but it will be simplified by next commits in the series. Link: http://lkml.kernel.org/r/20200623174037.3951353-9-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Johannes Weiner Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 3 + mm/memcontrol.c | 14 +++++++- mm/slab.c | 18 ++++++---- mm/slab.h | 60 +++++++++++++++++++++++++++++++---- mm/slub.c | 14 +++++--- 5 files changed, 88 insertions(+), 21 deletions(-) --- a/include/linux/memcontrol.h~mm-memcg-slab-save-obj_cgroup-for-non-root-slab-objects +++ a/include/linux/memcontrol.h @@ -1404,7 +1404,8 @@ static inline void memcg_set_shrinker_bi } #endif -struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep); +struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep, + struct obj_cgroup **objcgp); void memcg_kmem_put_cache(struct kmem_cache *cachep); #ifdef CONFIG_MEMCG_KMEM --- a/mm/memcontrol.c~mm-memcg-slab-save-obj_cgroup-for-non-root-slab-objects +++ a/mm/memcontrol.c @@ -2973,7 +2973,8 @@ static inline bool memcg_kmem_bypass(voi * done with it, memcg_kmem_put_cache() must be called to release the * reference. */ -struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) +struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep, + struct obj_cgroup **objcgp) { struct mem_cgroup *memcg; struct kmem_cache *memcg_cachep; @@ -3029,8 +3030,17 @@ struct kmem_cache *memcg_kmem_get_cache( */ if (unlikely(!memcg_cachep)) memcg_schedule_kmem_cache_create(memcg, cachep); - else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt)) + else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt)) { + struct obj_cgroup *objcg = rcu_dereference(memcg->objcg); + + if (!objcg || !obj_cgroup_tryget(objcg)) { + percpu_ref_put(&memcg_cachep->memcg_params.refcnt); + goto out_unlock; + } + + *objcgp = objcg; cachep = memcg_cachep; + } out_unlock: rcu_read_unlock(); return cachep; --- a/mm/slab.c~mm-memcg-slab-save-obj_cgroup-for-non-root-slab-objects +++ a/mm/slab.c @@ -3228,9 +3228,10 @@ slab_alloc_node(struct kmem_cache *cache unsigned long save_flags; void *ptr; int slab_node = numa_mem_id(); + struct obj_cgroup *objcg = NULL; flags &= gfp_allowed_mask; - cachep = slab_pre_alloc_hook(cachep, flags); + cachep = slab_pre_alloc_hook(cachep, &objcg, 1, flags); if (unlikely(!cachep)) return NULL; @@ -3266,7 +3267,7 @@ slab_alloc_node(struct kmem_cache *cache if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr) memset(ptr, 0, cachep->object_size); - slab_post_alloc_hook(cachep, flags, 1, &ptr); + slab_post_alloc_hook(cachep, objcg, flags, 1, &ptr); return ptr; } @@ -3307,9 +3308,10 @@ slab_alloc(struct kmem_cache *cachep, gf { unsigned long save_flags; void *objp; + struct obj_cgroup *objcg = NULL; flags &= gfp_allowed_mask; - cachep = slab_pre_alloc_hook(cachep, flags); + cachep = slab_pre_alloc_hook(cachep, &objcg, 1, flags); if (unlikely(!cachep)) return NULL; @@ -3323,7 +3325,7 @@ slab_alloc(struct kmem_cache *cachep, gf if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp) memset(objp, 0, cachep->object_size); - slab_post_alloc_hook(cachep, flags, 1, &objp); + slab_post_alloc_hook(cachep, objcg, flags, 1, &objp); return objp; } @@ -3450,6 +3452,7 @@ void ___cache_free(struct kmem_cache *ca memset(objp, 0, cachep->object_size); kmemleak_free_recursive(objp, cachep->flags); objp = cache_free_debugcheck(cachep, objp, caller); + memcg_slab_free_hook(cachep, virt_to_head_page(objp), objp); /* * Skip calling cache_free_alien() when the platform is not numa. @@ -3515,8 +3518,9 @@ int kmem_cache_alloc_bulk(struct kmem_ca void **p) { size_t i; + struct obj_cgroup *objcg = NULL; - s = slab_pre_alloc_hook(s, flags); + s = slab_pre_alloc_hook(s, &objcg, size, flags); if (!s) return 0; @@ -3539,13 +3543,13 @@ int kmem_cache_alloc_bulk(struct kmem_ca for (i = 0; i < size; i++) memset(p[i], 0, s->object_size); - slab_post_alloc_hook(s, flags, size, p); + slab_post_alloc_hook(s, objcg, flags, size, p); /* FIXME: Trace call missing. Christoph would like a bulk variant */ return size; error: local_irq_enable(); cache_alloc_debugcheck_after_bulk(s, flags, i, p, _RET_IP_); - slab_post_alloc_hook(s, flags, i, p); + slab_post_alloc_hook(s, objcg, flags, i, p); __kmem_cache_free_bulk(s, i, p); return 0; } --- a/mm/slab.h~mm-memcg-slab-save-obj_cgroup-for-non-root-slab-objects +++ a/mm/slab.h @@ -470,6 +470,41 @@ static inline void memcg_free_page_obj_c page->obj_cgroups = NULL; } +static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, + struct obj_cgroup *objcg, + size_t size, void **p) +{ + struct page *page; + unsigned long off; + size_t i; + + for (i = 0; i < size; i++) { + if (likely(p[i])) { + page = virt_to_head_page(p[i]); + off = obj_to_index(s, page, p[i]); + obj_cgroup_get(objcg); + page_obj_cgroups(page)[off] = objcg; + } + } + obj_cgroup_put(objcg); + memcg_kmem_put_cache(s); +} + +static inline void memcg_slab_free_hook(struct kmem_cache *s, struct page *page, + void *p) +{ + struct obj_cgroup *objcg; + unsigned int off; + + if (!memcg_kmem_enabled() || is_root_cache(s)) + return; + + off = obj_to_index(s, page, p); + objcg = page_obj_cgroups(page)[off]; + page_obj_cgroups(page)[off] = NULL; + obj_cgroup_put(objcg); +} + extern void slab_init_memcg_params(struct kmem_cache *); extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); @@ -529,6 +564,17 @@ static inline void memcg_free_page_obj_c { } +static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, + struct obj_cgroup *objcg, + size_t size, void **p) +{ +} + +static inline void memcg_slab_free_hook(struct kmem_cache *s, struct page *page, + void *p) +{ +} + static inline void slab_init_memcg_params(struct kmem_cache *s) { } @@ -631,7 +677,8 @@ static inline size_t slab_ksize(const st } static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, - gfp_t flags) + struct obj_cgroup **objcgp, + size_t size, gfp_t flags) { flags &= gfp_allowed_mask; @@ -645,13 +692,14 @@ static inline struct kmem_cache *slab_pr if (memcg_kmem_enabled() && ((flags & __GFP_ACCOUNT) || (s->flags & SLAB_ACCOUNT))) - return memcg_kmem_get_cache(s); + return memcg_kmem_get_cache(s, objcgp); return s; } -static inline void slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags, - size_t size, void **p) +static inline void slab_post_alloc_hook(struct kmem_cache *s, + struct obj_cgroup *objcg, + gfp_t flags, size_t size, void **p) { size_t i; @@ -663,8 +711,8 @@ static inline void slab_post_alloc_hook( s->flags, flags); } - if (memcg_kmem_enabled()) - memcg_kmem_put_cache(s); + if (memcg_kmem_enabled() && !is_root_cache(s)) + memcg_slab_post_alloc_hook(s, objcg, size, p); } #ifndef CONFIG_SLOB --- a/mm/slub.c~mm-memcg-slab-save-obj_cgroup-for-non-root-slab-objects +++ a/mm/slub.c @@ -2817,8 +2817,9 @@ static __always_inline void *slab_alloc_ struct kmem_cache_cpu *c; struct page *page; unsigned long tid; + struct obj_cgroup *objcg = NULL; - s = slab_pre_alloc_hook(s, gfpflags); + s = slab_pre_alloc_hook(s, &objcg, 1, gfpflags); if (!s) return NULL; redo: @@ -2894,7 +2895,7 @@ redo: if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object) memset(object, 0, s->object_size); - slab_post_alloc_hook(s, gfpflags, 1, &object); + slab_post_alloc_hook(s, objcg, gfpflags, 1, &object); return object; } @@ -3099,6 +3100,8 @@ static __always_inline void do_slab_free void *tail_obj = tail ? : head; struct kmem_cache_cpu *c; unsigned long tid; + + memcg_slab_free_hook(s, page, head); redo: /* * Determine the currently cpus per cpu slab. @@ -3278,9 +3281,10 @@ int kmem_cache_alloc_bulk(struct kmem_ca { struct kmem_cache_cpu *c; int i; + struct obj_cgroup *objcg = NULL; /* memcg and kmem_cache debug support */ - s = slab_pre_alloc_hook(s, flags); + s = slab_pre_alloc_hook(s, &objcg, size, flags); if (unlikely(!s)) return false; /* @@ -3334,11 +3338,11 @@ int kmem_cache_alloc_bulk(struct kmem_ca } /* memcg and kmem_cache debug support */ - slab_post_alloc_hook(s, flags, size, p); + slab_post_alloc_hook(s, objcg, flags, size, p); return i; error: local_irq_enable(); - slab_post_alloc_hook(s, flags, i, p); + slab_post_alloc_hook(s, objcg, flags, i, p); __kmem_cache_free_bulk(s, i, p); return 0; } From patchwork Fri Aug 7 06:21:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704817 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8791214B7 for ; Fri, 7 Aug 2020 06:21:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5569122C9F for ; Fri, 7 Aug 2020 06:21:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="h2Oibxo1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5569122C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 09E888D0052; Fri, 7 Aug 2020 02:21:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0755B8D0026; Fri, 7 Aug 2020 02:21:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF5618D0052; Fri, 7 Aug 2020 02:21:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0198.hostedemail.com [216.40.44.198]) by kanga.kvack.org (Postfix) with ESMTP id DAB468D0026 for ; Fri, 7 Aug 2020 02:21:05 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A48B8181AEF10 for ; Fri, 7 Aug 2020 06:21:05 +0000 (UTC) X-FDA: 77122774890.12.wrist44_1d10b3526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 8092A18009952 for ; Fri, 7 Aug 2020 06:21:05 +0000 (UTC) X-Spam-Summary: 1,0,0,132b851ea09361c9,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1500:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2693:2859:2892:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4470:5007:6261:6653:6737:7576:7903:8599:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12114:12294:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13229:14096:14181:14721:14849:21063:21080:21433:21451:21627:21939:21990:30029:30054:30062:30064:30070:30075,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y89n7bgxhrbdiqzqn6kohyyfzfxypbdxis3onqayiq4bcmrscke6k5371wf97.oe7ogk3iypx1gzt98afhbhyzd1zio7pq9cetu5ouoe58mfqbejqou6umx3eqdoe.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Baye sian:0.5 X-HE-Tag: wrist44_1d10b3526fbe X-Filterd-Recvd-Size: 4329 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:04 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C700C2177B; Fri, 7 Aug 2020 06:21:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781264; bh=kcYw7rdPT7nf8C8LR4PoU61MLtrCwNXJJnbBH6t2mX4=; h=Date:From:To:Subject:In-Reply-To:From; b=h2Oibxo1DIBLXBYBgK4jvJ8t64MYQPYJgdQMZAtyk5NwgIagFLwp1YI7u9UUxcOdu uUwaU0fzXfCgjd4ylzIBLXEQPFfKFbuK45u0zRU1yi/CiLd/U7TeRjpjTftlkx6flD 37CcQ9++8jHDqtZduEwH3F3IZ6aqqz6itwdWKInk= Date: Thu, 06 Aug 2020 23:21:03 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 071/163] mm: memcg/slab: deprecate memory.kmem.slabinfo Message-ID: <20200807062103.qn4835jCL%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 8092A18009952 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.004208, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: deprecate memory.kmem.slabinfo Deprecate memory.kmem.slabinfo. An empty file will be presented if corresponding config options are enabled. The interface is implementation dependent, isn't present in cgroup v2, and is generally useful only for core mm debugging purposes. In other words, it doesn't provide any value for the absolute majority of users. A drgn-based replacement can be found in tools/cgroup/memcg_slabinfo.py. It does support cgroup v1 and v2, mimics memory.kmem.slabinfo output and also allows to get any additional information without a need to recompile the kernel. If a drgn-based solution is too slow for a task, a bpf-based tracing tool can be used, which can easily keep track of all slab allocations belonging to a memory cgroup. Link: http://lkml.kernel.org/r/20200623174037.3951353-11-guro@fb.com Signed-off-by: Roman Gushchin Acked-by: Johannes Weiner Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- mm/memcontrol.c | 3 --- mm/slab_common.c | 31 ++++--------------------------- 2 files changed, 4 insertions(+), 30 deletions(-) --- a/mm/memcontrol.c~mm-memcg-slab-deprecate-memorykmemslabinfo +++ a/mm/memcontrol.c @@ -5114,9 +5114,6 @@ static struct cftype mem_cgroup_legacy_f (defined(CONFIG_SLAB) || defined(CONFIG_SLUB_DEBUG)) { .name = "kmem.slabinfo", - .seq_start = memcg_slab_start, - .seq_next = memcg_slab_next, - .seq_stop = memcg_slab_stop, .seq_show = memcg_slab_show, }, #endif --- a/mm/slab_common.c~mm-memcg-slab-deprecate-memorykmemslabinfo +++ a/mm/slab_common.c @@ -1561,35 +1561,12 @@ void dump_unreclaimable_slab(void) } #if defined(CONFIG_MEMCG_KMEM) -void *memcg_slab_start(struct seq_file *m, loff_t *pos) -{ - struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - - mutex_lock(&slab_mutex); - return seq_list_start(&memcg->kmem_caches, *pos); -} - -void *memcg_slab_next(struct seq_file *m, void *p, loff_t *pos) -{ - struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - - return seq_list_next(p, &memcg->kmem_caches, pos); -} - -void memcg_slab_stop(struct seq_file *m, void *p) -{ - mutex_unlock(&slab_mutex); -} - int memcg_slab_show(struct seq_file *m, void *p) { - struct kmem_cache *s = list_entry(p, struct kmem_cache, - memcg_params.kmem_caches_node); - struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - - if (p == memcg->kmem_caches.next) - print_slabinfo_header(m); - cache_show(s, m); + /* + * Deprecated. + * Please, take a look at tools/cgroup/slabinfo.py . + */ return 0; } #endif From patchwork Fri Aug 7 06:21:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704819 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F8C214B7 for ; Fri, 7 Aug 2020 06:21:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D0F5E2177B for ; Fri, 7 Aug 2020 06:21:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="0smpSvB0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D0F5E2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9032F8D0053; Fri, 7 Aug 2020 02:21:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8D8388D0026; Fri, 7 Aug 2020 02:21:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C6E18D0053; Fri, 7 Aug 2020 02:21:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0136.hostedemail.com [216.40.44.136]) by kanga.kvack.org (Postfix) with ESMTP id 651808D0026 for ; Fri, 7 Aug 2020 02:21:09 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2BD3E21F0 for ; Fri, 7 Aug 2020 06:21:09 +0000 (UTC) X-FDA: 77122775058.12.alarm35_5e11ad826fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id F319A18009B48 for ; Fri, 7 Aug 2020 06:21:08 +0000 (UTC) X-Spam-Summary: 1,0,0,028ac94f988fbb95,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2693:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6261:6653:6737:7576:8599:9025:9545:9592:10004:10913:11026:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12783:12986:14181:14721:14819:14849:14877:21080:21092:21433:21451:21627:21740:21939:21990:30012:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yr4uxticbp1jiot547b6zzhabg4opr8qp7s7d1f1a5usukrw6cq7mmxxabc9t.45eoubsd7k6dgrx5db347d6twodec7wb9a47ytx5jpd57345xduqnzxdye8sdp9.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk, SPF:fp,M X-HE-Tag: alarm35_5e11ad826fbe X-Filterd-Recvd-Size: 3608 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:08 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4F6DB2177B; Fri, 7 Aug 2020 06:21:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781267; bh=vSUkyCrxSJ4h5T+2raEVazvOOFp7gHy2SXsHVtILgd8=; h=Date:From:To:Subject:In-Reply-To:From; b=0smpSvB07S8vt+B4ycabX7NmPgSG6K8W0+XKS38hPao4ywxzZ4f+9sq7Em8QfqJjG Xhv+1FDM7Du+CoUfDqayeMlf/8JOoNV7bHBc3zuSOXAiQ4y6Fp78e4PQDlftbef3MG YNfrZZEDvhzwN58raBxbfCBY+pe+JsZ+MnSmjTDk= Date: Thu, 06 Aug 2020 23:21:06 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 072/163] mm: memcg/slab: move memcg_kmem_bypass() to memcontrol.h Message-ID: <20200807062106.N7mmmABPp%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: F319A18009B48 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000013, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: move memcg_kmem_bypass() to memcontrol.h To make the memcg_kmem_bypass() function available outside of the memcontrol.c, let's move it to memcontrol.h. The function is small and nicely fits into static inline sort of functions. It will be used from the slab code. Link: http://lkml.kernel.org/r/20200623174037.3951353-12-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Johannes Weiner Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 12 ++++++++++++ mm/memcontrol.c | 12 ------------ 2 files changed, 12 insertions(+), 12 deletions(-) --- a/include/linux/memcontrol.h~mm-memcg-slab-move-memcg_kmem_bypass-to-memcontrolh +++ a/include/linux/memcontrol.h @@ -1440,6 +1440,18 @@ static inline bool memcg_kmem_enabled(vo return static_branch_unlikely(&memcg_kmem_enabled_key); } +static inline bool memcg_kmem_bypass(void) +{ + if (in_interrupt()) + return true; + + /* Allow remote memcg charging in kthread contexts. */ + if ((!current->mm || (current->flags & PF_KTHREAD)) && + !current->active_memcg) + return true; + return false; +} + static inline int memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) { --- a/mm/memcontrol.c~mm-memcg-slab-move-memcg_kmem_bypass-to-memcontrolh +++ a/mm/memcontrol.c @@ -2945,18 +2945,6 @@ static void memcg_schedule_kmem_cache_cr queue_work(memcg_kmem_cache_wq, &cw->work); } -static inline bool memcg_kmem_bypass(void) -{ - if (in_interrupt()) - return true; - - /* Allow remote memcg charging in kthread contexts. */ - if ((!current->mm || (current->flags & PF_KTHREAD)) && - !current->active_memcg) - return true; - return false; -} - /** * memcg_kmem_get_cache: select the correct per-memcg cache for allocation * @cachep: the original global kmem cache From patchwork Fri Aug 7 06:21:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704821 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0467A138C for ; Fri, 7 Aug 2020 06:21:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8FAD4221E5 for ; Fri, 7 Aug 2020 06:21:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="UzjS4xoM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8FAD4221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1D4A58D0054; Fri, 7 Aug 2020 02:21:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 185CB8D0026; Fri, 7 Aug 2020 02:21:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 075C58D0054; Fri, 7 Aug 2020 02:21:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0254.hostedemail.com [216.40.44.254]) by kanga.kvack.org (Postfix) with ESMTP id CF5A88D0026 for ; Fri, 7 Aug 2020 02:21:13 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 901E8181AEF10 for ; Fri, 7 Aug 2020 06:21:13 +0000 (UTC) X-FDA: 77122775226.06.ants70_57100b726fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 5AC3F1003D0B7 for ; Fri, 7 Aug 2020 06:21:13 +0000 (UTC) X-Spam-Summary: 1,0,0,808125f5cd9498ad,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:69:327:355:379:421:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:2106:2194:2196:2198:2199:2200:2201:2393:2525:2538:2559:2563:2610:2682:2685:2689:2693:2731:2859:2892:2895:2897:2898:2901:2902:2903:2904:2914:2918:2922:2923:2924:2925:2926:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4605:5007:6261:6653:6737:7576:7875:7903:8599:8603:8660:8784:8957:9010:9025:9108:9121:9545:9592:10241:10394:10913:11026:11232:11914:12043:12048:12294:12295:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13148:13161:13229:13230:13255:13868:13972:14096:21063:21080:21222:21324:21433:21450:21451:21611:21627:21740:21789:21795:21796:21939:21972:21987:21990:30012:30029:30034:30036:30045:30046:30051:30054:30064:30070:30075:30080:30090,0, RBL:198. X-HE-Tag: ants70_57100b726fbe X-Filterd-Recvd-Size: 45105 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:12 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EE22122C9F; Fri, 7 Aug 2020 06:21:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781271; bh=KcopHGqUr2mGonBfoJwVhiMUn8mcJDhjMRwbQLawe/8=; h=Date:From:To:Subject:In-Reply-To:From; b=UzjS4xoM9hNBLs18UqusLVqFD6C7qCo+M7wIxDumkf/lIUmmjNvZM6FLk+V2UR3eO YY1amu6cmaHFw5cTMJSkR8SobY0vkF5Z6ow+tm1HpmdTFxumuXKRlgIwSp5IIy0mBr xKr4LeKg9886qtFS7KeMi72/YTi2L/Sv2OW3LHzM= Date: Thu, 06 Aug 2020 23:21:10 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 073/163] mm: memcg/slab: use a single set of kmem_caches for all accounted allocations Message-ID: <20200807062110.OsXinNSO_%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 5AC3F1003D0B7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: use a single set of kmem_caches for all accounted allocations This is fairly big but mostly red patch, which makes all accounted slab allocations use a single set of kmem_caches instead of creating a separate set for each memory cgroup. Because the number of non-root kmem_caches is now capped by the number of root kmem_caches, there is no need to shrink or destroy them prematurely. They can be perfectly destroyed together with their root counterparts. This allows to dramatically simplify the management of non-root kmem_caches and delete a ton of code. This patch performs the following changes: 1) introduces memcg_params.memcg_cache pointer to represent the kmem_cache which will be used for all non-root allocations 2) reuses the existing memcg kmem_cache creation mechanism to create memcg kmem_cache on the first allocation attempt 3) memcg kmem_caches are named -memcg, e.g. dentry-memcg 4) simplifies memcg_kmem_get_cache() to just return memcg kmem_cache or schedule it's creation and return the root cache 5) removes almost all non-root kmem_cache management code (separate refcounter, reparenting, shrinking, etc) 6) makes slab debugfs to display root_mem_cgroup css id and never show :dead and :deact flags in the memcg_slabinfo attribute. Following patches in the series will simplify the kmem_cache creation. Link: http://lkml.kernel.org/r/20200623174037.3951353-13-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Johannes Weiner Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 5 include/linux/slab.h | 5 mm/memcontrol.c | 165 ++---------- mm/slab.c | 16 - mm/slab.h | 146 +++-------- mm/slab_common.c | 461 +++-------------------------------- mm/slub.c | 38 -- 7 files changed, 136 insertions(+), 700 deletions(-) --- a/include/linux/memcontrol.h~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-accounted-allocations +++ a/include/linux/memcontrol.h @@ -317,7 +317,6 @@ struct mem_cgroup { /* Index in the kmem_cache->memcg_params.memcg_caches array */ int kmemcg_id; enum memcg_kmem_state kmem_state; - struct list_head kmem_caches; struct obj_cgroup __rcu *objcg; struct list_head objcg_list; /* list of inherited objcgs */ #endif @@ -1404,9 +1403,7 @@ static inline void memcg_set_shrinker_bi } #endif -struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep, - struct obj_cgroup **objcgp); -void memcg_kmem_put_cache(struct kmem_cache *cachep); +struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep); #ifdef CONFIG_MEMCG_KMEM int __memcg_kmem_charge(struct mem_cgroup *memcg, gfp_t gfp, --- a/include/linux/slab.h~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-accounted-allocations +++ a/include/linux/slab.h @@ -155,8 +155,7 @@ struct kmem_cache *kmem_cache_create_use void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); -void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); -void memcg_deactivate_kmem_caches(struct mem_cgroup *, struct mem_cgroup *); +void memcg_create_kmem_cache(struct kmem_cache *cachep); /* * Please use this macro to create slab caches. Simply specify the @@ -580,8 +579,6 @@ static __always_inline void *kmalloc_nod return __kmalloc_node(size, flags, node); } -int memcg_update_all_caches(int num_memcgs); - /** * kmalloc_array - allocate memory for an array. * @n: number of elements. --- a/mm/memcontrol.c~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-accounted-allocations +++ a/mm/memcontrol.c @@ -350,7 +350,7 @@ static void memcg_reparent_objcgs(struct } /* - * This will be the memcg's index in each cache's ->memcg_params.memcg_caches. + * This will be used as a shrinker list's index. * The main reason for not using cgroup id for this: * this works better in sparse environments, where we have a lot of memcgs, * but only a few kmem-limited. Or also, if we have, for instance, 200 @@ -569,20 +569,16 @@ ino_t page_cgroup_ino(struct page *page) unsigned long ino = 0; rcu_read_lock(); - if (PageSlab(page) && !PageTail(page)) { - memcg = memcg_from_slab_page(page); - } else { - memcg = page->mem_cgroup; + memcg = page->mem_cgroup; - /* - * The lowest bit set means that memcg isn't a valid - * memcg pointer, but a obj_cgroups pointer. - * In this case the page is shared and doesn't belong - * to any specific memory cgroup. - */ - if ((unsigned long) memcg & 0x1UL) - memcg = NULL; - } + /* + * The lowest bit set means that memcg isn't a valid + * memcg pointer, but a obj_cgroups pointer. + * In this case the page is shared and doesn't belong + * to any specific memory cgroup. + */ + if ((unsigned long) memcg & 0x1UL) + memcg = NULL; while (memcg && !(memcg->css.flags & CSS_ONLINE)) memcg = parent_mem_cgroup(memcg); @@ -2822,12 +2818,18 @@ struct mem_cgroup *mem_cgroup_from_obj(v page = virt_to_head_page(p); /* - * Slab pages don't have page->mem_cgroup set because corresponding - * kmem caches can be reparented during the lifetime. That's why - * memcg_from_slab_page() should be used instead. - */ - if (PageSlab(page)) - return memcg_from_slab_page(page); + * Slab objects are accounted individually, not per-page. + * Memcg membership data for each individual object is saved in + * the page->obj_cgroups. + */ + if (page_has_obj_cgroups(page)) { + struct obj_cgroup *objcg; + unsigned int off; + + off = obj_to_index(page->slab_cache, page, p); + objcg = page_obj_cgroups(page)[off]; + return obj_cgroup_memcg(objcg); + } /* All other pages use page->mem_cgroup */ return page->mem_cgroup; @@ -2882,9 +2884,7 @@ static int memcg_alloc_cache_id(void) else if (size > MEMCG_CACHES_MAX_SIZE) size = MEMCG_CACHES_MAX_SIZE; - err = memcg_update_all_caches(size); - if (!err) - err = memcg_update_all_list_lrus(size); + err = memcg_update_all_list_lrus(size); if (!err) memcg_nr_cache_ids = size; @@ -2903,7 +2903,6 @@ static void memcg_free_cache_id(int id) } struct memcg_kmem_cache_create_work { - struct mem_cgroup *memcg; struct kmem_cache *cachep; struct work_struct work; }; @@ -2912,33 +2911,24 @@ static void memcg_kmem_cache_create_func { struct memcg_kmem_cache_create_work *cw = container_of(w, struct memcg_kmem_cache_create_work, work); - struct mem_cgroup *memcg = cw->memcg; struct kmem_cache *cachep = cw->cachep; - memcg_create_kmem_cache(memcg, cachep); + memcg_create_kmem_cache(cachep); - css_put(&memcg->css); kfree(cw); } /* * Enqueue the creation of a per-memcg kmem_cache. */ -static void memcg_schedule_kmem_cache_create(struct mem_cgroup *memcg, - struct kmem_cache *cachep) +static void memcg_schedule_kmem_cache_create(struct kmem_cache *cachep) { struct memcg_kmem_cache_create_work *cw; - if (!css_tryget_online(&memcg->css)) - return; - cw = kmalloc(sizeof(*cw), GFP_NOWAIT | __GFP_NOWARN); - if (!cw) { - css_put(&memcg->css); + if (!cw) return; - } - cw->memcg = memcg; cw->cachep = cachep; INIT_WORK(&cw->work, memcg_kmem_cache_create_func); @@ -2946,102 +2936,26 @@ static void memcg_schedule_kmem_cache_cr } /** - * memcg_kmem_get_cache: select the correct per-memcg cache for allocation + * memcg_kmem_get_cache: select memcg or root cache for allocation * @cachep: the original global kmem cache * * Return the kmem_cache we're supposed to use for a slab allocation. - * We try to use the current memcg's version of the cache. * * If the cache does not exist yet, if we are the first user of it, we * create it asynchronously in a workqueue and let the current allocation * go through with the original cache. - * - * This function takes a reference to the cache it returns to assure it - * won't get destroyed while we are working with it. Once the caller is - * done with it, memcg_kmem_put_cache() must be called to release the - * reference. */ -struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep, - struct obj_cgroup **objcgp) +struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) { - struct mem_cgroup *memcg; struct kmem_cache *memcg_cachep; - struct memcg_cache_array *arr; - int kmemcg_id; - - VM_BUG_ON(!is_root_cache(cachep)); - if (memcg_kmem_bypass()) + memcg_cachep = READ_ONCE(cachep->memcg_params.memcg_cache); + if (unlikely(!memcg_cachep)) { + memcg_schedule_kmem_cache_create(cachep); return cachep; - - rcu_read_lock(); - - if (unlikely(current->active_memcg)) - memcg = current->active_memcg; - else - memcg = mem_cgroup_from_task(current); - - if (!memcg || memcg == root_mem_cgroup) - goto out_unlock; - - kmemcg_id = READ_ONCE(memcg->kmemcg_id); - if (kmemcg_id < 0) - goto out_unlock; - - arr = rcu_dereference(cachep->memcg_params.memcg_caches); - - /* - * Make sure we will access the up-to-date value. The code updating - * memcg_caches issues a write barrier to match the data dependency - * barrier inside READ_ONCE() (see memcg_create_kmem_cache()). - */ - memcg_cachep = READ_ONCE(arr->entries[kmemcg_id]); - - /* - * If we are in a safe context (can wait, and not in interrupt - * context), we could be be predictable and return right away. - * This would guarantee that the allocation being performed - * already belongs in the new cache. - * - * However, there are some clashes that can arrive from locking. - * For instance, because we acquire the slab_mutex while doing - * memcg_create_kmem_cache, this means no further allocation - * could happen with the slab_mutex held. So it's better to - * defer everything. - * - * If the memcg is dying or memcg_cache is about to be released, - * don't bother creating new kmem_caches. Because memcg_cachep - * is ZEROed as the fist step of kmem offlining, we don't need - * percpu_ref_tryget_live() here. css_tryget_online() check in - * memcg_schedule_kmem_cache_create() will prevent us from - * creation of a new kmem_cache. - */ - if (unlikely(!memcg_cachep)) - memcg_schedule_kmem_cache_create(memcg, cachep); - else if (percpu_ref_tryget(&memcg_cachep->memcg_params.refcnt)) { - struct obj_cgroup *objcg = rcu_dereference(memcg->objcg); - - if (!objcg || !obj_cgroup_tryget(objcg)) { - percpu_ref_put(&memcg_cachep->memcg_params.refcnt); - goto out_unlock; - } - - *objcgp = objcg; - cachep = memcg_cachep; } -out_unlock: - rcu_read_unlock(); - return cachep; -} -/** - * memcg_kmem_put_cache: drop reference taken by memcg_kmem_get_cache - * @cachep: the cache returned by memcg_kmem_get_cache - */ -void memcg_kmem_put_cache(struct kmem_cache *cachep) -{ - if (!is_root_cache(cachep)) - percpu_ref_put(&cachep->memcg_params.refcnt); + return memcg_cachep; } /** @@ -3731,7 +3645,6 @@ static int memcg_online_kmem(struct mem_ */ memcg->kmemcg_id = memcg_id; memcg->kmem_state = KMEM_ONLINE; - INIT_LIST_HEAD(&memcg->kmem_caches); return 0; } @@ -3744,22 +3657,13 @@ static void memcg_offline_kmem(struct me if (memcg->kmem_state != KMEM_ONLINE) return; - /* - * Clear the online state before clearing memcg_caches array - * entries. The slab_mutex in memcg_deactivate_kmem_caches() - * guarantees that no cache will be created for this cgroup - * after we are done (see memcg_create_kmem_cache()). - */ + memcg->kmem_state = KMEM_ALLOCATED; parent = parent_mem_cgroup(memcg); if (!parent) parent = root_mem_cgroup; - /* - * Deactivate and reparent kmem_caches and objcgs. - */ - memcg_deactivate_kmem_caches(memcg, parent); memcg_reparent_objcgs(memcg, parent); kmemcg_id = memcg->kmemcg_id; @@ -5384,9 +5288,6 @@ mem_cgroup_css_alloc(struct cgroup_subsy /* The following stuff does not apply to the root */ if (!parent) { -#ifdef CONFIG_MEMCG_KMEM - INIT_LIST_HEAD(&memcg->kmem_caches); -#endif root_mem_cgroup = memcg; return &memcg->css; } --- a/mm/slab.c~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-accounted-allocations +++ a/mm/slab.c @@ -1249,7 +1249,7 @@ void __init kmem_cache_init(void) nr_node_ids * sizeof(struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0); list_add(&kmem_cache->list, &slab_caches); - memcg_link_cache(kmem_cache, NULL); + memcg_link_cache(kmem_cache); slab_state = PARTIAL; /* @@ -2253,17 +2253,6 @@ int __kmem_cache_shrink(struct kmem_cach return (ret ? 1 : 0); } -#ifdef CONFIG_MEMCG -void __kmemcg_cache_deactivate(struct kmem_cache *cachep) -{ - __kmem_cache_shrink(cachep); -} - -void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) -{ -} -#endif - int __kmem_cache_shutdown(struct kmem_cache *cachep) { return __kmem_cache_shrink(cachep); @@ -3872,7 +3861,8 @@ static int do_tune_cpucache(struct kmem_ return ret; lockdep_assert_held(&slab_mutex); - for_each_memcg_cache(c, cachep) { + c = memcg_cache(cachep); + if (c) { /* return value determined by the root cache only */ __do_tune_cpucache(c, limit, batchcount, shared, gfp); } --- a/mm/slab_common.c~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-accounted-allocations +++ a/mm/slab_common.c @@ -133,141 +133,36 @@ int __kmem_cache_alloc_bulk(struct kmem_ #ifdef CONFIG_MEMCG_KMEM LIST_HEAD(slab_root_caches); -static DEFINE_SPINLOCK(memcg_kmem_wq_lock); - -static void kmemcg_cache_shutdown(struct percpu_ref *percpu_ref); void slab_init_memcg_params(struct kmem_cache *s) { s->memcg_params.root_cache = NULL; - RCU_INIT_POINTER(s->memcg_params.memcg_caches, NULL); - INIT_LIST_HEAD(&s->memcg_params.children); - s->memcg_params.dying = false; -} - -static int init_memcg_params(struct kmem_cache *s, - struct kmem_cache *root_cache) -{ - struct memcg_cache_array *arr; - - if (root_cache) { - int ret = percpu_ref_init(&s->memcg_params.refcnt, - kmemcg_cache_shutdown, - 0, GFP_KERNEL); - if (ret) - return ret; - - s->memcg_params.root_cache = root_cache; - INIT_LIST_HEAD(&s->memcg_params.children_node); - INIT_LIST_HEAD(&s->memcg_params.kmem_caches_node); - return 0; - } - - slab_init_memcg_params(s); - - if (!memcg_nr_cache_ids) - return 0; - - arr = kvzalloc(sizeof(struct memcg_cache_array) + - memcg_nr_cache_ids * sizeof(void *), - GFP_KERNEL); - if (!arr) - return -ENOMEM; - - RCU_INIT_POINTER(s->memcg_params.memcg_caches, arr); - return 0; -} - -static void destroy_memcg_params(struct kmem_cache *s) -{ - if (is_root_cache(s)) { - kvfree(rcu_access_pointer(s->memcg_params.memcg_caches)); - } else { - mem_cgroup_put(s->memcg_params.memcg); - WRITE_ONCE(s->memcg_params.memcg, NULL); - percpu_ref_exit(&s->memcg_params.refcnt); - } + s->memcg_params.memcg_cache = NULL; } -static void free_memcg_params(struct rcu_head *rcu) +static void init_memcg_params(struct kmem_cache *s, + struct kmem_cache *root_cache) { - struct memcg_cache_array *old; - - old = container_of(rcu, struct memcg_cache_array, rcu); - kvfree(old); -} - -static int update_memcg_params(struct kmem_cache *s, int new_array_size) -{ - struct memcg_cache_array *old, *new; - - new = kvzalloc(sizeof(struct memcg_cache_array) + - new_array_size * sizeof(void *), GFP_KERNEL); - if (!new) - return -ENOMEM; - - old = rcu_dereference_protected(s->memcg_params.memcg_caches, - lockdep_is_held(&slab_mutex)); - if (old) - memcpy(new->entries, old->entries, - memcg_nr_cache_ids * sizeof(void *)); - - rcu_assign_pointer(s->memcg_params.memcg_caches, new); - if (old) - call_rcu(&old->rcu, free_memcg_params); - return 0; -} - -int memcg_update_all_caches(int num_memcgs) -{ - struct kmem_cache *s; - int ret = 0; - - mutex_lock(&slab_mutex); - list_for_each_entry(s, &slab_root_caches, root_caches_node) { - ret = update_memcg_params(s, num_memcgs); - /* - * Instead of freeing the memory, we'll just leave the caches - * up to this point in an updated state. - */ - if (ret) - break; - } - mutex_unlock(&slab_mutex); - return ret; + if (root_cache) + s->memcg_params.root_cache = root_cache; + else + slab_init_memcg_params(s); } -void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg) +void memcg_link_cache(struct kmem_cache *s) { - if (is_root_cache(s)) { + if (is_root_cache(s)) list_add(&s->root_caches_node, &slab_root_caches); - } else { - css_get(&memcg->css); - s->memcg_params.memcg = memcg; - list_add(&s->memcg_params.children_node, - &s->memcg_params.root_cache->memcg_params.children); - list_add(&s->memcg_params.kmem_caches_node, - &s->memcg_params.memcg->kmem_caches); - } } static void memcg_unlink_cache(struct kmem_cache *s) { - if (is_root_cache(s)) { + if (is_root_cache(s)) list_del(&s->root_caches_node); - } else { - list_del(&s->memcg_params.children_node); - list_del(&s->memcg_params.kmem_caches_node); - } } #else -static inline int init_memcg_params(struct kmem_cache *s, - struct kmem_cache *root_cache) -{ - return 0; -} - -static inline void destroy_memcg_params(struct kmem_cache *s) +static inline void init_memcg_params(struct kmem_cache *s, + struct kmem_cache *root_cache) { } @@ -328,14 +223,6 @@ int slab_unmergeable(struct kmem_cache * if (s->refcount < 0) return 1; -#ifdef CONFIG_MEMCG_KMEM - /* - * Skip the dying kmem_cache. - */ - if (s->memcg_params.dying) - return 1; -#endif - return 0; } @@ -390,7 +277,7 @@ static struct kmem_cache *create_cache(c unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset, unsigned int usersize, void (*ctor)(void *), - struct mem_cgroup *memcg, struct kmem_cache *root_cache) + struct kmem_cache *root_cache) { struct kmem_cache *s; int err; @@ -410,24 +297,20 @@ static struct kmem_cache *create_cache(c s->useroffset = useroffset; s->usersize = usersize; - err = init_memcg_params(s, root_cache); - if (err) - goto out_free_cache; - + init_memcg_params(s, root_cache); err = __kmem_cache_create(s, flags); if (err) goto out_free_cache; s->refcount = 1; list_add(&s->list, &slab_caches); - memcg_link_cache(s, memcg); + memcg_link_cache(s); out: if (err) return ERR_PTR(err); return s; out_free_cache: - destroy_memcg_params(s); kmem_cache_free(kmem_cache, s); goto out; } @@ -514,7 +397,7 @@ kmem_cache_create_usercopy(const char *n s = create_cache(cache_name, size, calculate_alignment(flags, align, size), - flags, useroffset, usersize, ctor, NULL, NULL); + flags, useroffset, usersize, ctor, NULL); if (IS_ERR(s)) { err = PTR_ERR(s); kfree_const(cache_name); @@ -639,51 +522,27 @@ static int shutdown_cache(struct kmem_ca #ifdef CONFIG_MEMCG_KMEM /* - * memcg_create_kmem_cache - Create a cache for a memory cgroup. - * @memcg: The memory cgroup the new cache is for. + * memcg_create_kmem_cache - Create a cache for non-root memory cgroups. * @root_cache: The parent of the new cache. * * This function attempts to create a kmem cache that will serve allocation - * requests going from @memcg to @root_cache. The new cache inherits properties - * from its parent. + * requests going all non-root memory cgroups to @root_cache. The new cache + * inherits properties from its parent. */ -void memcg_create_kmem_cache(struct mem_cgroup *memcg, - struct kmem_cache *root_cache) +void memcg_create_kmem_cache(struct kmem_cache *root_cache) { - static char memcg_name_buf[NAME_MAX + 1]; /* protected by slab_mutex */ - struct cgroup_subsys_state *css = &memcg->css; - struct memcg_cache_array *arr; struct kmem_cache *s = NULL; char *cache_name; - int idx; get_online_cpus(); get_online_mems(); mutex_lock(&slab_mutex); - /* - * The memory cgroup could have been offlined while the cache - * creation work was pending. - */ - if (memcg->kmem_state != KMEM_ONLINE) + if (root_cache->memcg_params.memcg_cache) goto out_unlock; - idx = memcg_cache_id(memcg); - arr = rcu_dereference_protected(root_cache->memcg_params.memcg_caches, - lockdep_is_held(&slab_mutex)); - - /* - * Since per-memcg caches are created asynchronously on first - * allocation (see memcg_kmem_get_cache()), several threads can try to - * create the same cache, but only one of them may succeed. - */ - if (arr->entries[idx]) - goto out_unlock; - - cgroup_name(css->cgroup, memcg_name_buf, sizeof(memcg_name_buf)); - cache_name = kasprintf(GFP_KERNEL, "%s(%llu:%s)", root_cache->name, - css->serial_nr, memcg_name_buf); + cache_name = kasprintf(GFP_KERNEL, "%s-memcg", root_cache->name); if (!cache_name) goto out_unlock; @@ -691,7 +550,7 @@ void memcg_create_kmem_cache(struct mem_ root_cache->align, root_cache->flags & CACHE_CREATE_MASK, root_cache->useroffset, root_cache->usersize, - root_cache->ctor, memcg, root_cache); + root_cache->ctor, root_cache); /* * If we could not create a memcg cache, do not complain, because * that's not critical at all as we can always proceed with the root @@ -708,7 +567,7 @@ void memcg_create_kmem_cache(struct mem_ * initialized. */ smp_wmb(); - arr->entries[idx] = s; + root_cache->memcg_params.memcg_cache = s; out_unlock: mutex_unlock(&slab_mutex); @@ -717,231 +576,40 @@ out_unlock: put_online_cpus(); } -static void kmemcg_workfn(struct work_struct *work) -{ - struct kmem_cache *s = container_of(work, struct kmem_cache, - memcg_params.work); - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); - s->memcg_params.work_fn(s); - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); -} - -static void kmemcg_rcufn(struct rcu_head *head) -{ - struct kmem_cache *s = container_of(head, struct kmem_cache, - memcg_params.rcu_head); - - /* - * We need to grab blocking locks. Bounce to ->work. The - * work item shares the space with the RCU head and can't be - * initialized earlier. - */ - INIT_WORK(&s->memcg_params.work, kmemcg_workfn); - queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); -} - -static void kmemcg_cache_shutdown_fn(struct kmem_cache *s) -{ - WARN_ON(shutdown_cache(s)); -} - -static void kmemcg_cache_shutdown(struct percpu_ref *percpu_ref) -{ - struct kmem_cache *s = container_of(percpu_ref, struct kmem_cache, - memcg_params.refcnt); - unsigned long flags; - - spin_lock_irqsave(&memcg_kmem_wq_lock, flags); - if (s->memcg_params.root_cache->memcg_params.dying) - goto unlock; - - s->memcg_params.work_fn = kmemcg_cache_shutdown_fn; - INIT_WORK(&s->memcg_params.work, kmemcg_workfn); - queue_work(memcg_kmem_cache_wq, &s->memcg_params.work); - -unlock: - spin_unlock_irqrestore(&memcg_kmem_wq_lock, flags); -} - -static void kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) -{ - __kmemcg_cache_deactivate_after_rcu(s); - percpu_ref_kill(&s->memcg_params.refcnt); -} - -static void kmemcg_cache_deactivate(struct kmem_cache *s) -{ - if (WARN_ON_ONCE(is_root_cache(s))) - return; - - __kmemcg_cache_deactivate(s); - s->flags |= SLAB_DEACTIVATED; - - /* - * memcg_kmem_wq_lock is used to synchronize memcg_params.dying - * flag and make sure that no new kmem_cache deactivation tasks - * are queued (see flush_memcg_workqueue() ). - */ - spin_lock_irq(&memcg_kmem_wq_lock); - if (s->memcg_params.root_cache->memcg_params.dying) - goto unlock; - - s->memcg_params.work_fn = kmemcg_cache_deactivate_after_rcu; - call_rcu(&s->memcg_params.rcu_head, kmemcg_rcufn); -unlock: - spin_unlock_irq(&memcg_kmem_wq_lock); -} - -void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg, - struct mem_cgroup *parent) -{ - int idx; - struct memcg_cache_array *arr; - struct kmem_cache *s, *c; - unsigned int nr_reparented; - - idx = memcg_cache_id(memcg); - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); - list_for_each_entry(s, &slab_root_caches, root_caches_node) { - arr = rcu_dereference_protected(s->memcg_params.memcg_caches, - lockdep_is_held(&slab_mutex)); - c = arr->entries[idx]; - if (!c) - continue; - - kmemcg_cache_deactivate(c); - arr->entries[idx] = NULL; - } - nr_reparented = 0; - list_for_each_entry(s, &memcg->kmem_caches, - memcg_params.kmem_caches_node) { - WRITE_ONCE(s->memcg_params.memcg, parent); - css_put(&memcg->css); - nr_reparented++; - } - if (nr_reparented) { - list_splice_init(&memcg->kmem_caches, - &parent->kmem_caches); - css_get_many(&parent->css, nr_reparented); - } - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); -} - static int shutdown_memcg_caches(struct kmem_cache *s) { - struct memcg_cache_array *arr; - struct kmem_cache *c, *c2; - LIST_HEAD(busy); - int i; - BUG_ON(!is_root_cache(s)); - /* - * First, shutdown active caches, i.e. caches that belong to online - * memory cgroups. - */ - arr = rcu_dereference_protected(s->memcg_params.memcg_caches, - lockdep_is_held(&slab_mutex)); - for_each_memcg_cache_index(i) { - c = arr->entries[i]; - if (!c) - continue; - if (shutdown_cache(c)) - /* - * The cache still has objects. Move it to a temporary - * list so as not to try to destroy it for a second - * time while iterating over inactive caches below. - */ - list_move(&c->memcg_params.children_node, &busy); - else - /* - * The cache is empty and will be destroyed soon. Clear - * the pointer to it in the memcg_caches array so that - * it will never be accessed even if the root cache - * stays alive. - */ - arr->entries[i] = NULL; - } - - /* - * Second, shutdown all caches left from memory cgroups that are now - * offline. - */ - list_for_each_entry_safe(c, c2, &s->memcg_params.children, - memcg_params.children_node) - shutdown_cache(c); - - list_splice(&busy, &s->memcg_params.children); + if (s->memcg_params.memcg_cache) + WARN_ON(shutdown_cache(s->memcg_params.memcg_cache)); - /* - * A cache being destroyed must be empty. In particular, this means - * that all per memcg caches attached to it must be empty too. - */ - if (!list_empty(&s->memcg_params.children)) - return -EBUSY; return 0; } -static void memcg_set_kmem_cache_dying(struct kmem_cache *s) -{ - spin_lock_irq(&memcg_kmem_wq_lock); - s->memcg_params.dying = true; - spin_unlock_irq(&memcg_kmem_wq_lock); -} - static void flush_memcg_workqueue(struct kmem_cache *s) { /* - * SLAB and SLUB deactivate the kmem_caches through call_rcu. Make - * sure all registered rcu callbacks have been invoked. - */ - rcu_barrier(); - - /* * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB * deactivates the memcg kmem_caches through workqueue. Make sure all * previous workitems on workqueue are processed. */ if (likely(memcg_kmem_cache_wq)) flush_workqueue(memcg_kmem_cache_wq); - - /* - * If we're racing with children kmem_cache deactivation, it might - * take another rcu grace period to complete their destruction. - * At this moment the corresponding percpu_ref_kill() call should be - * done, but it might take another rcu grace period to complete - * switching to the atomic mode. - * Please, note that we check without grabbing the slab_mutex. It's safe - * because at this moment the children list can't grow. - */ - if (!list_empty(&s->memcg_params.children)) - rcu_barrier(); } #else static inline int shutdown_memcg_caches(struct kmem_cache *s) { return 0; } + +static inline void flush_memcg_workqueue(struct kmem_cache *s) +{ +} #endif /* CONFIG_MEMCG_KMEM */ void slab_kmem_cache_release(struct kmem_cache *s) { __kmem_cache_release(s); - destroy_memcg_params(s); kfree_const(s->name); kmem_cache_free(kmem_cache, s); } @@ -953,6 +621,8 @@ void kmem_cache_destroy(struct kmem_cach if (unlikely(!s)) return; + flush_memcg_workqueue(s); + get_online_cpus(); get_online_mems(); @@ -962,22 +632,6 @@ void kmem_cache_destroy(struct kmem_cach if (s->refcount) goto out_unlock; -#ifdef CONFIG_MEMCG_KMEM - memcg_set_kmem_cache_dying(s); - - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); - - flush_memcg_workqueue(s); - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); -#endif - err = shutdown_memcg_caches(s); if (!err) err = shutdown_cache(s); @@ -1019,7 +673,7 @@ int kmem_cache_shrink(struct kmem_cache EXPORT_SYMBOL(kmem_cache_shrink); /** - * kmem_cache_shrink_all - shrink a cache and all memcg caches for root cache + * kmem_cache_shrink_all - shrink root and memcg caches * @s: The cache pointer */ void kmem_cache_shrink_all(struct kmem_cache *s) @@ -1036,21 +690,11 @@ void kmem_cache_shrink_all(struct kmem_c kasan_cache_shrink(s); __kmem_cache_shrink(s); - /* - * We have to take the slab_mutex to protect from the memcg list - * modification. - */ - mutex_lock(&slab_mutex); - for_each_memcg_cache(c, s) { - /* - * Don't need to shrink deactivated memcg caches. - */ - if (s->flags & SLAB_DEACTIVATED) - continue; + c = memcg_cache(s); + if (c) { kasan_cache_shrink(c); __kmem_cache_shrink(c); } - mutex_unlock(&slab_mutex); put_online_mems(); put_online_cpus(); } @@ -1105,7 +749,7 @@ struct kmem_cache *__init create_kmalloc create_boot_cache(s, name, size, flags, useroffset, usersize); list_add(&s->list, &slab_caches); - memcg_link_cache(s, NULL); + memcg_link_cache(s); s->refcount = 1; return s; } @@ -1483,7 +1127,8 @@ memcg_accumulate_slabinfo(struct kmem_ca if (!is_root_cache(s)) return; - for_each_memcg_cache(c, s) { + c = memcg_cache(s); + if (c) { memset(&sinfo, 0, sizeof(sinfo)); get_slabinfo(c, &sinfo); @@ -1614,7 +1259,7 @@ module_init(slab_proc_init); #if defined(CONFIG_DEBUG_FS) && defined(CONFIG_MEMCG_KMEM) /* - * Display information about kmem caches that have child memcg caches. + * Display information about kmem caches that have memcg cache. */ static int memcg_slabinfo_show(struct seq_file *m, void *unused) { @@ -1626,9 +1271,9 @@ static int memcg_slabinfo_show(struct se seq_puts(m, " \n"); list_for_each_entry(s, &slab_root_caches, root_caches_node) { /* - * Skip kmem caches that don't have any memcg children. + * Skip kmem caches that don't have the memcg cache. */ - if (list_empty(&s->memcg_params.children)) + if (!s->memcg_params.memcg_cache) continue; memset(&sinfo, 0, sizeof(sinfo)); @@ -1637,23 +1282,13 @@ static int memcg_slabinfo_show(struct se cache_name(s), sinfo.active_objs, sinfo.num_objs, sinfo.active_slabs, sinfo.num_slabs); - for_each_memcg_cache(c, s) { - struct cgroup_subsys_state *css; - char *status = ""; - - css = &c->memcg_params.memcg->css; - if (!(css->flags & CSS_ONLINE)) - status = ":dead"; - else if (c->flags & SLAB_DEACTIVATED) - status = ":deact"; - - memset(&sinfo, 0, sizeof(sinfo)); - get_slabinfo(c, &sinfo); - seq_printf(m, "%-17s %4d%-6s %6lu %6lu %6lu %6lu\n", - cache_name(c), css->id, status, - sinfo.active_objs, sinfo.num_objs, - sinfo.active_slabs, sinfo.num_slabs); - } + c = s->memcg_params.memcg_cache; + memset(&sinfo, 0, sizeof(sinfo)); + get_slabinfo(c, &sinfo); + seq_printf(m, "%-17s %4d %6lu %6lu %6lu %6lu\n", + cache_name(c), root_mem_cgroup->css.id, + sinfo.active_objs, sinfo.num_objs, + sinfo.active_slabs, sinfo.num_slabs); } mutex_unlock(&slab_mutex); return 0; --- a/mm/slab.h~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-accounted-allocations +++ a/mm/slab.h @@ -32,66 +32,25 @@ struct kmem_cache { #else /* !CONFIG_SLOB */ -struct memcg_cache_array { - struct rcu_head rcu; - struct kmem_cache *entries[0]; -}; - /* * This is the main placeholder for memcg-related information in kmem caches. - * Both the root cache and the child caches will have it. For the root cache, - * this will hold a dynamically allocated array large enough to hold - * information about the currently limited memcgs in the system. To allow the - * array to be accessed without taking any locks, on relocation we free the old - * version only after a grace period. - * - * Root and child caches hold different metadata. + * Both the root cache and the child cache will have it. Some fields are used + * in both cases, other are specific to root caches. * * @root_cache: Common to root and child caches. NULL for root, pointer to * the root cache for children. * * The following fields are specific to root caches. * - * @memcg_caches: kmemcg ID indexed table of child caches. This table is - * used to index child cachces during allocation and cleared - * early during shutdown. - * - * @root_caches_node: List node for slab_root_caches list. - * - * @children: List of all child caches. While the child caches are also - * reachable through @memcg_caches, a child cache remains on - * this list until it is actually destroyed. - * - * The following fields are specific to child caches. - * - * @memcg: Pointer to the memcg this cache belongs to. - * - * @children_node: List node for @root_cache->children list. - * - * @kmem_caches_node: List node for @memcg->kmem_caches list. + * @memcg_cache: pointer to memcg kmem cache, used by all non-root memory + * cgroups. + * @root_caches_node: list node for slab_root_caches list. */ struct memcg_cache_params { struct kmem_cache *root_cache; - union { - struct { - struct memcg_cache_array __rcu *memcg_caches; - struct list_head __root_caches_node; - struct list_head children; - bool dying; - }; - struct { - struct mem_cgroup *memcg; - struct list_head children_node; - struct list_head kmem_caches_node; - struct percpu_ref refcnt; - - void (*work_fn)(struct kmem_cache *); - union { - struct rcu_head rcu_head; - struct work_struct work; - }; - }; - }; + + struct kmem_cache *memcg_cache; + struct list_head __root_caches_node; }; #endif /* CONFIG_SLOB */ @@ -236,8 +195,6 @@ bool __kmem_cache_empty(struct kmem_cach int __kmem_cache_shutdown(struct kmem_cache *); void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); -void __kmemcg_cache_deactivate(struct kmem_cache *s); -void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s); void slab_kmem_cache_release(struct kmem_cache *); void kmem_cache_shrink_all(struct kmem_cache *s); @@ -311,14 +268,6 @@ static inline bool kmem_cache_debug_flag extern struct list_head slab_root_caches; #define root_caches_node memcg_params.__root_caches_node -/* - * Iterate over all memcg caches of the given root cache. The caller must hold - * slab_mutex. - */ -#define for_each_memcg_cache(iter, root) \ - list_for_each_entry(iter, &(root)->memcg_params.children, \ - memcg_params.children_node) - static inline bool is_root_cache(struct kmem_cache *s) { return !s->memcg_params.root_cache; @@ -349,6 +298,13 @@ static inline struct kmem_cache *memcg_r return s->memcg_params.root_cache; } +static inline struct kmem_cache *memcg_cache(struct kmem_cache *s) +{ + if (is_root_cache(s)) + return s->memcg_params.memcg_cache; + return NULL; +} + static inline struct obj_cgroup **page_obj_cgroups(struct page *page) { /* @@ -361,25 +317,9 @@ static inline struct obj_cgroup **page_o ((unsigned long)page->obj_cgroups & ~0x1UL); } -/* - * Expects a pointer to a slab page. Please note, that PageSlab() check - * isn't sufficient, as it returns true also for tail compound slab pages, - * which do not have slab_cache pointer set. - * So this function assumes that the page can pass PageSlab() && !PageTail() - * check. - * - * The kmem_cache can be reparented asynchronously. The caller must ensure - * the memcg lifetime, e.g. by taking rcu_read_lock() or cgroup_mutex. - */ -static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +static inline bool page_has_obj_cgroups(struct page *page) { - struct kmem_cache *s; - - s = READ_ONCE(page->slab_cache); - if (s && !is_root_cache(s)) - return READ_ONCE(s->memcg_params.memcg); - - return NULL; + return ((unsigned long)page->obj_cgroups & 0x1UL); } static inline int memcg_alloc_page_obj_cgroups(struct page *page, @@ -418,17 +358,25 @@ static inline struct kmem_cache *memcg_s size_t objects, gfp_t flags) { struct kmem_cache *cachep; + struct obj_cgroup *objcg; + + if (memcg_kmem_bypass()) + return s; - cachep = memcg_kmem_get_cache(s, objcgp); + cachep = memcg_kmem_get_cache(s); if (is_root_cache(cachep)) return s; - if (obj_cgroup_charge(*objcgp, flags, objects * obj_full_size(s))) { - obj_cgroup_put(*objcgp); - memcg_kmem_put_cache(cachep); + objcg = get_obj_cgroup_from_current(); + if (!objcg) + return s; + + if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) { + obj_cgroup_put(objcg); cachep = NULL; } + *objcgp = objcg; return cachep; } @@ -467,7 +415,6 @@ static inline void memcg_slab_post_alloc } } obj_cgroup_put(objcg); - memcg_kmem_put_cache(s); } static inline void memcg_slab_free_hook(struct kmem_cache *s, struct page *page, @@ -491,7 +438,7 @@ static inline void memcg_slab_free_hook( } extern void slab_init_memcg_params(struct kmem_cache *); -extern void memcg_link_cache(struct kmem_cache *s, struct mem_cgroup *memcg); +extern void memcg_link_cache(struct kmem_cache *s); #else /* CONFIG_MEMCG_KMEM */ @@ -499,9 +446,6 @@ extern void memcg_link_cache(struct kmem #define slab_root_caches slab_caches #define root_caches_node list -#define for_each_memcg_cache(iter, root) \ - for ((void)(iter), (void)(root); 0; ) - static inline bool is_root_cache(struct kmem_cache *s) { return true; @@ -523,7 +467,17 @@ static inline struct kmem_cache *memcg_r return s; } -static inline struct mem_cgroup *memcg_from_slab_page(struct page *page) +static inline struct kmem_cache *memcg_cache(struct kmem_cache *s) +{ + return NULL; +} + +static inline bool page_has_obj_cgroups(struct page *page) +{ + return false; +} + +static inline struct mem_cgroup *memcg_from_slab_obj(void *ptr) { return NULL; } @@ -560,8 +514,7 @@ static inline void slab_init_memcg_param { } -static inline void memcg_link_cache(struct kmem_cache *s, - struct mem_cgroup *memcg) +static inline void memcg_link_cache(struct kmem_cache *s) { } @@ -582,17 +535,14 @@ static __always_inline int charge_slab_p gfp_t gfp, int order, struct kmem_cache *s) { -#ifdef CONFIG_MEMCG_KMEM if (memcg_kmem_enabled() && !is_root_cache(s)) { int ret; ret = memcg_alloc_page_obj_cgroups(page, s, gfp); if (ret) return ret; - - percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order); } -#endif + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), PAGE_SIZE << order); return 0; @@ -601,12 +551,9 @@ static __always_inline int charge_slab_p static __always_inline void uncharge_slab_page(struct page *page, int order, struct kmem_cache *s) { -#ifdef CONFIG_MEMCG_KMEM - if (memcg_kmem_enabled() && !is_root_cache(s)) { + if (memcg_kmem_enabled() && !is_root_cache(s)) memcg_free_page_obj_cgroups(page); - percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order); - } -#endif + mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), -(PAGE_SIZE << order)); } @@ -749,9 +696,6 @@ static inline struct kmem_cache_node *ge void *slab_start(struct seq_file *m, loff_t *pos); void *slab_next(struct seq_file *m, void *p, loff_t *pos); void slab_stop(struct seq_file *m, void *p); -void *memcg_slab_start(struct seq_file *m, loff_t *pos); -void *memcg_slab_next(struct seq_file *m, void *p, loff_t *pos); -void memcg_slab_stop(struct seq_file *m, void *p); int memcg_slab_show(struct seq_file *m, void *p); #if defined(CONFIG_SLAB) || defined(CONFIG_SLUB_DEBUG) --- a/mm/slub.c~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-accounted-allocations +++ a/mm/slub.c @@ -4204,36 +4204,6 @@ int __kmem_cache_shrink(struct kmem_cach return ret; } -#ifdef CONFIG_MEMCG -void __kmemcg_cache_deactivate_after_rcu(struct kmem_cache *s) -{ - /* - * Called with all the locks held after a sched RCU grace period. - * Even if @s becomes empty after shrinking, we can't know that @s - * doesn't have allocations already in-flight and thus can't - * destroy @s until the associated memcg is released. - * - * However, let's remove the sysfs files for empty caches here. - * Each cache has a lot of interface files which aren't - * particularly useful for empty draining caches; otherwise, we can - * easily end up with millions of unnecessary sysfs files on - * systems which have a lot of memory and transient cgroups. - */ - if (!__kmem_cache_shrink(s)) - sysfs_slab_remove(s); -} - -void __kmemcg_cache_deactivate(struct kmem_cache *s) -{ - /* - * Disable empty slabs caching. Used to avoid pinning offline - * memory cgroups by kmem pages that can be freed. - */ - slub_set_cpu_partial(s, 0); - s->min_partial = 0; -} -#endif /* CONFIG_MEMCG */ - static int slab_mem_going_offline_callback(void *arg) { struct kmem_cache *s; @@ -4390,7 +4360,7 @@ static struct kmem_cache * __init bootst } slab_init_memcg_params(s); list_add(&s->list, &slab_caches); - memcg_link_cache(s, NULL); + memcg_link_cache(s); return s; } @@ -4458,7 +4428,8 @@ __kmem_cache_alias(const char *name, uns s->object_size = max(s->object_size, size); s->inuse = max(s->inuse, ALIGN(size, sizeof(void *))); - for_each_memcg_cache(c, s) { + c = memcg_cache(s); + if (c) { c->object_size = s->object_size; c->inuse = max(c->inuse, ALIGN(size, sizeof(void *))); } @@ -5591,7 +5562,8 @@ static ssize_t slab_attr_store(struct ko * directly either failed or succeeded, in which case we loop * through the descendants with best-effort propagation. */ - for_each_memcg_cache(c, s) + c = memcg_cache(s); + if (c) attribute->store(c, buf, len); mutex_unlock(&slab_mutex); } From patchwork Fri Aug 7 06:21:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704823 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 06A8014B7 for ; Fri, 7 Aug 2020 06:21:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BF04B22CAF for ; Fri, 7 Aug 2020 06:21:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="jAShDOza" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF04B22CAF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3836A8D0055; Fri, 7 Aug 2020 02:21:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 333828D0026; Fri, 7 Aug 2020 02:21:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2704B8D0055; Fri, 7 Aug 2020 02:21:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0052.hostedemail.com [216.40.44.52]) by kanga.kvack.org (Postfix) with ESMTP id 11B768D0026 for ; Fri, 7 Aug 2020 02:21:17 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id CD9E8181AEF10 for ; Fri, 7 Aug 2020 06:21:16 +0000 (UTC) X-FDA: 77122775352.23.pail58_2b07a7526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id A514737606 for ; Fri, 7 Aug 2020 06:21:16 +0000 (UTC) X-Spam-Summary: 1,0,0,7db00370bd8c833a,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2525:2559:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4049:4119:4250:4321:4385:4423:4605:5007:6261:6653:6737:7576:7903:8599:8603:8660:9010:9025:9545:9592:10004:10913:11026:11232:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13148:13161:13229:13230:13868:21080:21324:21451:21611:21627:21740:21796:21939:21990:30034:30036:30054:30064:30080,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8sycuyujzbupesbd8e4tjiuej3op3nd8eq15pqaj4ggoahoj8hgcxumgtuct.1mi1uggs3zgopz9dzeewt457gbaqqd1umipsrh8w5bfyodh9krzqqi4cd9thkqh.q-lbl8.mai lshell.n X-HE-Tag: pail58_2b07a7526fbe X-Filterd-Recvd-Size: 8093 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:15 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 99CDC22CAF; Fri, 7 Aug 2020 06:21:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781275; bh=Fhu6cQTbKmBdAAVvI4mhNXkwC0uRFIVD/XvH3G4l5u0=; h=Date:From:To:Subject:In-Reply-To:From; b=jAShDOzajjZpL54sYHhpgdEJXHsHLoq5LlBpkHpcfXNBLYffXfvs8zGH6EKbjnPYL UGxh3C8dU0i5gMG12k6FTdMmfwDGincoqTGp9+MmneYeJDZI45MuP79+dmulPjdPk6 jYaoz3uZ0OAU9h1WGmo3e5d1fmWnKQTn4Age7+pg= Date: Thu, 06 Aug 2020 23:21:14 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 074/163] mm: memcg/slab: simplify memcg cache creation Message-ID: <20200807062114.A811_SKgY%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: A514737606 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: simplify memcg cache creation Because the number of non-root kmem_caches doesn't depend on the number of memory cgroups anymore and is generally not very big, there is no more need for a dedicated workqueue. Also, as there is no more need to pass any arguments to the memcg_create_kmem_cache() except the root kmem_cache, it's possible to just embed the work structure into the kmem_cache and avoid the dynamic allocation of the work structure. This will also simplify the synchronization: for each root kmem_cache there is only one work. So there will be no more concurrent attempts to create a non-root kmem_cache for a root kmem_cache: the second and all following attempts to queue the work will fail. On the kmem_cache destruction path there is no more need to call the expensive flush_workqueue() and wait for all pending works to be finished. Instead, cancel_work_sync() can be used to cancel/wait for only one work. Link: http://lkml.kernel.org/r/20200623174037.3951353-14-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Johannes Weiner Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 1 mm/memcontrol.c | 48 ----------------------------------- mm/slab.h | 2 + mm/slab_common.c | 22 ++++++++-------- 4 files changed, 15 insertions(+), 58 deletions(-) --- a/include/linux/memcontrol.h~mm-memcg-slab-simplify-memcg-cache-creation +++ a/include/linux/memcontrol.h @@ -1418,7 +1418,6 @@ int obj_cgroup_charge(struct obj_cgroup void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); extern struct static_key_false memcg_kmem_enabled_key; -extern struct workqueue_struct *memcg_kmem_cache_wq; extern int memcg_nr_cache_ids; void memcg_get_cache_ids(void); --- a/mm/memcontrol.c~mm-memcg-slab-simplify-memcg-cache-creation +++ a/mm/memcontrol.c @@ -399,8 +399,6 @@ void memcg_put_cache_ids(void) */ DEFINE_STATIC_KEY_FALSE(memcg_kmem_enabled_key); EXPORT_SYMBOL(memcg_kmem_enabled_key); - -struct workqueue_struct *memcg_kmem_cache_wq; #endif static int memcg_shrinker_map_size; @@ -2902,39 +2900,6 @@ static void memcg_free_cache_id(int id) ida_simple_remove(&memcg_cache_ida, id); } -struct memcg_kmem_cache_create_work { - struct kmem_cache *cachep; - struct work_struct work; -}; - -static void memcg_kmem_cache_create_func(struct work_struct *w) -{ - struct memcg_kmem_cache_create_work *cw = - container_of(w, struct memcg_kmem_cache_create_work, work); - struct kmem_cache *cachep = cw->cachep; - - memcg_create_kmem_cache(cachep); - - kfree(cw); -} - -/* - * Enqueue the creation of a per-memcg kmem_cache. - */ -static void memcg_schedule_kmem_cache_create(struct kmem_cache *cachep) -{ - struct memcg_kmem_cache_create_work *cw; - - cw = kmalloc(sizeof(*cw), GFP_NOWAIT | __GFP_NOWARN); - if (!cw) - return; - - cw->cachep = cachep; - INIT_WORK(&cw->work, memcg_kmem_cache_create_func); - - queue_work(memcg_kmem_cache_wq, &cw->work); -} - /** * memcg_kmem_get_cache: select memcg or root cache for allocation * @cachep: the original global kmem cache @@ -2951,7 +2916,7 @@ struct kmem_cache *memcg_kmem_get_cache( memcg_cachep = READ_ONCE(cachep->memcg_params.memcg_cache); if (unlikely(!memcg_cachep)) { - memcg_schedule_kmem_cache_create(cachep); + queue_work(system_wq, &cachep->memcg_params.work); return cachep; } @@ -7022,17 +6987,6 @@ static int __init mem_cgroup_init(void) { int cpu, node; -#ifdef CONFIG_MEMCG_KMEM - /* - * Kmem cache creation is mostly done with the slab_mutex held, - * so use a workqueue with limited concurrency to avoid stalling - * all worker threads in case lots of cgroups are created and - * destroyed simultaneously. - */ - memcg_kmem_cache_wq = alloc_workqueue("memcg_kmem_cache", 0, 1); - BUG_ON(!memcg_kmem_cache_wq); -#endif - cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL, memcg_hotplug_cpu_dead); --- a/mm/slab_common.c~mm-memcg-slab-simplify-memcg-cache-creation +++ a/mm/slab_common.c @@ -134,10 +134,18 @@ int __kmem_cache_alloc_bulk(struct kmem_ LIST_HEAD(slab_root_caches); +static void memcg_kmem_cache_create_func(struct work_struct *work) +{ + struct kmem_cache *cachep = container_of(work, struct kmem_cache, + memcg_params.work); + memcg_create_kmem_cache(cachep); +} + void slab_init_memcg_params(struct kmem_cache *s) { s->memcg_params.root_cache = NULL; s->memcg_params.memcg_cache = NULL; + INIT_WORK(&s->memcg_params.work, memcg_kmem_cache_create_func); } static void init_memcg_params(struct kmem_cache *s, @@ -586,15 +594,9 @@ static int shutdown_memcg_caches(struct return 0; } -static void flush_memcg_workqueue(struct kmem_cache *s) +static void cancel_memcg_cache_creation(struct kmem_cache *s) { - /* - * SLAB and SLUB create memcg kmem_caches through workqueue and SLUB - * deactivates the memcg kmem_caches through workqueue. Make sure all - * previous workitems on workqueue are processed. - */ - if (likely(memcg_kmem_cache_wq)) - flush_workqueue(memcg_kmem_cache_wq); + cancel_work_sync(&s->memcg_params.work); } #else static inline int shutdown_memcg_caches(struct kmem_cache *s) @@ -602,7 +604,7 @@ static inline int shutdown_memcg_caches( return 0; } -static inline void flush_memcg_workqueue(struct kmem_cache *s) +static inline void cancel_memcg_cache_creation(struct kmem_cache *s) { } #endif /* CONFIG_MEMCG_KMEM */ @@ -621,7 +623,7 @@ void kmem_cache_destroy(struct kmem_cach if (unlikely(!s)) return; - flush_memcg_workqueue(s); + cancel_memcg_cache_creation(s); get_online_cpus(); get_online_mems(); --- a/mm/slab.h~mm-memcg-slab-simplify-memcg-cache-creation +++ a/mm/slab.h @@ -45,12 +45,14 @@ struct kmem_cache { * @memcg_cache: pointer to memcg kmem cache, used by all non-root memory * cgroups. * @root_caches_node: list node for slab_root_caches list. + * @work: work struct used to create the non-root cache. */ struct memcg_cache_params { struct kmem_cache *root_cache; struct kmem_cache *memcg_cache; struct list_head __root_caches_node; + struct work_struct work; }; #endif /* CONFIG_SLOB */ From patchwork Fri Aug 7 06:21:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704825 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 568E7138C for ; Fri, 7 Aug 2020 06:21:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 29AF322CAF for ; Fri, 7 Aug 2020 06:21:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="PIXyP8vN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 29AF322CAF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3663E8D0056; Fri, 7 Aug 2020 02:21:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2E5E08D0026; Fri, 7 Aug 2020 02:21:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FCEB8D0056; Fri, 7 Aug 2020 02:21:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0021.hostedemail.com [216.40.44.21]) by kanga.kvack.org (Postfix) with ESMTP id 0B4118D0026 for ; Fri, 7 Aug 2020 02:21:20 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CC6CA3625 for ; Fri, 7 Aug 2020 06:21:19 +0000 (UTC) X-FDA: 77122775478.22.eggs07_460be9a26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id A79DB18038E67 for ; Fri, 7 Aug 2020 06:21:19 +0000 (UTC) X-Spam-Summary: 1,0,0,98b284c2034982be,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:334:355:368:369:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2693:2859:2902:2903:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:4605:5007:6261:6653:6737:7576:7903:8599:8660:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13148:13161:13229:13230:14181:14721:14849:21080:21451:21627:21740:21795:21939:21990:30012:30034:30051:30054:30064:30080:30089,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yf7bueb7j9yubqbx1g7jstxogwhocmupf1s8a17iza19t6j6n35ikiu5cqa6h.enrazgndyqooey43whxnzr15wpbhttuefdnbm51bos1yn7gf3787o9uhy7uqw8g.h-lbl8.mailshell.ne t-223.23 X-HE-Tag: eggs07_460be9a26fbe X-Filterd-Recvd-Size: 5306 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:19 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 156BB2177B; Fri, 7 Aug 2020 06:21:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781278; bh=TLUdxppDDB393KQgCT/k49c09HwP86rAwNffi0L8VdA=; h=Date:From:To:Subject:In-Reply-To:From; b=PIXyP8vNDfPHsWbeVrItcNMDW7PvdbThgTzAGqYzdTqEVgtGljZCoF0k+owhtA3E2 EnsY9J9MCrd0JE1U0QZvdLP+Vx8I+5p8PbtfsjvI1RrsH9PRQoA6vySKM5N0NJzVWe p5gpL5DjRrMPn02ytgzFrQWyCnq1XvEe0Cuh9mis= Date: Thu, 06 Aug 2020 23:21:17 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 075/163] mm: memcg/slab: remove memcg_kmem_get_cache() Message-ID: <20200807062117.-2gQ_1DHO%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: A79DB18038E67 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000113, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: remove memcg_kmem_get_cache() The memcg_kmem_get_cache() function became really trivial, so let's just inline it into the single call point: memcg_slab_pre_alloc_hook(). It will make the code less bulky and can also help the compiler to generate a better code. Link: http://lkml.kernel.org/r/20200623174037.3951353-15-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Johannes Weiner Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 2 -- mm/memcontrol.c | 25 +------------------------ mm/slab.h | 11 +++++++++-- mm/slab_common.c | 2 +- 4 files changed, 11 insertions(+), 29 deletions(-) --- a/include/linux/memcontrol.h~mm-memcg-slab-remove-memcg_kmem_get_cache +++ a/include/linux/memcontrol.h @@ -1403,8 +1403,6 @@ static inline void memcg_set_shrinker_bi } #endif -struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep); - #ifdef CONFIG_MEMCG_KMEM int __memcg_kmem_charge(struct mem_cgroup *memcg, gfp_t gfp, unsigned int nr_pages); --- a/mm/memcontrol.c~mm-memcg-slab-remove-memcg_kmem_get_cache +++ a/mm/memcontrol.c @@ -393,7 +393,7 @@ void memcg_put_cache_ids(void) /* * A lot of the calls to the cache allocation functions are expected to be - * inlined by the compiler. Since the calls to memcg_kmem_get_cache are + * inlined by the compiler. Since the calls to memcg_slab_pre_alloc_hook() are * conditional to this static branch, we'll have to allow modules that does * kmem_cache_alloc and the such to see this symbol as well */ @@ -2901,29 +2901,6 @@ static void memcg_free_cache_id(int id) } /** - * memcg_kmem_get_cache: select memcg or root cache for allocation - * @cachep: the original global kmem cache - * - * Return the kmem_cache we're supposed to use for a slab allocation. - * - * If the cache does not exist yet, if we are the first user of it, we - * create it asynchronously in a workqueue and let the current allocation - * go through with the original cache. - */ -struct kmem_cache *memcg_kmem_get_cache(struct kmem_cache *cachep) -{ - struct kmem_cache *memcg_cachep; - - memcg_cachep = READ_ONCE(cachep->memcg_params.memcg_cache); - if (unlikely(!memcg_cachep)) { - queue_work(system_wq, &cachep->memcg_params.work); - return cachep; - } - - return memcg_cachep; -} - -/** * __memcg_kmem_charge: charge a number of kernel pages to a memcg * @memcg: memory cgroup to charge * @gfp: reclaim mode --- a/mm/slab_common.c~mm-memcg-slab-remove-memcg_kmem_get_cache +++ a/mm/slab_common.c @@ -570,7 +570,7 @@ void memcg_create_kmem_cache(struct kmem } /* - * Since readers won't lock (see memcg_kmem_get_cache()), we need a + * Since readers won't lock (see memcg_slab_pre_alloc_hook()), we need a * barrier here to ensure nobody will see the kmem_cache partially * initialized. */ --- a/mm/slab.h~mm-memcg-slab-remove-memcg_kmem_get_cache +++ a/mm/slab.h @@ -365,9 +365,16 @@ static inline struct kmem_cache *memcg_s if (memcg_kmem_bypass()) return s; - cachep = memcg_kmem_get_cache(s); - if (is_root_cache(cachep)) + cachep = READ_ONCE(s->memcg_params.memcg_cache); + if (unlikely(!cachep)) { + /* + * If memcg cache does not exist yet, we schedule it's + * asynchronous creation and let the current allocation + * go through with the root cache. + */ + queue_work(system_wq, &s->memcg_params.work); return s; + } objcg = get_obj_cgroup_from_current(); if (!objcg) From patchwork Fri Aug 7 06:21:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704827 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9CB1214B7 for ; Fri, 7 Aug 2020 06:21:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 63B1422D03 for ; Fri, 7 Aug 2020 06:21:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="R8S8qkcG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 63B1422D03 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 94EEF8D0057; Fri, 7 Aug 2020 02:21:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 926CC8D0026; Fri, 7 Aug 2020 02:21:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 815118D0057; Fri, 7 Aug 2020 02:21:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0060.hostedemail.com [216.40.44.60]) by kanga.kvack.org (Postfix) with ESMTP id 66C3D8D0026 for ; Fri, 7 Aug 2020 02:21:23 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3DF783625 for ; Fri, 7 Aug 2020 06:21:23 +0000 (UTC) X-FDA: 77122775646.14.brush17_291757c26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 1155518229835 for ; Fri, 7 Aug 2020 06:21:23 +0000 (UTC) X-Spam-Summary: 1,0,0,b48e7ed4454e10ad,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2525:2559:2563:2610:2682:2685:2859:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4049:4119:4321:4385:4605:5007:6261:6653:6737:7576:7903:8599:8603:9010:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12294:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13141:13161:13229:13230:21080:21324:21451:21627:21740:21939:21990:30012:30029:30034:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfbyxc679qt5n6o33utewzbc5zoocpkwwsirpr5re9tpjmm87ujbiych6yjz3.hmuaq99swm6iu6dc56w87769scjp15u5icdc1ijjg3brckqejfpmwn1ikq448yf.o-lbl8.mailshell.net-223.238.255 .100,Cac X-HE-Tag: brush17_291757c26fbe X-Filterd-Recvd-Size: 8294 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:22 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 61997221E5; Fri, 7 Aug 2020 06:21:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781281; bh=XuIep6+ZGpnELZK23aWh6aSRIc+HZZFX3IRYj0Flt00=; h=Date:From:To:Subject:In-Reply-To:From; b=R8S8qkcG1APQ2FWSsisKsMeVsACFhzVBwcaL7uwBDya20g7RoOmq5obA1UfNPmFth 8hGa5FyJJlZJZCdGI9JX45v6hdr4SuC7HO20nuamUpv7JhRLcUl3HjgOoNKrZVVAWd +NQGHQCx+3pC0q/7FgekHxAJiWRoMTxdiwtIoenM= Date: Thu, 06 Aug 2020 23:21:20 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 076/163] mm: memcg/slab: deprecate slab_root_caches Message-ID: <20200807062120.4Io9E7e7i%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 1155518229835 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: deprecate slab_root_caches Currently there are two lists of kmem_caches: 1) slab_caches, which contains all kmem_caches, 2) slab_root_caches, which contains only root kmem_caches. And there is some preprocessor magic to have a single list if CONFIG_MEMCG_KMEM isn't enabled. It was required earlier because the number of non-root kmem_caches was proportional to the number of memory cgroups and could reach really big values. Now, when it cannot exceed the number of root kmem_caches, there is really no reason to maintain two lists. We never iterate over the slab_root_caches list on any hot paths, so it's perfectly fine to iterate over slab_caches and filter out non-root kmem_caches. It allows to remove a lot of config-dependent code and two pointers from the kmem_cache structure. Link: http://lkml.kernel.org/r/20200623174037.3951353-16-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Johannes Weiner Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- mm/slab.c | 1 - mm/slab.h | 17 ----------------- mm/slab_common.c | 37 ++++++++----------------------------- mm/slub.c | 1 - 4 files changed, 8 insertions(+), 48 deletions(-) --- a/mm/slab.c~mm-memcg-slab-deprecate-slab_root_caches +++ a/mm/slab.c @@ -1249,7 +1249,6 @@ void __init kmem_cache_init(void) nr_node_ids * sizeof(struct kmem_cache_node *), SLAB_HWCACHE_ALIGN, 0, 0); list_add(&kmem_cache->list, &slab_caches); - memcg_link_cache(kmem_cache); slab_state = PARTIAL; /* --- a/mm/slab_common.c~mm-memcg-slab-deprecate-slab_root_caches +++ a/mm/slab_common.c @@ -131,9 +131,6 @@ int __kmem_cache_alloc_bulk(struct kmem_ } #ifdef CONFIG_MEMCG_KMEM - -LIST_HEAD(slab_root_caches); - static void memcg_kmem_cache_create_func(struct work_struct *work) { struct kmem_cache *cachep = container_of(work, struct kmem_cache, @@ -156,27 +153,11 @@ static void init_memcg_params(struct kme else slab_init_memcg_params(s); } - -void memcg_link_cache(struct kmem_cache *s) -{ - if (is_root_cache(s)) - list_add(&s->root_caches_node, &slab_root_caches); -} - -static void memcg_unlink_cache(struct kmem_cache *s) -{ - if (is_root_cache(s)) - list_del(&s->root_caches_node); -} #else static inline void init_memcg_params(struct kmem_cache *s, struct kmem_cache *root_cache) { } - -static inline void memcg_unlink_cache(struct kmem_cache *s) -{ -} #endif /* CONFIG_MEMCG_KMEM */ /* @@ -253,7 +234,7 @@ struct kmem_cache *find_mergeable(unsign if (flags & SLAB_NEVER_MERGE) return NULL; - list_for_each_entry_reverse(s, &slab_root_caches, root_caches_node) { + list_for_each_entry_reverse(s, &slab_caches, list) { if (slab_unmergeable(s)) continue; @@ -312,7 +293,6 @@ static struct kmem_cache *create_cache(c s->refcount = 1; list_add(&s->list, &slab_caches); - memcg_link_cache(s); out: if (err) return ERR_PTR(err); @@ -507,7 +487,6 @@ static int shutdown_cache(struct kmem_ca if (__kmem_cache_shutdown(s) != 0) return -EBUSY; - memcg_unlink_cache(s); list_del(&s->list); if (s->flags & SLAB_TYPESAFE_BY_RCU) { @@ -751,7 +730,6 @@ struct kmem_cache *__init create_kmalloc create_boot_cache(s, name, size, flags, useroffset, usersize); list_add(&s->list, &slab_caches); - memcg_link_cache(s); s->refcount = 1; return s; } @@ -1107,12 +1085,12 @@ static void print_slabinfo_header(struct void *slab_start(struct seq_file *m, loff_t *pos) { mutex_lock(&slab_mutex); - return seq_list_start(&slab_root_caches, *pos); + return seq_list_start(&slab_caches, *pos); } void *slab_next(struct seq_file *m, void *p, loff_t *pos) { - return seq_list_next(p, &slab_root_caches, pos); + return seq_list_next(p, &slab_caches, pos); } void slab_stop(struct seq_file *m, void *p) @@ -1165,11 +1143,12 @@ static void cache_show(struct kmem_cache static int slab_show(struct seq_file *m, void *p) { - struct kmem_cache *s = list_entry(p, struct kmem_cache, root_caches_node); + struct kmem_cache *s = list_entry(p, struct kmem_cache, list); - if (p == slab_root_caches.next) + if (p == slab_caches.next) print_slabinfo_header(m); - cache_show(s, m); + if (is_root_cache(s)) + cache_show(s, m); return 0; } @@ -1271,7 +1250,7 @@ static int memcg_slabinfo_show(struct se mutex_lock(&slab_mutex); seq_puts(m, "# "); seq_puts(m, " \n"); - list_for_each_entry(s, &slab_root_caches, root_caches_node) { + list_for_each_entry(s, &slab_caches, list) { /* * Skip kmem caches that don't have the memcg cache. */ --- a/mm/slab.h~mm-memcg-slab-deprecate-slab_root_caches +++ a/mm/slab.h @@ -44,14 +44,12 @@ struct kmem_cache { * * @memcg_cache: pointer to memcg kmem cache, used by all non-root memory * cgroups. - * @root_caches_node: list node for slab_root_caches list. * @work: work struct used to create the non-root cache. */ struct memcg_cache_params { struct kmem_cache *root_cache; struct kmem_cache *memcg_cache; - struct list_head __root_caches_node; struct work_struct work; }; #endif /* CONFIG_SLOB */ @@ -265,11 +263,6 @@ static inline bool kmem_cache_debug_flag } #ifdef CONFIG_MEMCG_KMEM - -/* List of all root caches. */ -extern struct list_head slab_root_caches; -#define root_caches_node memcg_params.__root_caches_node - static inline bool is_root_cache(struct kmem_cache *s) { return !s->memcg_params.root_cache; @@ -447,14 +440,8 @@ static inline void memcg_slab_free_hook( } extern void slab_init_memcg_params(struct kmem_cache *); -extern void memcg_link_cache(struct kmem_cache *s); #else /* CONFIG_MEMCG_KMEM */ - -/* If !memcg, all caches are root. */ -#define slab_root_caches slab_caches -#define root_caches_node list - static inline bool is_root_cache(struct kmem_cache *s) { return true; @@ -523,10 +510,6 @@ static inline void slab_init_memcg_param { } -static inline void memcg_link_cache(struct kmem_cache *s) -{ -} - #endif /* CONFIG_MEMCG_KMEM */ static inline struct kmem_cache *virt_to_cache(const void *obj) --- a/mm/slub.c~mm-memcg-slab-deprecate-slab_root_caches +++ a/mm/slub.c @@ -4360,7 +4360,6 @@ static struct kmem_cache * __init bootst } slab_init_memcg_params(s); list_add(&s->list, &slab_caches); - memcg_link_cache(s); return s; } From patchwork Fri Aug 7 06:21:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704829 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BA7AE14B7 for ; Fri, 7 Aug 2020 06:21:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8033622D07 for ; Fri, 7 Aug 2020 06:21:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="xwGUHggN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8033622D07 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B03878D0058; Fri, 7 Aug 2020 02:21:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AAEFA8D0026; Fri, 7 Aug 2020 02:21:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A0448D0058; Fri, 7 Aug 2020 02:21:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0156.hostedemail.com [216.40.44.156]) by kanga.kvack.org (Postfix) with ESMTP id 827BA8D0026 for ; Fri, 7 Aug 2020 02:21:26 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 48B3182499A8 for ; Fri, 7 Aug 2020 06:21:26 +0000 (UTC) X-FDA: 77122775772.26.error44_031588d26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 1D7AE1804B66A for ; Fri, 7 Aug 2020 06:21:26 +0000 (UTC) X-Spam-Summary: 1,0,0,1ae143990acf101e,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2525:2538:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3868:3870:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:6737:7576:8599:9025:9545:9592:10004:10913:11026:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13069:13255:13311:13357:14181:14384:14721:14849:21080:21451:21627:21939:21990:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfpj8dhxttgqau1hssfw5dc19yyypyjk63nxygcobdeph1p79h5eukfc3eypt.re6c5fppjzgx5dinjmuwczebh4cwdc96yiw3uwy9ah5q8xr1qa3soo4nhh9s8ug.k-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL: 0,DNSBL: X-HE-Tag: error44_031588d26fbe X-Filterd-Recvd-Size: 2596 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:25 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AA56C2177B; Fri, 7 Aug 2020 06:21:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781285; bh=H58aq4rZBZxIDzcV4GGAxsC91DuutB//G1LzUWLK68M=; h=Date:From:To:Subject:In-Reply-To:From; b=xwGUHggNTHMlgi03GcRh+3SQT5Gy2R1e/5+Muw4nngZ35vkI9xj39YetpuWAvN+jK AQjkmAyykePNZhnyL6HVo3jR3XNhzNLxTx/jtMIqgG46iEySWjndICb234JsudDRDB ohG3xewR4mk+lzGF9LdWw96/gxipx7/+HBTUpXGI= Date: Thu, 06 Aug 2020 23:21:24 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 077/163] mm: memcg/slab: remove redundant check in memcg_accumulate_slabinfo() Message-ID: <20200807062124.HxrApR2M-%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 1D7AE1804B66A X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.001483, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: remove redundant check in memcg_accumulate_slabinfo() memcg_accumulate_slabinfo() is never called with a non-root kmem_cache as a first argument, so the is_root_cache(s) check is redundant and can be removed without any functional change. Link: http://lkml.kernel.org/r/20200623174037.3951353-17-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Johannes Weiner Cc: Michal Hocko Cc: Tejun Heo Signed-off-by: Andrew Morton --- mm/slab_common.c | 3 --- 1 file changed, 3 deletions(-) --- a/mm/slab_common.c~mm-memcg-slab-remove-redundant-check-in-memcg_accumulate_slabinfo +++ a/mm/slab_common.c @@ -1104,9 +1104,6 @@ memcg_accumulate_slabinfo(struct kmem_ca struct kmem_cache *c; struct slabinfo sinfo; - if (!is_root_cache(s)) - return; - c = memcg_cache(s); if (c) { memset(&sinfo, 0, sizeof(sinfo)); From patchwork Fri Aug 7 06:21:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704831 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CB467138C for ; Fri, 7 Aug 2020 06:21:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 74E3D22CAF for ; Fri, 7 Aug 2020 06:21:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="rRyYcvF9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 74E3D22CAF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8B47E8D0059; Fri, 7 Aug 2020 02:21:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 863558D0026; Fri, 7 Aug 2020 02:21:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72C878D0059; Fri, 7 Aug 2020 02:21:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0199.hostedemail.com [216.40.44.199]) by kanga.kvack.org (Postfix) with ESMTP id 589A98D0026 for ; Fri, 7 Aug 2020 02:21:30 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2C1533625 for ; Fri, 7 Aug 2020 06:21:30 +0000 (UTC) X-FDA: 77122775940.16.cave33_041622926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id F1F91100E6903 for ; Fri, 7 Aug 2020 06:21:29 +0000 (UTC) X-Spam-Summary: 1,0,0,6c54f75beba39e0b,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:69:327:355:379:421:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1801:2194:2196:2198:2199:2200:2201:2393:2525:2538:2559:2564:2610:2682:2685:2693:2731:2859:2892:2898:2902:2903:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4041:4321:4385:4423:4605:5007:6261:6653:6737:7576:7875:7903:7904:8599:8603:8957:9010:9025:9108:9121:9545:9592:10004:10241:10913:11026:11232:11233:11914:12043:12048:12257:12294:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13141:13161:13229:13230:13255:13972:14096:21080:21324:21450:21451:21611:21627:21740:21795:21939:21987:21990:30012:30029:30034:30045:30051:30054:30064:30070:30075:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfk3cqh9aqi7ia9iwqi11srfhk 4opk9e3z X-HE-Tag: cave33_041622926fbe X-Filterd-Recvd-Size: 34201 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:29 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E54C122CAF; Fri, 7 Aug 2020 06:21:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781288; bh=Ytg/xaOxSDq2xwD9yGX/WePaYUFiQ0Mo+IwglBvE9H4=; h=Date:From:To:Subject:In-Reply-To:From; b=rRyYcvF9vVxRd/YY5C4A6+M+jAqN/xdaPjbu5s+umqiAzekeUJdS7mvhM/YLCH0QS V5/o/TJR2xGS/I/fhqXnvG8RXAmfm+P1hM3qb0G+cqm+4rtdtjzGzBcDMtk1hlojoN xHOL+qv0tA553XHKr95QGy5bc99a0+GS90rJqkoo= Date: Thu, 06 Aug 2020 23:21:27 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, naresh.kamboju@linaro.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 078/163] mm: memcg/slab: use a single set of kmem_caches for all allocations Message-ID: <20200807062127.V3pcUyOtA%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: F1F91100E6903 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000330, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: use a single set of kmem_caches for all allocations Instead of having two sets of kmem_caches: one for system-wide and non-accounted allocations and the second one shared by all accounted allocations, we can use just one. The idea is simple: space for obj_cgroup metadata can be allocated on demand and filled only for accounted allocations. It allows to remove a bunch of code which is required to handle kmem_cache clones for accounted allocations. There is no more need to create them, accumulate statistics, propagate attributes, etc. It's a quite significant simplification. Also, because the total number of slab_caches is reduced almost twice (not all kmem_caches have a memcg clone), some additional memory savings are expected. On my devvm it additionally saves about 3.5% of slab memory. [guro@fb.com: fix build on MIPS] Link: http://lkml.kernel.org/r/20200717214810.3733082-1-guro@fb.com Link: http://lkml.kernel.org/r/20200623174037.3951353-18-guro@fb.com Suggested-by: Johannes Weiner Signed-off-by: Roman Gushchin Reviewed-by: Vlastimil Babka Reviewed-by: Shakeel Butt Cc: Christoph Lameter Cc: Michal Hocko Cc: Tejun Heo Cc: Naresh Kamboju Signed-off-by: Andrew Morton --- include/linux/slab.h | 2 include/linux/slab_def.h | 3 include/linux/slub_def.h | 10 - mm/memcontrol.c | 25 +++- mm/slab.c | 41 ------ mm/slab.h | 196 ++++++------------------------- mm/slab_common.c | 230 ------------------------------------- mm/slub.c | 163 -------------------------- 8 files changed, 79 insertions(+), 591 deletions(-) --- a/include/linux/slab_def.h~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-allocations +++ a/include/linux/slab_def.h @@ -72,9 +72,6 @@ struct kmem_cache { int obj_offset; #endif /* CONFIG_DEBUG_SLAB */ -#ifdef CONFIG_MEMCG - struct memcg_cache_params memcg_params; -#endif #ifdef CONFIG_KASAN struct kasan_cache kasan_info; #endif --- a/include/linux/slab.h~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-allocations +++ a/include/linux/slab.h @@ -155,8 +155,6 @@ struct kmem_cache *kmem_cache_create_use void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); -void memcg_create_kmem_cache(struct kmem_cache *cachep); - /* * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above. --- a/include/linux/slub_def.h~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-allocations +++ a/include/linux/slub_def.h @@ -108,17 +108,7 @@ struct kmem_cache { struct list_head list; /* List of slab caches */ #ifdef CONFIG_SYSFS struct kobject kobj; /* For sysfs */ - struct work_struct kobj_remove_work; #endif -#ifdef CONFIG_MEMCG - struct memcg_cache_params memcg_params; - /* For propagation, maximum size of a stored attr */ - unsigned int max_attr_size; -#ifdef CONFIG_SYSFS - struct kset *memcg_kset; -#endif -#endif - #ifdef CONFIG_SLAB_FREELIST_HARDENED unsigned long random; #endif --- a/mm/memcontrol.c~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-allocations +++ a/mm/memcontrol.c @@ -2800,6 +2800,26 @@ static void commit_charge(struct page *p } #ifdef CONFIG_MEMCG_KMEM +int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s, + gfp_t gfp) +{ + unsigned int objects = objs_per_slab_page(s, page); + void *vec; + + vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, + page_to_nid(page)); + if (!vec) + return -ENOMEM; + + if (cmpxchg(&page->obj_cgroups, NULL, + (struct obj_cgroup **) ((unsigned long)vec | 0x1UL))) + kfree(vec); + else + kmemleak_not_leak(vec); + + return 0; +} + /* * Returns a pointer to the memory cgroup to which the kernel object is charged. * @@ -2826,7 +2846,10 @@ struct mem_cgroup *mem_cgroup_from_obj(v off = obj_to_index(page->slab_cache, page, p); objcg = page_obj_cgroups(page)[off]; - return obj_cgroup_memcg(objcg); + if (objcg) + return obj_cgroup_memcg(objcg); + + return NULL; } /* All other pages use page->mem_cgroup */ --- a/mm/slab.c~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-allocations +++ a/mm/slab.c @@ -1379,11 +1379,7 @@ static struct page *kmem_getpages(struct return NULL; } - if (charge_slab_page(page, flags, cachep->gfporder, cachep)) { - __free_pages(page, cachep->gfporder); - return NULL; - } - + charge_slab_page(page, flags, cachep->gfporder, cachep); __SetPageSlab(page); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks() && page_is_pfmemalloc(page)) @@ -3799,8 +3795,8 @@ fail: } /* Always called with the slab_mutex held */ -static int __do_tune_cpucache(struct kmem_cache *cachep, int limit, - int batchcount, int shared, gfp_t gfp) +static int do_tune_cpucache(struct kmem_cache *cachep, int limit, + int batchcount, int shared, gfp_t gfp) { struct array_cache __percpu *cpu_cache, *prev; int cpu; @@ -3845,30 +3841,6 @@ setup_node: return setup_kmem_cache_nodes(cachep, gfp); } -static int do_tune_cpucache(struct kmem_cache *cachep, int limit, - int batchcount, int shared, gfp_t gfp) -{ - int ret; - struct kmem_cache *c; - - ret = __do_tune_cpucache(cachep, limit, batchcount, shared, gfp); - - if (slab_state < FULL) - return ret; - - if ((ret < 0) || !is_root_cache(cachep)) - return ret; - - lockdep_assert_held(&slab_mutex); - c = memcg_cache(cachep); - if (c) { - /* return value determined by the root cache only */ - __do_tune_cpucache(c, limit, batchcount, shared, gfp); - } - - return ret; -} - /* Called with slab_mutex held always */ static int enable_cpucache(struct kmem_cache *cachep, gfp_t gfp) { @@ -3881,13 +3853,6 @@ static int enable_cpucache(struct kmem_c if (err) goto end; - if (!is_root_cache(cachep)) { - struct kmem_cache *root = memcg_root_cache(cachep); - limit = root->limit; - shared = root->shared; - batchcount = root->batchcount; - } - if (limit && shared && batchcount) goto skip_setup; /* --- a/mm/slab_common.c~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-allocations +++ a/mm/slab_common.c @@ -130,36 +130,6 @@ int __kmem_cache_alloc_bulk(struct kmem_ return i; } -#ifdef CONFIG_MEMCG_KMEM -static void memcg_kmem_cache_create_func(struct work_struct *work) -{ - struct kmem_cache *cachep = container_of(work, struct kmem_cache, - memcg_params.work); - memcg_create_kmem_cache(cachep); -} - -void slab_init_memcg_params(struct kmem_cache *s) -{ - s->memcg_params.root_cache = NULL; - s->memcg_params.memcg_cache = NULL; - INIT_WORK(&s->memcg_params.work, memcg_kmem_cache_create_func); -} - -static void init_memcg_params(struct kmem_cache *s, - struct kmem_cache *root_cache) -{ - if (root_cache) - s->memcg_params.root_cache = root_cache; - else - slab_init_memcg_params(s); -} -#else -static inline void init_memcg_params(struct kmem_cache *s, - struct kmem_cache *root_cache) -{ -} -#endif /* CONFIG_MEMCG_KMEM */ - /* * Figure out what the alignment of the objects will be given a set of * flags, a user specified alignment and the size of the objects. @@ -197,9 +167,6 @@ int slab_unmergeable(struct kmem_cache * if (slab_nomerge || (s->flags & SLAB_NEVER_MERGE)) return 1; - if (!is_root_cache(s)) - return 1; - if (s->ctor) return 1; @@ -286,7 +253,6 @@ static struct kmem_cache *create_cache(c s->useroffset = useroffset; s->usersize = usersize; - init_memcg_params(s, root_cache); err = __kmem_cache_create(s, flags); if (err) goto out_free_cache; @@ -344,7 +310,6 @@ kmem_cache_create_usercopy(const char *n get_online_cpus(); get_online_mems(); - memcg_get_cache_ids(); mutex_lock(&slab_mutex); @@ -394,7 +359,6 @@ kmem_cache_create_usercopy(const char *n out_unlock: mutex_unlock(&slab_mutex); - memcg_put_cache_ids(); put_online_mems(); put_online_cpus(); @@ -507,87 +471,6 @@ static int shutdown_cache(struct kmem_ca return 0; } -#ifdef CONFIG_MEMCG_KMEM -/* - * memcg_create_kmem_cache - Create a cache for non-root memory cgroups. - * @root_cache: The parent of the new cache. - * - * This function attempts to create a kmem cache that will serve allocation - * requests going all non-root memory cgroups to @root_cache. The new cache - * inherits properties from its parent. - */ -void memcg_create_kmem_cache(struct kmem_cache *root_cache) -{ - struct kmem_cache *s = NULL; - char *cache_name; - - get_online_cpus(); - get_online_mems(); - - mutex_lock(&slab_mutex); - - if (root_cache->memcg_params.memcg_cache) - goto out_unlock; - - cache_name = kasprintf(GFP_KERNEL, "%s-memcg", root_cache->name); - if (!cache_name) - goto out_unlock; - - s = create_cache(cache_name, root_cache->object_size, - root_cache->align, - root_cache->flags & CACHE_CREATE_MASK, - root_cache->useroffset, root_cache->usersize, - root_cache->ctor, root_cache); - /* - * If we could not create a memcg cache, do not complain, because - * that's not critical at all as we can always proceed with the root - * cache. - */ - if (IS_ERR(s)) { - kfree(cache_name); - goto out_unlock; - } - - /* - * Since readers won't lock (see memcg_slab_pre_alloc_hook()), we need a - * barrier here to ensure nobody will see the kmem_cache partially - * initialized. - */ - smp_wmb(); - root_cache->memcg_params.memcg_cache = s; - -out_unlock: - mutex_unlock(&slab_mutex); - - put_online_mems(); - put_online_cpus(); -} - -static int shutdown_memcg_caches(struct kmem_cache *s) -{ - BUG_ON(!is_root_cache(s)); - - if (s->memcg_params.memcg_cache) - WARN_ON(shutdown_cache(s->memcg_params.memcg_cache)); - - return 0; -} - -static void cancel_memcg_cache_creation(struct kmem_cache *s) -{ - cancel_work_sync(&s->memcg_params.work); -} -#else -static inline int shutdown_memcg_caches(struct kmem_cache *s) -{ - return 0; -} - -static inline void cancel_memcg_cache_creation(struct kmem_cache *s) -{ -} -#endif /* CONFIG_MEMCG_KMEM */ - void slab_kmem_cache_release(struct kmem_cache *s) { __kmem_cache_release(s); @@ -602,8 +485,6 @@ void kmem_cache_destroy(struct kmem_cach if (unlikely(!s)) return; - cancel_memcg_cache_creation(s); - get_online_cpus(); get_online_mems(); @@ -613,10 +494,7 @@ void kmem_cache_destroy(struct kmem_cach if (s->refcount) goto out_unlock; - err = shutdown_memcg_caches(s); - if (!err) - err = shutdown_cache(s); - + err = shutdown_cache(s); if (err) { pr_err("kmem_cache_destroy %s: Slab cache still has objects\n", s->name); @@ -653,33 +531,6 @@ int kmem_cache_shrink(struct kmem_cache } EXPORT_SYMBOL(kmem_cache_shrink); -/** - * kmem_cache_shrink_all - shrink root and memcg caches - * @s: The cache pointer - */ -void kmem_cache_shrink_all(struct kmem_cache *s) -{ - struct kmem_cache *c; - - if (!IS_ENABLED(CONFIG_MEMCG_KMEM) || !is_root_cache(s)) { - kmem_cache_shrink(s); - return; - } - - get_online_cpus(); - get_online_mems(); - kasan_cache_shrink(s); - __kmem_cache_shrink(s); - - c = memcg_cache(s); - if (c) { - kasan_cache_shrink(c); - __kmem_cache_shrink(c); - } - put_online_mems(); - put_online_cpus(); -} - bool slab_is_available(void) { return slab_state >= UP; @@ -708,8 +559,6 @@ void __init create_boot_cache(struct kme s->useroffset = useroffset; s->usersize = usersize; - slab_init_memcg_params(s); - err = __kmem_cache_create(s, flags); if (err) @@ -1098,25 +947,6 @@ void slab_stop(struct seq_file *m, void mutex_unlock(&slab_mutex); } -static void -memcg_accumulate_slabinfo(struct kmem_cache *s, struct slabinfo *info) -{ - struct kmem_cache *c; - struct slabinfo sinfo; - - c = memcg_cache(s); - if (c) { - memset(&sinfo, 0, sizeof(sinfo)); - get_slabinfo(c, &sinfo); - - info->active_slabs += sinfo.active_slabs; - info->num_slabs += sinfo.num_slabs; - info->shared_avail += sinfo.shared_avail; - info->active_objs += sinfo.active_objs; - info->num_objs += sinfo.num_objs; - } -} - static void cache_show(struct kmem_cache *s, struct seq_file *m) { struct slabinfo sinfo; @@ -1124,10 +954,8 @@ static void cache_show(struct kmem_cache memset(&sinfo, 0, sizeof(sinfo)); get_slabinfo(s, &sinfo); - memcg_accumulate_slabinfo(s, &sinfo); - seq_printf(m, "%-17s %6lu %6lu %6u %4u %4d", - cache_name(s), sinfo.active_objs, sinfo.num_objs, s->size, + s->name, sinfo.active_objs, sinfo.num_objs, s->size, sinfo.objects_per_slab, (1 << sinfo.cache_order)); seq_printf(m, " : tunables %4u %4u %4u", @@ -1144,8 +972,7 @@ static int slab_show(struct seq_file *m, if (p == slab_caches.next) print_slabinfo_header(m); - if (is_root_cache(s)) - cache_show(s, m); + cache_show(s, m); return 0; } @@ -1170,13 +997,13 @@ void dump_unreclaimable_slab(void) pr_info("Name Used Total\n"); list_for_each_entry_safe(s, s2, &slab_caches, list) { - if (!is_root_cache(s) || (s->flags & SLAB_RECLAIM_ACCOUNT)) + if (s->flags & SLAB_RECLAIM_ACCOUNT) continue; get_slabinfo(s, &sinfo); if (sinfo.num_objs > 0) - pr_info("%-17s %10luKB %10luKB\n", cache_name(s), + pr_info("%-17s %10luKB %10luKB\n", s->name, (sinfo.active_objs * s->size) / 1024, (sinfo.num_objs * s->size) / 1024); } @@ -1235,53 +1062,6 @@ static int __init slab_proc_init(void) } module_init(slab_proc_init); -#if defined(CONFIG_DEBUG_FS) && defined(CONFIG_MEMCG_KMEM) -/* - * Display information about kmem caches that have memcg cache. - */ -static int memcg_slabinfo_show(struct seq_file *m, void *unused) -{ - struct kmem_cache *s, *c; - struct slabinfo sinfo; - - mutex_lock(&slab_mutex); - seq_puts(m, "# "); - seq_puts(m, " \n"); - list_for_each_entry(s, &slab_caches, list) { - /* - * Skip kmem caches that don't have the memcg cache. - */ - if (!s->memcg_params.memcg_cache) - continue; - - memset(&sinfo, 0, sizeof(sinfo)); - get_slabinfo(s, &sinfo); - seq_printf(m, "%-17s root %6lu %6lu %6lu %6lu\n", - cache_name(s), sinfo.active_objs, sinfo.num_objs, - sinfo.active_slabs, sinfo.num_slabs); - - c = s->memcg_params.memcg_cache; - memset(&sinfo, 0, sizeof(sinfo)); - get_slabinfo(c, &sinfo); - seq_printf(m, "%-17s %4d %6lu %6lu %6lu %6lu\n", - cache_name(c), root_mem_cgroup->css.id, - sinfo.active_objs, sinfo.num_objs, - sinfo.active_slabs, sinfo.num_slabs); - } - mutex_unlock(&slab_mutex); - return 0; -} -DEFINE_SHOW_ATTRIBUTE(memcg_slabinfo); - -static int __init memcg_slabinfo_init(void) -{ - debugfs_create_file("memcg_slabinfo", S_IFREG | S_IRUGO, - NULL, NULL, &memcg_slabinfo_fops); - return 0; -} - -late_initcall(memcg_slabinfo_init); -#endif /* CONFIG_DEBUG_FS && CONFIG_MEMCG_KMEM */ #endif /* CONFIG_SLAB || CONFIG_SLUB_DEBUG */ static __always_inline void *__do_krealloc(const void *p, size_t new_size, --- a/mm/slab.h~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-allocations +++ a/mm/slab.h @@ -30,28 +30,6 @@ struct kmem_cache { struct list_head list; /* List of all slab caches on the system */ }; -#else /* !CONFIG_SLOB */ - -/* - * This is the main placeholder for memcg-related information in kmem caches. - * Both the root cache and the child cache will have it. Some fields are used - * in both cases, other are specific to root caches. - * - * @root_cache: Common to root and child caches. NULL for root, pointer to - * the root cache for children. - * - * The following fields are specific to root caches. - * - * @memcg_cache: pointer to memcg kmem cache, used by all non-root memory - * cgroups. - * @work: work struct used to create the non-root cache. - */ -struct memcg_cache_params { - struct kmem_cache *root_cache; - - struct kmem_cache *memcg_cache; - struct work_struct work; -}; #endif /* CONFIG_SLOB */ #ifdef CONFIG_SLAB @@ -196,7 +174,6 @@ int __kmem_cache_shutdown(struct kmem_ca void __kmem_cache_release(struct kmem_cache *); int __kmem_cache_shrink(struct kmem_cache *); void slab_kmem_cache_release(struct kmem_cache *); -void kmem_cache_shrink_all(struct kmem_cache *s); struct seq_file; struct file; @@ -263,43 +240,6 @@ static inline bool kmem_cache_debug_flag } #ifdef CONFIG_MEMCG_KMEM -static inline bool is_root_cache(struct kmem_cache *s) -{ - return !s->memcg_params.root_cache; -} - -static inline bool slab_equal_or_root(struct kmem_cache *s, - struct kmem_cache *p) -{ - return p == s || p == s->memcg_params.root_cache; -} - -/* - * We use suffixes to the name in memcg because we can't have caches - * created in the system with the same name. But when we print them - * locally, better refer to them with the base name - */ -static inline const char *cache_name(struct kmem_cache *s) -{ - if (!is_root_cache(s)) - s = s->memcg_params.root_cache; - return s->name; -} - -static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) -{ - if (is_root_cache(s)) - return s; - return s->memcg_params.root_cache; -} - -static inline struct kmem_cache *memcg_cache(struct kmem_cache *s) -{ - if (is_root_cache(s)) - return s->memcg_params.memcg_cache; - return NULL; -} - static inline struct obj_cgroup **page_obj_cgroups(struct page *page) { /* @@ -317,21 +257,8 @@ static inline bool page_has_obj_cgroups( return ((unsigned long)page->obj_cgroups & 0x1UL); } -static inline int memcg_alloc_page_obj_cgroups(struct page *page, - struct kmem_cache *s, gfp_t gfp) -{ - unsigned int objects = objs_per_slab_page(s, page); - void *vec; - - vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, - page_to_nid(page)); - if (!vec) - return -ENOMEM; - - kmemleak_not_leak(vec); - page->obj_cgroups = (struct obj_cgroup **) ((unsigned long)vec | 0x1UL); - return 0; -} +int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s, + gfp_t gfp); static inline void memcg_free_page_obj_cgroups(struct page *page) { @@ -348,38 +275,25 @@ static inline size_t obj_full_size(struc return s->size + sizeof(struct obj_cgroup *); } -static inline struct kmem_cache *memcg_slab_pre_alloc_hook(struct kmem_cache *s, - struct obj_cgroup **objcgp, - size_t objects, gfp_t flags) +static inline struct obj_cgroup *memcg_slab_pre_alloc_hook(struct kmem_cache *s, + size_t objects, + gfp_t flags) { - struct kmem_cache *cachep; struct obj_cgroup *objcg; if (memcg_kmem_bypass()) - return s; - - cachep = READ_ONCE(s->memcg_params.memcg_cache); - if (unlikely(!cachep)) { - /* - * If memcg cache does not exist yet, we schedule it's - * asynchronous creation and let the current allocation - * go through with the root cache. - */ - queue_work(system_wq, &s->memcg_params.work); - return s; - } + return NULL; objcg = get_obj_cgroup_from_current(); if (!objcg) - return s; + return NULL; if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) { obj_cgroup_put(objcg); - cachep = NULL; + return NULL; } - *objcgp = objcg; - return cachep; + return objcg; } static inline void mod_objcg_state(struct obj_cgroup *objcg, @@ -398,15 +312,27 @@ static inline void mod_objcg_state(struc static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, - size_t size, void **p) + gfp_t flags, size_t size, + void **p) { struct page *page; unsigned long off; size_t i; + if (!objcg) + return; + + flags &= ~__GFP_ACCOUNT; for (i = 0; i < size; i++) { if (likely(p[i])) { page = virt_to_head_page(p[i]); + + if (!page_has_obj_cgroups(page) && + memcg_alloc_page_obj_cgroups(page, s, flags)) { + obj_cgroup_uncharge(objcg, obj_full_size(s)); + continue; + } + off = obj_to_index(s, page, p[i]); obj_cgroup_get(objcg); page_obj_cgroups(page)[off] = objcg; @@ -425,13 +351,19 @@ static inline void memcg_slab_free_hook( struct obj_cgroup *objcg; unsigned int off; - if (!memcg_kmem_enabled() || is_root_cache(s)) + if (!memcg_kmem_enabled()) + return; + + if (!page_has_obj_cgroups(page)) return; off = obj_to_index(s, page, p); objcg = page_obj_cgroups(page)[off]; page_obj_cgroups(page)[off] = NULL; + if (!objcg) + return; + obj_cgroup_uncharge(objcg, obj_full_size(s)); mod_objcg_state(objcg, page_pgdat(page), cache_vmstat_idx(s), -obj_full_size(s)); @@ -439,35 +371,7 @@ static inline void memcg_slab_free_hook( obj_cgroup_put(objcg); } -extern void slab_init_memcg_params(struct kmem_cache *); - #else /* CONFIG_MEMCG_KMEM */ -static inline bool is_root_cache(struct kmem_cache *s) -{ - return true; -} - -static inline bool slab_equal_or_root(struct kmem_cache *s, - struct kmem_cache *p) -{ - return s == p; -} - -static inline const char *cache_name(struct kmem_cache *s) -{ - return s->name; -} - -static inline struct kmem_cache *memcg_root_cache(struct kmem_cache *s) -{ - return s; -} - -static inline struct kmem_cache *memcg_cache(struct kmem_cache *s) -{ - return NULL; -} - static inline bool page_has_obj_cgroups(struct page *page) { return false; @@ -488,16 +392,17 @@ static inline void memcg_free_page_obj_c { } -static inline struct kmem_cache *memcg_slab_pre_alloc_hook(struct kmem_cache *s, - struct obj_cgroup **objcgp, - size_t objects, gfp_t flags) +static inline struct obj_cgroup *memcg_slab_pre_alloc_hook(struct kmem_cache *s, + size_t objects, + gfp_t flags) { return NULL; } static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, - size_t size, void **p) + gfp_t flags, size_t size, + void **p) { } @@ -505,11 +410,6 @@ static inline void memcg_slab_free_hook( void *p) { } - -static inline void slab_init_memcg_params(struct kmem_cache *s) -{ -} - #endif /* CONFIG_MEMCG_KMEM */ static inline struct kmem_cache *virt_to_cache(const void *obj) @@ -523,27 +423,18 @@ static inline struct kmem_cache *virt_to return page->slab_cache; } -static __always_inline int charge_slab_page(struct page *page, - gfp_t gfp, int order, - struct kmem_cache *s) -{ - if (memcg_kmem_enabled() && !is_root_cache(s)) { - int ret; - - ret = memcg_alloc_page_obj_cgroups(page, s, gfp); - if (ret) - return ret; - } - +static __always_inline void charge_slab_page(struct page *page, + gfp_t gfp, int order, + struct kmem_cache *s) +{ mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), PAGE_SIZE << order); - return 0; } static __always_inline void uncharge_slab_page(struct page *page, int order, struct kmem_cache *s) { - if (memcg_kmem_enabled() && !is_root_cache(s)) + if (memcg_kmem_enabled()) memcg_free_page_obj_cgroups(page); mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), @@ -555,12 +446,11 @@ static inline struct kmem_cache *cache_f struct kmem_cache *cachep; if (!IS_ENABLED(CONFIG_SLAB_FREELIST_HARDENED) && - !memcg_kmem_enabled() && !kmem_cache_debug_flags(s, SLAB_CONSISTENCY_CHECKS)) return s; cachep = virt_to_cache(x); - if (WARN(cachep && !slab_equal_or_root(cachep, s), + if (WARN(cachep && cachep != s, "%s: Wrong slab cache. %s but object is from %s\n", __func__, s->name, cachep->name)) print_tracking(cachep, x); @@ -613,7 +503,7 @@ static inline struct kmem_cache *slab_pr if (memcg_kmem_enabled() && ((flags & __GFP_ACCOUNT) || (s->flags & SLAB_ACCOUNT))) - return memcg_slab_pre_alloc_hook(s, objcgp, size, flags); + *objcgp = memcg_slab_pre_alloc_hook(s, size, flags); return s; } @@ -632,8 +522,8 @@ static inline void slab_post_alloc_hook( s->flags, flags); } - if (memcg_kmem_enabled() && !is_root_cache(s)) - memcg_slab_post_alloc_hook(s, objcg, size, p); + if (memcg_kmem_enabled()) + memcg_slab_post_alloc_hook(s, objcg, flags, size, p); } #ifndef CONFIG_SLOB --- a/mm/slub.c~mm-memcg-slab-use-a-single-set-of-kmem_caches-for-all-allocations +++ a/mm/slub.c @@ -218,14 +218,10 @@ enum track_item { TRACK_ALLOC, TRACK_FRE #ifdef CONFIG_SYSFS static int sysfs_slab_add(struct kmem_cache *); static int sysfs_slab_alias(struct kmem_cache *, const char *); -static void memcg_propagate_slab_attrs(struct kmem_cache *s); -static void sysfs_slab_remove(struct kmem_cache *s); #else static inline int sysfs_slab_add(struct kmem_cache *s) { return 0; } static inline int sysfs_slab_alias(struct kmem_cache *s, const char *p) { return 0; } -static inline void memcg_propagate_slab_attrs(struct kmem_cache *s) { } -static inline void sysfs_slab_remove(struct kmem_cache *s) { } #endif static inline void stat(const struct kmem_cache *s, enum stat_item si) @@ -1624,10 +1620,8 @@ static inline struct page *alloc_slab_pa else page = __alloc_pages_node(node, flags, order); - if (page && charge_slab_page(page, flags, order, s)) { - __free_pages(page, order); - page = NULL; - } + if (page) + charge_slab_page(page, flags, order, s); return page; } @@ -3920,7 +3914,6 @@ int __kmem_cache_shutdown(struct kmem_ca if (n->nr_partial || slabs_node(s, node)) return 1; } - sysfs_slab_remove(s); return 0; } @@ -4358,7 +4351,6 @@ static struct kmem_cache * __init bootst p->slab_cache = s; #endif } - slab_init_memcg_params(s); list_add(&s->list, &slab_caches); return s; } @@ -4414,7 +4406,7 @@ struct kmem_cache * __kmem_cache_alias(const char *name, unsigned int size, unsigned int align, slab_flags_t flags, void (*ctor)(void *)) { - struct kmem_cache *s, *c; + struct kmem_cache *s; s = find_mergeable(size, align, flags, name, ctor); if (s) { @@ -4427,12 +4419,6 @@ __kmem_cache_alias(const char *name, uns s->object_size = max(s->object_size, size); s->inuse = max(s->inuse, ALIGN(size, sizeof(void *))); - c = memcg_cache(s); - if (c) { - c->object_size = s->object_size; - c->inuse = max(c->inuse, ALIGN(size, sizeof(void *))); - } - if (sysfs_slab_alias(s, name)) { s->refcount--; s = NULL; @@ -4454,7 +4440,6 @@ int __kmem_cache_create(struct kmem_cach if (slab_state <= UP) return 0; - memcg_propagate_slab_attrs(s); err = sysfs_slab_add(s); if (err) __kmem_cache_release(s); @@ -5312,7 +5297,7 @@ static ssize_t shrink_store(struct kmem_ const char *buf, size_t length) { if (buf[0] == '1') - kmem_cache_shrink_all(s); + kmem_cache_shrink(s); else return -EINVAL; return length; @@ -5536,99 +5521,9 @@ static ssize_t slab_attr_store(struct ko return -EIO; err = attribute->store(s, buf, len); -#ifdef CONFIG_MEMCG - if (slab_state >= FULL && err >= 0 && is_root_cache(s)) { - struct kmem_cache *c; - - mutex_lock(&slab_mutex); - if (s->max_attr_size < len) - s->max_attr_size = len; - - /* - * This is a best effort propagation, so this function's return - * value will be determined by the parent cache only. This is - * basically because not all attributes will have a well - * defined semantics for rollbacks - most of the actions will - * have permanent effects. - * - * Returning the error value of any of the children that fail - * is not 100 % defined, in the sense that users seeing the - * error code won't be able to know anything about the state of - * the cache. - * - * Only returning the error code for the parent cache at least - * has well defined semantics. The cache being written to - * directly either failed or succeeded, in which case we loop - * through the descendants with best-effort propagation. - */ - c = memcg_cache(s); - if (c) - attribute->store(c, buf, len); - mutex_unlock(&slab_mutex); - } -#endif return err; } -static void memcg_propagate_slab_attrs(struct kmem_cache *s) -{ -#ifdef CONFIG_MEMCG - int i; - char *buffer = NULL; - struct kmem_cache *root_cache; - - if (is_root_cache(s)) - return; - - root_cache = s->memcg_params.root_cache; - - /* - * This mean this cache had no attribute written. Therefore, no point - * in copying default values around - */ - if (!root_cache->max_attr_size) - return; - - for (i = 0; i < ARRAY_SIZE(slab_attrs); i++) { - char mbuf[64]; - char *buf; - struct slab_attribute *attr = to_slab_attr(slab_attrs[i]); - ssize_t len; - - if (!attr || !attr->store || !attr->show) - continue; - - /* - * It is really bad that we have to allocate here, so we will - * do it only as a fallback. If we actually allocate, though, - * we can just use the allocated buffer until the end. - * - * Most of the slub attributes will tend to be very small in - * size, but sysfs allows buffers up to a page, so they can - * theoretically happen. - */ - if (buffer) - buf = buffer; - else if (root_cache->max_attr_size < ARRAY_SIZE(mbuf) && - !IS_ENABLED(CONFIG_SLUB_STATS)) - buf = mbuf; - else { - buffer = (char *) get_zeroed_page(GFP_KERNEL); - if (WARN_ON(!buffer)) - continue; - buf = buffer; - } - - len = attr->show(root_cache, buf); - if (len > 0) - attr->store(s, buf, len); - } - - if (buffer) - free_page((unsigned long)buffer); -#endif /* CONFIG_MEMCG */ -} - static void kmem_cache_release(struct kobject *k) { slab_kmem_cache_release(to_slab(k)); @@ -5648,10 +5543,6 @@ static struct kset *slab_kset; static inline struct kset *cache_kset(struct kmem_cache *s) { -#ifdef CONFIG_MEMCG - if (!is_root_cache(s)) - return s->memcg_params.root_cache->memcg_kset; -#endif return slab_kset; } @@ -5694,27 +5585,6 @@ static char *create_unique_id(struct kme return name; } -static void sysfs_slab_remove_workfn(struct work_struct *work) -{ - struct kmem_cache *s = - container_of(work, struct kmem_cache, kobj_remove_work); - - if (!s->kobj.state_in_sysfs) - /* - * For a memcg cache, this may be called during - * deactivation and again on shutdown. Remove only once. - * A cache is never shut down before deactivation is - * complete, so no need to worry about synchronization. - */ - goto out; - -#ifdef CONFIG_MEMCG - kset_unregister(s->memcg_kset); -#endif -out: - kobject_put(&s->kobj); -} - static int sysfs_slab_add(struct kmem_cache *s) { int err; @@ -5722,8 +5592,6 @@ static int sysfs_slab_add(struct kmem_ca struct kset *kset = cache_kset(s); int unmergeable = slab_unmergeable(s); - INIT_WORK(&s->kobj_remove_work, sysfs_slab_remove_workfn); - if (!kset) { kobject_init(&s->kobj, &slab_ktype); return 0; @@ -5760,16 +5628,6 @@ static int sysfs_slab_add(struct kmem_ca if (err) goto out_del_kobj; -#ifdef CONFIG_MEMCG - if (is_root_cache(s) && memcg_sysfs_enabled) { - s->memcg_kset = kset_create_and_add("cgroup", NULL, &s->kobj); - if (!s->memcg_kset) { - err = -ENOMEM; - goto out_del_kobj; - } - } -#endif - if (!unmergeable) { /* Setup first alias */ sysfs_slab_alias(s, s->name); @@ -5783,19 +5641,6 @@ out_del_kobj: goto out; } -static void sysfs_slab_remove(struct kmem_cache *s) -{ - if (slab_state < FULL) - /* - * Sysfs has not been setup yet so no need to remove the - * cache from sysfs. - */ - return; - - kobject_get(&s->kobj); - schedule_work(&s->kobj_remove_work); -} - void sysfs_slab_unlink(struct kmem_cache *s) { if (slab_state >= FULL) From patchwork Fri Aug 7 06:21:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704833 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AB5DC138C for ; Fri, 7 Aug 2020 06:21:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6E852221E5 for ; Fri, 7 Aug 2020 06:21:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="LruEwtSc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6E852221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 530A68D005A; Fri, 7 Aug 2020 02:21:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4E1898D0026; Fri, 7 Aug 2020 02:21:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F6528D005A; Fri, 7 Aug 2020 02:21:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0191.hostedemail.com [216.40.44.191]) by kanga.kvack.org (Postfix) with ESMTP id 299778D0026 for ; Fri, 7 Aug 2020 02:21:40 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id ECFE982499A8 for ; Fri, 7 Aug 2020 06:21:39 +0000 (UTC) X-FDA: 77122776318.02.boy32_2513b3826fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id CA40310097AA0 for ; Fri, 7 Aug 2020 06:21:39 +0000 (UTC) X-Spam-Summary: 1,0,0,403b8ee216b1a7df,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:41:69:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:2194:2196:2199:2200:2393:2525:2559:2563:2636:2682:2685:2693:2859:2890:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3308:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4042:4321:4385:4605:5007:6119:6261:6653:7576:7903:8599:8957:9025:9121:9545:9592:10004:10913:11026:11232:11233:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13227:13229:13846:14877:21063:21080:21433:21450:21451:21627:21660:21939:21987:21990:30001:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y85pejyc784wbjnk9baeq36tf5foctcnd7uaimkmrsfh8w1cckgt9wuj8m6om.gzf4p15sm3kb531whb4dgy115irfg8uq3f9namp9phkwc4o8tsfxib48nqeby96.g-lbl8.mailshell.net-223.238.255.10 0,CacheI X-HE-Tag: boy32_2513b3826fbe X-Filterd-Recvd-Size: 12134 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:39 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 24FF9221E5; Fri, 7 Aug 2020 06:21:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781298; bh=7lTOnYs2XH9AD5uvPszcKj3AytJenemPC3++jcHRye8=; h=Date:From:To:Subject:In-Reply-To:From; b=LruEwtSc2zzVz9lnL3PDTmki8rzswSFj3xcyeL14QQt4MwVoW3MUkNyEM3dXuUJoo FNmNiD2IEOAMw1LbP69XqbJ0B990ibaoIjRP1TmHlK/QHnQVTK8qghR3YqjHnW9+mf WxfAZEAoANLc5pzQP0LhEuJUH/xyhoIQ9ssLJfhM= Date: Thu, 06 Aug 2020 23:21:37 -0700 From: Andrew Morton To: akpm@linux-foundation.org, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, torvalds@linux-foundation.org Subject: [patch 081/163] mm: memcontrol: account kernel stack per node Message-ID: <20200807062137.o44Vg87wz%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: CA40310097AA0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Shakeel Butt Subject: mm: memcontrol: account kernel stack per node Currently the kernel stack is being accounted per-zone. There is no need to do that. In addition due to being per-zone, memcg has to keep a separate MEMCG_KERNEL_STACK_KB. Make the stat per-node and deprecate MEMCG_KERNEL_STACK_KB as memcg_stat_item is an extension of node_stat_item. In addition localize the kernel stack stats updates to account_kernel_stack(). Link: http://lkml.kernel.org/r/20200630161539.1759185-1-shakeelb@google.com Signed-off-by: Shakeel Butt Reviewed-by: Roman Gushchin Cc: Johannes Weiner Cc: Michal Hocko Signed-off-by: Andrew Morton --- drivers/base/node.c | 4 +- fs/proc/meminfo.c | 4 +- include/linux/memcontrol.h | 21 +++++++++++++- include/linux/mmzone.h | 8 ++--- kernel/fork.c | 51 +++++++++-------------------------- kernel/scs.c | 2 - mm/memcontrol.c | 2 - mm/page_alloc.c | 16 +++++----- mm/vmstat.c | 8 ++--- 9 files changed, 55 insertions(+), 61 deletions(-) --- a/drivers/base/node.c~mm-memcontrol-account-kernel-stack-per-node +++ a/drivers/base/node.c @@ -440,9 +440,9 @@ static ssize_t node_read_meminfo(struct nid, K(node_page_state(pgdat, NR_FILE_MAPPED)), nid, K(node_page_state(pgdat, NR_ANON_MAPPED)), nid, K(i.sharedram), - nid, sum_zone_node_page_state(nid, NR_KERNEL_STACK_KB), + nid, node_page_state(pgdat, NR_KERNEL_STACK_KB), #ifdef CONFIG_SHADOW_CALL_STACK - nid, sum_zone_node_page_state(nid, NR_KERNEL_SCS_KB), + nid, node_page_state(pgdat, NR_KERNEL_SCS_KB), #endif nid, K(sum_zone_node_page_state(nid, NR_PAGETABLE)), nid, 0UL, --- a/fs/proc/meminfo.c~mm-memcontrol-account-kernel-stack-per-node +++ a/fs/proc/meminfo.c @@ -101,10 +101,10 @@ static int meminfo_proc_show(struct seq_ show_val_kb(m, "SReclaimable: ", sreclaimable); show_val_kb(m, "SUnreclaim: ", sunreclaim); seq_printf(m, "KernelStack: %8lu kB\n", - global_zone_page_state(NR_KERNEL_STACK_KB)); + global_node_page_state(NR_KERNEL_STACK_KB)); #ifdef CONFIG_SHADOW_CALL_STACK seq_printf(m, "ShadowCallStack:%8lu kB\n", - global_zone_page_state(NR_KERNEL_SCS_KB)); + global_node_page_state(NR_KERNEL_SCS_KB)); #endif show_val_kb(m, "PageTables: ", global_zone_page_state(NR_PAGETABLE)); --- a/include/linux/memcontrol.h~mm-memcontrol-account-kernel-stack-per-node +++ a/include/linux/memcontrol.h @@ -32,8 +32,6 @@ struct kmem_cache; enum memcg_stat_item { MEMCG_SWAP = NR_VM_NODE_STAT_ITEMS, MEMCG_SOCK, - /* XXX: why are these zone and not node counters? */ - MEMCG_KERNEL_STACK_KB, MEMCG_NR_STAT, }; @@ -729,8 +727,19 @@ void __mod_memcg_lruvec_state(struct lru void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); void __mod_lruvec_slab_state(void *p, enum node_stat_item idx, int val); + void mod_memcg_obj_state(void *p, int idx, int val); +static inline void mod_lruvec_slab_state(void *p, enum node_stat_item idx, + int val) +{ + unsigned long flags; + + local_irq_save(flags); + __mod_lruvec_slab_state(p, idx, val); + local_irq_restore(flags); +} + static inline void mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val) { @@ -1151,6 +1160,14 @@ static inline void __mod_lruvec_slab_sta __mod_node_page_state(page_pgdat(page), idx, val); } +static inline void mod_lruvec_slab_state(void *p, enum node_stat_item idx, + int val) +{ + struct page *page = virt_to_head_page(p); + + mod_node_page_state(page_pgdat(page), idx, val); +} + static inline void mod_memcg_obj_state(void *p, int idx, int val) { } --- a/include/linux/mmzone.h~mm-memcontrol-account-kernel-stack-per-node +++ a/include/linux/mmzone.h @@ -155,10 +155,6 @@ enum zone_stat_item { NR_ZONE_WRITE_PENDING, /* Count of dirty, writeback and unstable pages */ NR_MLOCK, /* mlock()ed pages found and moved off LRU */ NR_PAGETABLE, /* used for pagetables */ - NR_KERNEL_STACK_KB, /* measured in KiB */ -#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) - NR_KERNEL_SCS_KB, /* measured in KiB */ -#endif /* Second 128 byte cacheline */ NR_BOUNCE, #if IS_ENABLED(CONFIG_ZSMALLOC) @@ -203,6 +199,10 @@ enum node_stat_item { NR_KERNEL_MISC_RECLAIMABLE, /* reclaimable non-slab kernel pages */ NR_FOLL_PIN_ACQUIRED, /* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED, /* pages returned via unpin_user_page() */ + NR_KERNEL_STACK_KB, /* measured in KiB */ +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) + NR_KERNEL_SCS_KB, /* measured in KiB */ +#endif NR_VM_NODE_STAT_ITEMS }; --- a/kernel/fork.c~mm-memcontrol-account-kernel-stack-per-node +++ a/kernel/fork.c @@ -276,13 +276,8 @@ static inline void free_thread_stack(str if (vm) { int i; - for (i = 0; i < THREAD_SIZE / PAGE_SIZE; i++) { - mod_memcg_page_state(vm->pages[i], - MEMCG_KERNEL_STACK_KB, - -(int)(PAGE_SIZE / 1024)); - + for (i = 0; i < THREAD_SIZE / PAGE_SIZE; i++) memcg_kmem_uncharge_page(vm->pages[i], 0); - } for (i = 0; i < NR_CACHED_STACKS; i++) { if (this_cpu_cmpxchg(cached_stacks[i], @@ -382,31 +377,14 @@ static void account_kernel_stack(struct void *stack = task_stack_page(tsk); struct vm_struct *vm = task_stack_vm_area(tsk); - BUILD_BUG_ON(IS_ENABLED(CONFIG_VMAP_STACK) && PAGE_SIZE % 1024 != 0); - if (vm) { - int i; - - BUG_ON(vm->nr_pages != THREAD_SIZE / PAGE_SIZE); - - for (i = 0; i < THREAD_SIZE / PAGE_SIZE; i++) { - mod_zone_page_state(page_zone(vm->pages[i]), - NR_KERNEL_STACK_KB, - PAGE_SIZE / 1024 * account); - } - } else { - /* - * All stack pages are in the same zone and belong to the - * same memcg. - */ - struct page *first_page = virt_to_page(stack); - - mod_zone_page_state(page_zone(first_page), NR_KERNEL_STACK_KB, - THREAD_SIZE / 1024 * account); - - mod_memcg_obj_state(stack, MEMCG_KERNEL_STACK_KB, - account * (THREAD_SIZE / 1024)); - } + /* All stack pages are in the same node. */ + if (vm) + mod_lruvec_page_state(vm->pages[0], NR_KERNEL_STACK_KB, + account * (THREAD_SIZE / 1024)); + else + mod_lruvec_slab_state(stack, NR_KERNEL_STACK_KB, + account * (THREAD_SIZE / 1024)); } static int memcg_charge_kernel_stack(struct task_struct *tsk) @@ -415,24 +393,23 @@ static int memcg_charge_kernel_stack(str struct vm_struct *vm = task_stack_vm_area(tsk); int ret; + BUILD_BUG_ON(IS_ENABLED(CONFIG_VMAP_STACK) && PAGE_SIZE % 1024 != 0); + if (vm) { int i; + BUG_ON(vm->nr_pages != THREAD_SIZE / PAGE_SIZE); + for (i = 0; i < THREAD_SIZE / PAGE_SIZE; i++) { /* * If memcg_kmem_charge_page() fails, page->mem_cgroup - * pointer is NULL, and both memcg_kmem_uncharge_page() - * and mod_memcg_page_state() in free_thread_stack() - * will ignore this page. So it's safe. + * pointer is NULL, and memcg_kmem_uncharge_page() in + * free_thread_stack() will ignore this page. */ ret = memcg_kmem_charge_page(vm->pages[i], GFP_KERNEL, 0); if (ret) return ret; - - mod_memcg_page_state(vm->pages[i], - MEMCG_KERNEL_STACK_KB, - PAGE_SIZE / 1024); } } #endif --- a/kernel/scs.c~mm-memcontrol-account-kernel-stack-per-node +++ a/kernel/scs.c @@ -17,7 +17,7 @@ static void __scs_account(void *s, int a { struct page *scs_page = virt_to_page(s); - mod_zone_page_state(page_zone(scs_page), NR_KERNEL_SCS_KB, + mod_node_page_state(page_pgdat(scs_page), NR_KERNEL_SCS_KB, account * (SCS_SIZE / SZ_1K)); } --- a/mm/memcontrol.c~mm-memcontrol-account-kernel-stack-per-node +++ a/mm/memcontrol.c @@ -1485,7 +1485,7 @@ static char *memory_stat_format(struct m (u64)memcg_page_state(memcg, NR_FILE_PAGES) * PAGE_SIZE); seq_buf_printf(&s, "kernel_stack %llu\n", - (u64)memcg_page_state(memcg, MEMCG_KERNEL_STACK_KB) * + (u64)memcg_page_state(memcg, NR_KERNEL_STACK_KB) * 1024); seq_buf_printf(&s, "slab %llu\n", (u64)(memcg_page_state(memcg, NR_SLAB_RECLAIMABLE_B) + --- a/mm/page_alloc.c~mm-memcontrol-account-kernel-stack-per-node +++ a/mm/page_alloc.c @@ -5396,6 +5396,10 @@ void show_free_areas(unsigned int filter " anon_thp: %lukB" #endif " writeback_tmp:%lukB" + " kernel_stack:%lukB" +#ifdef CONFIG_SHADOW_CALL_STACK + " shadow_call_stack:%lukB" +#endif " all_unreclaimable? %s" "\n", pgdat->node_id, @@ -5417,6 +5421,10 @@ void show_free_areas(unsigned int filter K(node_page_state(pgdat, NR_ANON_THPS) * HPAGE_PMD_NR), #endif K(node_page_state(pgdat, NR_WRITEBACK_TEMP)), + node_page_state(pgdat, NR_KERNEL_STACK_KB), +#ifdef CONFIG_SHADOW_CALL_STACK + node_page_state(pgdat, NR_KERNEL_SCS_KB), +#endif pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES ? "yes" : "no"); } @@ -5448,10 +5456,6 @@ void show_free_areas(unsigned int filter " present:%lukB" " managed:%lukB" " mlocked:%lukB" - " kernel_stack:%lukB" -#ifdef CONFIG_SHADOW_CALL_STACK - " shadow_call_stack:%lukB" -#endif " pagetables:%lukB" " bounce:%lukB" " free_pcp:%lukB" @@ -5473,10 +5477,6 @@ void show_free_areas(unsigned int filter K(zone->present_pages), K(zone_managed_pages(zone)), K(zone_page_state(zone, NR_MLOCK)), - zone_page_state(zone, NR_KERNEL_STACK_KB), -#ifdef CONFIG_SHADOW_CALL_STACK - zone_page_state(zone, NR_KERNEL_SCS_KB), -#endif K(zone_page_state(zone, NR_PAGETABLE)), K(zone_page_state(zone, NR_BOUNCE)), K(free_pcp), --- a/mm/vmstat.c~mm-memcontrol-account-kernel-stack-per-node +++ a/mm/vmstat.c @@ -1140,10 +1140,6 @@ const char * const vmstat_text[] = { "nr_zone_write_pending", "nr_mlock", "nr_page_table_pages", - "nr_kernel_stack", -#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) - "nr_shadow_call_stack", -#endif "nr_bounce", #if IS_ENABLED(CONFIG_ZSMALLOC) "nr_zspages", @@ -1194,6 +1190,10 @@ const char * const vmstat_text[] = { "nr_kernel_misc_reclaimable", "nr_foll_pin_acquired", "nr_foll_pin_released", + "nr_kernel_stack", +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) + "nr_shadow_call_stack", +#endif /* enum writeback_stat_item counters */ "nr_dirty_threshold", From patchwork Fri Aug 7 06:21:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704835 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EBEB714B7 for ; Fri, 7 Aug 2020 06:21:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B0C2A2177B for ; Fri, 7 Aug 2020 06:21:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="cwhzIOQo" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B0C2A2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7C8448D005B; Fri, 7 Aug 2020 02:21:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 777FE8D0026; Fri, 7 Aug 2020 02:21:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68E688D005B; Fri, 7 Aug 2020 02:21:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id 537998D0026 for ; Fri, 7 Aug 2020 02:21:43 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 26517180AD801 for ; Fri, 7 Aug 2020 06:21:43 +0000 (UTC) X-FDA: 77122776486.15.snail41_4d003d326fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id E9F5B1814B0C7 for ; Fri, 7 Aug 2020 06:21:42 +0000 (UTC) X-Spam-Summary: 1,0,0,bb384bc11b15fd59,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4419:5007:6119:6261:6653:6737:7576:7901:7903:8599:8957:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12517:12519:12555:12679:12783:12986:14181:14721:14849:21080:21451:21627:21939:21990:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8447pgcywkktoonbkdp7cs7cr3ycm9ojnkwut6m66osp8csmo4mxntg9apt4.bamabmrj1k1s3nx19dmpquy8ishyxnbkmu9xf97csjc7hx3q8u5aeswd8soutdm.g-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rule s:0:0:0, X-HE-Tag: snail41_4d003d326fbe X-Filterd-Recvd-Size: 3547 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:42 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4955C22CAF; Fri, 7 Aug 2020 06:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781301; bh=sCu/LXHeNRB3b7T1bXZGPdiJpSQ0GevMbRfzdCd1y8k=; h=Date:From:To:Subject:In-Reply-To:From; b=cwhzIOQos2CPex616yp3EZv9cVfFH7Ss3tk6Oj+DZPkhqn10ypvZWoxV1OC2gLWBL 5N9kuvXRxf8jtAAx0jXBX1xd+OTRCZPFbGiC/jxKlocY1R7/iDzT4QpwRFTRJCdxvq 5aBMH1RqrniwOkyIKsSULpKimYNBf8fglxD3DvP0= Date: Thu, 06 Aug 2020 23:21:40 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, shakeelb@google.com, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 082/163] mm: memcg/slab: remove unused argument by charge_slab_page() Message-ID: <20200807062140.bVEg7TBZa%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E9F5B1814B0C7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcg/slab: remove unused argument by charge_slab_page() charge_slab_page() is not using the gfp argument anymore, remove it. Link: http://lkml.kernel.org/r/20200707173612.124425-1-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Johannes Weiner Cc: Michal Hocko Signed-off-by: Andrew Morton --- mm/slab.c | 2 +- mm/slab.h | 3 +-- mm/slub.c | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-) --- a/mm/slab.c~mm-memcg-slab-remove-unused-argument-by-charge_slab_page +++ a/mm/slab.c @@ -1379,7 +1379,7 @@ static struct page *kmem_getpages(struct return NULL; } - charge_slab_page(page, flags, cachep->gfporder, cachep); + charge_slab_page(page, cachep->gfporder, cachep); __SetPageSlab(page); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks() && page_is_pfmemalloc(page)) --- a/mm/slab.h~mm-memcg-slab-remove-unused-argument-by-charge_slab_page +++ a/mm/slab.h @@ -423,8 +423,7 @@ static inline struct kmem_cache *virt_to return page->slab_cache; } -static __always_inline void charge_slab_page(struct page *page, - gfp_t gfp, int order, +static __always_inline void charge_slab_page(struct page *page, int order, struct kmem_cache *s) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), --- a/mm/slub.c~mm-memcg-slab-remove-unused-argument-by-charge_slab_page +++ a/mm/slub.c @@ -1621,7 +1621,7 @@ static inline struct page *alloc_slab_pa page = __alloc_pages_node(node, flags, order); if (page) - charge_slab_page(page, flags, order, s); + charge_slab_page(page, order, s); return page; } From patchwork Fri Aug 7 06:21:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704837 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9486F14B7 for ; Fri, 7 Aug 2020 06:21:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5CFAB22D08 for ; Fri, 7 Aug 2020 06:21:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ah1kSBz/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5CFAB22D08 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1D4D38D005C; Fri, 7 Aug 2020 02:21:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1857B8D0026; Fri, 7 Aug 2020 02:21:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09C1D8D005C; Fri, 7 Aug 2020 02:21:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0091.hostedemail.com [216.40.44.91]) by kanga.kvack.org (Postfix) with ESMTP id E7B678D0026 for ; Fri, 7 Aug 2020 02:21:46 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B60A83625 for ; Fri, 7 Aug 2020 06:21:46 +0000 (UTC) X-FDA: 77122776612.03.chess34_5104be126fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 8C6C428A4E8 for ; Fri, 7 Aug 2020 06:21:46 +0000 (UTC) X-Spam-Summary: 1,0,0,b18c7ae096cf261a,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3867:3868:3870:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:4605:5007:6261:6653:6737:7576:8599:8957:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13255:14181:14721:14849:21080:21450:21451:21627:21939:21990:30030:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrmwdq1ywbcp5gs9fzgh6ssuha9opoq318yn659unezu6fd514hdnsgm1e11a.7hkbqbb54uep639qmpcqf1sfrjwys5twmemptr6dzxrgwnijutn8ht6u45crf11.e-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF: fp,MSBL: X-HE-Tag: chess34_5104be126fbe X-Filterd-Recvd-Size: 4638 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:46 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C0FE022CF6; Fri, 7 Aug 2020 06:21:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781305; bh=xOWgJD48Gb7TDVGrHga5xs7hMO5A84SRp7XarTCaMc0=; h=Date:From:To:Subject:In-Reply-To:From; b=Ah1kSBz/aC9BleI6c2YS2Eyb8twHCDtbzNouf0LAafDwB2eIwiAjFRhAdZXLTM1Zi 7RhQclh2XACdOzK6VDn+v4QXOBma0ntIAkDJjiB/oELpfity0MUeSrIM2fm3sdA6zm ZWcJLWsav3srzOaVNFA+GhoSeo2QK7uhOHjAHq8I= Date: Thu, 06 Aug 2020 23:21:44 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, shakeelb@google.com, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 083/163] mm: slab: rename (un)charge_slab_page() to (un)account_slab_page() Message-ID: <20200807062144.9neae9BKk%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 8C6C428A4E8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.001012, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: slab: rename (un)charge_slab_page() to (un)account_slab_page() charge_slab_page() and uncharge_slab_page() are not related anymore to memcg charging and uncharging. In order to make their names less confusing, let's rename them to account_slab_page() and unaccount_slab_page() respectively. Link: http://lkml.kernel.org/r/20200707173612.124425-2-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Vlastimil Babka Cc: Christoph Lameter Cc: David Rientjes Cc: Johannes Weiner Cc: Joonsoo Kim Cc: Michal Hocko Cc: Pekka Enberg Signed-off-by: Andrew Morton --- mm/slab.c | 4 ++-- mm/slab.h | 8 ++++---- mm/slub.c | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-) --- a/mm/slab.c~mm-slab-rename-uncharge_slab_page-to-unaccount_slab_page +++ a/mm/slab.c @@ -1379,7 +1379,7 @@ static struct page *kmem_getpages(struct return NULL; } - charge_slab_page(page, cachep->gfporder, cachep); + account_slab_page(page, cachep->gfporder, cachep); __SetPageSlab(page); /* Record if ALLOC_NO_WATERMARKS was set when allocating the slab */ if (sk_memalloc_socks() && page_is_pfmemalloc(page)) @@ -1403,7 +1403,7 @@ static void kmem_freepages(struct kmem_c if (current->reclaim_state) current->reclaim_state->reclaimed_slab += 1 << order; - uncharge_slab_page(page, order, cachep); + unaccount_slab_page(page, order, cachep); __free_pages(page, order); } --- a/mm/slab.h~mm-slab-rename-uncharge_slab_page-to-unaccount_slab_page +++ a/mm/slab.h @@ -423,15 +423,15 @@ static inline struct kmem_cache *virt_to return page->slab_cache; } -static __always_inline void charge_slab_page(struct page *page, int order, - struct kmem_cache *s) +static __always_inline void account_slab_page(struct page *page, int order, + struct kmem_cache *s) { mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s), PAGE_SIZE << order); } -static __always_inline void uncharge_slab_page(struct page *page, int order, - struct kmem_cache *s) +static __always_inline void unaccount_slab_page(struct page *page, int order, + struct kmem_cache *s) { if (memcg_kmem_enabled()) memcg_free_page_obj_cgroups(page); --- a/mm/slub.c~mm-slab-rename-uncharge_slab_page-to-unaccount_slab_page +++ a/mm/slub.c @@ -1621,7 +1621,7 @@ static inline struct page *alloc_slab_pa page = __alloc_pages_node(node, flags, order); if (page) - charge_slab_page(page, order, s); + account_slab_page(page, order, s); return page; } @@ -1844,7 +1844,7 @@ static void __free_slab(struct kmem_cach page->mapping = NULL; if (current->reclaim_state) current->reclaim_state->reclaimed_slab += pages; - uncharge_slab_page(page, order, s); + unaccount_slab_page(page, order, s); __free_pages(page, order); } From patchwork Fri Aug 7 06:21:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704839 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 04DA7138C for ; Fri, 7 Aug 2020 06:21:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C0A6422D03 for ; Fri, 7 Aug 2020 06:21:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="lCqRl/zT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C0A6422D03 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8B3248D005D; Fri, 7 Aug 2020 02:21:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 88AD78D0026; Fri, 7 Aug 2020 02:21:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C7718D005D; Fri, 7 Aug 2020 02:21:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0232.hostedemail.com [216.40.44.232]) by kanga.kvack.org (Postfix) with ESMTP id 684E68D0026 for ; Fri, 7 Aug 2020 02:21:50 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 36BD23625 for ; Fri, 7 Aug 2020 06:21:50 +0000 (UTC) X-FDA: 77122776780.28.base69_5b15adc26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 0F4126D74 for ; Fri, 7 Aug 2020 06:21:50 +0000 (UTC) X-Spam-Summary: 1,0,0,913f34db0004cfef,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2740:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:6737:7576:7903:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12114:12297:12438:12517:12519:12555:12679:12783:12986:13069:13227:13229:13255:13311:13357:14181:14384:14721:14849:21080:21433:21450:21451:21627:21939:21990:30012:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yr4ib9x9ee7dqgxwjak9ohskrnwyp4fzxhy9abztbmhuxnjopo5pxpdz14q91.5h37iwwxj1tiipsqso6qrg9rqua4dh3pxporqdi3ajojwkm7zm8ccj41n35wtj7.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,Domai nCache:0 X-HE-Tag: base69_5b15adc26fbe X-Filterd-Recvd-Size: 3231 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:49 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3D2C42177B; Fri, 7 Aug 2020 06:21:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781308; bh=5EErJWA38wl42unJIOii+c8EWiSGmCmFx+nSjMA4Av4=; h=Date:From:To:Subject:In-Reply-To:From; b=lCqRl/zTeKDz1KHviVYbGaN6PcVajkxMg9ZXwrKoCj2RXIjHAGQkrD7ZjCvzFYzAB lbPAFoHQdE+WrJOUtaWZVF0+XUz8ZzROinIIwsQsMEMl+RJaFJCRNrBV5MZHyNeDxs +Po0+Lpq51FOKVJbz5V4y1TMLJcrpvxMpLnXvqbA= Date: Thu, 06 Aug 2020 23:21:47 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, guro@fb.com, hannes@cmpxchg.org, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, shakeelb@google.com, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 084/163] mm: kmem: switch to static_branch_likely() in memcg_kmem_enabled() Message-ID: <20200807062147.3pKXVD4Kh%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 0F4126D74 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000010, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: kmem: switch to static_branch_likely() in memcg_kmem_enabled() Currently memcg_kmem_enabled() is optimized for the kernel memory accounting being off. It was so for a long time, and arguably the reason behind was that the kernel memory accounting was initially an opt-in feature. However, now it's on by default on both cgroup v1 and cgroup v2, and it's on for all cgroups. So let's switch over to static_branch_likely() to reflect this fact. Unlikely there is a significant performance difference, as the cost of a memory allocation and its accounting significantly exceeds the cost of a jump. However, the conversion makes the code look more logically. Link: http://lkml.kernel.org/r/20200707173612.124425-3-guro@fb.com Signed-off-by: Roman Gushchin Reviewed-by: Shakeel Butt Acked-by: Vlastimil Babka Cc: Johannes Weiner Cc: Michal Hocko Cc: Christoph Lameter Cc: David Rientjes Cc: Joonsoo Kim Cc: Pekka Enberg Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/include/linux/memcontrol.h~mm-kmem-switch-to-static_branch_likely-in-memcg_kmem_enabled +++ a/include/linux/memcontrol.h @@ -1448,7 +1448,7 @@ void memcg_put_cache_ids(void); static inline bool memcg_kmem_enabled(void) { - return static_branch_unlikely(&memcg_kmem_enabled_key); + return static_branch_likely(&memcg_kmem_enabled_key); } static inline bool memcg_kmem_bypass(void) From patchwork Fri Aug 7 06:21:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704841 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8DB28138C for ; Fri, 7 Aug 2020 06:21:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 525832177B for ; Fri, 7 Aug 2020 06:21:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="j3iOVIdv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 525832177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1135F8D005E; Fri, 7 Aug 2020 02:21:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0EA7E8D0026; Fri, 7 Aug 2020 02:21:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 025C28D005E; Fri, 7 Aug 2020 02:21:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0119.hostedemail.com [216.40.44.119]) by kanga.kvack.org (Postfix) with ESMTP id E1A858D0026 for ; Fri, 7 Aug 2020 02:21:53 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id ACD0B180AD804 for ; Fri, 7 Aug 2020 06:21:53 +0000 (UTC) X-FDA: 77122776906.09.form94_1e0732026fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 87FD2180AD802 for ; Fri, 7 Aug 2020 06:21:53 +0000 (UTC) X-Spam-Summary: 1,0,0,a96ec4e0036fee57,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2525:2559:2563:2682:2685:2731:2859:2895:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6119:6261:6653:6737:7576:7903:8531:8599:9025:9545:9592:9707:10004:10913:11026:11658:11914:12043:12048:12264:12295:12296:12297:12438:12517:12519:12555:12679:12783:12986:13221:13229:14181:14721:14849:21080:21451:21627:21740:21795:21939:21990:30051:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrfwj7asx3qg5mundz11nate7chyc4pys768rxomrsfwy7isi7dznjzpemkcn.x59yknx3rkuiuco9t5zsjxaq7m5g9ukws7fyg4bkir5y1fdk6794wur5fpjppa7.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netche ck:none, X-HE-Tag: form94_1e0732026fbe X-Filterd-Recvd-Size: 3855 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:53 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9F81D221E5; Fri, 7 Aug 2020 06:21:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781312; bh=Dl3Dm/Eu5wjiTaKK45eDk/3qF/7YCH/X+eekyMacs4s=; h=Date:From:To:Subject:In-Reply-To:From; b=j3iOVIdvXBQzyDWvr08wtiWnnW4Cg4M1PSsNy8TOkRUJPHJSYcHRDg4DUG9NnbznD 3e03V4qTC82SCn4p7A5ZxNgcaXzSmQwnlOE2qBR38FWVncDrAVoYYtFC0fAAkufe9f WWYOeoKDAYkITKBdRmBo9mSykeG69z/wXzZAVBGo= Date: Thu, 06 Aug 2020 23:21:51 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, domas@fb.com, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org Subject: [patch 085/163] mm: memcontrol: avoid workload stalls when lowering memory.high Message-ID: <20200807062151.2JYr5jJgy%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 87FD2180AD802 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Subject: mm: memcontrol: avoid workload stalls when lowering memory.high Memory.high limit is implemented in a way such that the kernel penalizes all threads which are allocating a memory over the limit. Forcing all threads into the synchronous reclaim and adding some artificial delays allows to slow down the memory consumption and potentially give some time for userspace oom handlers/resource control agents to react. It works nicely if the memory usage is hitting the limit from below, however it works sub-optimal if a user adjusts memory.high to a value way below the current memory usage. It basically forces all workload threads (doing any memory allocations) into the synchronous reclaim and sleep. This makes the workload completely unresponsive for a long period of time and can also lead to a system-wide contention on lru locks. It can happen even if the workload is not actually tight on memory and has, for example, a ton of cold pagecache. In the current implementation writing to memory.high causes an atomic update of page counter's high value followed by an attempt to reclaim enough memory to fit into the new limit. To fix the problem described above, all we need is to change the order of execution: try to push the memory usage under the limit first, and only then set the new high limit. Link: http://lkml.kernel.org/r/20200709194718.189231-1-guro@fb.com Signed-off-by: Roman Gushchin Reported-by: Domas Mituzas Acked-by: Michal Hocko Reviewed-by: Shakeel Butt Cc: Johannes Weiner Cc: Tejun Heo Cc: Chris Down Signed-off-by: Andrew Morton --- mm/memcontrol.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --- a/mm/memcontrol.c~mm-memcontrol-avoid-workload-stalls-when-lowering-memoryhigh +++ a/mm/memcontrol.c @@ -6213,8 +6213,6 @@ static ssize_t memory_high_write(struct if (err) return err; - page_counter_set_high(&memcg->memory, high); - for (;;) { unsigned long nr_pages = page_counter_read(&memcg->memory); unsigned long reclaimed; @@ -6238,6 +6236,8 @@ static ssize_t memory_high_write(struct break; } + page_counter_set_high(&memcg->memory, high); + return nbytes; } From patchwork Fri Aug 7 06:21:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704843 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 29EB1138C for ; Fri, 7 Aug 2020 06:21:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EFAED22D03 for ; Fri, 7 Aug 2020 06:21:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="0Ytp3ECg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EFAED22D03 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B4C3A8D005F; Fri, 7 Aug 2020 02:21:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AFAFB8D0026; Fri, 7 Aug 2020 02:21:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E9938D005F; Fri, 7 Aug 2020 02:21:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id 8A8738D0026 for ; Fri, 7 Aug 2020 02:21:57 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6002D2C79 for ; Fri, 7 Aug 2020 06:21:57 +0000 (UTC) X-FDA: 77122777074.13.sheet71_2206ea926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 2F72018140B67 for ; Fri, 7 Aug 2020 06:21:57 +0000 (UTC) X-Spam-Summary: 1,0,0,ce6db6e040f4815a,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1606:1730:1747:1777:1792:1969:2195:2196:2199:2200:2393:2525:2559:2564:2682:2685:2859:2890:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4042:4118:4321:4385:5007:6119:6261:6299:6653:6671:7576:7903:8599:8603:9025:9545:10004:11026:11257:11473:11658:11914:12043:12048:12291:12295:12296:12297:12438:12517:12519:12555:12679:12683:12740:12783:12895:12986:13007:13161:13229:13255:13846:13869:14096:21063:21080:21324:21450:21451:21627:21740:21795:21939:21966:21990:22013:30001:30005:30034:30051:30054:30064:30066:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yffscwfpk38o4dzzb4919ptcfddycpa44cdq3fjcxq1cditx893ot1xndip1r.i6ehdgs79hmie9hg6jf9xj5ea 7sdsqymq X-HE-Tag: sheet71_2206ea926fbe X-Filterd-Recvd-Size: 7278 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:21:56 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7436F22CB3; Fri, 7 Aug 2020 06:21:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781316; bh=M/0MyN8poq5VJes0iC5bnxr2WvF5QR4caUDBSUFC6pU=; h=Date:From:To:Subject:In-Reply-To:From; b=0Ytp3ECg0XdJ04xAqpzNCft9Mg6DFoPw9J/UoYWc6TYjYDi6So894qXsBcyQmIdmm SBD8GDU9+J+AYdf/ZJ5w4wWdlWGG+QkBJLSJXDWtGrRHojMv31xIT55YTwJZnVFBxp 7ERx1IK4YWqm3hMBnptdzHSm2naJ8pjlzjfbjo+o= Date: Thu, 06 Aug 2020 23:21:54 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org Subject: [patch 086/163] mm, memcg: reclaim more aggressively before high allocator throttling Message-ID: <20200807062154.skuN7Pgbg%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 2F72018140B67 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chris Down Subject: mm, memcg: reclaim more aggressively before high allocator throttling Patch series "mm, memcg: reclaim harder before high throttling", v2. This patch (of 2): In Facebook production, we've seen cases where cgroups have been put into allocator throttling even when they appear to have a lot of slack file caches which should be trivially reclaimable. Looking more closely, the problem is that we only try a single cgroup reclaim walk for each return to usermode before calculating whether or not we should throttle. This single attempt doesn't produce enough pressure to shrink for cgroups with a rapidly growing amount of file caches prior to entering allocator throttling. As an example, we see that threads in an affected cgroup are stuck in allocator throttling: # for i in $(cat cgroup.threads); do > grep over_high "/proc/$i/stack" > done [<0>] mem_cgroup_handle_over_high+0x10b/0x150 [<0>] mem_cgroup_handle_over_high+0x10b/0x150 [<0>] mem_cgroup_handle_over_high+0x10b/0x150 ...however, there is no I/O pressure reported by PSI, despite a lot of slack file pages: # cat memory.pressure some avg10=78.50 avg60=84.99 avg300=84.53 total=5702440903 full avg10=78.50 avg60=84.99 avg300=84.53 total=5702116959 # cat io.pressure some avg10=0.00 avg60=0.00 avg300=0.00 total=78051391 full avg10=0.00 avg60=0.00 avg300=0.00 total=78049640 # grep _file memory.stat inactive_file 1370939392 active_file 661635072 This patch changes the behaviour to retry reclaim either until the current task goes below the 10ms grace period, or we are making no reclaim progress at all. In the latter case, we enter reclaim throttling as before. To a user, there's no intuitive reason for the reclaim behaviour to differ from hitting memory.high as part of a new allocation, as opposed to hitting memory.high because someone lowered its value. As such this also brings an added benefit: it unifies the reclaim behaviour between the two. There's precedent for this behaviour: we already do reclaim retries when writing to memory.{high,max}, in max reclaim, and in the page allocator itself. Link: http://lkml.kernel.org/r/cover.1594640214.git.chris@chrisdown.name Link: http://lkml.kernel.org/r/a4e23b59e9ef499b575ae73a8120ee089b7d3373.1594640214.git.chris@chrisdown.name Signed-off-by: Chris Down Reviewed-by: Shakeel Butt Acked-by: Johannes Weiner Cc: Tejun Heo Cc: Michal Hocko Cc: Roman Gushchin Signed-off-by: Andrew Morton --- mm/memcontrol.c | 42 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 5 deletions(-) --- a/mm/memcontrol.c~mm-memcg-reclaim-more-aggressively-before-high-allocator-throttling +++ a/mm/memcontrol.c @@ -73,6 +73,7 @@ EXPORT_SYMBOL(memory_cgrp_subsys); struct mem_cgroup *root_mem_cgroup __read_mostly; +/* The number of times we should retry reclaim failures before giving up. */ #define MEM_CGROUP_RECLAIM_RETRIES 5 /* Socket memory accounting disabled? */ @@ -2363,18 +2364,23 @@ static int memcg_hotplug_cpu_dead(unsign return 0; } -static void reclaim_high(struct mem_cgroup *memcg, - unsigned int nr_pages, - gfp_t gfp_mask) +static unsigned long reclaim_high(struct mem_cgroup *memcg, + unsigned int nr_pages, + gfp_t gfp_mask) { + unsigned long nr_reclaimed = 0; + do { if (page_counter_read(&memcg->memory) <= READ_ONCE(memcg->memory.high)) continue; memcg_memory_event(memcg, MEMCG_HIGH); - try_to_free_mem_cgroup_pages(memcg, nr_pages, gfp_mask, true); + nr_reclaimed += try_to_free_mem_cgroup_pages(memcg, nr_pages, + gfp_mask, true); } while ((memcg = parent_mem_cgroup(memcg)) && !mem_cgroup_is_root(memcg)); + + return nr_reclaimed; } static void high_work_func(struct work_struct *work) @@ -2530,16 +2536,32 @@ void mem_cgroup_handle_over_high(void) { unsigned long penalty_jiffies; unsigned long pflags; + unsigned long nr_reclaimed; unsigned int nr_pages = current->memcg_nr_pages_over_high; + int nr_retries = MEM_CGROUP_RECLAIM_RETRIES; struct mem_cgroup *memcg; + bool in_retry = false; if (likely(!nr_pages)) return; memcg = get_mem_cgroup_from_mm(current->mm); - reclaim_high(memcg, nr_pages, GFP_KERNEL); current->memcg_nr_pages_over_high = 0; +retry_reclaim: + /* + * The allocating task should reclaim at least the batch size, but for + * subsequent retries we only want to do what's necessary to prevent oom + * or breaching resource isolation. + * + * This is distinct from memory.max or page allocator behaviour because + * memory.high is currently batched, whereas memory.max and the page + * allocator run every time an allocation is made. + */ + nr_reclaimed = reclaim_high(memcg, + in_retry ? SWAP_CLUSTER_MAX : nr_pages, + GFP_KERNEL); + /* * memory.high is breached and reclaim is unable to keep up. Throttle * allocators proactively to slow down excessive growth. @@ -2567,6 +2589,16 @@ void mem_cgroup_handle_over_high(void) goto out; /* + * If reclaim is making forward progress but we're still over + * memory.high, we want to encourage that rather than doing allocator + * throttling. + */ + if (nr_reclaimed || nr_retries--) { + in_retry = true; + goto retry_reclaim; + } + + /* * If we exit early, we're guaranteed to die (since * schedule_timeout_killable sets TASK_KILLABLE). This means we don't * need to account for any ill-begotten jiffies to pay them off later. From patchwork Fri Aug 7 06:21:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704845 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1C5C14B7 for ; Fri, 7 Aug 2020 06:22:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 71C3222D03 for ; Fri, 7 Aug 2020 06:22:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="DKYi6H1K" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 71C3222D03 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1D5E48D0060; Fri, 7 Aug 2020 02:22:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 186888D0026; Fri, 7 Aug 2020 02:22:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09A5C8D0060; Fri, 7 Aug 2020 02:22:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id E92058D0026 for ; Fri, 7 Aug 2020 02:22:00 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B8B30180AD801 for ; Fri, 7 Aug 2020 06:22:00 +0000 (UTC) X-FDA: 77122777200.02.fruit72_300f59d26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 8F87D10097AA0 for ; Fri, 7 Aug 2020 06:22:00 +0000 (UTC) X-Spam-Summary: 1,0,0,e662629a9728b939,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4605:5007:6261:6653:7576:8660:9025:9121:9168:9545:9592:10004:11026:11233:11257:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13148:13230:13846:14096:14181:14721:14849:21080:21433:21450:21451:21627:21740:21939:21966:30034:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygxmuep4rr5iasu37tj1yzrjnx1ocgm4b6tpq8x5ndz1anrjy37ggps9eostu.riwohan7up648qup6equsz8bjxmgpotcdi1d3nj37cob9jebo8xd3bsqsqopmxs.k-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5, Netcheck X-HE-Tag: fruit72_300f59d26fbe X-Filterd-Recvd-Size: 5097 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:00 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id DFE952177B; Fri, 7 Aug 2020 06:21:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781319; bh=RuN4u55pMCP/fBpF68DHXV1C3Eb7qRzSwcB34rO+s+c=; h=Date:From:To:Subject:In-Reply-To:From; b=DKYi6H1Kw9pOfNuEBDRISKKLjKD5aGAepmBrcBmle7bQGvJEmS9tQrNrSHtNW3+ae pktumNxolF8OIXH4B8xWX3JcSZrYdZVeJNmfKCzceEoYd89uYTAUa7hgMcf4pbNEyc 2lCjOV11jg62u+cHxNTduHsYVsvyJr6gfzHufLmc= Date: Thu, 06 Aug 2020 23:21:58 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, shakeelb@google.com, tj@kernel.org, torvalds@linux-foundation.org Subject: [patch 087/163] mm, memcg: unify reclaim retry limits with page allocator Message-ID: <20200807062158.KIdlvMFeD%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 8F87D10097AA0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chris Down Subject: mm, memcg: unify reclaim retry limits with page allocator Reclaim retries have been set to 5 since the beginning of time in commit 66e1707bc346 ("Memory controller: add per cgroup LRU and reclaim"). However, we now have a generally agreed-upon standard for page reclaim: MAX_RECLAIM_RETRIES (currently 16), added many years later in commit 0a0337e0d1d1 ("mm, oom: rework oom detection"). In the absence of a compelling reason to declare an OOM earlier in memcg context than page allocator context, it seems reasonable to supplant MEM_CGROUP_RECLAIM_RETRIES with MAX_RECLAIM_RETRIES, making the page allocator and memcg internals more similar in semantics when reclaim fails to produce results, avoiding premature OOMs or throttling. Link: http://lkml.kernel.org/r/da557856c9c7654308eaff4eedc1952a95e8df5f.1594640214.git.chris@chrisdown.name Signed-off-by: Chris Down Reviewed-by: Shakeel Butt Acked-by: Michal Hocko Cc: Johannes Weiner Cc: Tejun Heo Cc: Roman Gushchin Signed-off-by: Andrew Morton --- mm/memcontrol.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) --- a/mm/memcontrol.c~mm-memcg-unify-reclaim-retry-limits-with-page-allocator +++ a/mm/memcontrol.c @@ -73,9 +73,6 @@ EXPORT_SYMBOL(memory_cgrp_subsys); struct mem_cgroup *root_mem_cgroup __read_mostly; -/* The number of times we should retry reclaim failures before giving up. */ -#define MEM_CGROUP_RECLAIM_RETRIES 5 - /* Socket memory accounting disabled? */ static bool cgroup_memory_nosocket; @@ -2538,7 +2535,7 @@ void mem_cgroup_handle_over_high(void) unsigned long pflags; unsigned long nr_reclaimed; unsigned int nr_pages = current->memcg_nr_pages_over_high; - int nr_retries = MEM_CGROUP_RECLAIM_RETRIES; + int nr_retries = MAX_RECLAIM_RETRIES; struct mem_cgroup *memcg; bool in_retry = false; @@ -2615,7 +2612,7 @@ static int try_charge(struct mem_cgroup unsigned int nr_pages) { unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages); - int nr_retries = MEM_CGROUP_RECLAIM_RETRIES; + int nr_retries = MAX_RECLAIM_RETRIES; struct mem_cgroup *mem_over_limit; struct page_counter *counter; unsigned long nr_reclaimed; @@ -2734,7 +2731,7 @@ retry: get_order(nr_pages * PAGE_SIZE)); switch (oom_status) { case OOM_SUCCESS: - nr_retries = MEM_CGROUP_RECLAIM_RETRIES; + nr_retries = MAX_RECLAIM_RETRIES; goto retry; case OOM_FAILED: goto force; @@ -3414,7 +3411,7 @@ static inline bool memcg_has_children(st */ static int mem_cgroup_force_empty(struct mem_cgroup *memcg) { - int nr_retries = MEM_CGROUP_RECLAIM_RETRIES; + int nr_retries = MAX_RECLAIM_RETRIES; /* we call try-to-free pages for make this cgroup empty */ lru_add_drain_all(); @@ -6235,7 +6232,7 @@ static ssize_t memory_high_write(struct char *buf, size_t nbytes, loff_t off) { struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); - unsigned int nr_retries = MEM_CGROUP_RECLAIM_RETRIES; + unsigned int nr_retries = MAX_RECLAIM_RETRIES; bool drained = false; unsigned long high; int err; @@ -6283,7 +6280,7 @@ static ssize_t memory_max_write(struct k char *buf, size_t nbytes, loff_t off) { struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); - unsigned int nr_reclaims = MEM_CGROUP_RECLAIM_RETRIES; + unsigned int nr_reclaims = MAX_RECLAIM_RETRIES; bool drained = false; unsigned long max; int err; From patchwork Fri Aug 7 06:22:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704847 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 734EC14B7 for ; Fri, 7 Aug 2020 06:22:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 391832177B for ; Fri, 7 Aug 2020 06:22:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="iFtudIPM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 391832177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AFE488D0061; Fri, 7 Aug 2020 02:22:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AB58F8D0026; Fri, 7 Aug 2020 02:22:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EAB48D0061; Fri, 7 Aug 2020 02:22:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0214.hostedemail.com [216.40.44.214]) by kanga.kvack.org (Postfix) with ESMTP id 8810B8D0026 for ; Fri, 7 Aug 2020 02:22:04 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4F8CB181AEF10 for ; Fri, 7 Aug 2020 06:22:04 +0000 (UTC) X-FDA: 77122777368.26.money25_1b0830c26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 1AC641804B667 for ; Fri, 7 Aug 2020 06:22:04 +0000 (UTC) X-Spam-Summary: 1,0,0,b17532f037449e22,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:2393:2525:2553:2559:2564:2682:2685:2859:2897:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4050:4119:4250:4321:4470:4605:5007:6261:6653:7514:7576:7875:7903:7904:8599:9025:9545:10004:11026:11257:11473:11658:11914:12043:12048:12219:12291:12295:12296:12297:12438:12517:12519:12555:12664:12679:12683:12783:12986:13149:13161:13229:13230:13255:13846:21080:21433:21451:21627:21740:21939:21966:21990:30054:30056:30064:30074:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfe7rp9r9omqw1qh4iwmpmgihqiycr6myqswj1bjfx46ca48sfzmwaehe7yk9.xkhkny3dn8r9pauo8psx7ibrtteiqe7713nx6qg78gd67uci5p6ttsu14cwhgaa.g-lbl8.mailshell.net-223.238.255.100,C acheIP:n X-HE-Tag: money25_1b0830c26fbe X-Filterd-Recvd-Size: 8680 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:03 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 62B8B221E5; Fri, 7 Aug 2020 06:22:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781322; bh=oQ8voHX5dfQMAOwgZvfX9mLPjnZLDynr9IV4MlZ0KUI=; h=Date:From:To:Subject:In-Reply-To:From; b=iFtudIPMk3G+AR1dQzn3PNGudcGimpz2TCm1HRcJhH6Br8LWtCA2/D8wp9/z9pU5m PdfZ4G8TwESptY+cStueZPkma7NnI9fytIy95+dXFagl5JtQKdgG/5zyk/cJgFQvyB 0o0cMeWzbwSIKezWGM+3CGRqz5aO5l38qAWnzKao= Date: Thu, 06 Aug 2020 23:22:01 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, guro@fb.com, hannes@cmpxchg.org, laoar.shao@gmail.com, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 088/163] mm, memcg: avoid stale protection values when cgroup is above protection Message-ID: <20200807062201.4toCRHwgc%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 1AC641804B667 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yafang Shao Subject: mm, memcg: avoid stale protection values when cgroup is above protection Patch series "mm, memcg: memory.{low,min} reclaim fix & cleanup", v4. This series contains a fix for a edge case in my earlier protection calculation patches, and a patch to make the area overall a little more robust to hopefully help avoid this in future. This patch (of 2): A cgroup can have both memory protection and a memory limit to isolate it from its siblings in both directions - for example, to prevent it from being shrunk below 2G under high pressure from outside, but also from growing beyond 4G under low pressure. Commit 9783aa9917f8 ("mm, memcg: proportional memory.{low,min} reclaim") implemented proportional scan pressure so that multiple siblings in excess of their protection settings don't get reclaimed equally but instead in accordance to their unprotected portion. During limit reclaim, this proportionality shouldn't apply of course: there is no competition, all pressure is from within the cgroup and should be applied as such. Reclaim should operate at full efficiency. However, mem_cgroup_protected() never expected anybody to look at the effective protection values when it indicated that the cgroup is above its protection. As a result, a query during limit reclaim may return stale protection values that were calculated by a previous reclaim cycle in which the cgroup did have siblings. When this happens, reclaim is unnecessarily hesitant and potentially slow to meet the desired limit. In theory this could lead to premature OOM kills, although it's not obvious this has occurred in practice. Workaround the problem by special casing reclaim roots in mem_cgroup_protection. These memcgs are never participating in the reclaim protection because the reclaim is internal. We have to ignore effective protection values for reclaim roots because mem_cgroup_protected might be called from racing reclaim contexts with different roots. Calculation is relying on root -> leaf tree traversal therefore top-down reclaim protection invariants should hold. The only exception is the reclaim root which should have effective protection set to 0 but that would be problematic for the following setup: Let's have global and A's reclaim in parallel: | A (low=2G, usage = 3G, max = 3G, children_low_usage = 1.5G) |\ | C (low = 1G, usage = 2.5G) B (low = 1G, usage = 0.5G) for A reclaim we have B.elow = B.low C.elow = C.low For the global reclaim A.elow = A.low B.elow = min(B.usage, B.low) because children_low_usage <= A.elow C.elow = min(C.usage, C.low) With the effective values resetting we have A reclaim A.elow = 0 B.elow = B.low C.elow = C.low and global reclaim could see the above and then B.elow = C.elow = 0 because children_low_usage > A.elow Which means that protected memcgs would get reclaimed. In future we would like to make mem_cgroup_protected more robust against racing reclaim contexts but that is likely more complex solution than this simple workaround. [hannes@cmpxchg.org - large part of the changelog] [mhocko@suse.com - workaround explanation] [chris@chrisdown.name - retitle] Link: http://lkml.kernel.org/r/cover.1594638158.git.chris@chrisdown.name Link: http://lkml.kernel.org/r/044fb8ecffd001c7905d27c0c2ad998069fdc396.1594638158.git.chris@chrisdown.name Fixes: 9783aa9917f8 ("mm, memcg: proportional memory.{low,min} reclaim") Signed-off-by: Yafang Shao Signed-off-by: Chris Down Acked-by: Michal Hocko Acked-by: Johannes Weiner Acked-by: Chris Down Acked-by: Roman Gushchin Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 42 +++++++++++++++++++++++++++++++++-- mm/memcontrol.c | 8 ++++++ mm/vmscan.c | 3 +- 3 files changed, 50 insertions(+), 3 deletions(-) --- a/include/linux/memcontrol.h~mm-memcg-avoid-stale-protection-values-when-cgroup-is-above-protection +++ a/include/linux/memcontrol.h @@ -355,12 +355,49 @@ static inline bool mem_cgroup_disabled(v return !cgroup_subsys_enabled(memory_cgrp_subsys); } -static inline unsigned long mem_cgroup_protection(struct mem_cgroup *memcg, +static inline unsigned long mem_cgroup_protection(struct mem_cgroup *root, + struct mem_cgroup *memcg, bool in_low_reclaim) { if (mem_cgroup_disabled()) return 0; + /* + * There is no reclaim protection applied to a targeted reclaim. + * We are special casing this specific case here because + * mem_cgroup_protected calculation is not robust enough to keep + * the protection invariant for calculated effective values for + * parallel reclaimers with different reclaim target. This is + * especially a problem for tail memcgs (as they have pages on LRU) + * which would want to have effective values 0 for targeted reclaim + * but a different value for external reclaim. + * + * Example + * Let's have global and A's reclaim in parallel: + * | + * A (low=2G, usage = 3G, max = 3G, children_low_usage = 1.5G) + * |\ + * | C (low = 1G, usage = 2.5G) + * B (low = 1G, usage = 0.5G) + * + * For the global reclaim + * A.elow = A.low + * B.elow = min(B.usage, B.low) because children_low_usage <= A.elow + * C.elow = min(C.usage, C.low) + * + * With the effective values resetting we have A reclaim + * A.elow = 0 + * B.elow = B.low + * C.elow = C.low + * + * If the global reclaim races with A's reclaim then + * B.elow = C.elow = 0 because children_low_usage > A.elow) + * is possible and reclaiming B would be violating the protection. + * + */ + if (root == memcg) + return 0; + if (in_low_reclaim) return READ_ONCE(memcg->memory.emin); @@ -891,7 +928,8 @@ static inline void memcg_memory_event_mm { } -static inline unsigned long mem_cgroup_protection(struct mem_cgroup *memcg, +static inline unsigned long mem_cgroup_protection(struct mem_cgroup *root, + struct mem_cgroup *memcg, bool in_low_reclaim) { return 0; --- a/mm/memcontrol.c~mm-memcg-avoid-stale-protection-values-when-cgroup-is-above-protection +++ a/mm/memcontrol.c @@ -6605,6 +6605,14 @@ enum mem_cgroup_protection mem_cgroup_pr if (!root) root = root_mem_cgroup; + + /* + * Effective values of the reclaim targets are ignored so they + * can be stale. Have a look at mem_cgroup_protection for more + * details. + * TODO: calculation should be more robust so that we do not need + * that special casing. + */ if (memcg == root) return MEMCG_PROT_NONE; --- a/mm/vmscan.c~mm-memcg-avoid-stale-protection-values-when-cgroup-is-above-protection +++ a/mm/vmscan.c @@ -2331,7 +2331,8 @@ out: unsigned long protection; lruvec_size = lruvec_lru_size(lruvec, lru, sc->reclaim_idx); - protection = mem_cgroup_protection(memcg, + protection = mem_cgroup_protection(sc->target_mem_cgroup, + memcg, sc->memcg_low_reclaim); if (protection) { From patchwork Fri Aug 7 06:22:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704849 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E172B138C for ; Fri, 7 Aug 2020 06:22:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A76222177B for ; Fri, 7 Aug 2020 06:22:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="e9RBW9WT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A76222177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 383C18D0062; Fri, 7 Aug 2020 02:22:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3596D8D0026; Fri, 7 Aug 2020 02:22:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2499F8D0062; Fri, 7 Aug 2020 02:22:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0044.hostedemail.com [216.40.44.44]) by kanga.kvack.org (Postfix) with ESMTP id 0F49A8D0026 for ; Fri, 7 Aug 2020 02:22:08 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CE0642489 for ; Fri, 7 Aug 2020 06:22:07 +0000 (UTC) X-FDA: 77122777494.30.tramp56_0a1043a26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id A3237180B3C83 for ; Fri, 7 Aug 2020 06:22:07 +0000 (UTC) X-Spam-Summary: 1,0,0,cd3e6a1d0ac0f79b,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:2198:2199:2393:2525:2553:2559:2563:2682:2685:2731:2859:2890:2897:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4042:4049:4119:4321:4605:5007:6261:6653:7514:7576:7903:9025:9040:9545:9592:10004:11026:11232:11257:11473:11658:11914:12043:12048:12291:12295:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13255:13846:14096:21063:21080:21450:21451:21627:21795:21939:21966:21990:30001:30041:30045:30051:30054:30064:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8txpmq6r3wf9agxwjrrtf37c6focptbh7ruzfjpd739bu8fbmfocssumqoci.ungb8oextncugbezt9dhabjn84emm1erqjtqenb8ixm3dy3nwckiw7gu3b7pmjc.6-lbl8.mailshell.net -223.238 X-HE-Tag: tramp56_0a1043a26fbe X-Filterd-Recvd-Size: 8194 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:07 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D6A5222C9F; Fri, 7 Aug 2020 06:22:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781326; bh=Xl/qgk7bGxv2hcZoa4sFHFFj+X2a7VZedKrg964YjGE=; h=Date:From:To:Subject:In-Reply-To:From; b=e9RBW9WT1ghvckN2BajGS8KneVQzJB4LGYhOdc+NwvXObJ9ULy1ncWb08osRd2nSl PL9HpwhmxwuU+Jb7cdVkJEffb344NxeT5LFoElQl2/0sMfM1mESie1geMQ4mwo+spF eybwk6i5aEawIxC7qRU1b1nei1gH9T/jvmzlyWi4= Date: Thu, 06 Aug 2020 23:22:05 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, guro@fb.com, hannes@cmpxchg.org, laoar.shao@gmail.com, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 089/163] mm, memcg: decouple e{low,min} state mutations from protection checks Message-ID: <20200807062205.giFveKsjG%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: A3237180B3C83 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chris Down Subject: mm, memcg: decouple e{low,min} state mutations from protection checks mem_cgroup_protected currently is both used to set effective low and min and return a mem_cgroup_protection based on the result. As a user, this can be a little unexpected: it appears to be a simple predicate function, if not for the big warning in the comment above about the order in which it must be executed. This change makes it so that we separate the state mutations from the actual protection checks, which makes it more obvious where we need to be careful mutating internal state, and where we are simply checking and don't need to worry about that. [mhocko@suse.com - don't check protection on root memcgs] Link: http://lkml.kernel.org/r/ff3f915097fcee9f6d7041c084ef92d16aaeb56a.1594638158.git.chris@chrisdown.name Signed-off-by: Chris Down Suggested-by: Johannes Weiner Acked-by: Johannes Weiner Acked-by: Michal Hocko Cc: Roman Gushchin Cc: Yafang Shao Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 53 +++++++++++++++++++++++++++-------- mm/memcontrol.c | 28 ++++-------------- mm/vmscan.c | 17 ++--------- 3 files changed, 53 insertions(+), 45 deletions(-) --- a/include/linux/memcontrol.h~mm-memcg-decouple-elowmin-state-mutations-from-protection-checks +++ a/include/linux/memcontrol.h @@ -47,12 +47,6 @@ enum memcg_memory_event { MEMCG_NR_MEMORY_EVENTS, }; -enum mem_cgroup_protection { - MEMCG_PROT_NONE, - MEMCG_PROT_LOW, - MEMCG_PROT_MIN, -}; - struct mem_cgroup_reclaim_cookie { pg_data_t *pgdat; unsigned int generation; @@ -405,8 +399,36 @@ static inline unsigned long mem_cgroup_p READ_ONCE(memcg->memory.elow)); } -enum mem_cgroup_protection mem_cgroup_protected(struct mem_cgroup *root, - struct mem_cgroup *memcg); +void mem_cgroup_calculate_protection(struct mem_cgroup *root, + struct mem_cgroup *memcg); + +static inline bool mem_cgroup_supports_protection(struct mem_cgroup *memcg) +{ + /* + * The root memcg doesn't account charges, and doesn't support + * protection. + */ + return !mem_cgroup_disabled() && !mem_cgroup_is_root(memcg); + +} + +static inline bool mem_cgroup_below_low(struct mem_cgroup *memcg) +{ + if (!mem_cgroup_supports_protection(memcg)) + return false; + + return READ_ONCE(memcg->memory.elow) >= + page_counter_read(&memcg->memory); +} + +static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg) +{ + if (!mem_cgroup_supports_protection(memcg)) + return false; + + return READ_ONCE(memcg->memory.emin) >= + page_counter_read(&memcg->memory); +} int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask); @@ -935,10 +957,19 @@ static inline unsigned long mem_cgroup_p return 0; } -static inline enum mem_cgroup_protection mem_cgroup_protected( - struct mem_cgroup *root, struct mem_cgroup *memcg) +static inline void mem_cgroup_calculate_protection(struct mem_cgroup *root, + struct mem_cgroup *memcg) +{ +} + +static inline bool mem_cgroup_below_low(struct mem_cgroup *memcg) +{ + return false; +} + +static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg) { - return MEMCG_PROT_NONE; + return false; } static inline int mem_cgroup_charge(struct page *page, struct mm_struct *mm, --- a/mm/memcontrol.c~mm-memcg-decouple-elowmin-state-mutations-from-protection-checks +++ a/mm/memcontrol.c @@ -6587,21 +6587,15 @@ static unsigned long effective_protectio * * WARNING: This function is not stateless! It can only be used as part * of a top-down tree iteration, not for isolated queries. - * - * Returns one of the following: - * MEMCG_PROT_NONE: cgroup memory is not protected - * MEMCG_PROT_LOW: cgroup memory is protected as long there is - * an unprotected supply of reclaimable memory from other cgroups. - * MEMCG_PROT_MIN: cgroup memory is protected */ -enum mem_cgroup_protection mem_cgroup_protected(struct mem_cgroup *root, - struct mem_cgroup *memcg) +void mem_cgroup_calculate_protection(struct mem_cgroup *root, + struct mem_cgroup *memcg) { unsigned long usage, parent_usage; struct mem_cgroup *parent; if (mem_cgroup_disabled()) - return MEMCG_PROT_NONE; + return; if (!root) root = root_mem_cgroup; @@ -6614,21 +6608,21 @@ enum mem_cgroup_protection mem_cgroup_pr * that special casing. */ if (memcg == root) - return MEMCG_PROT_NONE; + return; usage = page_counter_read(&memcg->memory); if (!usage) - return MEMCG_PROT_NONE; + return; parent = parent_mem_cgroup(memcg); /* No parent means a non-hierarchical mode on v1 memcg */ if (!parent) - return MEMCG_PROT_NONE; + return; if (parent == root) { memcg->memory.emin = READ_ONCE(memcg->memory.min); memcg->memory.elow = READ_ONCE(memcg->memory.low); - goto out; + return; } parent_usage = page_counter_read(&parent->memory); @@ -6642,14 +6636,6 @@ enum mem_cgroup_protection mem_cgroup_pr READ_ONCE(memcg->memory.low), READ_ONCE(parent->memory.elow), atomic_long_read(&parent->memory.children_low_usage))); - -out: - if (usage <= memcg->memory.emin) - return MEMCG_PROT_MIN; - else if (usage <= memcg->memory.elow) - return MEMCG_PROT_LOW; - else - return MEMCG_PROT_NONE; } /** --- a/mm/vmscan.c~mm-memcg-decouple-elowmin-state-mutations-from-protection-checks +++ a/mm/vmscan.c @@ -2620,14 +2620,15 @@ static void shrink_node_memcgs(pg_data_t unsigned long reclaimed; unsigned long scanned; - switch (mem_cgroup_protected(target_memcg, memcg)) { - case MEMCG_PROT_MIN: + mem_cgroup_calculate_protection(target_memcg, memcg); + + if (mem_cgroup_below_min(memcg)) { /* * Hard protection. * If there is no reclaimable memory, OOM. */ continue; - case MEMCG_PROT_LOW: + } else if (mem_cgroup_below_low(memcg)) { /* * Soft protection. * Respect the protection only as long as @@ -2639,16 +2640,6 @@ static void shrink_node_memcgs(pg_data_t continue; } memcg_memory_event(memcg, MEMCG_LOW); - break; - case MEMCG_PROT_NONE: - /* - * All protection thresholds breached. We may - * still choose to vary the scan pressure - * applied based on by how much the cgroup in - * question has exceeded its protection - * thresholds (see get_scan_count). - */ - break; } reclaimed = sc->nr_reclaimed; From patchwork Fri Aug 7 06:22:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704851 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 37E2914B7 for ; Fri, 7 Aug 2020 06:22:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 007CC221E5 for ; Fri, 7 Aug 2020 06:22:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="OD9SJB/p" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 007CC221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 753498D0063; Fri, 7 Aug 2020 02:22:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 72AFF8D0026; Fri, 7 Aug 2020 02:22:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 668018D0063; Fri, 7 Aug 2020 02:22:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0165.hostedemail.com [216.40.44.165]) by kanga.kvack.org (Postfix) with ESMTP id 5260A8D0026 for ; Fri, 7 Aug 2020 02:22:11 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 12F79180AD801 for ; Fri, 7 Aug 2020 06:22:11 +0000 (UTC) X-FDA: 77122777662.28.meat36_010fc4e26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id DE8896D74 for ; Fri, 7 Aug 2020 06:22:10 +0000 (UTC) X-Spam-Summary: 1,0,0,a956df2a2775b412,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:1801:2198:2199:2393:2525:2559:2563:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6119:6261:6653:6737:7514:7576:7903:8660:9025:9121:9149:9391:9545:10004:10913:11026:11233:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12533:12550:12555:12679:12783:12986:13148:13161:13229:13230:13846:14181:14721:14849:21080:21324:21433:21451:21627:21819:21939:21990:30054:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8aw6kgteiz8uo9f16casq65yiwypj99qtu4upm9jnuimbkboiwsd3h54th17.h4bo4ebymg3ej184toeb5ycb7rurjmpg63xqh9exxdj1r9nakinfpyhc44hsfbi.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bay esian:0. X-HE-Tag: meat36_010fc4e26fbe X-Filterd-Recvd-Size: 4019 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:10 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5039F221E5; Fri, 7 Aug 2020 06:22:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781329; bh=8buI4U/A+YZMwTmUmzwOuFWCSMd7tVNO5fQy6sTm0yU=; h=Date:From:To:Subject:In-Reply-To:From; b=OD9SJB/p0NMRrUKp1k4SI+XY+ZSKnBu7FeMQfntAefLDV6E257gXhwXxcIMDBVfWI dpV1ALFh0J/1hSIw6k1yixYMyI0rb5j3EJ1I4EFwJgNQZ3YcBoffsIETg88i4l1UtV Ei/hoGbNuFh0u+z2Hb6WQqA670Fn6RVz1glNHhg8= Date: Thu, 06 Aug 2020 23:22:08 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, hannes@cmpxchg.org, laoar.shao@gmail.com, linux-mm@kvack.org, mhocko@kernel.org, mhocko@suse.com, mm-commits@vger.kernel.org, penguin-kernel@i-love.sakura.ne.jp, rientjes@google.com, torvalds@linux-foundation.org Subject: [patch 090/163] memcg, oom: check memcg margin for parallel oom Message-ID: <20200807062208.OPynQN011%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: DE8896D74 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yafang Shao Subject: memcg, oom: check memcg margin for parallel oom Memcg oom killer invocation is synchronized by the global oom_lock and tasks are sleeping on the lock while somebody is selecting the victim or potentially race with the oom_reaper is releasing the victim's memory. This can result in a pointless oom killer invocation because a waiter might be racing with the oom_reaper P1 oom_reaper P2 oom_reap_task mutex_lock(oom_lock) out_of_memory # no victim because we have one already __oom_reap_task_mm mute_unlock(oom_lock) mutex_lock(oom_lock) set MMF_OOM_SKIP select_bad_process # finds a new victim The page allocator prevents from this race by trying to allocate after the lock can be acquired (in __alloc_pages_may_oom) which acts as a last minute check. Moreover page allocator simply doesn't block on the oom_lock and simply retries the whole reclaim process. Memcg oom killer should do the last minute check as well. Call mem_cgroup_margin to do that. Trylock on the oom_lock could be done as well but this doesn't seem to be necessary at this stage. [mhocko@kernel.org: commit log] Link: http://lkml.kernel.org/r/1594735034-19190-1-git-send-email-laoar.shao@gmail.com Suggested-by: Michal Hocko Signed-off-by: Yafang Shao Acked-by: Michal Hocko Acked-by: Chris Down Cc: Tetsuo Handa Cc: David Rientjes Cc: Johannes Weiner Signed-off-by: Andrew Morton --- mm/memcontrol.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) --- a/mm/memcontrol.c~memcg-oom-check-memcg-margin-for-parallel-oom +++ a/mm/memcontrol.c @@ -1663,15 +1663,21 @@ static bool mem_cgroup_out_of_memory(str .gfp_mask = gfp_mask, .order = order, }; - bool ret; + bool ret = true; if (mutex_lock_killable(&oom_lock)) return true; + + if (mem_cgroup_margin(memcg) >= (1 << order)) + goto unlock; + /* * A few threads which were not waiting at mutex_lock_killable() can * fail to bail out. Therefore, check again after holding oom_lock. */ ret = should_force_charge() || out_of_memory(&oc); + +unlock: mutex_unlock(&oom_lock); return ret; } From patchwork Fri Aug 7 06:22:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704853 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 69860138C for ; Fri, 7 Aug 2020 06:22:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3A2AD221E5 for ; Fri, 7 Aug 2020 06:22:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="wwNq4cSW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3A2AD221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AD5F78D0064; Fri, 7 Aug 2020 02:22:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AACB88D0026; Fri, 7 Aug 2020 02:22:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E9518D0064; Fri, 7 Aug 2020 02:22:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0148.hostedemail.com [216.40.44.148]) by kanga.kvack.org (Postfix) with ESMTP id 8195B8D0026 for ; Fri, 7 Aug 2020 02:22:14 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 48B4B2489 for ; Fri, 7 Aug 2020 06:22:14 +0000 (UTC) X-FDA: 77122777788.09.party82_12156d926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 244AC180AD801 for ; Fri, 7 Aug 2020 06:22:14 +0000 (UTC) X-Spam-Summary: 1,0,0,08fdbfe6e7f47f09,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2553:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3870:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:5007:6119:6261:6653:7576:7903:8599:8603:9025:9545:10004:10913:11026:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12783:12986:13069:13255:13311:13357:13846:14093:14130:14181:14384:14721:14849:21080:21451:21627:21939:21990:30012:30054:30064:30069:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8qzohenyusnp8r6i67or4qheqzop18pyp6uxykkzc6p56hguqqjnniaznewy.exsmcf33swbas8a7wc6bqmogt6jemhkbunapp8gjhi4aw1tamfdtouiqo4cpg9n.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MS F:not bu X-HE-Tag: party82_12156d926fbe X-Filterd-Recvd-Size: 2878 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:13 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A579E2177B; Fri, 7 Aug 2020 06:22:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781333; bh=4GM+OdBs020kSyD8E3UZo77r6Tblr8fQbRSIP0O90fY=; h=Date:From:To:Subject:In-Reply-To:From; b=wwNq4cSW+rOfQ09zKGk0FlKNjl7J9BgbgYstyDUlJBIuUsQOXaZN+WnV3EkYpg6Nt OEgjDC4pacZNYuaFxBXJGzCHbyH7tu0wN38Qr62Lccza7vM/BRkHH7p0/iQZ+B7ENu YnYLvhkdp+GZz8QukJ8mK7NaALF4KineU865wo3M= Date: Thu, 06 Aug 2020 23:22:12 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, shakeelb@google.com, torvalds@linux-foundation.org Subject: [patch 091/163] mm: memcontrol: restore proper dirty throttling when memory.high changes Message-ID: <20200807062212.z-72hz3Ek%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 244AC180AD801 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000009, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Johannes Weiner Subject: mm: memcontrol: restore proper dirty throttling when memory.high changes Commit 8c8c383c04f6 ("mm: memcontrol: try harder to set a new memory.high") inadvertently removed a callback to recalculate the writeback cache size in light of a newly configured memory.high limit. Without letting the writeback cache know about a potentially heavily reduced limit, it may permit too many dirty pages, which can cause unnecessary reclaim latencies or even avoidable OOM situations. This was spotted while reading the code, it hasn't knowingly caused any problems in practice so far. Link: http://lkml.kernel.org/r/20200728135210.379885-1-hannes@cmpxchg.org Fixes: 8c8c383c04f6 ("mm: memcontrol: try harder to set a new memory.high") Signed-off-by: Johannes Weiner Acked-by: Chris Down Reviewed-by: Shakeel Butt Acked-by: Michal Hocko Cc: Roman Gushchin Signed-off-by: Andrew Morton --- mm/memcontrol.c | 2 ++ 1 file changed, 2 insertions(+) --- a/mm/memcontrol.c~mm-memcontrol-restore-proper-dirty-throttling-when-memoryhigh-changes +++ a/mm/memcontrol.c @@ -6273,6 +6273,8 @@ static ssize_t memory_high_write(struct page_counter_set_high(&memcg->memory, high); + memcg_wb_domain_size_changed(memcg); + return nbytes; } From patchwork Fri Aug 7 06:22:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704855 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DDADA14B7 for ; Fri, 7 Aug 2020 06:22:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A84F9221E5 for ; Fri, 7 Aug 2020 06:22:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="EzfEXjag" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A84F9221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3D0F88D0065; Fri, 7 Aug 2020 02:22:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 382748D0026; Fri, 7 Aug 2020 02:22:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26F8C8D0065; Fri, 7 Aug 2020 02:22:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0128.hostedemail.com [216.40.44.128]) by kanga.kvack.org (Postfix) with ESMTP id 108388D0026 for ; Fri, 7 Aug 2020 02:22:18 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CAF971F0A for ; Fri, 7 Aug 2020 06:22:17 +0000 (UTC) X-FDA: 77122777914.20.key56_131220726fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id A43FD180C07A3 for ; Fri, 7 Aug 2020 06:22:17 +0000 (UTC) X-Spam-Summary: 1,0,0,3b09d2cbb7ac8e53,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1544:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2564:2682:2685:2693:2731:2859:2898:2902:2916:2933:2937:2939:2942:2945:2947:2951:2954:2987:3022:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6261:6653:7576:7903:8599:9025:9121:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13229:13255:13846:14096:14181:14721:14849:21064:21080:21433:21450:21451:21627:21740:21939:21990:22013:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygbwgqq87rsoxiam3gwsnsujjgeyprdnr7sw3qauie86g94g5yxhkh1ezki6q.3wmkp7qof8fxrq8b8pmcpkmujxx4tdexgbcrfesz63qtbuh7551sp53bnswyo9d.r-lbl8.mailshell.net-223.238.255.1 00,Cache X-HE-Tag: key56_131220726fbe X-Filterd-Recvd-Size: 5527 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:17 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E17A92177B; Fri, 7 Aug 2020 06:22:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781336; bh=57YwXGxS5eIpgfsMtX1iESNczBzRJgaBM2J+QLcJ8mw=; h=Date:From:To:Subject:In-Reply-To:From; b=EzfEXjag1a8vgc3bbaGHAP7muVKYiyHDUTboSj7tOeNxWLuyaU4f3zToafL1nKJ+r kZgne3bvIH/RpmHDg2GLEhFx0AAwtbI1WgZo8JmX21oKghMIUABiho8H1PIPsg/PBx xBSuZgjciN3mPdvxf+jb1wfxAPlqcBX/A/Szhqa4= Date: Thu, 06 Aug 2020 23:22:15 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@suse.com, mm-commits@vger.kernel.org, shakeelb@google.com, torvalds@linux-foundation.org Subject: [patch 092/163] mm: memcontrol: don't count limit-setting reclaim as memory pressure Message-ID: <20200807062215.hpQOJtMdn%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: A43FD180C07A3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000120, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Johannes Weiner Subject: mm: memcontrol: don't count limit-setting reclaim as memory pressure When an outside process lowers one of the memory limits of a cgroup (or uses the force_empty knob in cgroup1), direct reclaim is performed in the context of the write(), in order to directly enforce the new limit and have it being met by the time the write() returns. Currently, this reclaim activity is accounted as memory pressure in the cgroup that the writer(!) belongs to. This is unexpected. It specifically causes problems for senpai (https://github.com/facebookincubator/senpai), which is an agent that routinely adjusts the memory limits and performs associated reclaim work in tens or even hundreds of cgroups running on the host. The cgroup that senpai is running in itself will report elevated levels of memory pressure, even though it itself is under no memory shortage or any sort of distress. Move the psi annotation from the central cgroup reclaim function to callsites in the allocation context, and thereby no longer count any limit-setting reclaim as memory pressure. If the newly set limit causes the workload inside the cgroup into direct reclaim, that of course will continue to count as memory pressure. Link: http://lkml.kernel.org/r/20200728135210.379885-2-hannes@cmpxchg.org Signed-off-by: Johannes Weiner Reviewed-by: Shakeel Butt Reviewed-by: Roman Gushchin Acked-by: Chris Down Acked-by: Michal Hocko Signed-off-by: Andrew Morton --- mm/memcontrol.c | 11 ++++++++++- mm/vmscan.c | 6 ------ 2 files changed, 10 insertions(+), 7 deletions(-) --- a/mm/memcontrol.c~mm-memcontrol-dont-count-limit-setting-reclaim-as-memory-pressure +++ a/mm/memcontrol.c @@ -2374,12 +2374,18 @@ static unsigned long reclaim_high(struct unsigned long nr_reclaimed = 0; do { + unsigned long pflags; + if (page_counter_read(&memcg->memory) <= READ_ONCE(memcg->memory.high)) continue; + memcg_memory_event(memcg, MEMCG_HIGH); + + psi_memstall_enter(&pflags); nr_reclaimed += try_to_free_mem_cgroup_pages(memcg, nr_pages, gfp_mask, true); + psi_memstall_leave(&pflags); } while ((memcg = parent_mem_cgroup(memcg)) && !mem_cgroup_is_root(memcg)); @@ -2621,10 +2627,11 @@ static int try_charge(struct mem_cgroup int nr_retries = MAX_RECLAIM_RETRIES; struct mem_cgroup *mem_over_limit; struct page_counter *counter; + enum oom_status oom_status; unsigned long nr_reclaimed; bool may_swap = true; bool drained = false; - enum oom_status oom_status; + unsigned long pflags; if (mem_cgroup_is_root(memcg)) return 0; @@ -2684,8 +2691,10 @@ retry: memcg_memory_event(mem_over_limit, MEMCG_MAX); + psi_memstall_enter(&pflags); nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages, gfp_mask, may_swap); + psi_memstall_leave(&pflags); if (mem_cgroup_margin(mem_over_limit) >= nr_pages) goto retry; --- a/mm/vmscan.c~mm-memcontrol-dont-count-limit-setting-reclaim-as-memory-pressure +++ a/mm/vmscan.c @@ -3310,7 +3310,6 @@ unsigned long try_to_free_mem_cgroup_pag bool may_swap) { unsigned long nr_reclaimed; - unsigned long pflags; unsigned int noreclaim_flag; struct scan_control sc = { .nr_to_reclaim = max(nr_pages, SWAP_CLUSTER_MAX), @@ -3331,17 +3330,12 @@ unsigned long try_to_free_mem_cgroup_pag struct zonelist *zonelist = node_zonelist(numa_node_id(), sc.gfp_mask); set_task_reclaim_state(current, &sc.reclaim_state); - trace_mm_vmscan_memcg_reclaim_begin(0, sc.gfp_mask); - - psi_memstall_enter(&pflags); noreclaim_flag = memalloc_noreclaim_save(); nr_reclaimed = do_try_to_free_pages(zonelist, &sc); memalloc_noreclaim_restore(noreclaim_flag); - psi_memstall_leave(&pflags); - trace_mm_vmscan_memcg_reclaim_end(nr_reclaimed); set_task_reclaim_state(current, NULL); From patchwork Fri Aug 7 06:22:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704857 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E3D00138C for ; Fri, 7 Aug 2020 06:22:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B7BE12177B for ; Fri, 7 Aug 2020 06:22:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="XX7RS8XN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B7BE12177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6B8508D0066; Fri, 7 Aug 2020 02:22:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 63FBB8D0026; Fri, 7 Aug 2020 02:22:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 557608D0066; Fri, 7 Aug 2020 02:22:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0145.hostedemail.com [216.40.44.145]) by kanga.kvack.org (Postfix) with ESMTP id 3A29C8D0026 for ; Fri, 7 Aug 2020 02:22:21 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 024FA181AEF10 for ; Fri, 7 Aug 2020 06:22:21 +0000 (UTC) X-FDA: 77122778082.28.grain08_1e0d9f926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id C995D6D74 for ; Fri, 7 Aug 2020 06:22:20 +0000 (UTC) X-Spam-Summary: 1,0,0,a514fad50c76c018,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:152:355:379:800:960:967:973:988:989:1260:1263:1277:1311:1313:1314:1345:1359:1381:1431:1437:1513:1515:1516:1518:1521:1534:1542:1593:1594:1711:1730:1747:1777:1792:2393:2525:2553:2559:2563:2682:2685:2859:2897:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4041:4321:5007:6261:6653:7576:8599:9025:9545:10004:10400:10450:10455:10913:11026:11658:11914:12043:12048:12295:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13229:13846:13870:14093:14096:14097:14181:14721:14849:19904:19999:21067:21080:21451:21627:21740:21939:30054:30064:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yg6whfqpphspd3n7m4fu1jyu5pjyp9noef3npn1hmrfpoy7pamhuw1e7p483w.1waxj9hr3hagpuhz6ozfqqo3eoah86epy8am4miab5oduxtm4ctpfs47z77qs5g.4-lbl8.mail shell.ne X-HE-Tag: grain08_1e0d9f926fbe X-Filterd-Recvd-Size: 4293 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:20 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 33CC122C9F; Fri, 7 Aug 2020 06:22:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781339; bh=oSdNly9aIeyibSd9/S4eMSMD5jPf3sak6mS/2Aqks1A=; h=Date:From:To:Subject:In-Reply-To:From; b=XX7RS8XNlUeUGQWfrxFpo/0uT/lGY26j6IKgkZFLpueXaBLdAiIAABur7zi8FVCzf X+Ll7btLzEg5Vtp8Je/89OT1Mo6vwK+wKDRRfAnjetFf6NoKI+axYh8c4husXYnb8N i7rENz9gdA3cZhTqQnGazowo0YSklM5Shbt6IGhw= Date: Thu, 06 Aug 2020 23:22:18 -0700 From: Andrew Morton To: akpm@linux-foundation.org, guro@fb.com, hannes@cmpxchg.org, linux-mm@kvack.org, mhocko@suse.com, mkoutny@suse.com, mm-commits@vger.kernel.org, stable@vger.kernel.org, tj@kernel.org, torvalds@linux-foundation.org Subject: [patch 093/163] mm/page_counter.c: fix protection usage propagation Message-ID: <20200807062218.U7OzBph9t%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 MIME-Version: 1.0 X-Rspamd-Queue-Id: C995D6D74 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Michal Koutný Subject: mm/page_counter.c: fix protection usage propagation When workload runs in cgroups that aren't directly below root cgroup and their parent specifies reclaim protection, it may end up ineffective. The reason is that propagate_protected_usage() is not called in all hierarchy up. All the protected usage is incorrectly accumulated in the workload's parent. This means that siblings_low_usage is overestimated and effective protection underestimated. Even though it is transitional phenomenon (uncharge path does correct propagation and fixes the wrong children_low_usage), it can undermine the intended protection unexpectedly. We have noticed this problem while seeing a swap out in a descendant of a protected memcg (intermediate node) while the parent was conveniently under its protection limit and the memory pressure was external to that hierarchy. Michal has pinpointed this down to the wrong siblings_low_usage which led to the unwanted reclaim. The fix is simply updating children_low_usage in respective ancestors also in the charging path. Link: http://lkml.kernel.org/r/20200803153231.15477-1-mhocko@kernel.org Fixes: 230671533d64 ("mm: memory.low hierarchical behavior") Signed-off-by: Michal Koutný Signed-off-by: Michal Hocko Acked-by: Michal Hocko Acked-by: Roman Gushchin Cc: Johannes Weiner Cc: Tejun Heo Cc: [4.18+] Signed-off-by: Andrew Morton --- mm/page_counter.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --- a/mm/page_counter.c~mm-fix-protection-usage-propagation +++ a/mm/page_counter.c @@ -72,7 +72,7 @@ void page_counter_charge(struct page_cou long new; new = atomic_long_add_return(nr_pages, &c->usage); - propagate_protected_usage(counter, new); + propagate_protected_usage(c, new); /* * This is indeed racy, but we can live with some * inaccuracy in the watermark. @@ -116,7 +116,7 @@ bool page_counter_try_charge(struct page new = atomic_long_add_return(nr_pages, &c->usage); if (new > c->max) { atomic_long_sub(nr_pages, &c->usage); - propagate_protected_usage(counter, new); + propagate_protected_usage(c, new); /* * This is racy, but we can live with some * inaccuracy in the failcnt. @@ -125,7 +125,7 @@ bool page_counter_try_charge(struct page *fail = c; goto failed; } - propagate_protected_usage(counter, new); + propagate_protected_usage(c, new); /* * Just like with failcnt, we can live with some * inaccuracy in the watermark. From patchwork Fri Aug 7 06:22:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704859 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CAD95138C for ; Fri, 7 Aug 2020 06:22:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 94FE822D00 for ; Fri, 7 Aug 2020 06:22:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="gI5gry7F" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 94FE822D00 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 46DF38D0067; Fri, 7 Aug 2020 02:22:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 444BA8D0026; Fri, 7 Aug 2020 02:22:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35A0D8D0067; Fri, 7 Aug 2020 02:22:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0228.hostedemail.com [216.40.44.228]) by kanga.kvack.org (Postfix) with ESMTP id 1ED7C8D0026 for ; Fri, 7 Aug 2020 02:22:24 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E32D3181AEF10 for ; Fri, 7 Aug 2020 06:22:23 +0000 (UTC) X-FDA: 77122778166.11.bikes17_2f0128226fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id B0166180F8B81 for ; Fri, 7 Aug 2020 06:22:23 +0000 (UTC) X-Spam-Summary: 1,0,0,ac730ec8f6045c28,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3867:3868:3870:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12219:12297:12438:12517:12519:12555:12679:12783:12986:13069:13311:13357:13846:14094:14181:14384:14721:14849:21080:21451:21627:21939:30012:30054:30064:30070:30074,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y89nwxpnn6o97ac4tpkjqngdqftocwnmyxdg16ys63nizrkwdxfs7capcfifp.uciudinzgpty1ea4ixfbcxr3tesxt7ww7r6c5zdctni76zkbhr8p54bpka9jhcz.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutra l,Custom X-HE-Tag: bikes17_2f0128226fbe X-Filterd-Recvd-Size: 2475 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:23 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 56144221E5; Fri, 7 Aug 2020 06:22:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781342; bh=OL8oqQ2stk7opHjsLnLzeXyTEWilQEc9tZV1tv1GZv4=; h=Date:From:To:Subject:In-Reply-To:From; b=gI5gry7Ff4w8WcIdQzDNk2zmIeW9HzTGQ2FFn3Wag6KwBRxacU8JdKT5CFfeEyFjq ht9Cc/4tB6zTA3J2LudZoNMllvAeKPbjcqMfzvAr9RKi/4PpTNgr4GmS2ZjaijABO3 W2mNszdK++VPEfMR+4krN6IOrFicXICwyyZ5sK5U= Date: Thu, 06 Aug 2020 23:22:21 -0700 From: Andrew Morton To: akpm@linux-foundation.org, david@redhat.com, jgg@mellanox.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, rcampbell@nvidia.com, torvalds@linux-foundation.org Subject: [patch 094/163] mm: remove redundant check non_swap_entry() Message-ID: <20200807062221.EfR5XGPgf%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: B0166180F8B81 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Ralph Campbell Subject: mm: remove redundant check non_swap_entry() In zap_pte_range(), the check for non_swap_entry() and is_device_private_entry() is unnecessary since the latter is sufficient to determine if the page is a device private page. Remove the test for non_swap_entry() to simplify the code and for clarity. Link: http://lkml.kernel.org/r/20200615175405.4613-1-rcampbell@nvidia.com Signed-off-by: Ralph Campbell Reviewed-by: Jason Gunthorpe Acked-by: David Hildenbrand Signed-off-by: Andrew Morton --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/mm/memory.c~mm-remove-redundant-check-non_swap_entry +++ a/mm/memory.c @@ -1098,7 +1098,7 @@ again: } entry = pte_to_swp_entry(ptent); - if (non_swap_entry(entry) && is_device_private_entry(entry)) { + if (is_device_private_entry(entry)) { struct page *page = device_private_entry_to_page(entry); if (unlikely(details && details->check_mapping)) { From patchwork Fri Aug 7 06:22:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704861 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CE4F3138C for ; Fri, 7 Aug 2020 06:22:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9F6B122D00 for ; Fri, 7 Aug 2020 06:22:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="2euGcCb0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F6B122D00 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3D0798D0068; Fri, 7 Aug 2020 02:22:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3816F8D0026; Fri, 7 Aug 2020 02:22:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 271C18D0068; Fri, 7 Aug 2020 02:22:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0197.hostedemail.com [216.40.44.197]) by kanga.kvack.org (Postfix) with ESMTP id 101858D0026 for ; Fri, 7 Aug 2020 02:22:27 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C916382499A8 for ; Fri, 7 Aug 2020 06:22:26 +0000 (UTC) X-FDA: 77122778292.28.hall09_1a0c76826fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id AFD7E6D74 for ; Fri, 7 Aug 2020 06:22:26 +0000 (UTC) X-Spam-Summary: 1,0,0,b865893365e8744a,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2553:2559:2563:2682:2685:2859:2897:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:7875:8599:8603:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12517:12519:12555:12679:12783:12986:13069:13311:13357:13846:14096:14181:14384:14721:14849:21080:21451:21627:21795:21939:21990:30003:30034:30051:30054:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrgf9rmwwgcwxfdgzjn6oii5eneyp1q7qj3zj1ti9ouuyizdpxrx4j57c51gy.hginsnp89e9chz5yng4jepzbbnmdz4gprgtw81bjn33a4xkz1gx68ms6ba8pm69.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCac he:0,MSF X-HE-Tag: hall09_1a0c76826fbe X-Filterd-Recvd-Size: 2867 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:26 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6495F2177B; Fri, 7 Aug 2020 06:22:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781345; bh=ZKPDNveFMOs5PNNaKoZIbOLuceh7796kjd4EhLCYqY4=; h=Date:From:To:Subject:In-Reply-To:From; b=2euGcCb0N92qpHG0GSBFy6FzLbMUkXKI9sWd+nA/c4FF1nTOJsZ2aNyTrPbuLw/Qn GbxF7j0b03v8NlrJUbOhxtyyvWx4DiEGTUMxvbD2m3rgAVL5LHe6uEFaFRGwCTpRSt xFHkOk0MopAxtJTOEdw8vsSMKhlHYDs1yR4SGlxA= Date: Thu, 06 Aug 2020 23:22:24 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, torvalds@linux-foundation.org, zhangalex@google.com Subject: [patch 095/163] mm/memory.c: make remap_pfn_range() reject unaligned addr Message-ID: <20200807062224.b7xjNHWMw%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: AFD7E6D74 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alex Zhang Subject: mm/memory.c: make remap_pfn_range() reject unaligned addr This function implicitly assumes that the addr passed in is page aligned. A non page aligned addr could ultimately cause a kernel bug in remap_pte_range as the exit condition in the logic loop may never be satisfied. This patch documents the need for the requirement, as well as explicitly adds a check for it. Link: http://lkml.kernel.org/r/20200617233512.177519-1-zhangalex@google.com Signed-off-by: Alex Zhang Reviewed-by: Andrew Morton Signed-off-by: Andrew Morton --- mm/memory.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) --- a/mm/memory.c~mm-memoryc-make-remap_pfn_range-reject-unaligned-addr +++ a/mm/memory.c @@ -2082,7 +2082,7 @@ static inline int remap_p4d_range(struct /** * remap_pfn_range - remap kernel memory to userspace * @vma: user vma to map to - * @addr: target user address to start at + * @addr: target page aligned user address to start at * @pfn: page frame number of kernel physical memory address * @size: size of mapping area * @prot: page protection flags for this mapping @@ -2101,6 +2101,9 @@ int remap_pfn_range(struct vm_area_struc unsigned long remap_pfn = pfn; int err; + if (WARN_ON_ONCE(!PAGE_ALIGNED(addr))) + return -EINVAL; + /* * Physically remapped pages are special. Tell the * rest of the world about it: From patchwork Fri Aug 7 06:22:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704863 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 16589138C for ; Fri, 7 Aug 2020 06:22:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A347722CAF for ; Fri, 7 Aug 2020 06:22:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="izWCESX/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A347722CAF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2C1238D0069; Fri, 7 Aug 2020 02:22:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 296608D0026; Fri, 7 Aug 2020 02:22:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 160E38D0069; Fri, 7 Aug 2020 02:22:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0040.hostedemail.com [216.40.44.40]) by kanga.kvack.org (Postfix) with ESMTP id ED1288D0026 for ; Fri, 7 Aug 2020 02:22:31 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B2C7A3625 for ; Fri, 7 Aug 2020 06:22:31 +0000 (UTC) X-FDA: 77122778502.09.fight04_0d0436926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 8B6FB180AD802 for ; Fri, 7 Aug 2020 06:22:31 +0000 (UTC) X-Spam-Summary: 1,0,0,3ebf974aa118804f,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:327:355:379:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1434:1437:1605:1730:1747:1777:1792:1801:1981:2194:2196:2198:2199:2200:2201:2393:2525:2559:2564:2682:2685:2859:2894:2896:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4605:5007:6261:6653:6737:7514:7576:7903:7974:8599:8603:8784:9010:9025:9036:9149:9545:9592:10004:10913:11026:11657:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13141:13161:13229:13230:13846:21080:21094:21323:21433:21451:21611:21627:21740:21789:21939:30003:30029:30054:30056:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrr6zumznce9o1agtf9xrjs7a5jycfc1nt1hik3j869eq7sxxxxdyiueig9ep.hfyc4ehd4oa5ksz8zciwjowx1c8fehohb3od689anit4nwcoe6oehyjw6hndu5q.r-lbl 8.mailsh X-HE-Tag: fight04_0d0436926fbe X-Filterd-Recvd-Size: 49847 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:30 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id CE6D22177B; Fri, 7 Aug 2020 06:22:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781349; bh=Bgz6L+BykRZpW9FtE0b8rbrPMS0gUdHFshXErraIHOg=; h=Date:From:To:Subject:In-Reply-To:From; b=izWCESX/ue5YbUiAksGru+x9qEh17+p7XvXF/8Q4FvsOu9UxPUi9sCWua6DB/+Hf7 YiA/SbXY/QHImOFCPxX07Nk3Sc7rNrCljYSON403SJm2ILggoIFxO0x6NhRRhX870m sCHhIR5CAzo3XRS5WC/ssihWrlQBMbYg8piHiPIY= Date: Thu, 06 Aug 2020 23:22:28 -0700 From: Andrew Morton To: abdhalee@linux.vnet.ibm.com, akpm@linux-foundation.org, arnd@arndb.de, christophe.leroy@csgroup.eu, geert@linux-m68k.org, jcmvbkbc@gmail.com, joro@8bytes.org, jroedel@suse.de, linux-mm@kvack.org, luto@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, peterz@infradead.org, rostedt@goodmis.org, rppt@linux.ibm.com, sathnaga@linux.vnet.ibm.com, sfr@canb.auug.org.au, shorne@gmail.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 096/163] mm: remove unneeded includes of Message-ID: <20200807062228.ABou53Sje%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 8B6FB180AD802 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Subject: mm: remove unneeded includes of Patch series "mm: cleanup usage of " Most architectures have very similar versions of pXd_alloc_one() and pXd_free_one() for intermediate levels of page table. These patches add generic versions of these functions in and enable use of the generic functions where appropriate. In addition, functions declared and defined in headers are used mostly by core mm and early mm initialization in arch and there is no actual reason to have the included all over the place. The first patch in this series removes unneeded includes of In the end it didn't work out as neatly as I hoped and moving pXd_alloc_track() definitions to would require unnecessary changes to arches that have custom page table allocations, so I've decided to move lib/ioremap.c to mm/ and make pgalloc-track.h local to mm/. This patch (of 8): In most cases header is required only for allocations of page table memory. Most of the .c files that include that header do not use symbols declared in and do not require that header. As for the other header files that used to include , it is possible to move that include into the .c file that actually uses symbols from and drop the include from the header file. The process was somewhat automated using sed -i -E '/[<"]asm\/pgalloc\.h/d' \ $(grep -L -w -f /tmp/xx \ $(git grep -E -l '[<"]asm/pgalloc\.h')) where /tmp/xx contains all the symbols defined in arch/*/include/asm/pgalloc.h. [rppt@linux.ibm.com: fix powerpc warning] Link: http://lkml.kernel.org/r/20200627143453.31835-1-rppt@kernel.org Link: http://lkml.kernel.org/r/20200627143453.31835-2-rppt@kernel.org Signed-off-by: Mike Rapoport Acked-by: Geert Uytterhoeven [m68k] Reviewed-by: Pekka Enberg Cc: Abdul Haleem Cc: Andy Lutomirski Cc: Arnd Bergmann Cc: Christophe Leroy Cc: Joerg Roedel Cc: Max Filippov Cc: Peter Zijlstra Cc: Satheesh Rajendran Cc: Stafford Horne Cc: Stephen Rothwell Cc: Steven Rostedt Cc: Joerg Roedel Cc: Matthew Wilcox Signed-off-by: Andrew Morton --- arch/alpha/include/asm/tlbflush.h | 1 - arch/alpha/kernel/core_irongate.c | 1 - arch/alpha/kernel/core_marvel.c | 1 - arch/alpha/kernel/core_titan.c | 1 - arch/alpha/kernel/machvec_impl.h | 2 -- arch/alpha/kernel/smp.c | 1 - arch/alpha/mm/numa.c | 1 - arch/arc/mm/fault.c | 1 - arch/arc/mm/init.c | 1 - arch/arm/include/asm/tlb.h | 1 - arch/arm/kernel/machine_kexec.c | 1 - arch/arm/kernel/smp.c | 1 - arch/arm/kernel/suspend.c | 1 - arch/arm/mach-omap2/omap-mpuss-lowpower.c | 1 - arch/arm/mm/hugetlbpage.c | 1 - arch/arm/mm/mmu.c | 1 + arch/arm64/kernel/smp.c | 1 - arch/arm64/mm/hugetlbpage.c | 1 - arch/arm64/mm/ioremap.c | 1 - arch/arm64/mm/mmu.c | 1 + arch/csky/kernel/smp.c | 1 - arch/ia64/include/asm/tlb.h | 1 - arch/ia64/kernel/process.c | 1 - arch/ia64/kernel/smp.c | 1 - arch/ia64/kernel/smpboot.c | 1 - arch/ia64/mm/contig.c | 1 - arch/ia64/mm/discontig.c | 1 - arch/ia64/mm/hugetlbpage.c | 1 - arch/ia64/mm/tlb.c | 1 - arch/m68k/include/asm/mmu_context.h | 2 +- arch/m68k/kernel/dma.c | 2 +- arch/m68k/kernel/traps.c | 3 +-- arch/m68k/mm/cache.c | 2 +- arch/m68k/mm/fault.c | 1 - arch/m68k/mm/kmap.c | 2 +- arch/m68k/mm/mcfmmu.c | 1 + arch/m68k/mm/memory.c | 1 - arch/m68k/sun3x/dvma.c | 2 +- arch/microblaze/include/asm/tlbflush.h | 1 - arch/microblaze/kernel/process.c | 1 - arch/microblaze/kernel/signal.c | 1 - arch/mips/sgi-ip32/ip32-memory.c | 1 - arch/openrisc/include/asm/tlbflush.h | 1 - arch/openrisc/kernel/or32_ksyms.c | 1 - arch/parisc/include/asm/mmu_context.h | 1 - arch/parisc/kernel/cache.c | 1 - arch/parisc/kernel/pci-dma.c | 1 - arch/parisc/kernel/process.c | 1 - arch/parisc/kernel/signal.c | 1 - arch/parisc/kernel/smp.c | 1 - arch/parisc/mm/hugetlbpage.c | 1 - arch/parisc/mm/ioremap.c | 2 +- arch/powerpc/include/asm/tlb.h | 1 - arch/powerpc/mm/book3s64/hash_hugetlbpage.c | 1 - arch/powerpc/mm/book3s64/hash_pgtable.c | 1 - arch/powerpc/mm/book3s64/hash_tlb.c | 1 - arch/powerpc/mm/book3s64/radix_hugetlbpage.c | 1 - arch/powerpc/mm/init_32.c | 1 - arch/powerpc/mm/kasan/8xx.c | 1 - arch/powerpc/mm/kasan/book3s_32.c | 1 - arch/powerpc/mm/mem.c | 1 - arch/powerpc/mm/nohash/40x.c | 1 - arch/powerpc/mm/nohash/8xx.c | 1 - arch/powerpc/mm/nohash/fsl_booke.c | 1 - arch/powerpc/mm/nohash/kaslr_booke.c | 1 - arch/powerpc/mm/nohash/tlb.c | 1 + arch/powerpc/mm/pgtable.c | 1 - arch/powerpc/mm/pgtable_64.c | 1 - arch/powerpc/mm/ptdump/hashpagetable.c | 2 +- arch/powerpc/mm/ptdump/ptdump.c | 1 - arch/powerpc/platforms/pseries/cmm.c | 1 - arch/riscv/mm/fault.c | 1 - arch/s390/include/asm/tlb.h | 1 - arch/s390/include/asm/tlbflush.h | 1 - arch/s390/kernel/machine_kexec.c | 1 - arch/s390/kernel/ptrace.c | 1 - arch/s390/kvm/diag.c | 1 - arch/s390/kvm/priv.c | 1 - arch/s390/kvm/pv.c | 1 - arch/s390/mm/cmm.c | 1 - arch/s390/mm/mmap.c | 1 - arch/s390/mm/pgtable.c | 1 - arch/sh/kernel/idle.c | 1 - arch/sh/kernel/machine_kexec.c | 1 - arch/sh/mm/cache-sh3.c | 1 - arch/sh/mm/cache-sh7705.c | 1 - arch/sh/mm/hugetlbpage.c | 1 - arch/sh/mm/init.c | 1 + arch/sh/mm/ioremap_fixed.c | 1 - arch/sh/mm/tlb-sh3.c | 1 - arch/sparc/include/asm/ide.h | 1 - arch/sparc/include/asm/tlb_64.h | 1 - arch/sparc/kernel/leon_smp.c | 1 - arch/sparc/kernel/process_32.c | 1 - arch/sparc/kernel/signal_32.c | 1 - arch/sparc/kernel/smp_32.c | 1 - arch/sparc/kernel/smp_64.c | 1 + arch/sparc/kernel/sun4m_irq.c | 1 - arch/sparc/mm/highmem.c | 1 - arch/sparc/mm/io-unit.c | 1 - arch/sparc/mm/iommu.c | 1 - arch/sparc/mm/tlb.c | 1 - arch/x86/ia32/ia32_aout.c | 1 - arch/x86/include/asm/mmu_context.h | 1 - arch/x86/kernel/alternative.c | 1 + arch/x86/kernel/apic/apic.c | 1 - arch/x86/kernel/mpparse.c | 1 - arch/x86/kernel/traps.c | 1 - arch/x86/mm/fault.c | 1 - arch/x86/mm/hugetlbpage.c | 1 - arch/x86/mm/kaslr.c | 1 - arch/x86/mm/pgtable_32.c | 1 - arch/x86/mm/pti.c | 1 - arch/x86/platform/uv/bios_uv.c | 1 + arch/xtensa/kernel/xtensa_ksyms.c | 1 - arch/xtensa/mm/cache.c | 1 - arch/xtensa/mm/fault.c | 1 - drivers/block/xen-blkback/common.h | 1 - drivers/iommu/ipmmu-vmsa.c | 1 - drivers/xen/balloon.c | 1 - drivers/xen/privcmd.c | 1 - fs/binfmt_elf_fdpic.c | 1 - include/asm-generic/tlb.h | 1 - mm/hugetlb.c | 1 + mm/sparse.c | 1 - 125 files changed, 17 insertions(+), 118 deletions(-) --- a/arch/alpha/include/asm/tlbflush.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/alpha/include/asm/tlbflush.h @@ -5,7 +5,6 @@ #include #include #include -#include #ifndef __EXTERN_INLINE #define __EXTERN_INLINE extern inline --- a/arch/alpha/kernel/core_irongate.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/alpha/kernel/core_irongate.c @@ -302,7 +302,6 @@ irongate_init_arch(void) #include #include #include -#include #define GET_PAGE_DIR_OFF(addr) (addr >> 22) #define GET_PAGE_DIR_IDX(addr) (GET_PAGE_DIR_OFF(addr)) --- a/arch/alpha/kernel/core_marvel.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/alpha/kernel/core_marvel.c @@ -23,7 +23,6 @@ #include #include #include -#include #include #include --- a/arch/alpha/kernel/core_titan.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/alpha/kernel/core_titan.c @@ -20,7 +20,6 @@ #include #include -#include #include #include --- a/arch/alpha/kernel/machvec_impl.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/alpha/kernel/machvec_impl.h @@ -7,8 +7,6 @@ * This file has goodies to help simplify instantiation of machine vectors. */ -#include - /* Whee. These systems don't have an HAE: IRONGATE, MARVEL, POLARIS, TSUNAMI, TITAN, WILDFIRE Fix things up for the GENERIC kernel by defining the HAE address --- a/arch/alpha/kernel/smp.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/alpha/kernel/smp.c @@ -36,7 +36,6 @@ #include #include -#include #include #include --- a/arch/alpha/mm/numa.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/alpha/mm/numa.c @@ -17,7 +17,6 @@ #include #include -#include #include pg_data_t node_data[MAX_NUMNODES]; --- a/arch/arc/mm/fault.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arc/mm/fault.c @@ -13,7 +13,6 @@ #include #include #include -#include #include /* --- a/arch/arc/mm/init.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arc/mm/init.c @@ -14,7 +14,6 @@ #include #include #include -#include #include #include --- a/arch/arm64/kernel/smp.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm64/kernel/smp.c @@ -43,7 +43,6 @@ #include #include #include -#include #include #include #include --- a/arch/arm64/mm/hugetlbpage.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm64/mm/hugetlbpage.c @@ -17,7 +17,6 @@ #include #include #include -#include /* * HugeTLB Support Matrix --- a/arch/arm64/mm/ioremap.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm64/mm/ioremap.c @@ -16,7 +16,6 @@ #include #include -#include static void __iomem *__ioremap_caller(phys_addr_t phys_addr, size_t size, pgprot_t prot, void *caller) --- a/arch/arm64/mm/mmu.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm64/mm/mmu.c @@ -35,6 +35,7 @@ #include #include #include +#include #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) --- a/arch/arm/include/asm/tlb.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm/include/asm/tlb.h @@ -27,7 +27,6 @@ #else /* !CONFIG_MMU */ #include -#include #include static inline void __tlb_remove_table(void *_table) --- a/arch/arm/kernel/machine_kexec.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm/kernel/machine_kexec.c @@ -11,7 +11,6 @@ #include #include #include -#include #include #include #include --- a/arch/arm/kernel/smp.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm/kernel/smp.c @@ -37,7 +37,6 @@ #include #include #include -#include #include #include #include --- a/arch/arm/kernel/suspend.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm/kernel/suspend.c @@ -7,7 +7,6 @@ #include #include #include -#include #include #include #include --- a/arch/arm/mach-omap2/omap-mpuss-lowpower.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm/mach-omap2/omap-mpuss-lowpower.c @@ -42,7 +42,6 @@ #include #include #include -#include #include #include #include --- a/arch/arm/mm/hugetlbpage.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm/mm/hugetlbpage.c @@ -17,7 +17,6 @@ #include #include #include -#include /* * On ARM, huge pages are backed by pmd's rather than pte's, so we do a lot --- a/arch/arm/mm/mmu.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/arm/mm/mmu.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include --- a/arch/csky/kernel/smp.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/csky/kernel/smp.c @@ -23,7 +23,6 @@ #include #include #include -#include #ifdef CONFIG_CPU_HAS_FPU #include #endif --- a/arch/ia64/include/asm/tlb.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/ia64/include/asm/tlb.h @@ -42,7 +42,6 @@ #include #include -#include #include #include --- a/arch/ia64/kernel/process.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/ia64/kernel/process.c @@ -40,7 +40,6 @@ #include #include #include -#include #include #include #include --- a/arch/ia64/kernel/smpboot.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/ia64/kernel/smpboot.c @@ -49,7 +49,6 @@ #include #include #include -#include #include #include #include --- a/arch/ia64/kernel/smp.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/ia64/kernel/smp.c @@ -39,7 +39,6 @@ #include #include #include -#include #include #include #include --- a/arch/ia64/mm/contig.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/ia64/mm/contig.c @@ -21,7 +21,6 @@ #include #include -#include #include #include --- a/arch/ia64/mm/discontig.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/ia64/mm/discontig.c @@ -24,7 +24,6 @@ #include #include #include -#include #include #include #include --- a/arch/ia64/mm/hugetlbpage.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/ia64/mm/hugetlbpage.c @@ -18,7 +18,6 @@ #include #include #include -#include #include #include --- a/arch/ia64/mm/tlb.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/ia64/mm/tlb.c @@ -27,7 +27,6 @@ #include #include -#include #include #include #include --- a/arch/m68k/include/asm/mmu_context.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/m68k/include/asm/mmu_context.h @@ -222,7 +222,7 @@ static inline void activate_mm(struct mm #include #include -#include +#include static inline int init_new_context(struct task_struct *tsk, struct mm_struct *mm) --- a/arch/m68k/kernel/dma.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/m68k/kernel/dma.c @@ -15,7 +15,7 @@ #include #include -#include +#include #if defined(CONFIG_MMU) && !defined(CONFIG_COLDFIRE) void arch_dma_prep_coherent(struct page *page, size_t size) --- a/arch/m68k/kernel/traps.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/m68k/kernel/traps.c @@ -35,10 +35,9 @@ #include #include #include -#include #include #include - +#include static const char *vec_names[] = { [VEC_RESETSP] = "RESET SP", --- a/arch/m68k/mm/cache.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/m68k/mm/cache.c @@ -8,7 +8,7 @@ */ #include -#include +#include #include --- a/arch/m68k/mm/fault.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/m68k/mm/fault.c @@ -15,7 +15,6 @@ #include #include -#include extern void die_if_kernel(char *, struct pt_regs *, long); --- a/arch/m68k/mm/kmap.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/m68k/mm/kmap.c @@ -19,8 +19,8 @@ #include #include #include -#include #include +#include #undef DEBUG --- a/arch/m68k/mm/mcfmmu.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/m68k/mm/mcfmmu.c @@ -20,6 +20,7 @@ #include #include #include +#include #define KMAPAREA(x) ((x >= VMALLOC_START) && (x < KMAP_END)) --- a/arch/m68k/mm/memory.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/m68k/mm/memory.c @@ -17,7 +17,6 @@ #include #include #include -#include #include #include --- a/arch/m68k/sun3x/dvma.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/m68k/sun3x/dvma.c @@ -22,7 +22,7 @@ #include #include #include -#include +#include /* IOMMU support */ --- a/arch/microblaze/include/asm/tlbflush.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/microblaze/include/asm/tlbflush.h @@ -15,7 +15,6 @@ #include /* For TASK_SIZE */ #include #include -#include extern void _tlbie(unsigned long address); extern void _tlbia(void); --- a/arch/microblaze/kernel/process.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/microblaze/kernel/process.c @@ -18,7 +18,6 @@ #include #include #include -#include #include /* for USER_DS macros */ #include --- a/arch/microblaze/kernel/signal.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/microblaze/kernel/signal.c @@ -35,7 +35,6 @@ #include #include #include -#include #include #include #include --- a/arch/mips/sgi-ip32/ip32-memory.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/mips/sgi-ip32/ip32-memory.c @@ -14,7 +14,6 @@ #include #include #include -#include extern void crime_init(void); --- a/arch/openrisc/include/asm/tlbflush.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/openrisc/include/asm/tlbflush.h @@ -17,7 +17,6 @@ #include #include -#include #include #include --- a/arch/openrisc/kernel/or32_ksyms.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/openrisc/kernel/or32_ksyms.c @@ -26,7 +26,6 @@ #include #include #include -#include #define DECLARE_EXPORT(name) extern void name(void); EXPORT_SYMBOL(name) --- a/arch/parisc/include/asm/mmu_context.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/parisc/include/asm/mmu_context.h @@ -5,7 +5,6 @@ #include #include #include -#include #include static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) --- a/arch/parisc/kernel/cache.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/parisc/kernel/cache.c @@ -24,7 +24,6 @@ #include #include #include -#include #include #include #include --- a/arch/parisc/kernel/pci-dma.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/parisc/kernel/pci-dma.c @@ -32,7 +32,6 @@ #include /* for DMA_CHUNK_SIZE */ #include #include /* get_order */ -#include #include #include /* for purge_tlb_*() macros */ --- a/arch/parisc/kernel/process.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/parisc/kernel/process.c @@ -47,7 +47,6 @@ #include #include #include -#include #include #include --- a/arch/parisc/kernel/signal.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/parisc/kernel/signal.c @@ -30,7 +30,6 @@ #include #include #include -#include #include #include --- a/arch/parisc/kernel/smp.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/parisc/kernel/smp.c @@ -39,7 +39,6 @@ #include /* for CPU_IRQ_REGION and friends */ #include #include -#include #include #include #include --- a/arch/parisc/mm/hugetlbpage.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/parisc/mm/hugetlbpage.c @@ -15,7 +15,6 @@ #include #include -#include #include #include #include --- a/arch/parisc/mm/ioremap.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/parisc/mm/ioremap.c @@ -11,7 +11,7 @@ #include #include #include -#include +#include /* * Generic mapping function (not visible outside): --- a/arch/powerpc/include/asm/tlb.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/include/asm/tlb.h @@ -12,7 +12,6 @@ #ifndef __powerpc64__ #include #endif -#include #ifndef __powerpc64__ #include #include --- a/arch/powerpc/mm/book3s64/hash_hugetlbpage.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/book3s64/hash_hugetlbpage.c @@ -10,7 +10,6 @@ #include #include -#include #include #include --- a/arch/powerpc/mm/book3s64/hash_pgtable.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/book3s64/hash_pgtable.c @@ -9,7 +9,6 @@ #include #include -#include #include #include #include --- a/arch/powerpc/mm/book3s64/hash_tlb.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/book3s64/hash_tlb.c @@ -21,7 +21,6 @@ #include #include #include -#include #include #include #include --- a/arch/powerpc/mm/book3s64/radix_hugetlbpage.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/book3s64/radix_hugetlbpage.c @@ -2,7 +2,6 @@ #include #include #include -#include #include #include #include --- a/arch/powerpc/mm/init_32.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/init_32.c @@ -29,7 +29,6 @@ #include #include -#include #include #include #include --- a/arch/powerpc/mm/kasan/8xx.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/kasan/8xx.c @@ -5,7 +5,6 @@ #include #include #include -#include static int __init kasan_init_shadow_8M(unsigned long k_start, unsigned long k_end, void *block) --- a/arch/powerpc/mm/kasan/book3s_32.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/kasan/book3s_32.c @@ -4,7 +4,6 @@ #include #include -#include #include int __init kasan_init_region(void *start, size_t size) --- a/arch/powerpc/mm/mem.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/mem.c @@ -34,7 +34,6 @@ #include #include -#include #include #include #include --- a/arch/powerpc/mm/nohash/40x.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/nohash/40x.c @@ -32,7 +32,6 @@ #include #include -#include #include #include #include --- a/arch/powerpc/mm/nohash/8xx.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/nohash/8xx.c @@ -13,7 +13,6 @@ #include #include #include -#include #include --- a/arch/powerpc/mm/nohash/fsl_booke.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/nohash/fsl_booke.c @@ -37,7 +37,6 @@ #include #include -#include #include #include #include --- a/arch/powerpc/mm/nohash/kaslr_booke.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/nohash/kaslr_booke.c @@ -15,7 +15,6 @@ #include #include #include -#include #include #include #include --- a/arch/powerpc/mm/nohash/tlb.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/nohash/tlb.c @@ -34,6 +34,7 @@ #include #include +#include #include #include #include --- a/arch/powerpc/mm/pgtable_64.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/pgtable_64.c @@ -31,7 +31,6 @@ #include #include -#include #include #include #include --- a/arch/powerpc/mm/pgtable.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/pgtable.c @@ -23,7 +23,6 @@ #include #include #include -#include #include #include #include --- a/arch/powerpc/mm/ptdump/hashpagetable.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/ptdump/hashpagetable.c @@ -17,10 +17,10 @@ #include #include #include -#include #include #include #include +#include struct pg_state { struct seq_file *seq; --- a/arch/powerpc/mm/ptdump/ptdump.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/mm/ptdump/ptdump.c @@ -21,7 +21,6 @@ #include #include #include -#include #include #include --- a/arch/powerpc/platforms/pseries/cmm.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/powerpc/platforms/pseries/cmm.c @@ -26,7 +26,6 @@ #include #include #include -#include #include #include #include --- a/arch/riscv/mm/fault.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/riscv/mm/fault.c @@ -14,7 +14,6 @@ #include #include -#include #include #include --- a/arch/s390/include/asm/tlbflush.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/include/asm/tlbflush.h @@ -5,7 +5,6 @@ #include #include #include -#include /* * Flush all TLB entries on the local CPU. --- a/arch/s390/include/asm/tlb.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/include/asm/tlb.h @@ -36,7 +36,6 @@ static inline bool __tlb_remove_page_siz #define p4d_free_tlb p4d_free_tlb #define pud_free_tlb pud_free_tlb -#include #include #include --- a/arch/s390/kernel/machine_kexec.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/kernel/machine_kexec.c @@ -16,7 +16,6 @@ #include #include #include -#include #include #include #include --- a/arch/s390/kernel/ptrace.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/kernel/ptrace.c @@ -25,7 +25,6 @@ #include #include #include -#include #include #include #include --- a/arch/s390/kvm/diag.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/kvm/diag.c @@ -10,7 +10,6 @@ #include #include -#include #include #include #include "kvm-s390.h" --- a/arch/s390/kvm/priv.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/kvm/priv.c @@ -22,7 +22,6 @@ #include #include #include -#include #include #include #include --- a/arch/s390/kvm/pv.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/kvm/pv.c @@ -9,7 +9,6 @@ #include #include #include -#include #include #include #include --- a/arch/s390/mm/cmm.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/mm/cmm.c @@ -21,7 +21,6 @@ #include #include -#include #include #ifdef CONFIG_CMM_IUCV --- a/arch/s390/mm/mmap.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/mm/mmap.c @@ -17,7 +17,6 @@ #include #include #include -#include #include static unsigned long stack_maxrandom_size(void) --- a/arch/s390/mm/pgtable.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/s390/mm/pgtable.c @@ -19,7 +19,6 @@ #include #include -#include #include #include #include --- a/arch/sh/kernel/idle.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sh/kernel/idle.c @@ -14,7 +14,6 @@ #include #include #include -#include #include #include --- a/arch/sh/kernel/machine_kexec.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sh/kernel/machine_kexec.c @@ -14,7 +14,6 @@ #include #include #include -#include #include #include #include --- a/arch/sh/mm/cache-sh3.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sh/mm/cache-sh3.c @@ -16,7 +16,6 @@ #include #include #include -#include #include #include --- a/arch/sh/mm/cache-sh7705.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sh/mm/cache-sh7705.c @@ -20,7 +20,6 @@ #include #include #include -#include #include #include --- a/arch/sh/mm/hugetlbpage.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sh/mm/hugetlbpage.c @@ -17,7 +17,6 @@ #include #include -#include #include #include #include --- a/arch/sh/mm/init.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sh/mm/init.c @@ -27,6 +27,7 @@ #include #include #include +#include #include pgd_t swapper_pg_dir[PTRS_PER_PGD]; --- a/arch/sh/mm/ioremap_fixed.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sh/mm/ioremap_fixed.c @@ -18,7 +18,6 @@ #include #include #include -#include #include #include #include --- a/arch/sh/mm/tlb-sh3.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sh/mm/tlb-sh3.c @@ -21,7 +21,6 @@ #include #include -#include #include #include --- a/arch/sparc/include/asm/ide.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/include/asm/ide.h @@ -13,7 +13,6 @@ #include #ifdef CONFIG_SPARC64 -#include #include #include #include --- a/arch/sparc/include/asm/tlb_64.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/include/asm/tlb_64.h @@ -4,7 +4,6 @@ #include #include -#include #include #include --- a/arch/sparc/kernel/leon_smp.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/kernel/leon_smp.c @@ -38,7 +38,6 @@ #include #include #include -#include #include #include #include --- a/arch/sparc/kernel/process_32.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/kernel/process_32.c @@ -34,7 +34,6 @@ #include #include #include -#include #include #include #include --- a/arch/sparc/kernel/signal_32.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/kernel/signal_32.c @@ -23,7 +23,6 @@ #include #include -#include #include /* flush_sig_insns */ #include --- a/arch/sparc/kernel/smp_32.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/kernel/smp_32.c @@ -29,7 +29,6 @@ #include #include -#include #include #include #include --- a/arch/sparc/kernel/smp_64.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/kernel/smp_64.c @@ -47,6 +47,7 @@ #include #include #include +#include #include #include #include --- a/arch/sparc/kernel/sun4m_irq.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/kernel/sun4m_irq.c @@ -16,7 +16,6 @@ #include #include -#include #include #include #include --- a/arch/sparc/mm/highmem.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/mm/highmem.c @@ -29,7 +29,6 @@ #include #include -#include #include static pte_t *kmap_pte; --- a/arch/sparc/mm/iommu.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/mm/iommu.c @@ -16,7 +16,6 @@ #include #include -#include #include #include #include --- a/arch/sparc/mm/io-unit.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/mm/io-unit.c @@ -15,7 +15,6 @@ #include #include -#include #include #include #include --- a/arch/sparc/mm/tlb.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/sparc/mm/tlb.c @@ -10,7 +10,6 @@ #include #include -#include #include #include #include --- a/arch/x86/ia32/ia32_aout.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/ia32/ia32_aout.c @@ -30,7 +30,6 @@ #include #include -#include #include #include #include --- a/arch/x86/include/asm/mmu_context.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/include/asm/mmu_context.h @@ -9,7 +9,6 @@ #include -#include #include #include #include --- a/arch/x86/kernel/alternative.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/kernel/alternative.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include --- a/arch/x86/kernel/apic/apic.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/kernel/apic/apic.c @@ -40,7 +40,6 @@ #include #include #include -#include #include #include #include --- a/arch/x86/kernel/mpparse.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/kernel/mpparse.c @@ -22,7 +22,6 @@ #include #include #include -#include #include #include #include --- a/arch/x86/kernel/traps.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/kernel/traps.c @@ -62,7 +62,6 @@ #ifdef CONFIG_X86_64 #include -#include #include #else #include --- a/arch/x86/mm/fault.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/mm/fault.c @@ -21,7 +21,6 @@ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ -#include /* pgd_*(), ... */ #include /* VSYSCALL_ADDR */ #include /* emulate_vsyscall */ #include /* struct vm86 */ --- a/arch/x86/mm/hugetlbpage.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/mm/hugetlbpage.c @@ -17,7 +17,6 @@ #include #include #include -#include #include #if 0 /* This is just for testing */ --- a/arch/x86/mm/kaslr.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/mm/kaslr.c @@ -26,7 +26,6 @@ #include #include -#include #include #include --- a/arch/x86/mm/pgtable_32.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/mm/pgtable_32.c @@ -11,7 +11,6 @@ #include #include -#include #include #include #include --- a/arch/x86/mm/pti.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/mm/pti.c @@ -34,7 +34,6 @@ #include #include #include -#include #include #include #include --- a/arch/x86/platform/uv/bios_uv.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/x86/platform/uv/bios_uv.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include --- a/arch/xtensa/kernel/xtensa_ksyms.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/xtensa/kernel/xtensa_ksyms.c @@ -25,7 +25,6 @@ #include #include #include -#include #include #ifdef CONFIG_BLK_DEV_FD #include --- a/arch/xtensa/mm/cache.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/xtensa/mm/cache.c @@ -31,7 +31,6 @@ #include #include #include -#include /* * Note: --- a/arch/xtensa/mm/fault.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/arch/xtensa/mm/fault.c @@ -20,7 +20,6 @@ #include #include #include -#include DEFINE_PER_CPU(unsigned long, asid_cache) = ASID_USER_FIRST; void bad_page_fault(struct pt_regs*, unsigned long, int); --- a/drivers/block/xen-blkback/common.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/drivers/block/xen-blkback/common.h @@ -36,7 +36,6 @@ #include #include #include -#include #include #include #include --- a/drivers/iommu/ipmmu-vmsa.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/drivers/iommu/ipmmu-vmsa.c @@ -28,7 +28,6 @@ #if defined(CONFIG_ARM) && !defined(CONFIG_IOMMU_DMA) #include -#include #else #define arm_iommu_create_mapping(...) NULL #define arm_iommu_attach_device(...) -ENODEV --- a/drivers/xen/balloon.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/drivers/xen/balloon.c @@ -58,7 +58,6 @@ #include #include -#include #include #include --- a/drivers/xen/privcmd.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/drivers/xen/privcmd.c @@ -25,7 +25,6 @@ #include #include -#include #include #include --- a/fs/binfmt_elf_fdpic.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/fs/binfmt_elf_fdpic.c @@ -38,7 +38,6 @@ #include #include -#include typedef char *elf_caddr_t; --- a/include/asm-generic/tlb.h~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/include/asm-generic/tlb.h @@ -14,7 +14,6 @@ #include #include #include -#include #include #include --- a/mm/hugetlb.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/mm/hugetlb.c @@ -31,6 +31,7 @@ #include #include +#include #include #include --- a/mm/sparse.c~mm-remove-unneeded-includes-of-asm-pgalloch +++ a/mm/sparse.c @@ -16,7 +16,6 @@ #include "internal.h" #include -#include /* * Permanent SPARSEMEM data: From patchwork Fri Aug 7 06:22:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704865 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1FC36138C for ; Fri, 7 Aug 2020 06:22:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E6C57221E5 for ; Fri, 7 Aug 2020 06:22:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="YwQOeEbY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E6C57221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 902438D006A; Fri, 7 Aug 2020 02:22:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8B0D18D0026; Fri, 7 Aug 2020 02:22:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F08D8D006A; Fri, 7 Aug 2020 02:22:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0141.hostedemail.com [216.40.44.141]) by kanga.kvack.org (Postfix) with ESMTP id 694F98D0026 for ; Fri, 7 Aug 2020 02:22:35 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 2AC3D82499A8 for ; Fri, 7 Aug 2020 06:22:35 +0000 (UTC) X-FDA: 77122778670.15.boot66_2717a4926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id EFE221814B0C1 for ; Fri, 7 Aug 2020 06:22:34 +0000 (UTC) X-Spam-Summary: 1,0,0,e570c38015a81edf,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:1978:1981:2194:2196:2199:2200:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6261:6653:6737:7514:7576:8599:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13229:13846:14181:14721:14849:21080:21451:21627:21795:21939:21990:30051:30054:30055:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yfki7wowbbmyu8mmdm65riw4tu9ypq595y95ist9dcxyhpe7red6kwjyucwpq.ep49j3t3dzrxdr7eyyyhoadh45nmjkuxiyapot8rz657g7qcxtf8e6qsiss4rd8.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,Domain Cache:0, X-HE-Tag: boot66_2717a4926fbe X-Filterd-Recvd-Size: 4325 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:34 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id DA73A22C9F; Fri, 7 Aug 2020 06:22:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781353; bh=1Ig4RehoHuoccqQrfVl0xTQN5UF60fiYe1h00zAkXW4=; h=Date:From:To:Subject:In-Reply-To:From; b=YwQOeEbYKyO+K5v+iS6T5RX8fSr6Ey17hG91bs4AzvbtNSD/ooy8n7PsFSCRkJV2S AKeJ0MBMOCNFT5JwEgn48XrkLqkciaY498gjDx9Dk112vs67WkgCODic7kRSE+i6kl YkTdd4+QaYNuD3H6N2hihiN4CXn1fFhJ1tllSLFQ= Date: Thu, 06 Aug 2020 23:22:32 -0700 From: Andrew Morton To: abdhalee@linux.vnet.ibm.com, akpm@linux-foundation.org, arnd@arndb.de, christophe.leroy@csgroup.eu, geert@linux-m68k.org, jcmvbkbc@gmail.com, joro@8bytes.org, jroedel@suse.de, linux-mm@kvack.org, luto@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, peterz@infradead.org, rostedt@goodmis.org, rppt@linux.ibm.com, sathnaga@linux.vnet.ibm.com, sfr@canb.auug.org.au, shorne@gmail.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 097/163] opeinrisc: switch to generic version of pte allocation Message-ID: <20200807062232.OczdEdm4i%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: EFE221814B0C1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Subject: opeinrisc: switch to generic version of pte allocation Replace pte_alloc_one(), pte_free() and pte_free_kernel() with the generic implementation. The only actual functional change is the addition of __GFP_ACCOUT for the allocation of the user page tables. The pte_alloc_one_kernel() is kept back because its implementation on openrisc is different than the generic one. Link: http://lkml.kernel.org/r/20200627143453.31835-3-rppt@kernel.org Signed-off-by: Mike Rapoport Acked-by: Stafford Horne Reviewed-by: Pekka Enberg Cc: Abdul Haleem Cc: Andy Lutomirski Cc: Arnd Bergmann Cc: Christophe Leroy Cc: Joerg Roedel Cc: Joerg Roedel Cc: Max Filippov Cc: Peter Zijlstra (Intel) Cc: Satheesh Rajendran Cc: Stephen Rothwell Cc: Steven Rostedt Cc: Geert Uytterhoeven Cc: Matthew Wilcox Signed-off-by: Andrew Morton --- arch/openrisc/include/asm/pgalloc.h | 33 ++------------------------ 1 file changed, 3 insertions(+), 30 deletions(-) --- a/arch/openrisc/include/asm/pgalloc.h~opeinrisc-switch-to-generic-version-of-pte-allocation +++ a/arch/openrisc/include/asm/pgalloc.h @@ -20,6 +20,9 @@ #include #include +#define __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL +#include + extern int mem_init_done; #define pmd_populate_kernel(mm, pmd, pte) \ @@ -61,38 +64,8 @@ extern inline pgd_t *pgd_alloc(struct mm } #endif -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_page((unsigned long)pgd); -} - extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); -static inline struct page *pte_alloc_one(struct mm_struct *mm) -{ - struct page *pte; - pte = alloc_pages(GFP_KERNEL, 0); - if (!pte) - return NULL; - clear_page(page_address(pte)); - if (!pgtable_pte_page_ctor(pte)) { - __free_page(pte); - return NULL; - } - return pte; -} - -static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) -{ - free_page((unsigned long)pte); -} - -static inline void pte_free(struct mm_struct *mm, struct page *pte) -{ - pgtable_pte_page_dtor(pte); - __free_page(pte); -} - #define __pte_free_tlb(tlb, pte, addr) \ do { \ pgtable_pte_page_dtor(pte); \ From patchwork Fri Aug 7 06:22:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704867 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B8F6138C for ; Fri, 7 Aug 2020 06:22:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C7EB122D08 for ; Fri, 7 Aug 2020 06:22:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="D5skGTmX" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7EB122D08 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 448078D006B; Fri, 7 Aug 2020 02:22:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3F61B8D0026; Fri, 7 Aug 2020 02:22:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30C358D006B; Fri, 7 Aug 2020 02:22:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0092.hostedemail.com [216.40.44.92]) by kanga.kvack.org (Postfix) with ESMTP id 1CA2A8D0026 for ; Fri, 7 Aug 2020 02:22:39 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id DEB8C180AD801 for ; Fri, 7 Aug 2020 06:22:38 +0000 (UTC) X-FDA: 77122778796.11.pump25_3e1672226fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id AE7B1180F8B81 for ; Fri, 7 Aug 2020 06:22:38 +0000 (UTC) X-Spam-Summary: 1,0,0,ccbba9826724306c,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1543:1711:1730:1747:1777:1792:1978:1981:2194:2196:2199:2200:2393:2525:2559:2564:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:5007:6119:6261:6653:6737:7514:7576:7903:8599:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13229:13846:14096:14181:14721:14849:21080:21451:21627:21939:21990:30054:30055,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yr6eg5is75b5u7hzud71eef68iuycdc4sg7dymuxo11dmbzctaikrdihotzpx.y5wikgs11t5n77nsqz1hwhupakwwqgi3ow58mogtrc4omdhod61xeqxqxi8i166.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:non e,Domain X-HE-Tag: pump25_3e1672226fbe X-Filterd-Recvd-Size: 5246 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:38 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A851222CB3; Fri, 7 Aug 2020 06:22:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781357; bh=KLBYiFochOq64XZ35lZYcd4x31nq5E1Ks780kMS4its=; h=Date:From:To:Subject:In-Reply-To:From; b=D5skGTmXgXgBqaRiEZ7/K0mfzm3CXzB11dA1UlICatmoOuJvDlupL8hiKEJ3Rk6Fa BC2NUjt5SHu8zCtwOg1QQiHsR7JXuPc62kLLIapaWTLNgvardThd3IZsRhLxlk60U7 7wyjN9HGPGeOsWedAutqo4iqfVtm9OyAN1XJjE9I= Date: Thu, 06 Aug 2020 23:22:36 -0700 From: Andrew Morton To: abdhalee@linux.vnet.ibm.com, akpm@linux-foundation.org, arnd@arndb.de, christophe.leroy@csgroup.eu, geert@linux-m68k.org, jcmvbkbc@gmail.com, joro@8bytes.org, jroedel@suse.de, linux-mm@kvack.org, luto@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, peterz@infradead.org, rostedt@goodmis.org, rppt@linux.ibm.com, sathnaga@linux.vnet.ibm.com, sfr@canb.auug.org.au, shorne@gmail.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 098/163] xtensa: switch to generic version of pte allocation Message-ID: <20200807062236.8Y5USD56Y%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: AE7B1180F8B81 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Subject: xtensa: switch to generic version of pte allocation xtensa clears PTEs during allocation of the page tables and pte_clear() sets the PTE to a non-zero value. Splitting ptes_clear() helper out of pte_alloc_one() and pte_alloc_one_kernel() allows reuse of base generic allocation methods (__pte_alloc_one() and __pte_alloc_one_kernel()) and the common GFP mask for page table allocations. The pte_free() and pte_free_kernel() implementations on xtensa are identical to the generic ones and can be dropped. [jcmvbkbc@gmail.com: xtensa: fix closing endif comment] Link: http://lkml.kernel.org/r/20200721024751.1257-1-jcmvbkbc@gmail.com Link: http://lkml.kernel.org/r/20200627143453.31835-4-rppt@kernel.org Signed-off-by: Mike Rapoport Signed-off-by: Max Filippov Reviewed-by: Pekka Enberg Cc: Abdul Haleem Cc: Andy Lutomirski Cc: Arnd Bergmann Cc: Christophe Leroy Cc: Joerg Roedel Cc: Joerg Roedel Cc: Max Filippov Cc: Peter Zijlstra (Intel) Cc: Satheesh Rajendran Cc: Stafford Horne Cc: Stephen Rothwell Cc: Steven Rostedt Cc: Geert Uytterhoeven Cc: Matthew Wilcox Signed-off-by: Andrew Morton --- arch/xtensa/include/asm/pgalloc.h | 41 ++++++++++++---------------- 1 file changed, 19 insertions(+), 22 deletions(-) --- a/arch/xtensa/include/asm/pgalloc.h~xtensa-switch-to-generic-version-of-pte-allocation +++ a/arch/xtensa/include/asm/pgalloc.h @@ -8,9 +8,14 @@ #ifndef _XTENSA_PGALLOC_H #define _XTENSA_PGALLOC_H +#ifdef CONFIG_MMU #include #include +#define __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL +#define __HAVE_ARCH_PTE_ALLOC_ONE +#include + /* * Allocating and freeing a pmd is trivial: the 1-entry pmd is * inside the pgd, so has no extra memory associated with it. @@ -33,45 +38,37 @@ static inline void pgd_free(struct mm_st free_page((unsigned long)pgd); } +static inline void ptes_clear(pte_t *ptep) +{ + int i; + + for (i = 0; i < PTRS_PER_PTE; i++) + pte_clear(NULL, 0, ptep + i); +} + static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *ptep; - int i; - ptep = (pte_t *)__get_free_page(GFP_KERNEL); + ptep = (pte_t *)__pte_alloc_one_kernel(mm); if (!ptep) return NULL; - for (i = 0; i < 1024; i++) - pte_clear(NULL, 0, ptep + i); + ptes_clear(ptep); return ptep; } static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { - pte_t *pte; struct page *page; - pte = pte_alloc_one_kernel(mm); - if (!pte) - return NULL; - page = virt_to_page(pte); - if (!pgtable_pte_page_ctor(page)) { - __free_page(page); + page = __pte_alloc_one(mm, GFP_PGTABLE_USER); + if (!page) return NULL; - } + ptes_clear(page_address(page)); return page; } -static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) -{ - free_page((unsigned long)pte); -} - -static inline void pte_free(struct mm_struct *mm, pgtable_t pte) -{ - pgtable_pte_page_dtor(pte); - __free_page(pte); -} #define pmd_pgtable(pmd) pmd_page(pmd) +#endif /* CONFIG_MMU */ #endif /* _XTENSA_PGALLOC_H */ From patchwork Fri Aug 7 06:22:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704869 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5D69814B7 for ; Fri, 7 Aug 2020 06:22:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1058422D00 for ; Fri, 7 Aug 2020 06:22:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="wN2dVOLU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1058422D00 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 99D538D006C; Fri, 7 Aug 2020 02:22:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 94D7E8D0026; Fri, 7 Aug 2020 02:22:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 863828D006C; Fri, 7 Aug 2020 02:22:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0032.hostedemail.com [216.40.44.32]) by kanga.kvack.org (Postfix) with ESMTP id 7075E8D0026 for ; Fri, 7 Aug 2020 02:22:43 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 32E1D21F0 for ; Fri, 7 Aug 2020 06:22:43 +0000 (UTC) X-FDA: 77122779006.10.work04_450425f26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id E28B016A0DE for ; Fri, 7 Aug 2020 06:22:42 +0000 (UTC) X-Spam-Summary: 1,0,0,0071404659bf03f1,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:41:69:355:379:800:960:965:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1978:1981:2194:2196:2198:2199:2200:2201:2393:2525:2559:2563:2638:2682:2685:2859:2890:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4042:4250:4321:4385:4390:4395:4605:5007:6117:6119:6261:6653:6737:7514:7576:7875:7903:8599:8603:8660:9025:9036:9545:9592:10004:10394:10913:11026:11232:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12783:12986:13148:13161:13229:13230:13255:13846:14096:21080:21433:21450:21451:21611:21627:21789:21795:21939:21990:30001:30003:30051:30054:30055:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yr6omezcoxhk7ddw5k3gktcki8aoc3mru58tye3wmb94kpwdn6qx1kksaumyx.1pfqia4s5d 3arz67bn X-HE-Tag: work04_450425f26fbe X-Filterd-Recvd-Size: 14671 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:42 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 875B522CAF; Fri, 7 Aug 2020 06:22:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781361; bh=Ky9K22cILxeJXz+6/X/d4wO3hQ0Isera3xxd34Y9DWw=; h=Date:From:To:Subject:In-Reply-To:From; b=wN2dVOLUMoOEH39s6cWuu43s2OlXY9B4Y6buW9rQpvNTcrB/FJ1iieCUzpp0HxyNy HBb7Czq33ZGkNldDMr+n1UTjfFI/SQ7WvgTKvqRWPGekIzieK1YBpDSeNUd9wRA1RX HwX45ibGo3qEuV7I+i+iF6Of10RIhHmwo7xaBs/Y= Date: Thu, 06 Aug 2020 23:22:39 -0700 From: Andrew Morton To: abdhalee@linux.vnet.ibm.com, akpm@linux-foundation.org, arnd@arndb.de, christophe.leroy@csgroup.eu, geert@linux-m68k.org, jcmvbkbc@gmail.com, joro@8bytes.org, jroedel@suse.de, linux-mm@kvack.org, luto@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, peterz@infradead.org, rostedt@goodmis.org, rppt@linux.ibm.com, sathnaga@linux.vnet.ibm.com, sfr@canb.auug.org.au, shorne@gmail.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 099/163] asm-generic: pgalloc: provide generic pmd_alloc_one() and pmd_free_one() Message-ID: <20200807062239.8Nlm6Xcrw%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: E28B016A0DE X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Subject: asm-generic: pgalloc: provide generic pmd_alloc_one() and pmd_free_one() For most architectures that support >2 levels of page tables, pmd_alloc_one() is a wrapper for __get_free_pages(), sometimes with __GFP_ZERO and sometimes followed by memset(0) instead. More elaborate versions on arm64 and x86 account memory for the user page tables and call to pgtable_pmd_page_ctor() as the part of PMD page initialization. Move the arm64 version to include/asm-generic/pgalloc.h and use the generic version on several architectures. The pgtable_pmd_page_ctor() is a NOP when ARCH_ENABLE_SPLIT_PMD_PTLOCK is not enabled, so there is no functional change for most architectures except of the addition of __GFP_ACCOUNT for allocation of user page tables. The pmd_free() is a wrapper for free_page() in all the cases, so no functional change here. Link: http://lkml.kernel.org/r/20200627143453.31835-5-rppt@kernel.org Signed-off-by: Mike Rapoport Reviewed-by: Pekka Enberg Cc: Matthew Wilcox Cc: Abdul Haleem Cc: Andy Lutomirski Cc: Arnd Bergmann Cc: Christophe Leroy Cc: Joerg Roedel Cc: Joerg Roedel Cc: Max Filippov Cc: Peter Zijlstra (Intel) Cc: Satheesh Rajendran Cc: Stafford Horne Cc: Stephen Rothwell Cc: Steven Rostedt Cc: Geert Uytterhoeven Signed-off-by: Andrew Morton --- arch/alpha/include/asm/pgalloc.h | 15 -------- arch/arm/include/asm/pgalloc.h | 11 ------ arch/arm64/include/asm/pgalloc.h | 27 --------------- arch/ia64/include/asm/pgalloc.h | 10 ----- arch/mips/include/asm/pgalloc.h | 8 +--- arch/parisc/include/asm/pgalloc.h | 11 +----- arch/riscv/include/asm/pgalloc.h | 13 ------- arch/sh/include/asm/pgalloc.h | 3 + arch/um/include/asm/pgalloc.h | 8 ---- arch/um/include/asm/pgtable-3level.h | 3 - arch/um/kernel/mem.c | 12 ------ arch/x86/include/asm/pgalloc.h | 26 --------------- include/asm-generic/pgalloc.h | 43 +++++++++++++++++++++++++ 13 files changed, 55 insertions(+), 135 deletions(-) --- a/arch/alpha/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/alpha/include/asm/pgalloc.h @@ -5,7 +5,7 @@ #include #include -#include /* for pte_{alloc,free}_one */ +#include /* * Allocate and free page tables. The xxx_kernel() versions are @@ -40,17 +40,4 @@ pgd_free(struct mm_struct *mm, pgd_t *pg free_page((unsigned long)pgd); } -static inline pmd_t * -pmd_alloc_one(struct mm_struct *mm, unsigned long address) -{ - pmd_t *ret = (pmd_t *)__get_free_page(GFP_PGTABLE_USER); - return ret; -} - -static inline void -pmd_free(struct mm_struct *mm, pmd_t *pmd) -{ - free_page((unsigned long)pmd); -} - #endif /* _ALPHA_PGALLOC_H */ --- a/arch/arm64/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/arm64/include/asm/pgalloc.h @@ -13,37 +13,12 @@ #include #include -#include /* for pte_{alloc,free}_one */ +#include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) #if CONFIG_PGTABLE_LEVELS > 2 -static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_PGTABLE_USER; - struct page *page; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - - page = alloc_page(gfp); - if (!page) - return NULL; - if (!pgtable_pmd_page_ctor(page)) { - __free_page(page); - return NULL; - } - return page_address(page); -} - -static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) -{ - BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); - pgtable_pmd_page_dtor(virt_to_page(pmdp)); - free_page((unsigned long)pmdp); -} - static inline void __pud_populate(pud_t *pudp, phys_addr_t pmdp, pudval_t prot) { set_pud(pudp, __pud(__phys_to_pud_val(pmdp) | prot)); --- a/arch/arm/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/arm/include/asm/pgalloc.h @@ -22,17 +22,6 @@ #ifdef CONFIG_ARM_LPAE -static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - return (pmd_t *)get_zeroed_page(GFP_KERNEL); -} - -static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) -{ - BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - free_page((unsigned long)pmd); -} - static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd) { set_pud(pud, __pud(__pa(pmd) | PMD_TYPE_TABLE)); --- a/arch/ia64/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/ia64/include/asm/pgalloc.h @@ -59,16 +59,6 @@ pud_populate(struct mm_struct *mm, pud_t pud_val(*pud_entry) = __pa(pmd); } -static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - return (pmd_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); -} - -static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) -{ - free_page((unsigned long)pmd); -} - #define __pmd_free_tlb(tlb, pmd, address) pmd_free((tlb)->mm, pmd) static inline void --- a/arch/mips/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/mips/include/asm/pgalloc.h @@ -13,7 +13,8 @@ #include #include -#include /* for pte_{alloc,free}_one */ +#define __HAVE_ARCH_PMD_ALLOC_ONE +#include static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) @@ -70,11 +71,6 @@ static inline pmd_t *pmd_alloc_one(struc return pmd; } -static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) -{ - free_pages((unsigned long)pmd, PMD_ORDER); -} - #define __pmd_free_tlb(tlb, x, addr) pmd_free((tlb)->mm, x) #endif --- a/arch/parisc/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/parisc/include/asm/pgalloc.h @@ -10,7 +10,8 @@ #include -#include /* for pte_{alloc,free}_one */ +#define __HAVE_ARCH_PMD_FREE +#include /* Allocate the top level pgd (page directory) * @@ -65,14 +66,6 @@ static inline void pud_populate(struct m (__u32)(__pa((unsigned long)pmd) >> PxD_VALUE_SHIFT))); } -static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) -{ - pmd_t *pmd = (pmd_t *)__get_free_pages(GFP_KERNEL, PMD_ORDER); - if (pmd) - memset(pmd, 0, PAGE_SIZE< #ifdef CONFIG_MMU -#include /* for pte_{alloc,free}_one */ +#include static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) @@ -62,17 +62,6 @@ static inline void pgd_free(struct mm_st #ifndef __PAGETABLE_PMD_FOLDED -static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - return (pmd_t *)__get_free_page( - GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_ZERO); -} - -static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) -{ - free_page((unsigned long)pmd); -} - #define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd) #endif /* __PAGETABLE_PMD_FOLDED */ --- a/arch/sh/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/sh/include/asm/pgalloc.h @@ -3,6 +3,9 @@ #define __ASM_SH_PGALLOC_H #include + +#define __HAVE_ARCH_PMD_ALLOC_ONE +#define __HAVE_ARCH_PMD_FREE #include extern pgd_t *pgd_alloc(struct mm_struct *); --- a/arch/um/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/um/include/asm/pgalloc.h @@ -10,7 +10,7 @@ #include -#include /* for pte_{alloc,free}_one */ +#include #define pmd_populate_kernel(mm, pmd, pte) \ set_pmd(pmd, __pmd(_PAGE_TABLE + (unsigned long) __pa(pte))) @@ -34,12 +34,6 @@ do { \ } while (0) #ifdef CONFIG_3_LEVEL_PGTABLES - -static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) -{ - free_page((unsigned long)pmd); -} - #define __pmd_free_tlb(tlb,x, address) tlb_remove_page((tlb),virt_to_page(x)) #endif --- a/arch/um/include/asm/pgtable-3level.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/um/include/asm/pgtable-3level.h @@ -78,9 +78,6 @@ static inline void pgd_mkuptodate(pgd_t #define set_pmd(pmdptr, pmdval) (*(pmdptr) = (pmdval)) #endif -struct mm_struct; -extern pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address); - static inline void pud_clear (pud_t *pud) { set_pud(pud, __pud(_PAGE_NEWPAGE)); --- a/arch/um/kernel/mem.c~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/um/kernel/mem.c @@ -201,18 +201,6 @@ void pgd_free(struct mm_struct *mm, pgd_ free_page((unsigned long) pgd); } -#ifdef CONFIG_3_LEVEL_PGTABLES -pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) -{ - pmd_t *pmd = (pmd_t *) __get_free_page(GFP_KERNEL); - - if (pmd) - memset(pmd, 0, PAGE_SIZE); - - return pmd; -} -#endif - void *uml_kmalloc(int size, int flags) { return kmalloc(size, flags); --- a/arch/x86/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/arch/x86/include/asm/pgalloc.h @@ -7,7 +7,7 @@ #include #define __HAVE_ARCH_PTE_ALLOC_ONE -#include /* for pte_{alloc,free}_one */ +#include static inline int __paravirt_pgd_alloc(struct mm_struct *mm) { return 0; } @@ -86,30 +86,6 @@ static inline void pmd_populate(struct m #define pmd_pgtable(pmd) pmd_page(pmd) #if CONFIG_PGTABLE_LEVELS > 2 -static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - struct page *page; - gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO; - - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - page = alloc_pages(gfp, 0); - if (!page) - return NULL; - if (!pgtable_pmd_page_ctor(page)) { - __free_pages(page, 0); - return NULL; - } - return (pmd_t *)page_address(page); -} - -static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) -{ - BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); - pgtable_pmd_page_dtor(virt_to_page(pmd)); - free_page((unsigned long)pmd); -} - extern void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd); static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, --- a/include/asm-generic/pgalloc.h~asm-generic-pgalloc-provide-generic-pmd_alloc_one-and-pmd_free_one +++ a/include/asm-generic/pgalloc.h @@ -102,6 +102,49 @@ static inline void pte_free(struct mm_st __free_page(pte_page); } + +#if CONFIG_PGTABLE_LEVELS > 2 + +#ifndef __HAVE_ARCH_PMD_ALLOC_ONE +/** + * pmd_alloc_one - allocate a page for PMD-level page table + * @mm: the mm_struct of the current context + * + * Allocates a page and runs the pgtable_pmd_page_ctor(). + * Allocations use %GFP_PGTABLE_USER in user context and + * %GFP_PGTABLE_KERNEL in kernel context. + * + * Return: pointer to the allocated memory or %NULL on error + */ +static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) +{ + struct page *page; + gfp_t gfp = GFP_PGTABLE_USER; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + page = alloc_pages(gfp, 0); + if (!page) + return NULL; + if (!pgtable_pmd_page_ctor(page)) { + __free_pages(page, 0); + return NULL; + } + return (pmd_t *)page_address(page); +} +#endif + +#ifndef __HAVE_ARCH_PMD_FREE +static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) +{ + BUG_ON((unsigned long)pmd & (PAGE_SIZE-1)); + pgtable_pmd_page_dtor(virt_to_page(pmd)); + free_page((unsigned long)pmd); +} +#endif + +#endif /* CONFIG_PGTABLE_LEVELS > 2 */ + #endif /* CONFIG_MMU */ #endif /* __ASM_GENERIC_PGALLOC_H */ From patchwork Fri Aug 7 06:22:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704871 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1701B138C for ; Fri, 7 Aug 2020 06:22:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D5613221E5 for ; Fri, 7 Aug 2020 06:22:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ni5aapW+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D5613221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2E5BB8D006D; Fri, 7 Aug 2020 02:22:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 295B58D0026; Fri, 7 Aug 2020 02:22:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 185C58D006D; Fri, 7 Aug 2020 02:22:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0172.hostedemail.com [216.40.44.172]) by kanga.kvack.org (Postfix) with ESMTP id 037858D0026 for ; Fri, 7 Aug 2020 02:22:46 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B8DB9181AEF10 for ; Fri, 7 Aug 2020 06:22:46 +0000 (UTC) X-FDA: 77122779132.05.skate28_361105826fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 929FA1802912B for ; Fri, 7 Aug 2020 06:22:46 +0000 (UTC) X-Spam-Summary: 1,0,0,95855ca136cf295d,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:965:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1606:1730:1747:1777:1792:1978:1981:2194:2196:2198:2199:2200:2201:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4118:4321:4385:4390:4395:5007:6119:6261:6653:6737:7514:7576:7875:7903:8599:9025:9545:9592:10004:10913:11026:11232:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12783:12986:13255:13846:14096:21080:21433:21450:21451:21627:21795:21939:21990:30003:30051:30054:30055,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yrbu9efedimaysnmjw8nmdrh4jbop3g13569ctm7zkfy8ieiu993mjh99mktz.7y896ftu33einhierzska6sr6swffw1837f7f5561a7eeorguiqoziupedbxdck.o-lbl8.mailshell.net-223.238.255.100,Ca cheIP:no X-HE-Tag: skate28_361105826fbe X-Filterd-Recvd-Size: 7132 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:45 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8027622D03; Fri, 7 Aug 2020 06:22:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781365; bh=DTb5jAxRUZ6+ae1MDWomt3btWeDBYbFnGbdiwjJqvq8=; h=Date:From:To:Subject:In-Reply-To:From; b=Ni5aapW+RwHsLGMa6EJzebQKaW1ndO20GKzQ5g0tsftX06xnDc4I3CXTFQ/Stawrn JwyTzMG37Ind6kmzT0R5EoxscqPpdQlzhNU8I6T1I5m8XtGxwNFbiGiHVZQbDO/wpq bQXPYDOBQ4JGT5sDBrslvFPgPzXiUjlWm8kNJFxc= Date: Thu, 06 Aug 2020 23:22:44 -0700 From: Andrew Morton To: abdhalee@linux.vnet.ibm.com, akpm@linux-foundation.org, arnd@arndb.de, christophe.leroy@csgroup.eu, geert@linux-m68k.org, jcmvbkbc@gmail.com, joro@8bytes.org, jroedel@suse.de, linux-mm@kvack.org, luto@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, peterz@infradead.org, rostedt@goodmis.org, rppt@linux.ibm.com, sathnaga@linux.vnet.ibm.com, sfr@canb.auug.org.au, shorne@gmail.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 100/163] asm-generic: pgalloc: provide generic pud_alloc_one() and pud_free_one() Message-ID: <20200807062244.lcljBq8Tg%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 929FA1802912B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Subject: asm-generic: pgalloc: provide generic pud_alloc_one() and pud_free_one() Several architectures define pud_alloc_one() as a wrapper for __get_free_page() and pud_free() as a wrapper for free_page(). Provide a generic implementation in asm-generic/pgalloc.h and use it where appropriate. Link: http://lkml.kernel.org/r/20200627143453.31835-6-rppt@kernel.org Signed-off-by: Mike Rapoport Reviewed-by: Pekka Enberg Cc: Abdul Haleem Cc: Andy Lutomirski Cc: Arnd Bergmann Cc: Christophe Leroy Cc: Joerg Roedel Cc: Joerg Roedel Cc: Max Filippov Cc: Peter Zijlstra (Intel) Cc: Satheesh Rajendran Cc: Stafford Horne Cc: Stephen Rothwell Cc: Steven Rostedt Cc: Geert Uytterhoeven Cc: Matthew Wilcox Signed-off-by: Andrew Morton --- arch/arm64/include/asm/pgalloc.h | 11 ---------- arch/ia64/include/asm/pgalloc.h | 9 -------- arch/mips/include/asm/pgalloc.h | 6 ----- arch/x86/include/asm/pgalloc.h | 15 -------------- include/asm-generic/pgalloc.h | 30 +++++++++++++++++++++++++++++ 5 files changed, 31 insertions(+), 40 deletions(-) --- a/arch/arm64/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pud_alloc_one-and-pud_free_one +++ a/arch/arm64/include/asm/pgalloc.h @@ -37,17 +37,6 @@ static inline void __pud_populate(pud_t #if CONFIG_PGTABLE_LEVELS > 3 -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - return (pud_t *)__get_free_page(GFP_PGTABLE_USER); -} - -static inline void pud_free(struct mm_struct *mm, pud_t *pudp) -{ - BUG_ON((unsigned long)pudp & (PAGE_SIZE-1)); - free_page((unsigned long)pudp); -} - static inline void __p4d_populate(p4d_t *p4dp, phys_addr_t pudp, p4dval_t prot) { set_p4d(p4dp, __p4d(__phys_to_p4d_val(pudp) | prot)); --- a/arch/ia64/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pud_alloc_one-and-pud_free_one +++ a/arch/ia64/include/asm/pgalloc.h @@ -41,15 +41,6 @@ p4d_populate(struct mm_struct *mm, p4d_t p4d_val(*p4d_entry) = __pa(pud); } -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - return (pud_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); -} - -static inline void pud_free(struct mm_struct *mm, pud_t *pud) -{ - free_page((unsigned long)pud); -} #define __pud_free_tlb(tlb, pud, address) pud_free((tlb)->mm, pud) #endif /* CONFIG_PGTABLE_LEVELS == 4 */ --- a/arch/mips/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pud_alloc_one-and-pud_free_one +++ a/arch/mips/include/asm/pgalloc.h @@ -14,6 +14,7 @@ #include #define __HAVE_ARCH_PMD_ALLOC_ONE +#define __HAVE_ARCH_PUD_ALLOC_ONE #include static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, @@ -87,11 +88,6 @@ static inline pud_t *pud_alloc_one(struc return pud; } -static inline void pud_free(struct mm_struct *mm, pud_t *pud) -{ - free_pages((unsigned long)pud, PUD_ORDER); -} - static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4d, pud_t *pud) { set_p4d(p4d, __p4d((unsigned long)pud)); --- a/arch/x86/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pud_alloc_one-and-pud_free_one +++ a/arch/x86/include/asm/pgalloc.h @@ -123,21 +123,6 @@ static inline void p4d_populate_safe(str set_p4d_safe(p4d, __p4d(_PAGE_TABLE | __pa(pud))); } -static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) -{ - gfp_t gfp = GFP_KERNEL_ACCOUNT; - - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - return (pud_t *)get_zeroed_page(gfp); -} - -static inline void pud_free(struct mm_struct *mm, pud_t *pud) -{ - BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - free_page((unsigned long)pud); -} - extern void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud); static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, --- a/include/asm-generic/pgalloc.h~asm-generic-pgalloc-provide-generic-pud_alloc_one-and-pud_free_one +++ a/include/asm-generic/pgalloc.h @@ -145,6 +145,36 @@ static inline void pmd_free(struct mm_st #endif /* CONFIG_PGTABLE_LEVELS > 2 */ +#if CONFIG_PGTABLE_LEVELS > 3 + +#ifndef __HAVE_ARCH_PUD_FREE +/** + * pud_alloc_one - allocate a page for PUD-level page table + * @mm: the mm_struct of the current context + * + * Allocates a page using %GFP_PGTABLE_USER for user context and + * %GFP_PGTABLE_KERNEL for kernel context. + * + * Return: pointer to the allocated memory or %NULL on error + */ +static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (pud_t *)get_zeroed_page(gfp); +} +#endif + +static inline void pud_free(struct mm_struct *mm, pud_t *pud) +{ + BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); + free_page((unsigned long)pud); +} + +#endif /* CONFIG_PGTABLE_LEVELS > 3 */ + #endif /* CONFIG_MMU */ #endif /* __ASM_GENERIC_PGALLOC_H */ From patchwork Fri Aug 7 06:22:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704873 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1D174138C for ; Fri, 7 Aug 2020 06:22:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CF4FF22D2C for ; Fri, 7 Aug 2020 06:22:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="GaEPC3TU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF4FF22D2C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 299E38D006E; Fri, 7 Aug 2020 02:22:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 248098D0026; Fri, 7 Aug 2020 02:22:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15F3C8D006E; Fri, 7 Aug 2020 02:22:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0198.hostedemail.com [216.40.44.198]) by kanga.kvack.org (Postfix) with ESMTP id 014CB8D0026 for ; Fri, 7 Aug 2020 02:22:50 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C725E3628 for ; Fri, 7 Aug 2020 06:22:50 +0000 (UTC) X-FDA: 77122779300.25.milk62_090bad626fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 839061804E3A0 for ; Fri, 7 Aug 2020 06:22:50 +0000 (UTC) X-Spam-Summary: 1,0,0,7ca5cfef7aa44580,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:41:355:379:800:960:965:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1978:1981:2194:2196:2198:2199:2200:2201:2393:2525:2538:2559:2563:2636:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3868:3870:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:4390:4395:4605:5007:6119:6261:6653:6737:7514:7576:7903:8599:8603:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13846:21080:21451:21524:21627:21939:21990:30003:30054:30055:30067,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yfaqye1dnt6poj47iya9mioqfnpocobmnshn7ea4xdjba55u78tbra9e9br4o.yiquxggssmahymrf5zm1na3x89hzfqisktnc4ioq51498rg7woqgbmsd9rrxfwo.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCa che:0,MS X-HE-Tag: milk62_090bad626fbe X-Filterd-Recvd-Size: 12350 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:49 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5CB7C22D00; Fri, 7 Aug 2020 06:22:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781369; bh=5fd7DQcn4eXmY2ykRrT4G89HhfeTLw9r6xnI1VlvRCU=; h=Date:From:To:Subject:In-Reply-To:From; b=GaEPC3TU4KRRzapq6i7XJ3Y01cPUNQVuQvRpMlC64RkiZT3F0+iE85Udk9QrD9Gcr ywYHdBkcLBRePsck12NuBEvM0sS4sxpVwvF/uDIXfZNPYQeeu9spz9mms4C3ZfLULs 3bMz4dyBns2Wp8GoMnpr9b1IBUr1wxcymxrwE07w= Date: Thu, 06 Aug 2020 23:22:47 -0700 From: Andrew Morton To: abdhalee@linux.vnet.ibm.com, akpm@linux-foundation.org, arnd@arndb.de, christophe.leroy@csgroup.eu, geert@linux-m68k.org, jcmvbkbc@gmail.com, joro@8bytes.org, jroedel@suse.de, linux-mm@kvack.org, luto@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, peterz@infradead.org, rostedt@goodmis.org, rppt@linux.ibm.com, sathnaga@linux.vnet.ibm.com, sfr@canb.auug.org.au, shorne@gmail.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 101/163] asm-generic: pgalloc: provide generic pgd_free() Message-ID: <20200807062247.fTrAYwgx5%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 839061804E3A0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Subject: asm-generic: pgalloc: provide generic pgd_free() Most architectures define pgd_free() as a wrapper for free_page(). Provide a generic version in asm-generic/pgalloc.h and enable its use for most architectures. Link: http://lkml.kernel.org/r/20200627143453.31835-7-rppt@kernel.org Signed-off-by: Mike Rapoport Acked-by: Geert Uytterhoeven [m68k] Reviewed-by: Pekka Enberg Cc: Abdul Haleem Cc: Andy Lutomirski Cc: Arnd Bergmann Cc: Christophe Leroy Cc: Joerg Roedel Cc: Joerg Roedel Cc: Max Filippov Cc: Peter Zijlstra (Intel) Cc: Satheesh Rajendran Cc: Stafford Horne Cc: Stephen Rothwell Cc: Steven Rostedt Cc: Matthew Wilcox Signed-off-by: Andrew Morton --- arch/alpha/include/asm/pgalloc.h | 6 ------ arch/arm/include/asm/pgalloc.h | 1 + arch/arm64/include/asm/pgalloc.h | 1 + arch/csky/include/asm/pgalloc.h | 7 +------ arch/hexagon/include/asm/pgalloc.h | 7 +------ arch/ia64/include/asm/pgalloc.h | 5 ----- arch/m68k/include/asm/sun3_pgalloc.h | 7 +------ arch/microblaze/include/asm/pgalloc.h | 6 ------ arch/mips/include/asm/pgalloc.h | 5 ----- arch/nds32/mm/mm-nds32.c | 2 ++ arch/nios2/include/asm/pgalloc.h | 7 +------ arch/parisc/include/asm/pgalloc.h | 1 + arch/riscv/include/asm/pgalloc.h | 5 ----- arch/sh/include/asm/pgalloc.h | 1 + arch/um/include/asm/pgalloc.h | 1 - arch/um/kernel/mem.c | 5 ----- arch/x86/include/asm/pgalloc.h | 1 + arch/xtensa/include/asm/pgalloc.h | 5 ----- include/asm-generic/pgalloc.h | 7 +++++++ 19 files changed, 18 insertions(+), 62 deletions(-) --- a/arch/alpha/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/alpha/include/asm/pgalloc.h @@ -34,10 +34,4 @@ pud_populate(struct mm_struct *mm, pud_t extern pgd_t *pgd_alloc(struct mm_struct *mm); -static inline void -pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_page((unsigned long)pgd); -} - #endif /* _ALPHA_PGALLOC_H */ --- a/arch/arm64/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/arm64/include/asm/pgalloc.h @@ -13,6 +13,7 @@ #include #include +#define __HAVE_ARCH_PGD_FREE #include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) --- a/arch/arm/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/arm/include/asm/pgalloc.h @@ -65,6 +65,7 @@ static inline void clean_pte_table(pte_t #define __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL #define __HAVE_ARCH_PTE_ALLOC_ONE +#define __HAVE_ARCH_PGD_FREE #include static inline pte_t * --- a/arch/csky/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/csky/include/asm/pgalloc.h @@ -9,7 +9,7 @@ #include #define __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL -#include /* for pte_{alloc,free}_one */ +#include static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) @@ -42,11 +42,6 @@ static inline pte_t *pte_alloc_one_kerne return pte; } -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_pages((unsigned long)pgd, PGD_ORDER); -} - static inline pgd_t *pgd_alloc(struct mm_struct *mm) { pgd_t *ret; --- a/arch/hexagon/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/hexagon/include/asm/pgalloc.h @@ -11,7 +11,7 @@ #include #include -#include /* for pte_{alloc,free}_one */ +#include extern unsigned long long kmap_generation; @@ -41,11 +41,6 @@ static inline pgd_t *pgd_alloc(struct mm return pgd; } -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_page((unsigned long) pgd); -} - static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t pte) { --- a/arch/ia64/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/ia64/include/asm/pgalloc.h @@ -29,11 +29,6 @@ static inline pgd_t *pgd_alloc(struct mm return (pgd_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); } -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_page((unsigned long)pgd); -} - #if CONFIG_PGTABLE_LEVELS == 4 static inline void p4d_populate(struct mm_struct *mm, p4d_t * p4d_entry, pud_t * pud) --- a/arch/m68k/include/asm/sun3_pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/m68k/include/asm/sun3_pgalloc.h @@ -13,7 +13,7 @@ #include -#include /* for pte_{alloc,free}_one */ +#include extern const char bad_pmd_string[]; @@ -40,11 +40,6 @@ static inline void pmd_populate(struct m */ #define pmd_free(mm, x) do { } while (0) -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_page((unsigned long) pgd); -} - static inline pgd_t * pgd_alloc(struct mm_struct *mm) { pgd_t *new_pgd; --- a/arch/microblaze/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/microblaze/include/asm/pgalloc.h @@ -28,12 +28,6 @@ static inline pgd_t *get_pgd(void) return (pgd_t *)__get_free_pages(GFP_KERNEL|__GFP_ZERO, 0); } -static inline void free_pgd(pgd_t *pgd) -{ - free_page((unsigned long)pgd); -} - -#define pgd_free(mm, pgd) free_pgd(pgd) #define pgd_alloc(mm) get_pgd() #define pmd_pgtable(pmd) pmd_page(pmd) --- a/arch/mips/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/mips/include/asm/pgalloc.h @@ -49,11 +49,6 @@ static inline void pud_populate(struct m extern void pgd_init(unsigned long page); extern pgd_t *pgd_alloc(struct mm_struct *mm); -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_pages((unsigned long)pgd, PGD_ORDER); -} - #define __pte_free_tlb(tlb,pte,address) \ do { \ pgtable_pte_page_dtor(pte); \ --- a/arch/nds32/mm/mm-nds32.c~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/nds32/mm/mm-nds32.c @@ -2,6 +2,8 @@ // Copyright (C) 2005-2017 Andes Technology Corporation #include + +#define __HAVE_ARCH_PGD_FREE #include #define FIRST_KERNEL_PGD_NR (USER_PTRS_PER_PGD) --- a/arch/nios2/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/nios2/include/asm/pgalloc.h @@ -12,7 +12,7 @@ #include -#include /* for pte_{alloc,free}_one */ +#include static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) @@ -34,11 +34,6 @@ extern void pmd_init(unsigned long page, extern pgd_t *pgd_alloc(struct mm_struct *mm); -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_pages((unsigned long)pgd, PGD_ORDER); -} - #define __pte_free_tlb(tlb, pte, addr) \ do { \ pgtable_pte_page_dtor(pte); \ --- a/arch/parisc/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/parisc/include/asm/pgalloc.h @@ -11,6 +11,7 @@ #include #define __HAVE_ARCH_PMD_FREE +#define __HAVE_ARCH_PGD_FREE #include /* Allocate the top level pgd (page directory) --- a/arch/riscv/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/riscv/include/asm/pgalloc.h @@ -55,11 +55,6 @@ static inline pgd_t *pgd_alloc(struct mm return pgd; } -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_page((unsigned long)pgd); -} - #ifndef __PAGETABLE_PMD_FOLDED #define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd) --- a/arch/sh/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/sh/include/asm/pgalloc.h @@ -6,6 +6,7 @@ #define __HAVE_ARCH_PMD_ALLOC_ONE #define __HAVE_ARCH_PMD_FREE +#define __HAVE_ARCH_PGD_FREE #include extern pgd_t *pgd_alloc(struct mm_struct *); --- a/arch/um/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/um/include/asm/pgalloc.h @@ -25,7 +25,6 @@ * Allocate and free page tables. */ extern pgd_t *pgd_alloc(struct mm_struct *); -extern void pgd_free(struct mm_struct *mm, pgd_t *pgd); #define __pte_free_tlb(tlb,pte, address) \ do { \ --- a/arch/um/kernel/mem.c~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/um/kernel/mem.c @@ -196,11 +196,6 @@ pgd_t *pgd_alloc(struct mm_struct *mm) return pgd; } -void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_page((unsigned long) pgd); -} - void *uml_kmalloc(int size, int flags) { return kmalloc(size, flags); --- a/arch/x86/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/x86/include/asm/pgalloc.h @@ -7,6 +7,7 @@ #include #define __HAVE_ARCH_PTE_ALLOC_ONE +#define __HAVE_ARCH_PGD_FREE #include static inline int __paravirt_pgd_alloc(struct mm_struct *mm) { return 0; } --- a/arch/xtensa/include/asm/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/arch/xtensa/include/asm/pgalloc.h @@ -33,11 +33,6 @@ pgd_alloc(struct mm_struct *mm) return (pgd_t*) __get_free_pages(GFP_KERNEL | __GFP_ZERO, PGD_ORDER); } -static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) -{ - free_page((unsigned long)pgd); -} - static inline void ptes_clear(pte_t *ptep) { int i; --- a/include/asm-generic/pgalloc.h~asm-generic-pgalloc-provide-generic-pgd_free +++ a/include/asm-generic/pgalloc.h @@ -175,6 +175,13 @@ static inline void pud_free(struct mm_st #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#ifndef __HAVE_ARCH_PGD_FREE +static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) +{ + free_page((unsigned long)pgd); +} +#endif + #endif /* CONFIG_MMU */ #endif /* __ASM_GENERIC_PGALLOC_H */ From patchwork Fri Aug 7 06:22:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704875 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 41C96138C for ; Fri, 7 Aug 2020 06:22:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D888C22D0B for ; Fri, 7 Aug 2020 06:22:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="XTbs8alG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D888C22D0B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3F82E8D006F; Fri, 7 Aug 2020 02:22:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3A81D8D0026; Fri, 7 Aug 2020 02:22:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C7DB8D006F; Fri, 7 Aug 2020 02:22:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0144.hostedemail.com [216.40.44.144]) by kanga.kvack.org (Postfix) with ESMTP id 1008C8D0026 for ; Fri, 7 Aug 2020 02:22:55 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D55093628 for ; Fri, 7 Aug 2020 06:22:54 +0000 (UTC) X-FDA: 77122779468.18.brass13_150558e26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id AC9F6100EC678 for ; Fri, 7 Aug 2020 06:22:54 +0000 (UTC) X-Spam-Summary: 1,0,0,7444e1304d275488,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:4:41:69:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1981:2194:2196:2198:2199:2200:2201:2393:2525:2538:2559:2563:2682:2685:2693:2736:2859:2898:2901:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3165:3865:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4605:5007:6119:6261:6653:6737:7514:7576:7875:8599:8603:9025:9036:9545:9592:10004:10913:11026:11232:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13229:13846:21080:21451:21627:21789:21939:21966:21990:30003:30054:30055:30067:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8ehgmh5d7ozn71zsx99z8h9dcnopw7akmzj1sca1xqm1gzpqexb8g74nw8kb.f7kj7bjkm386zywibi491k86fp5iu9ze3i53w6mts6zkjnmeankcud1uct9ofeo.q-lbl8.mailshell. net-223. X-HE-Tag: brass13_150558e26fbe X-Filterd-Recvd-Size: 18309 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:54 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5CDE222CF6; Fri, 7 Aug 2020 06:22:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781373; bh=gVETXYlKHv2E8QuEZrQkIsoDwgn5P4hrNaJQvpjovbU=; h=Date:From:To:Subject:In-Reply-To:From; b=XTbs8alG56FDFlDEx6+mvXPO8MiSyB7cLdwreqyzgMCrNWF/5W5MxmcEg1yy4EbzH GvMkuS/FIVZ3fZm7DSw78D50DeDeVWAq2H7KcA11GDkc+QG7EbUXYsLhjz3Pn9ELrL gd3iZx8cavqP9H4q4fNt+iqaPSTZ/Iztk45do56g= Date: Thu, 06 Aug 2020 23:22:51 -0700 From: Andrew Morton To: abdhalee@linux.vnet.ibm.com, akpm@linux-foundation.org, arnd@arndb.de, christophe.leroy@csgroup.eu, geert@linux-m68k.org, jcmvbkbc@gmail.com, joro@8bytes.org, jroedel@suse.de, linux-mm@kvack.org, luto@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, peterz@infradead.org, rostedt@goodmis.org, rppt@linux.ibm.com, sathnaga@linux.vnet.ibm.com, sfr@canb.auug.org.au, shorne@gmail.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 102/163] mm: move lib/ioremap.c to mm/ Message-ID: <20200807062251.amozo-C91%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: AC9F6100EC678 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Subject: mm: move lib/ioremap.c to mm/ The functionality in lib/ioremap.c deals with pagetables, vmalloc and caches, so it naturally belongs to mm/ Moving it there will also allow declaring p?d_alloc_track functions in an header file inside mm/ rather than having those declarations in include/linux/mm.h Link: http://lkml.kernel.org/r/20200627143453.31835-8-rppt@kernel.org Signed-off-by: Mike Rapoport Suggested-by: Andrew Morton Reviewed-by: Pekka Enberg Cc: Abdul Haleem Cc: Andy Lutomirski Cc: Arnd Bergmann Cc: Christophe Leroy Cc: Joerg Roedel Cc: Joerg Roedel Cc: Max Filippov Cc: Peter Zijlstra (Intel) Cc: Satheesh Rajendran Cc: Stafford Horne Cc: Stephen Rothwell Cc: Steven Rostedt Cc: Geert Uytterhoeven Cc: Matthew Wilcox Signed-off-by: Andrew Morton --- lib/Makefile | 1 lib/ioremap.c | 287 ------------------------------------------------ mm/Makefile | 2 mm/ioremap.c | 287 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 288 insertions(+), 289 deletions(-) --- a/lib/ioremap.c +++ /dev/null @@ -1,287 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* - * Re-map IO memory to kernel address space so that we can access it. - * This is needed for high PCI addresses that aren't mapped in the - * 640k-1MB IO memory area on PC's - * - * (C) Copyright 1995 1996 Linus Torvalds - */ -#include -#include -#include -#include -#include -#include - -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static int __read_mostly ioremap_p4d_capable; -static int __read_mostly ioremap_pud_capable; -static int __read_mostly ioremap_pmd_capable; -static int __read_mostly ioremap_huge_disabled; - -static int __init set_nohugeiomap(char *str) -{ - ioremap_huge_disabled = 1; - return 0; -} -early_param("nohugeiomap", set_nohugeiomap); - -void __init ioremap_huge_init(void) -{ - if (!ioremap_huge_disabled) { - if (arch_ioremap_p4d_supported()) - ioremap_p4d_capable = 1; - if (arch_ioremap_pud_supported()) - ioremap_pud_capable = 1; - if (arch_ioremap_pmd_supported()) - ioremap_pmd_capable = 1; - } -} - -static inline int ioremap_p4d_enabled(void) -{ - return ioremap_p4d_capable; -} - -static inline int ioremap_pud_enabled(void) -{ - return ioremap_pud_capable; -} - -static inline int ioremap_pmd_enabled(void) -{ - return ioremap_pmd_capable; -} - -#else /* !CONFIG_HAVE_ARCH_HUGE_VMAP */ -static inline int ioremap_p4d_enabled(void) { return 0; } -static inline int ioremap_pud_enabled(void) { return 0; } -static inline int ioremap_pmd_enabled(void) { return 0; } -#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ - -static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) -{ - pte_t *pte; - u64 pfn; - - pfn = phys_addr >> PAGE_SHIFT; - pte = pte_alloc_kernel_track(pmd, addr, mask); - if (!pte) - return -ENOMEM; - do { - BUG_ON(!pte_none(*pte)); - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; - } while (pte++, addr += PAGE_SIZE, addr != end); - *mask |= PGTBL_PTE_MODIFIED; - return 0; -} - -static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) -{ - if (!ioremap_pmd_enabled()) - return 0; - - if ((end - addr) != PMD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PMD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PMD_SIZE)) - return 0; - - if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) - return 0; - - return pmd_set_huge(pmd, phys_addr, prot); -} - -static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) -{ - pmd_t *pmd; - unsigned long next; - - pmd = pmd_alloc_track(&init_mm, pud, addr, mask); - if (!pmd) - return -ENOMEM; - do { - next = pmd_addr_end(addr, end); - - if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { - *mask |= PGTBL_PMD_MODIFIED; - continue; - } - - if (ioremap_pte_range(pmd, addr, next, phys_addr, prot, mask)) - return -ENOMEM; - } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) -{ - if (!ioremap_pud_enabled()) - return 0; - - if ((end - addr) != PUD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PUD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PUD_SIZE)) - return 0; - - if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) - return 0; - - return pud_set_huge(pud, phys_addr, prot); -} - -static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) -{ - pud_t *pud; - unsigned long next; - - pud = pud_alloc_track(&init_mm, p4d, addr, mask); - if (!pud) - return -ENOMEM; - do { - next = pud_addr_end(addr, end); - - if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) { - *mask |= PGTBL_PUD_MODIFIED; - continue; - } - - if (ioremap_pmd_range(pud, addr, next, phys_addr, prot, mask)) - return -ENOMEM; - } while (pud++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) -{ - if (!ioremap_p4d_enabled()) - return 0; - - if ((end - addr) != P4D_SIZE) - return 0; - - if (!IS_ALIGNED(addr, P4D_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, P4D_SIZE)) - return 0; - - if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) - return 0; - - return p4d_set_huge(p4d, phys_addr, prot); -} - -static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) -{ - p4d_t *p4d; - unsigned long next; - - p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); - if (!p4d) - return -ENOMEM; - do { - next = p4d_addr_end(addr, end); - - if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { - *mask |= PGTBL_P4D_MODIFIED; - continue; - } - - if (ioremap_pud_range(p4d, addr, next, phys_addr, prot, mask)) - return -ENOMEM; - } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -int ioremap_page_range(unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot) -{ - pgd_t *pgd; - unsigned long start; - unsigned long next; - int err; - pgtbl_mod_mask mask = 0; - - might_sleep(); - BUG_ON(addr >= end); - - start = addr; - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot, - &mask); - if (err) - break; - } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - - flush_cache_vmap(start, end); - - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) - arch_sync_kernel_mappings(start, end); - - return err; -} - -#ifdef CONFIG_GENERIC_IOREMAP -void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) -{ - unsigned long offset, vaddr; - phys_addr_t last_addr; - struct vm_struct *area; - - /* Disallow wrap-around or zero size */ - last_addr = addr + size - 1; - if (!size || last_addr < addr) - return NULL; - - /* Page-align mappings */ - offset = addr & (~PAGE_MASK); - addr -= offset; - size = PAGE_ALIGN(size + offset); - - area = get_vm_area_caller(size, VM_IOREMAP, - __builtin_return_address(0)); - if (!area) - return NULL; - vaddr = (unsigned long)area->addr; - - if (ioremap_page_range(vaddr, vaddr + size, addr, __pgprot(prot))) { - free_vm_area(area); - return NULL; - } - - return (void __iomem *)(vaddr + offset); -} -EXPORT_SYMBOL(ioremap_prot); - -void iounmap(volatile void __iomem *addr) -{ - vunmap((void *)((unsigned long)addr & PAGE_MASK)); -} -EXPORT_SYMBOL(iounmap); -#endif /* CONFIG_GENERIC_IOREMAP */ --- a/lib/Makefile~mm-move-lib-ioremapc-to-mm +++ a/lib/Makefile @@ -37,7 +37,6 @@ lib-y := ctype.o string.o vsprintf.o cmd nmi_backtrace.o nodemask.o win_minmax.o memcat_p.o lib-$(CONFIG_PRINTK) += dump_stack.o -lib-$(CONFIG_MMU) += ioremap.o lib-$(CONFIG_SMP) += cpumask.o lib-y += kobject.o klist.o --- /dev/null +++ a/mm/ioremap.c @@ -0,0 +1,287 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Re-map IO memory to kernel address space so that we can access it. + * This is needed for high PCI addresses that aren't mapped in the + * 640k-1MB IO memory area on PC's + * + * (C) Copyright 1995 1996 Linus Torvalds + */ +#include +#include +#include +#include +#include +#include + +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +static int __read_mostly ioremap_p4d_capable; +static int __read_mostly ioremap_pud_capable; +static int __read_mostly ioremap_pmd_capable; +static int __read_mostly ioremap_huge_disabled; + +static int __init set_nohugeiomap(char *str) +{ + ioremap_huge_disabled = 1; + return 0; +} +early_param("nohugeiomap", set_nohugeiomap); + +void __init ioremap_huge_init(void) +{ + if (!ioremap_huge_disabled) { + if (arch_ioremap_p4d_supported()) + ioremap_p4d_capable = 1; + if (arch_ioremap_pud_supported()) + ioremap_pud_capable = 1; + if (arch_ioremap_pmd_supported()) + ioremap_pmd_capable = 1; + } +} + +static inline int ioremap_p4d_enabled(void) +{ + return ioremap_p4d_capable; +} + +static inline int ioremap_pud_enabled(void) +{ + return ioremap_pud_capable; +} + +static inline int ioremap_pmd_enabled(void) +{ + return ioremap_pmd_capable; +} + +#else /* !CONFIG_HAVE_ARCH_HUGE_VMAP */ +static inline int ioremap_p4d_enabled(void) { return 0; } +static inline int ioremap_pud_enabled(void) { return 0; } +static inline int ioremap_pmd_enabled(void) { return 0; } +#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) +{ + pte_t *pte; + u64 pfn; + + pfn = phys_addr >> PAGE_SHIFT; + pte = pte_alloc_kernel_track(pmd, addr, mask); + if (!pte) + return -ENOMEM; + do { + BUG_ON(!pte_none(*pte)); + set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); + pfn++; + } while (pte++, addr += PAGE_SIZE, addr != end); + *mask |= PGTBL_PTE_MODIFIED; + return 0; +} + +static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, + unsigned long end, phys_addr_t phys_addr, + pgprot_t prot) +{ + if (!ioremap_pmd_enabled()) + return 0; + + if ((end - addr) != PMD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PMD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PMD_SIZE)) + return 0; + + if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) + return 0; + + return pmd_set_huge(pmd, phys_addr, prot); +} + +static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, + unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) +{ + pmd_t *pmd; + unsigned long next; + + pmd = pmd_alloc_track(&init_mm, pud, addr, mask); + if (!pmd) + return -ENOMEM; + do { + next = pmd_addr_end(addr, end); + + if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + *mask |= PGTBL_PMD_MODIFIED; + continue; + } + + if (ioremap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + return -ENOMEM; + } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, + unsigned long end, phys_addr_t phys_addr, + pgprot_t prot) +{ + if (!ioremap_pud_enabled()) + return 0; + + if ((end - addr) != PUD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PUD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PUD_SIZE)) + return 0; + + if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) + return 0; + + return pud_set_huge(pud, phys_addr, prot); +} + +static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, + unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) +{ + pud_t *pud; + unsigned long next; + + pud = pud_alloc_track(&init_mm, p4d, addr, mask); + if (!pud) + return -ENOMEM; + do { + next = pud_addr_end(addr, end); + + if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + *mask |= PGTBL_PUD_MODIFIED; + continue; + } + + if (ioremap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + return -ENOMEM; + } while (pud++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, + unsigned long end, phys_addr_t phys_addr, + pgprot_t prot) +{ + if (!ioremap_p4d_enabled()) + return 0; + + if ((end - addr) != P4D_SIZE) + return 0; + + if (!IS_ALIGNED(addr, P4D_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, P4D_SIZE)) + return 0; + + if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) + return 0; + + return p4d_set_huge(p4d, phys_addr, prot); +} + +static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, + unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) +{ + p4d_t *p4d; + unsigned long next; + + p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); + if (!p4d) + return -ENOMEM; + do { + next = p4d_addr_end(addr, end); + + if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + *mask |= PGTBL_P4D_MODIFIED; + continue; + } + + if (ioremap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + return -ENOMEM; + } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +int ioremap_page_range(unsigned long addr, + unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +{ + pgd_t *pgd; + unsigned long start; + unsigned long next; + int err; + pgtbl_mod_mask mask = 0; + + might_sleep(); + BUG_ON(addr >= end); + + start = addr; + pgd = pgd_offset_k(addr); + do { + next = pgd_addr_end(addr, end); + err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot, + &mask); + if (err) + break; + } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); + + flush_cache_vmap(start, end); + + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, end); + + return err; +} + +#ifdef CONFIG_GENERIC_IOREMAP +void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) +{ + unsigned long offset, vaddr; + phys_addr_t last_addr; + struct vm_struct *area; + + /* Disallow wrap-around or zero size */ + last_addr = addr + size - 1; + if (!size || last_addr < addr) + return NULL; + + /* Page-align mappings */ + offset = addr & (~PAGE_MASK); + addr -= offset; + size = PAGE_ALIGN(size + offset); + + area = get_vm_area_caller(size, VM_IOREMAP, + __builtin_return_address(0)); + if (!area) + return NULL; + vaddr = (unsigned long)area->addr; + + if (ioremap_page_range(vaddr, vaddr + size, addr, __pgprot(prot))) { + free_vm_area(area); + return NULL; + } + + return (void __iomem *)(vaddr + offset); +} +EXPORT_SYMBOL(ioremap_prot); + +void iounmap(volatile void __iomem *addr) +{ + vunmap((void *)((unsigned long)addr & PAGE_MASK)); +} +EXPORT_SYMBOL(iounmap); +#endif /* CONFIG_GENERIC_IOREMAP */ --- a/mm/Makefile~mm-move-lib-ioremapc-to-mm +++ a/mm/Makefile @@ -38,7 +38,7 @@ mmu-y := nommu.o mmu-$(CONFIG_MMU) := highmem.o memory.o mincore.o \ mlock.o mmap.o mmu_gather.o mprotect.o mremap.o \ msync.o page_vma_mapped.o pagewalk.o \ - pgtable-generic.o rmap.o vmalloc.o + pgtable-generic.o rmap.o vmalloc.o ioremap.o ifdef CONFIG_CROSS_MEMORY_ATTACH From patchwork Fri Aug 7 06:22:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704877 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB617138C for ; Fri, 7 Aug 2020 06:23:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B2EF82177B for ; Fri, 7 Aug 2020 06:23:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="G9iu0OzN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B2EF82177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 493658D0070; Fri, 7 Aug 2020 02:22:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4447A8D0026; Fri, 7 Aug 2020 02:22:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 381838D0070; Fri, 7 Aug 2020 02:22:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0136.hostedemail.com [216.40.44.136]) by kanga.kvack.org (Postfix) with ESMTP id 2305C8D0026 for ; Fri, 7 Aug 2020 02:22:59 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E8BBA82499A8 for ; Fri, 7 Aug 2020 06:22:58 +0000 (UTC) X-FDA: 77122779636.20.show54_25038c926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id BB275180C07A3 for ; Fri, 7 Aug 2020 06:22:58 +0000 (UTC) X-Spam-Summary: 1,0,0,86e1e19443107d8c,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1606:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4117:4321:4605:5007:6261:6653:6737:7514:7576:8599:9025:9545:9592:10004:10913:11026:11232:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13229:13846:21080:21451:21627:21939:21990:30003:30054:30055,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygu36zps477btffcsic3b8x993coc6eh7sbewu4kmr8959bm7eausbge5ypuf.y6zidno1j9x3d93cjpzh6stzwc5h1phe4jxidh7k5atndzw3r5nxj5m9b95wnhr.r-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSB L:neutra X-HE-Tag: show54_25038c926fbe X-Filterd-Recvd-Size: 6966 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:22:58 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4719522D07; Fri, 7 Aug 2020 06:22:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781377; bh=b9vlsC15PBDYbHL0nX7UTcDt389GW8p9k4aVdgtUkCQ=; h=Date:From:To:Subject:In-Reply-To:From; b=G9iu0OzNtutzi6OC8OoFEF1ZscbWxiXTq1Ds30iOSWQs/HNFXz32aP+0q+0CeuSD4 Jo+v93B+M6INxZkkXjg9QDRGJXqPOMdY3VG6OBOdQv+eMPD5pcIqL6btWYhEZOpZ0e ih0H8v4bmVP1fi7Mz4YVV/JqfxjEpYuTwRUPGbGw= Date: Thu, 06 Aug 2020 23:22:55 -0700 From: Andrew Morton To: abdhalee@linux.vnet.ibm.com, akpm@linux-foundation.org, arnd@arndb.de, christophe.leroy@csgroup.eu, geert@linux-m68k.org, jcmvbkbc@gmail.com, jroedel@suse.de, linux-mm@kvack.org, luto@kernel.org, mm-commits@vger.kernel.org, penberg@kernel.org, peterz@infradead.org, rostedt@goodmis.org, rppt@linux.ibm.com, sathnaga@linux.vnet.ibm.com, sfr@canb.auug.org.au, shorne@gmail.com, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 103/163] mm: move p?d_alloc_track to separate header file Message-ID: <20200807062255.fIx5o1Ivv%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: BB275180C07A3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joerg Roedel Subject: mm: move p?d_alloc_track to separate header file The functions are only used in two source files, so there is no need for them to be in the global header. Move them to the new header and include it only where needed. Link: http://lkml.kernel.org/r/20200609120533.25867-1-joro@8bytes.org Signed-off-by: Joerg Roedel Reviewed-by: Pekka Enberg Cc: Peter Zijlstra (Intel) Cc: Andy Lutomirski Cc: Abdul Haleem Cc: Satheesh Rajendran Cc: Stephen Rothwell Cc: Steven Rostedt (VMware) Cc: Mike Rapoport Cc: Christophe Leroy Cc: Arnd Bergmann Cc: Max Filippov Cc: Stafford Horne Cc: Geert Uytterhoeven Cc: Matthew Wilcox Signed-off-by: Andrew Morton --- include/linux/mm.h | 45 ------------------------------------- mm/ioremap.c | 2 + mm/pgalloc-track.h | 51 +++++++++++++++++++++++++++++++++++++++++++ mm/vmalloc.c | 1 4 files changed, 54 insertions(+), 45 deletions(-) --- a/include/linux/mm.h~mm-move-pd_alloc_track-to-separate-header-file +++ a/include/linux/mm.h @@ -2103,51 +2103,11 @@ static inline pud_t *pud_alloc(struct mm NULL : pud_offset(p4d, address); } -static inline p4d_t *p4d_alloc_track(struct mm_struct *mm, pgd_t *pgd, - unsigned long address, - pgtbl_mod_mask *mod_mask) - -{ - if (unlikely(pgd_none(*pgd))) { - if (__p4d_alloc(mm, pgd, address)) - return NULL; - *mod_mask |= PGTBL_PGD_MODIFIED; - } - - return p4d_offset(pgd, address); -} - -static inline pud_t *pud_alloc_track(struct mm_struct *mm, p4d_t *p4d, - unsigned long address, - pgtbl_mod_mask *mod_mask) -{ - if (unlikely(p4d_none(*p4d))) { - if (__pud_alloc(mm, p4d, address)) - return NULL; - *mod_mask |= PGTBL_P4D_MODIFIED; - } - - return pud_offset(p4d, address); -} - static inline pmd_t *pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) { return (unlikely(pud_none(*pud)) && __pmd_alloc(mm, pud, address))? NULL: pmd_offset(pud, address); } - -static inline pmd_t *pmd_alloc_track(struct mm_struct *mm, pud_t *pud, - unsigned long address, - pgtbl_mod_mask *mod_mask) -{ - if (unlikely(pud_none(*pud))) { - if (__pmd_alloc(mm, pud, address)) - return NULL; - *mod_mask |= PGTBL_PUD_MODIFIED; - } - - return pmd_offset(pud, address); -} #endif /* CONFIG_MMU */ #if USE_SPLIT_PTE_PTLOCKS @@ -2263,11 +2223,6 @@ static inline void pgtable_pte_page_dtor ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd))? \ NULL: pte_offset_kernel(pmd, address)) -#define pte_alloc_kernel_track(pmd, address, mask) \ - ((unlikely(pmd_none(*(pmd))) && \ - (__pte_alloc_kernel(pmd) || ({*(mask)|=PGTBL_PMD_MODIFIED;0;})))?\ - NULL: pte_offset_kernel(pmd, address)) - #if USE_SPLIT_PMD_PTLOCKS static struct page *pmd_to_page(pmd_t *pmd) --- a/mm/ioremap.c~mm-move-pd_alloc_track-to-separate-header-file +++ a/mm/ioremap.c @@ -13,6 +13,8 @@ #include #include +#include "pgalloc-track.h" + #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP static int __read_mostly ioremap_p4d_capable; static int __read_mostly ioremap_pud_capable; --- /dev/null +++ a/mm/pgalloc-track.h @@ -0,0 +1,51 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_PGALLLC_TRACK_H +#define _LINUX_PGALLLC_TRACK_H + +#if defined(CONFIG_MMU) +static inline p4d_t *p4d_alloc_track(struct mm_struct *mm, pgd_t *pgd, + unsigned long address, + pgtbl_mod_mask *mod_mask) +{ + if (unlikely(pgd_none(*pgd))) { + if (__p4d_alloc(mm, pgd, address)) + return NULL; + *mod_mask |= PGTBL_PGD_MODIFIED; + } + + return p4d_offset(pgd, address); +} + +static inline pud_t *pud_alloc_track(struct mm_struct *mm, p4d_t *p4d, + unsigned long address, + pgtbl_mod_mask *mod_mask) +{ + if (unlikely(p4d_none(*p4d))) { + if (__pud_alloc(mm, p4d, address)) + return NULL; + *mod_mask |= PGTBL_P4D_MODIFIED; + } + + return pud_offset(p4d, address); +} + +static inline pmd_t *pmd_alloc_track(struct mm_struct *mm, pud_t *pud, + unsigned long address, + pgtbl_mod_mask *mod_mask) +{ + if (unlikely(pud_none(*pud))) { + if (__pmd_alloc(mm, pud, address)) + return NULL; + *mod_mask |= PGTBL_PUD_MODIFIED; + } + + return pmd_offset(pud, address); +} +#endif /* CONFIG_MMU */ + +#define pte_alloc_kernel_track(pmd, address, mask) \ + ((unlikely(pmd_none(*(pmd))) && \ + (__pte_alloc_kernel(pmd) || ({*(mask)|=PGTBL_PMD_MODIFIED;0;})))?\ + NULL: pte_offset_kernel(pmd, address)) + +#endif /* _LINUX_PGALLLC_TRACK_H */ --- a/mm/vmalloc.c~mm-move-pd_alloc_track-to-separate-header-file +++ a/mm/vmalloc.c @@ -41,6 +41,7 @@ #include #include "internal.h" +#include "pgalloc-track.h" bool is_vmalloc_addr(const void *x) { From patchwork Fri Aug 7 06:22:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704879 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E208B138C for ; Fri, 7 Aug 2020 06:23:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A771C2177B for ; Fri, 7 Aug 2020 06:23:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Cj30MXfi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A771C2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 41E3B8D0071; Fri, 7 Aug 2020 02:23:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3CF4F8D0026; Fri, 7 Aug 2020 02:23:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E8778D0071; Fri, 7 Aug 2020 02:23:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0082.hostedemail.com [216.40.44.82]) by kanga.kvack.org (Postfix) with ESMTP id 188758D0026 for ; Fri, 7 Aug 2020 02:23:02 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D69A7180AD801 for ; Fri, 7 Aug 2020 06:23:01 +0000 (UTC) X-FDA: 77122779762.25.sign44_2017cd226fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id A69C61804E3A0 for ; Fri, 7 Aug 2020 06:23:01 +0000 (UTC) X-Spam-Summary: 1,0,0,f4654f53e2c17b43,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2393:2525:2553:2559:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6119:6261:6653:7576:7903:8599:8603:9025:9545:10004:10913:11026:11658:11914:12043:12048:12295:12296:12297:12517:12519:12555:12679:12783:12986:13190:13229:13548:13846:14093:14181:14721:14849:21080:21433:21451:21627:21795:21939:21990:30012:30051:30054:30056:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8sxfencf1te766z51ibsyqkuwiycnjo7d38b6whn3hsbdzg46dsh5gtbyznx.pmyqm6zkdjtjm14agnguj5ig4fmxzgo6kfqsz9aton819c13fzwrkpdzm8qiqxs.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none ,DomainC X-HE-Tag: sign44_2017cd226fbe X-Filterd-Recvd-Size: 3419 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:01 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 585CB22C9F; Fri, 7 Aug 2020 06:23:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781380; bh=AAmWkQvvZBKmqp4aVbr8oXCk1LrZkBLh/2YaYAxgK/U=; h=Date:From:To:Subject:In-Reply-To:From; b=Cj30MXfiy9vJN/28rFSojY1S7+uKUeVky0Nee9JE3nGxA+jXmfIWGvsJRxTDZPyXI wUsiVETGTDVNMS9QQqwrH04DudmDsq0X2hovoxkyHsOFGgfcbFRhY9wkkQtSk5Km1P 6wSoPu2LEm3aYm8+mvPaDH+83jgJASTNaChtHx7s= Date: Thu, 06 Aug 2020 23:22:59 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, thunder.leizhen@huawei.com, torvalds@linux-foundation.org Subject: [patch 104/163] mm/mmap: optimize a branch judgment in ksys_mmap_pgoff() Message-ID: <20200807062259.JNE-ENp6G%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: A69C61804E3A0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zhen Lei Subject: mm/mmap: optimize a branch judgment in ksys_mmap_pgoff() Look at the pseudo code below. It's very clear that, the judgement "!is_file_hugepages(file)" at 3) is duplicated to the one at 1), we can use "else if" to avoid it. And the assignment "retval = -EINVAL" at 2) is only needed by the branch 3), because "retval" will be overwritten at 4). No functional change, but it can reduce the code size. Maybe more clearer? Before: text data bss dec hex filename 28733 1590 1 30324 7674 mm/mmap.o After: text data bss dec hex filename 28701 1590 1 30292 7654 mm/mmap.o ====pseudo code====: if (!(flags & MAP_ANONYMOUS)) { ... 1) if (is_file_hugepages(file)) len = ALIGN(len, huge_page_size(hstate_file(file))); 2) retval = -EINVAL; 3) if (unlikely(flags & MAP_HUGETLB && !is_file_hugepages(file))) goto out_fput; } else if (flags & MAP_HUGETLB) { ... } ... 4) retval = vm_mmap_pgoff(file, addr, len, prot, flags, pgoff); out_fput: ... return retval; Link: http://lkml.kernel.org/r/20200705080112.1405-1-thunder.leizhen@huawei.com Signed-off-by: Zhen Lei Reviewed-by: Andrew Morton Signed-off-by: Andrew Morton --- mm/mmap.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) --- a/mm/mmap.c~mm-mmap-optimize-a-branch-judgment-in-ksys_mmap_pgoff +++ a/mm/mmap.c @@ -1562,11 +1562,12 @@ unsigned long ksys_mmap_pgoff(unsigned l file = fget(fd); if (!file) return -EBADF; - if (is_file_hugepages(file)) + if (is_file_hugepages(file)) { len = ALIGN(len, huge_page_size(hstate_file(file))); - retval = -EINVAL; - if (unlikely(flags & MAP_HUGETLB && !is_file_hugepages(file))) + } else if (unlikely(flags & MAP_HUGETLB)) { + retval = -EINVAL; goto out_fput; + } } else if (flags & MAP_HUGETLB) { struct user_struct *user = NULL; struct hstate *hs; From patchwork Fri Aug 7 06:23:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704881 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 16A6914B7 for ; Fri, 7 Aug 2020 06:23:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D9900221E5 for ; Fri, 7 Aug 2020 06:23:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="x/tYvjZG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D9900221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AAE258D0072; Fri, 7 Aug 2020 02:23:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A861E8D0026; Fri, 7 Aug 2020 02:23:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 99C598D0072; Fri, 7 Aug 2020 02:23:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0139.hostedemail.com [216.40.44.139]) by kanga.kvack.org (Postfix) with ESMTP id 855DE8D0026 for ; Fri, 7 Aug 2020 02:23:06 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 59490181AEF10 for ; Fri, 7 Aug 2020 06:23:06 +0000 (UTC) X-FDA: 77122779972.15.food57_531668626fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 279CC1814B0C7 for ; Fri, 7 Aug 2020 06:23:06 +0000 (UTC) X-Spam-Summary: 1,0,0,832e3cfd65b6fa40,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1543:1711:1730:1747:1777:1792:2393:2525:2559:2565:2682:2685:2691:2693:2840:2859:2902:2910:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6261:6653:6737:6738:7576:7688:7903:8555:8603:9025:9168:9391:9545:10004:10913:11026:11473:11658:11914:12043:12048:12050:12296:12297:12438:12517:12519:12555:12679:12783:12986:13149:13180:13229:13230:13846:14040:14181:14721:14849:21080:21433:21451:21627:21740:21811:21819:21939:30034:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04ygnw1361a3rkegk4aprwe9zq86goceuqs76uyyx9jwt7uibyy8r99ufmpf5nm.pe18tp5d9fet1kh4go9fqgkebz9hcn7fe14yesp4xit3h6e6gp84sqf16w5jtx1.4-lbl8.mailshell.net-223.238.255.100, CacheIP: X-HE-Tag: food57_531668626fbe X-Filterd-Recvd-Size: 5119 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:05 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id DABDE22CB3; Fri, 7 Aug 2020 06:23:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781384; bh=x5XI0e+PD1K5/WMcEyJMto6bUXiu0canCXug7o/7e5c=; h=Date:From:To:Subject:In-Reply-To:From; b=x/tYvjZG1wKGzolsPEDng4fNoyvkxd3ccMUGGo9qqG34YzY89Y5w3w9fvJLa/Ovk7 RGcIt5zQ0qreuE0uEpla7+JuYhefdtNd7JHt3XB8+zUDbDZ0quPtK5uwVxjSUZ9lBo /vV/z4XY3XkT+Gkqi8ovZnchFTdIi2CVl9nUV4fw= Date: Thu, 06 Aug 2020 23:23:03 -0700 From: Andrew Morton To: akpm@linux-foundation.org, andi.kleen@intel.com, cai@lca.pw, cl@linux.com, dave.hansen@intel.com, dennis@kernel.org, feng.tang@intel.com, haiyangz@microsoft.com, hannes@cmpxchg.org, keescook@chromium.org, kys@microsoft.com, linux-mm@kvack.org, mgorman@suse.de, mhocko@suse.com, mm-commits@vger.kernel.org, rong.a.chen@intel.com, tim.c.chen@intel.com, tj@kernel.org, torvalds@linux-foundation.org, willy@infradead.org, ying.huang@intel.com Subject: [patch 105/163] proc/meminfo: avoid open coded reading of vm_committed_as Message-ID: <20200807062303.lSAmVBTlH%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 279CC1814B0C7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Feng Tang Subject: proc/meminfo: avoid open coded reading of vm_committed_as Patch series "make vm_committed_as_batch aware of vm overcommit policy", v6. When checking a performance change for will-it-scale scalability mmap test [1], we found very high lock contention for spinlock of percpu counter 'vm_committed_as': 94.14% 0.35% [kernel.kallsyms] [k] _raw_spin_lock_irqsave 48.21% _raw_spin_lock_irqsave;percpu_counter_add_batch;__vm_enough_memory;mmap_region;do_mmap; 45.91% _raw_spin_lock_irqsave;percpu_counter_add_batch;__do_munmap; Actually this heavy lock contention is not always necessary. The 'vm_committed_as' needs to be very precise when the strict OVERCOMMIT_NEVER policy is set, which requires a rather small batch number for the percpu counter. So keep 'batch' number unchanged for strict OVERCOMMIT_NEVER policy, and enlarge it for not-so-strict OVERCOMMIT_ALWAYS and OVERCOMMIT_GUESS policies. Benchmark with the same testcase in [1] shows 53% improvement on a 8C/16T desktop, and 2097%(20X) on a 4S/72C/144T server. And for that case, whether it shows improvements depends on if the test mmap size is bigger than the batch number computed. We tested 10+ platforms in 0day (server, desktop and laptop). If we lift it to 64X, 80%+ platforms show improvements, and for 16X lift, 1/3 of the platforms will show improvements. And generally it should help the mmap/unmap usage,as Michal Hocko mentioned: : I believe that there are non-synthetic worklaods which would benefit : from a larger batch. E.g. large in memory databases which do large : mmaps during startups from multiple threads. Note: There are some style complain from checkpatch for patch 4, as sysctl handler declaration follows the similar format of sibling functions [1] https://lore.kernel.org/lkml/20200305062138.GI5972@shao2-debian/ This patch (of 4): Use the existing vm_memory_committed() instead, which is also convenient for future change. Link: http://lkml.kernel.org/r/1594389708-60781-1-git-send-email-feng.tang@intel.com Link: http://lkml.kernel.org/r/1594389708-60781-2-git-send-email-feng.tang@intel.com Signed-off-by: Feng Tang Acked-by: Michal Hocko Cc: Matthew Wilcox (Oracle) Cc: Johannes Weiner Cc: Mel Gorman Cc: Qian Cai Cc: Kees Cook Cc: Andi Kleen Cc: Tim Chen Cc: Dave Hansen Cc: Huang Ying Cc: Christoph Lameter Cc: Dennis Zhou Cc: Haiyang Zhang Cc: kernel test robot Cc: "K. Y. Srinivasan" Cc: Tejun Heo Signed-off-by: Andrew Morton --- fs/proc/meminfo.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/fs/proc/meminfo.c~proc-meminfo-avoid-open-coded-reading-of-vm_committed_as +++ a/fs/proc/meminfo.c @@ -41,7 +41,7 @@ static int meminfo_proc_show(struct seq_ si_meminfo(&i); si_swapinfo(&i); - committed = percpu_counter_read_positive(&vm_committed_as); + committed = vm_memory_committed(); cached = global_node_page_state(NR_FILE_PAGES) - total_swapcache_pages() - i.bufferram; From patchwork Fri Aug 7 06:23:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704883 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 58D83138C for ; Fri, 7 Aug 2020 06:23:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 246232177B for ; Fri, 7 Aug 2020 06:23:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="EBkUrnH0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 246232177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E314F8D0073; Fri, 7 Aug 2020 02:23:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DE16F8D0026; Fri, 7 Aug 2020 02:23:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCF9C8D0073; Fri, 7 Aug 2020 02:23:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0071.hostedemail.com [216.40.44.71]) by kanga.kvack.org (Postfix) with ESMTP id B84DD8D0026 for ; Fri, 7 Aug 2020 02:23:10 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8392382499A8 for ; Fri, 7 Aug 2020 06:23:10 +0000 (UTC) X-FDA: 77122780140.12.wave67_220869e26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 5ACB518011E4B for ; Fri, 7 Aug 2020 06:23:10 +0000 (UTC) X-Spam-Summary: 1,0,0,7fd7b321b4324074,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2393:2525:2559:2564:2682:2685:2693:2740:2859:2902:2910:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:6737:6738:7576:7688:9025:9168:9391:9545:10004:10913:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13846:14181:14721:14849:21080:21433:21451:21627:21819:21939:21990:30046:30054:30064:30075,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8ob54ai6a1b6y47hd6bjqqdnsfycjefg33k6mhfog1xnojg4gcqmwdnounem.huqy983e9a1pbkn1k5idc88prj6pu4jmmbpszn91wtwmuwc5eddyksbq1zmqx4b.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF: not bulk X-HE-Tag: wave67_220869e26fbe X-Filterd-Recvd-Size: 4208 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:09 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0835E22D3E; Fri, 7 Aug 2020 06:23:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781389; bh=3kwIVbG0Je9mRVpk/AEhSvKQxgMLHoOizB7KRNYvYc8=; h=Date:From:To:Subject:In-Reply-To:From; b=EBkUrnH0xaDhmBrYTv427ZCXqFuwvCRw7allzjt9rlkaHUlI3IdcorLEWYrib5egy SmlIeTXF7n4CY88RlCr3EzkW9etnnxnhTlb5VaytdvBD612OM5JazEE2NHEs2cbQA4 QyalS5NKKWhjPrSTeQfM6utAOVSgf6UhjH7Lk28A= Date: Thu, 06 Aug 2020 23:23:07 -0700 From: Andrew Morton To: akpm@linux-foundation.org, andi.kleen@intel.com, cai@lca.pw, cl@linux.com, dave.hansen@intel.com, dennis@kernel.org, feng.tang@intel.com, haiyangz@microsoft.com, hannes@cmpxchg.org, keescook@chromium.org, kys@microsoft.com, linux-mm@kvack.org, mgorman@suse.de, mhocko@suse.com, mm-commits@vger.kernel.org, rong.a.chen@intel.com, tim.c.chen@intel.com, tj@kernel.org, torvalds@linux-foundation.org, willy@infradead.org, ying.huang@intel.com Subject: [patch 106/163] mm/util.c: make vm_memory_committed() more accurate Message-ID: <20200807062307.YSHh3UQuU%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 5ACB518011E4B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Feng Tang Subject: mm/util.c: make vm_memory_committed() more accurate percpu_counter_sum_positive() will provide more accurate info. As with percpu_counter_read_positive(), in worst case the deviation could be 'batch * nr_cpus', which is totalram_pages/256 for now, and will be more when the batch gets enlarged. Its time cost is about 800 nanoseconds on a 2C/4T platform and 2~3 microseconds on a 2S/36C/72T Skylake server in normal case, and in worst case where vm_committed_as's spinlock is under severe contention, it costs 30~40 microseconds for the 2S/36C/72T Skylake sever, which should be fine for its only two users: /proc/meminfo and HyperV balloon driver's status trace per second. Link: http://lkml.kernel.org/r/1592725000-73486-3-git-send-email-feng.tang@intel.com Link: http://lkml.kernel.org/r/1594389708-60781-3-git-send-email-feng.tang@intel.com Signed-off-by: Feng Tang Acked-by: Michal Hocko # for /proc/meminfo Cc: "K. Y. Srinivasan" Cc: Haiyang Zhang Cc: Matthew Wilcox (Oracle) Cc: Johannes Weiner Cc: Mel Gorman Cc: Qian Cai Cc: Andi Kleen Cc: Tim Chen Cc: Dave Hansen Cc: Huang Ying Cc: Christoph Lameter Cc: Dennis Zhou Cc: Kees Cook Cc: kernel test robot Cc: Tejun Heo Signed-off-by: Andrew Morton --- mm/util.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) --- a/mm/util.c~mm-utilc-make-vm_memory_committed-more-accurate +++ a/mm/util.c @@ -787,10 +787,15 @@ struct percpu_counter vm_committed_as __ * balancing memory across competing virtual machines that are hosted. * Several metrics drive this policy engine including the guest reported * memory commitment. + * + * The time cost of this is very low for small platforms, and for big + * platform like a 2S/36C/72T Skylake server, in worst case where + * vm_committed_as's spinlock is under severe contention, the time cost + * could be about 30~40 microseconds. */ unsigned long vm_memory_committed(void) { - return percpu_counter_read_positive(&vm_committed_as); + return percpu_counter_sum_positive(&vm_committed_as); } EXPORT_SYMBOL_GPL(vm_memory_committed); From patchwork Fri Aug 7 06:23:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704885 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2BB5F138C for ; Fri, 7 Aug 2020 06:23:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E33C8221E5 for ; Fri, 7 Aug 2020 06:23:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="WMw2xD7N" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E33C8221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C5B9F8D0074; Fri, 7 Aug 2020 02:23:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C0C368D0026; Fri, 7 Aug 2020 02:23:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B49A68D0074; Fri, 7 Aug 2020 02:23:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id 9F7AE8D0026 for ; Fri, 7 Aug 2020 02:23:14 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 77C1E3644 for ; Fri, 7 Aug 2020 06:23:14 +0000 (UTC) X-FDA: 77122780308.20.scarf50_241372326fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 4DBBD180C07A3 for ; Fri, 7 Aug 2020 06:23:14 +0000 (UTC) X-Spam-Summary: 1,0,0,f758ddfdcf6338e8,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1543:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4605:5007:6117:6261:6653:6737:6738:7576:7688:7903:8603:9025:9121:9391:9545:10004:10913:11026:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13221:13229:13846:14096:14181:14721:14849:21080:21433:21451:21627:21819:21939:21990:30054:30064,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04y8n8juoqu9e4qmpmiy1dku7g6qoopa43ogndggs3w7zijdj9xiy5gdgkttthb.w4jqsy5paqidbbgxxusyoxi16gd463j171obfhkcisf461ua9hkbx5anhxebj8k.k-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF :not bul X-HE-Tag: scarf50_241372326fbe X-Filterd-Recvd-Size: 4807 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:13 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 349A522D08; Fri, 7 Aug 2020 06:23:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781393; bh=sm6Eco5XINRSqKtQDLCisOXZ/S6wNW0C+Zo4rThmMQs=; h=Date:From:To:Subject:In-Reply-To:From; b=WMw2xD7NAWRBAvxbY29x8p3SysFQa059yOHbZB0U9YgWGHKuzpicJHJTwBs+ECgzz vDFuCNr71gKypf+5OT8ZSaDSvNP/s01lSszBMXGCgxsOPnqigvzwJFiVJ5E+QHILkX p3FYtjBxzS89goAKbM7UobwDk74irpt/LfY5wND8= Date: Thu, 06 Aug 2020 23:23:11 -0700 From: Andrew Morton To: akpm@linux-foundation.org, andi.kleen@intel.com, cai@lca.pw, cl@linux.com, dave.hansen@intel.com, dennis@kernel.org, feng.tang@intel.com, haiyangz@microsoft.com, hannes@cmpxchg.org, keescook@chromium.org, kys@microsoft.com, linux-mm@kvack.org, mgorman@suse.de, mhocko@suse.com, mm-commits@vger.kernel.org, rong.a.chen@intel.com, tim.c.chen@intel.com, tj@kernel.org, torvalds@linux-foundation.org, willy@infradead.org, ying.huang@intel.com Subject: [patch 107/163] percpu_counter: add percpu_counter_sync() Message-ID: <20200807062311.3Piln41qG%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 4DBBD180C07A3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Feng Tang Subject: percpu_counter: add percpu_counter_sync() percpu_counter's accuracy is related to its batch size. For a percpu_counter with a big batch, its deviation could be big, so when the counter's batch is runtime changed to a smaller value for better accuracy, there could also be requirment to reduce the big deviation. So add a percpu-counter sync function to be run on each CPU. Link: http://lkml.kernel.org/r/1594389708-60781-4-git-send-email-feng.tang@intel.com Signed-off-by: Feng Tang Reported-by: kernel test robot Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Michal Hocko Cc: Qian Cai Cc: Andi Kleen Cc: Huang Ying Cc: Dave Hansen Cc: Haiyang Zhang Cc: Johannes Weiner Cc: Kees Cook Cc: "K. Y. Srinivasan" Cc: Matthew Wilcox (Oracle) Cc: Mel Gorman Cc: Tim Chen Signed-off-by: Andrew Morton --- include/linux/percpu_counter.h | 4 ++++ lib/percpu_counter.c | 19 +++++++++++++++++++ 2 files changed, 23 insertions(+) --- a/include/linux/percpu_counter.h~percpu_counter-add-percpu_counter_sync +++ a/include/linux/percpu_counter.h @@ -44,6 +44,7 @@ void percpu_counter_add_batch(struct per s32 batch); s64 __percpu_counter_sum(struct percpu_counter *fbc); int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch); +void percpu_counter_sync(struct percpu_counter *fbc); static inline int percpu_counter_compare(struct percpu_counter *fbc, s64 rhs) { @@ -172,6 +173,9 @@ static inline bool percpu_counter_initia return true; } +static inline void percpu_counter_sync(struct percpu_counter *fbc) +{ +} #endif /* CONFIG_SMP */ static inline void percpu_counter_inc(struct percpu_counter *fbc) --- a/lib/percpu_counter.c~percpu_counter-add-percpu_counter_sync +++ a/lib/percpu_counter.c @@ -99,6 +99,25 @@ void percpu_counter_add_batch(struct per EXPORT_SYMBOL(percpu_counter_add_batch); /* + * For percpu_counter with a big batch, the devication of its count could + * be big, and there is requirement to reduce the deviation, like when the + * counter's batch could be runtime decreased to get a better accuracy, + * which can be achieved by running this sync function on each CPU. + */ +void percpu_counter_sync(struct percpu_counter *fbc) +{ + unsigned long flags; + s64 count; + + raw_spin_lock_irqsave(&fbc->lock, flags); + count = __this_cpu_read(*fbc->counters); + fbc->count += count; + __this_cpu_sub(*fbc->counters, count); + raw_spin_unlock_irqrestore(&fbc->lock, flags); +} +EXPORT_SYMBOL(percpu_counter_sync); + +/* * Add up all the per-cpu counts, return the result. This is a more accurate * but much slower version of percpu_counter_read_positive() */ From patchwork Fri Aug 7 06:23:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704887 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8A14A14B7 for ; Fri, 7 Aug 2020 06:23:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4C0B42177B for ; Fri, 7 Aug 2020 06:23:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="b54NoXL7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4C0B42177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 185F58D0075; Fri, 7 Aug 2020 02:23:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1378A8D0026; Fri, 7 Aug 2020 02:23:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 025AA8D0075; Fri, 7 Aug 2020 02:23:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0099.hostedemail.com [216.40.44.99]) by kanga.kvack.org (Postfix) with ESMTP id E0AF08D0026 for ; Fri, 7 Aug 2020 02:23:18 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id AA885180AD802 for ; Fri, 7 Aug 2020 06:23:18 +0000 (UTC) X-FDA: 77122780476.28.queen12_370f0cb26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 7E7816D74 for ; Fri, 7 Aug 2020 06:23:18 +0000 (UTC) X-Spam-Summary: 1,0,0,1dcb3e84f3fc8d6b,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:2194:2198:2199:2200:2393:2525:2559:2566:2682:2685:2691:2693:2731:2859:2902:2910:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4050:4120:4321:4605:5007:6117:6261:6653:6737:6738:7576:7688:7901:7903:8603:9025:9168:9391:9545:10004:10913:11026:11473:11658:11914:12043:12048:12050:12114:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13255:13846:13972:14096:21080:21433:21451:21627:21740:21795:21811:21819:21939:21990:30034:30051:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrwjnt3x7icaat9rue5snh3wg5wopc9i8axj6a11ni3cdtg566zd88kn9scne.5qs3eb7dqz96gwhr1foa1ihehif51ejtbo317rthni1gne43qh7dp9s5abuqgup.n-lb l8.mails X-HE-Tag: queen12_370f0cb26fbe X-Filterd-Recvd-Size: 9379 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:17 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 40D60221E5; Fri, 7 Aug 2020 06:23:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781397; bh=wr7hbxBJUgPqvbbeDGCkgzNhXXmdkej7C4QWaoE1IlY=; h=Date:From:To:Subject:In-Reply-To:From; b=b54NoXL7RJdVXY0fJ6QJtZQfNWAjI65DuPgvLMwsTVei9cfLft9Ty/7ZDNv16rcJ7 ziwlrjQMImNYds9Hm8PCW5/tI3+VJwaY2inHcYhv044/VLrwpvQL3K5TiNPyDWwbdA 5ilywuuZk1g5BLdwPPafi1C7mSYl//sbiLi/2n7c= Date: Thu, 06 Aug 2020 23:23:15 -0700 From: Andrew Morton To: akpm@linux-foundation.org, andi.kleen@intel.com, cai@lca.pw, cl@linux.com, dave.hansen@intel.com, dennis@kernel.org, feng.tang@intel.com, haiyangz@microsoft.com, hannes@cmpxchg.org, keescook@chromium.org, kys@microsoft.com, linux-mm@kvack.org, mgorman@suse.de, mhocko@suse.com, mm-commits@vger.kernel.org, rong.a.chen@intel.com, tim.c.chen@intel.com, tj@kernel.org, torvalds@linux-foundation.org, willy@infradead.org, ying.huang@intel.com Subject: [patch 108/163] mm: adjust vm_committed_as_batch according to vm overcommit policy Message-ID: <20200807062315.nuQITHzCh%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 7E7816D74 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Feng Tang Subject: mm: adjust vm_committed_as_batch according to vm overcommit policy When checking a performance change for will-it-scale scalability mmap test [1], we found very high lock contention for spinlock of percpu counter 'vm_committed_as': 94.14% 0.35% [kernel.kallsyms] [k] _raw_spin_lock_irqsave 48.21% _raw_spin_lock_irqsave;percpu_counter_add_batch;__vm_enough_memory;mmap_region;do_mmap; 45.91% _raw_spin_lock_irqsave;percpu_counter_add_batch;__do_munmap; Actually this heavy lock contention is not always necessary. The 'vm_committed_as' needs to be very precise when the strict OVERCOMMIT_NEVER policy is set, which requires a rather small batch number for the percpu counter. So keep 'batch' number unchanged for strict OVERCOMMIT_NEVER policy, and lift it to 64X for OVERCOMMIT_ALWAYS and OVERCOMMIT_GUESS policies. Also add a sysctl handler to adjust it when the policy is reconfigured. Benchmark with the same testcase in [1] shows 53% improvement on a 8C/16T desktop, and 2097%(20X) on a 4S/72C/144T server. We tested with test platforms in 0day (server, desktop and laptop), and 80%+ platforms shows improvements with that test. And whether it shows improvements depends on if the test mmap size is bigger than the batch number computed. And if the lift is 16X, 1/3 of the platforms will show improvements, though it should help the mmap/unmap usage generally, as Michal Hocko mentioned: : I believe that there are non-synthetic worklaods which would benefit from : a larger batch. E.g. large in memory databases which do large mmaps : during startups from multiple threads. [1] https://lore.kernel.org/lkml/20200305062138.GI5972@shao2-debian/ Link: http://lkml.kernel.org/r/1589611660-89854-4-git-send-email-feng.tang@intel.com Link: http://lkml.kernel.org/r/1592725000-73486-4-git-send-email-feng.tang@intel.com Link: http://lkml.kernel.org/r/1594389708-60781-5-git-send-email-feng.tang@intel.com Signed-off-by: Feng Tang Acked-by: Michal Hocko Cc: Matthew Wilcox (Oracle) Cc: Johannes Weiner Cc: Mel Gorman Cc: Qian Cai Cc: Kees Cook Cc: Andi Kleen Cc: Tim Chen Cc: Dave Hansen Cc: Huang Ying Cc: Christoph Lameter Cc: Dennis Zhou Cc: Haiyang Zhang Cc: kernel test robot Cc: "K. Y. Srinivasan" Cc: Tejun Heo Signed-off-by: Andrew Morton --- include/linux/mm.h | 2 ++ include/linux/mman.h | 4 ++++ kernel/sysctl.c | 2 +- mm/mm_init.c | 20 +++++++++++++++----- mm/util.c | 41 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 63 insertions(+), 6 deletions(-) --- a/include/linux/mman.h~mm-adjust-vm_committed_as_batch-according-to-vm-overcommit-policy +++ a/include/linux/mman.h @@ -57,8 +57,12 @@ extern struct percpu_counter vm_committe #ifdef CONFIG_SMP extern s32 vm_committed_as_batch; +extern void mm_compute_batch(int overcommit_policy); #else #define vm_committed_as_batch 0 +static inline void mm_compute_batch(int overcommit_policy) +{ +} #endif unsigned long vm_memory_committed(void); --- a/include/linux/mm.h~mm-adjust-vm_committed_as_batch-according-to-vm-overcommit-policy +++ a/include/linux/mm.h @@ -206,6 +206,8 @@ int overcommit_ratio_handler(struct ctl_ loff_t *); int overcommit_kbytes_handler(struct ctl_table *, int, void *, size_t *, loff_t *); +int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *, + loff_t *); #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n)) --- a/kernel/sysctl.c~mm-adjust-vm_committed_as_batch-according-to-vm-overcommit-policy +++ a/kernel/sysctl.c @@ -2671,7 +2671,7 @@ static struct ctl_table vm_table[] = { .data = &sysctl_overcommit_memory, .maxlen = sizeof(sysctl_overcommit_memory), .mode = 0644, - .proc_handler = proc_dointvec_minmax, + .proc_handler = overcommit_policy_handler, .extra1 = SYSCTL_ZERO, .extra2 = &two, }, --- a/mm/mm_init.c~mm-adjust-vm_committed_as_batch-according-to-vm-overcommit-policy +++ a/mm/mm_init.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "internal.h" #ifdef CONFIG_DEBUG_MEMORY_INIT @@ -144,14 +145,23 @@ EXPORT_SYMBOL_GPL(mm_kobj); #ifdef CONFIG_SMP s32 vm_committed_as_batch = 32; -static void __meminit mm_compute_batch(void) +void mm_compute_batch(int overcommit_policy) { u64 memsized_batch; s32 nr = num_present_cpus(); s32 batch = max_t(s32, nr*2, 32); + unsigned long ram_pages = totalram_pages(); - /* batch size set to 0.4% of (total memory/#cpus), or max int32 */ - memsized_batch = min_t(u64, (totalram_pages()/nr)/256, 0x7fffffff); + /* + * For policy OVERCOMMIT_NEVER, set batch size to 0.4% of + * (total memory/#cpus), and lift it to 25% for other policies + * to easy the possible lock contention for percpu_counter + * vm_committed_as, while the max limit is INT_MAX + */ + if (overcommit_policy == OVERCOMMIT_NEVER) + memsized_batch = min_t(u64, ram_pages/nr/256, INT_MAX); + else + memsized_batch = min_t(u64, ram_pages/nr/4, INT_MAX); vm_committed_as_batch = max_t(s32, memsized_batch, batch); } @@ -162,7 +172,7 @@ static int __meminit mm_compute_batch_no switch (action) { case MEM_ONLINE: case MEM_OFFLINE: - mm_compute_batch(); + mm_compute_batch(sysctl_overcommit_memory); default: break; } @@ -176,7 +186,7 @@ static struct notifier_block compute_bat static int __init mm_compute_batch_init(void) { - mm_compute_batch(); + mm_compute_batch(sysctl_overcommit_memory); register_hotmemory_notifier(&compute_batch_nb); return 0; --- a/mm/util.c~mm-adjust-vm_committed_as_batch-according-to-vm-overcommit-policy +++ a/mm/util.c @@ -746,6 +746,47 @@ int overcommit_ratio_handler(struct ctl_ return ret; } +static void sync_overcommit_as(struct work_struct *dummy) +{ + percpu_counter_sync(&vm_committed_as); +} + +int overcommit_policy_handler(struct ctl_table *table, int write, void *buffer, + size_t *lenp, loff_t *ppos) +{ + struct ctl_table t; + int new_policy; + int ret; + + /* + * The deviation of sync_overcommit_as could be big with loose policy + * like OVERCOMMIT_ALWAYS/OVERCOMMIT_GUESS. When changing policy to + * strict OVERCOMMIT_NEVER, we need to reduce the deviation to comply + * with the strict "NEVER", and to avoid possible race condtion (even + * though user usually won't too frequently do the switching to policy + * OVERCOMMIT_NEVER), the switch is done in the following order: + * 1. changing the batch + * 2. sync percpu count on each CPU + * 3. switch the policy + */ + if (write) { + t = *table; + t.data = &new_policy; + ret = proc_dointvec_minmax(&t, write, buffer, lenp, ppos); + if (ret) + return ret; + + mm_compute_batch(new_policy); + if (new_policy == OVERCOMMIT_NEVER) + schedule_on_each_cpu(sync_overcommit_as); + sysctl_overcommit_memory = new_policy; + } else { + ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos); + } + + return ret; +} + int overcommit_kbytes_handler(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { From patchwork Fri Aug 7 06:23:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704889 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B61E138C for ; Fri, 7 Aug 2020 06:23:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EE4DF22C9F for ; Fri, 7 Aug 2020 06:23:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="r5YUKC5I" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EE4DF22C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C84008D0076; Fri, 7 Aug 2020 02:23:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C5A5A8D0026; Fri, 7 Aug 2020 02:23:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B98B58D0076; Fri, 7 Aug 2020 02:23:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id A499F8D0026 for ; Fri, 7 Aug 2020 02:23:23 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 693D12466 for ; Fri, 7 Aug 2020 06:23:23 +0000 (UTC) X-FDA: 77122780686.29.scene95_6313e2326fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 3C79E180868E6 for ; Fri, 7 Aug 2020 06:23:23 +0000 (UTC) X-Spam-Summary: 1,0,0,e13f9fa335ce6190,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:2:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1963:2196:2198:2199:2200:2393:2525:2559:2564:2682:2685:2693:2731:2859:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3167:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4043:4051:4321:4385:4605:5007:6119:6261:6653:6737:6738:7576:7875:8603:9025:9036:9545:10004:10913:11026:11473:11657:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13255:13846:14096:21063:21080:21433:21451:21627:21819:21939:21966:21990:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygn6y3mkhhr67diakdd3sktiqi1ocpg8g4gq1ou4yj1setg3pedkijatydjdn.f56t5aw6bm3kj31rdrktw89tt7i3ucxjp3ozpwob1bfci91uzp8zra5psju3iek.q-lbl8.mailshell.net-223.238.255.100,CacheIP :none,Ba X-HE-Tag: scene95_6313e2326fbe X-Filterd-Recvd-Size: 10922 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:22 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8553B221E5; Fri, 7 Aug 2020 06:23:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781401; bh=uvQwcFTIf4q/HJ34D7U3/XoAfS+lyL5UaaUr9QsjiuY=; h=Date:From:To:Subject:In-Reply-To:From; b=r5YUKC5IGVXB1tXaAtwWKMDjtjO449VXBKPKG6ogaiDT9tfxXrfyKk4YlO42/o8Gh uKaX3wsO9wA7ZURI3zKkk55ecwFrhpl/0eAG6J8CN+XYgs2zA9d3BJXaD64sEt/HF8 RWxlSNQ277VQd8RA9VqIj6SFOcdKCk66LSMXkCl4= Date: Thu, 06 Aug 2020 23:23:19 -0700 From: Andrew Morton To: akpm@linux-foundation.org, anshuman.khandual@arm.com, benh@kernel.crashing.org, bp@alien8.de, catalin.marinas@arm.com, corbet@lwn.net, dan.j.williams@intel.com, dave.hansen@linux.intel.com, david@redhat.com, fenghua.yu@intel.com, hpa@zytor.com, hsinyi@chromium.org, justin.he@arm.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, luto@kernel.org, mark.rutland@arm.com, mhocko@suse.com, mingo@redhat.com, mm-commits@vger.kernel.org, mpe@ellerman.id.au, palmer@dabbelt.com, pasha.tatashin@soleen.com, paul.walmsley@sifive.com, paulus@samba.org, peterz@infradead.org, robin.murphy@arm.com, rppt@linux.ibm.com, steve.capper@arm.com, tglx@linutronix.de, tony.luck@intel.com, torvalds@linux-foundation.org, will@kernel.org, willy@infradead.org, yuzhao@google.com Subject: [patch 109/163] mm/sparsemem: enable vmem_altmap support in vmemmap_populate_basepages() Message-ID: <20200807062319.DkSvcHBWW%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 3C79E180868E6 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Anshuman Khandual Subject: mm/sparsemem: enable vmem_altmap support in vmemmap_populate_basepages() Patch series "arm64: Enable vmemmap mapping from device memory", v4. This series enables vmemmap backing memory allocation from device memory ranges on arm64. But before that, it enables vmemmap_populate_basepages() and vmemmap_alloc_block_buf() to accommodate struct vmem_altmap based alocation requests. This patch (of 3): vmemmap_populate_basepages() is used across platforms to allocate backing memory for vmemmap mapping. This is used as a standard default choice or as a fallback when intended huge pages allocation fails. This just creates entire vmemmap mapping with base pages (PAGE_SIZE). On arm64 platforms, vmemmap_populate_basepages() is called instead of the platform specific vmemmap_populate() when ARM64_SWAPPER_USES_SECTION_MAPS is not enabled as in case for ARM64_16K_PAGES and ARM64_64K_PAGES configs. At present vmemmap_populate_basepages() does not support allocating from driver defined struct vmem_altmap while trying to create vmemmap mapping for a device memory range. It prevents ARM64_16K_PAGES and ARM64_64K_PAGES configs on arm64 from supporting device memory with vmemap_altmap request. This enables vmem_altmap support in vmemmap_populate_basepages() unlocking device memory allocation for vmemap mapping on arm64 platforms with 16K or 64K base page configs. Each architecture should evaluate and decide on subscribing device memory based base page allocation through vmemmap_populate_basepages(). Hence lets keep it disabled on all archs in order to preserve the existing semantics. A subsequent patch enables it on arm64. Link: http://lkml.kernel.org/r/1594004178-8861-1-git-send-email-anshuman.khandual@arm.com Link: http://lkml.kernel.org/r/1594004178-8861-2-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual Acked-by: Will Deacon Acked-by: Catalin Marinas Reviewed-by: David Hildenbrand Tested-by: Jia He Cc: Mark Rutland Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Tony Luck Cc: Fenghua Yu Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Mike Rapoport Cc: Michal Hocko Cc: "Matthew Wilcox (Oracle)" Cc: "Kirill A. Shutemov" Cc: Dan Williams Cc: Pavel Tatashin Cc: Benjamin Herrenschmidt Cc: Borislav Petkov Cc: "H. Peter Anvin" Cc: Hsin-Yi Wang Cc: Jonathan Corbet Cc: Michael Ellerman Cc: Paul Mackerras Cc: Robin Murphy Cc: Steve Capper Cc: Yu Zhao Signed-off-by: Andrew Morton --- arch/arm64/mm/mmu.c | 2 +- arch/ia64/mm/discontig.c | 2 +- arch/riscv/mm/init.c | 2 +- arch/x86/mm/init_64.c | 6 +++--- include/linux/mm.h | 5 +++-- mm/sparse-vmemmap.c | 16 +++++++++++----- 6 files changed, 20 insertions(+), 13 deletions(-) --- a/arch/arm64/mm/mmu.c~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_populate_basepages +++ a/arch/arm64/mm/mmu.c @@ -1070,7 +1070,7 @@ static void free_empty_tables(unsigned l int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { - return vmemmap_populate_basepages(start, end, node); + return vmemmap_populate_basepages(start, end, node, NULL); } #else /* !ARM64_SWAPPER_USES_SECTION_MAPS */ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, --- a/arch/ia64/mm/discontig.c~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_populate_basepages +++ a/arch/ia64/mm/discontig.c @@ -655,7 +655,7 @@ void arch_refresh_nodedata(int update_no int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { - return vmemmap_populate_basepages(start, end, node); + return vmemmap_populate_basepages(start, end, node, NULL); } void vmemmap_free(unsigned long start, unsigned long end, --- a/arch/riscv/mm/init.c~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_populate_basepages +++ a/arch/riscv/mm/init.c @@ -554,6 +554,6 @@ void __init paging_init(void) int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { - return vmemmap_populate_basepages(start, end, node); + return vmemmap_populate_basepages(start, end, node, NULL); } #endif --- a/arch/x86/mm/init_64.c~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_populate_basepages +++ a/arch/x86/mm/init_64.c @@ -1545,7 +1545,7 @@ static int __meminit vmemmap_populate_hu vmemmap_verify((pte_t *)pmd, node, addr, next); continue; } - if (vmemmap_populate_basepages(addr, next, node)) + if (vmemmap_populate_basepages(addr, next, node, NULL)) return -ENOMEM; } return 0; @@ -1557,7 +1557,7 @@ int __meminit vmemmap_populate(unsigned int err; if (end - start < PAGES_PER_SECTION * sizeof(struct page)) - err = vmemmap_populate_basepages(start, end, node); + err = vmemmap_populate_basepages(start, end, node, NULL); else if (boot_cpu_has(X86_FEATURE_PSE)) err = vmemmap_populate_hugepages(start, end, node, altmap); else if (altmap) { @@ -1565,7 +1565,7 @@ int __meminit vmemmap_populate(unsigned __func__); err = -ENOMEM; } else - err = vmemmap_populate_basepages(start, end, node); + err = vmemmap_populate_basepages(start, end, node, NULL); if (!err) sync_global_pgds(start, end - 1); return err; --- a/include/linux/mm.h~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_populate_basepages +++ a/include/linux/mm.h @@ -2978,14 +2978,15 @@ pgd_t *vmemmap_pgd_populate(unsigned lon p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); -pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node); +pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, + struct vmem_altmap *altmap); void *vmemmap_alloc_block(unsigned long size, int node); struct vmem_altmap; void *vmemmap_alloc_block_buf(unsigned long size, int node); void *altmap_alloc_block_buf(unsigned long size, struct vmem_altmap *altmap); void vmemmap_verify(pte_t *, int, unsigned long, unsigned long); int vmemmap_populate_basepages(unsigned long start, unsigned long end, - int node); + int node, struct vmem_altmap *altmap); int vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); void vmemmap_populate_print_last(void); --- a/mm/sparse-vmemmap.c~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_populate_basepages +++ a/mm/sparse-vmemmap.c @@ -139,12 +139,18 @@ void __meminit vmemmap_verify(pte_t *pte start, end - 1); } -pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node) +pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, + struct vmem_altmap *altmap) { pte_t *pte = pte_offset_kernel(pmd, addr); if (pte_none(*pte)) { pte_t entry; - void *p = vmemmap_alloc_block_buf(PAGE_SIZE, node); + void *p; + + if (altmap) + p = altmap_alloc_block_buf(PAGE_SIZE, altmap); + else + p = vmemmap_alloc_block_buf(PAGE_SIZE, node); if (!p) return NULL; entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); @@ -212,8 +218,8 @@ pgd_t * __meminit vmemmap_pgd_populate(u return pgd; } -int __meminit vmemmap_populate_basepages(unsigned long start, - unsigned long end, int node) +int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap) { unsigned long addr = start; pgd_t *pgd; @@ -235,7 +241,7 @@ int __meminit vmemmap_populate_basepages pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return -ENOMEM; - pte = vmemmap_pte_populate(pmd, addr, node); + pte = vmemmap_pte_populate(pmd, addr, node, altmap); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); From patchwork Fri Aug 7 06:23:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704891 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3120C14B7 for ; Fri, 7 Aug 2020 06:23:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E2A6A22C9F for ; Fri, 7 Aug 2020 06:23:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="KNSrLPmW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E2A6A22C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C0D908D0077; Fri, 7 Aug 2020 02:23:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B962C8D0026; Fri, 7 Aug 2020 02:23:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD4F58D0077; Fri, 7 Aug 2020 02:23:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0060.hostedemail.com [216.40.44.60]) by kanga.kvack.org (Postfix) with ESMTP id 9762B8D0026 for ; Fri, 7 Aug 2020 02:23:28 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 658BD82499A8 for ; Fri, 7 Aug 2020 06:23:28 +0000 (UTC) X-FDA: 77122780896.10.care64_2405e5026fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 287EA16A0DE for ; Fri, 7 Aug 2020 06:23:28 +0000 (UTC) X-Spam-Summary: 1,0,0,7ac5df27ee91bbc7,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1730:1747:1777:1792:1963:2393:2525:2553:2559:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3167:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4043:4050:4120:4321:4605:5007:6119:6261:6653:6737:6738:7576:7875:8603:8660:9025:9036:9545:10913:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13148:13161:13229:13230:13255:13846:14096:21080:21433:21451:21611:21627:21819:21939:21990:30054:30064:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygan3ow7uazxr4wpb89sboesh87yppp4mtyd7yr6mszw5t59izg3p9h4ciim3.smdn5iisn9ps9nokaifcaebfpa55c4z8p1stt6gqujhkene4au3o6n8k68erfj8.e-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesi an:0.5,0 X-HE-Tag: care64_2405e5026fbe X-Filterd-Recvd-Size: 9507 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:27 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1576E22CAF; Fri, 7 Aug 2020 06:23:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781406; bh=TQ5Ff4Ar0kztfOqZ4wnyCQsV6UFDRMfaNs+xp0sXu6k=; h=Date:From:To:Subject:In-Reply-To:From; b=KNSrLPmWoSPraKgP7hcEkgVacQM3XgZENAQeADvg8usCLyuKFDZ8ae1OO1y7+o9OM uSn5g3+qttrrgNudgfPmI4gHLWs4azlrOQztbE10s20BJMkFkJ+TYj0IyLvBtdRfvw m+ZfLSGZgGFBPtTg8ja6PjvMpnjbEKOaFzGRwo+4= Date: Thu, 06 Aug 2020 23:23:24 -0700 From: Andrew Morton To: akpm@linux-foundation.org, anshuman.khandual@arm.com, benh@kernel.crashing.org, bp@alien8.de, catalin.marinas@arm.com, corbet@lwn.net, dan.j.williams@intel.com, dave.hansen@linux.intel.com, david@redhat.com, fenghua.yu@intel.com, hpa@zytor.com, hsinyi@chromium.org, justin.he@arm.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, luto@kernel.org, mark.rutland@arm.com, mhocko@suse.com, mingo@redhat.com, mm-commits@vger.kernel.org, mpe@ellerman.id.au, palmer@dabbelt.com, pasha.tatashin@soleen.com, paul.walmsley@sifive.com, paulus@samba.org, peterz@infradead.org, robin.murphy@arm.com, rppt@linux.ibm.com, steve.capper@arm.com, tglx@linutronix.de, tony.luck@intel.com, torvalds@linux-foundation.org, will@kernel.org, willy@infradead.org, yuzhao@google.com Subject: [patch 110/163] mm/sparsemem: enable vmem_altmap support in vmemmap_alloc_block_buf() Message-ID: <20200807062324.zHDQnOlwz%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 287EA16A0DE X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Anshuman Khandual Subject: mm/sparsemem: enable vmem_altmap support in vmemmap_alloc_block_buf() There are many instances where vmemap allocation is often switched between regular memory and device memory just based on whether altmap is available or not. vmemmap_alloc_block_buf() is used in various platforms to allocate vmemmap mappings. Lets also enable it to handle altmap based device memory allocation along with existing regular memory allocations. This will help in avoiding the altmap based allocation switch in many places. To summarize there are two different methods to call vmemmap_alloc_block_buf(). vmemmap_alloc_block_buf(size, node, NULL) /* Allocate from system RAM */ vmemmap_alloc_block_buf(size, node, altmap) /* Allocate from altmap */ This converts altmap_alloc_block_buf() into a static function, drops it's entry from the header and updates Documentation/vm/memory-model.rst. Link: http://lkml.kernel.org/r/1594004178-8861-3-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual Suggested-by: Robin Murphy Tested-by: Jia He Reviewed-by: Catalin Marinas Cc: Jonathan Corbet Cc: Will Deacon Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: Michael Ellerman Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: "H. Peter Anvin" Cc: Dan Williams Cc: David Hildenbrand Cc: Fenghua Yu Cc: Hsin-Yi Wang Cc: "Kirill A. Shutemov" Cc: Mark Rutland Cc: "Matthew Wilcox (Oracle)" Cc: Michal Hocko Cc: Mike Rapoport Cc: Palmer Dabbelt Cc: Paul Walmsley Cc: Pavel Tatashin Cc: Steve Capper Cc: Tony Luck Cc: Yu Zhao Signed-off-by: Andrew Morton --- Documentation/vm/memory-model.rst | 2 +- arch/arm64/mm/mmu.c | 2 +- arch/powerpc/mm/init_64.c | 4 ++-- arch/x86/mm/init_64.c | 5 +---- include/linux/mm.h | 4 ++-- mm/sparse-vmemmap.c | 28 +++++++++++++--------------- 6 files changed, 20 insertions(+), 25 deletions(-) --- a/arch/arm64/mm/mmu.c~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_alloc_block_buf +++ a/arch/arm64/mm/mmu.c @@ -1102,7 +1102,7 @@ int __meminit vmemmap_populate(unsigned if (pmd_none(READ_ONCE(*pmdp))) { void *p = NULL; - p = vmemmap_alloc_block_buf(PMD_SIZE, node); + p = vmemmap_alloc_block_buf(PMD_SIZE, node, NULL); if (!p) return -ENOMEM; --- a/arch/powerpc/mm/init_64.c~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_alloc_block_buf +++ a/arch/powerpc/mm/init_64.c @@ -225,12 +225,12 @@ int __meminit vmemmap_populate(unsigned * fall back to system memory if the altmap allocation fail. */ if (altmap && !altmap_cross_boundary(altmap, start, page_size)) { - p = altmap_alloc_block_buf(page_size, altmap); + p = vmemmap_alloc_block_buf(page_size, node, altmap); if (!p) pr_debug("altmap block allocation failed, falling back to system memory"); } if (!p) - p = vmemmap_alloc_block_buf(page_size, node); + p = vmemmap_alloc_block_buf(page_size, node, NULL); if (!p) return -ENOMEM; --- a/arch/x86/mm/init_64.c~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_alloc_block_buf +++ a/arch/x86/mm/init_64.c @@ -1515,10 +1515,7 @@ static int __meminit vmemmap_populate_hu if (pmd_none(*pmd)) { void *p; - if (altmap) - p = altmap_alloc_block_buf(PMD_SIZE, altmap); - else - p = vmemmap_alloc_block_buf(PMD_SIZE, node); + p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); if (p) { pte_t entry; --- a/Documentation/vm/memory-model.rst~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_alloc_block_buf +++ a/Documentation/vm/memory-model.rst @@ -178,7 +178,7 @@ for persistent memory devices in pre-all devices. This storage is represented with :c:type:`struct vmem_altmap` that is eventually passed to vmemmap_populate() through a long chain of function calls. The vmemmap_populate() implementation may use the -`vmem_altmap` along with :c:func:`altmap_alloc_block_buf` helper to +`vmem_altmap` along with :c:func:`vmemmap_alloc_block_buf` helper to allocate memory map on the persistent memory device. ZONE_DEVICE --- a/include/linux/mm.h~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_alloc_block_buf +++ a/include/linux/mm.h @@ -2982,8 +2982,8 @@ pte_t *vmemmap_pte_populate(pmd_t *pmd, struct vmem_altmap *altmap); void *vmemmap_alloc_block(unsigned long size, int node); struct vmem_altmap; -void *vmemmap_alloc_block_buf(unsigned long size, int node); -void *altmap_alloc_block_buf(unsigned long size, struct vmem_altmap *altmap); +void *vmemmap_alloc_block_buf(unsigned long size, int node, + struct vmem_altmap *altmap); void vmemmap_verify(pte_t *, int, unsigned long, unsigned long); int vmemmap_populate_basepages(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); --- a/mm/sparse-vmemmap.c~mm-sparsemem-enable-vmem_altmap-support-in-vmemmap_alloc_block_buf +++ a/mm/sparse-vmemmap.c @@ -69,11 +69,19 @@ void * __meminit vmemmap_alloc_block(uns __pa(MAX_DMA_ADDRESS)); } +static void * __meminit altmap_alloc_block_buf(unsigned long size, + struct vmem_altmap *altmap); + /* need to make sure size is all the same during early stage */ -void * __meminit vmemmap_alloc_block_buf(unsigned long size, int node) +void * __meminit vmemmap_alloc_block_buf(unsigned long size, int node, + struct vmem_altmap *altmap) { - void *ptr = sparse_buffer_alloc(size); + void *ptr; + + if (altmap) + return altmap_alloc_block_buf(size, altmap); + ptr = sparse_buffer_alloc(size); if (!ptr) ptr = vmemmap_alloc_block(size, node); return ptr; @@ -94,15 +102,8 @@ static unsigned long __meminit vmem_altm return 0; } -/** - * altmap_alloc_block_buf - allocate pages from the device page map - * @altmap: device page map - * @size: size (in bytes) of the allocation - * - * Allocations are aligned to the size of the request. - */ -void * __meminit altmap_alloc_block_buf(unsigned long size, - struct vmem_altmap *altmap) +static void * __meminit altmap_alloc_block_buf(unsigned long size, + struct vmem_altmap *altmap) { unsigned long pfn, nr_pfns, nr_align; @@ -147,10 +148,7 @@ pte_t * __meminit vmemmap_pte_populate(p pte_t entry; void *p; - if (altmap) - p = altmap_alloc_block_buf(PAGE_SIZE, altmap); - else - p = vmemmap_alloc_block_buf(PAGE_SIZE, node); + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); if (!p) return NULL; entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); From patchwork Fri Aug 7 06:23:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704893 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0CB7916B1 for ; Fri, 7 Aug 2020 06:23:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C711F2177B for ; Fri, 7 Aug 2020 06:23:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="W4GaR8dd" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C711F2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 99C7A8D0078; Fri, 7 Aug 2020 02:23:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 94D568D0026; Fri, 7 Aug 2020 02:23:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8630F8D0078; Fri, 7 Aug 2020 02:23:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6FC4A8D0026 for ; Fri, 7 Aug 2020 02:23:33 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 39ACA180AD802 for ; Fri, 7 Aug 2020 06:23:33 +0000 (UTC) X-FDA: 77122781106.12.fact17_491515026fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 08E9018011E4B for ; Fri, 7 Aug 2020 06:23:33 +0000 (UTC) X-Spam-Summary: 1,0,0,1c51bc1db5c21412,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:2:41:69:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1730:1747:1777:1792:1963:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2693:2859:2890:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3167:3354:3865:3866:3867:3868:3870:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4042:4043:4051:4321:4385:5007:6117:6261:6653:6737:6738:7576:8603:8784:8957:9025:9036:9545:10004:10913:11026:11232:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13846:14096:21080:21433:21450:21451:21611:21627:21819:21939:21990:30001:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygchb6pmfqctqbf69m9au8waj39ypkmq3c9ccp3yp346f99zkd75s3d4ik774.wciey173opggcmmwoqpguacz99sbojwxyze4khga79aoqjhfy94g3oifjj8x8s6.4-lbl8.mailshell.net-223.238.255.100 ,CacheIP X-HE-Tag: fact17_491515026fbe X-Filterd-Recvd-Size: 10078 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:32 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1D19B22D08; Fri, 7 Aug 2020 06:23:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781411; bh=BYgXQkk0YpzEpZs+H3DKrPqJ+d4D9s9TZ2WNsBZkKLc=; h=Date:From:To:Subject:In-Reply-To:From; b=W4GaR8ddiWZDruadX5ttkBapGc0cNgg/r1aSsJi5AA+MbQWj69HHWqBi0/CY42/X7 Kgrgdx/JhCSGCH8WO4sbFh4yn/A4paKw7UMa85aSIxyh25uyGVskCP0BYzS/owGkV9 UcQKNUNLlP8GoK+m5ixI9CK6p5TAKuCovDr613pg= Date: Thu, 06 Aug 2020 23:23:29 -0700 From: Andrew Morton To: akpm@linux-foundation.org, anshuman.khandual@arm.com, benh@kernel.crashing.org, bp@alien8.de, catalin.marinas@arm.com, corbet@lwn.net, dan.j.williams@intel.com, dave.hansen@linux.intel.com, david@redhat.com, fenghua.yu@intel.com, hpa@zytor.com, hsinyi@chromium.org, justin.he@arm.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, luto@kernel.org, mark.rutland@arm.com, mhocko@suse.com, mingo@redhat.com, mm-commits@vger.kernel.org, mpe@ellerman.id.au, palmer@dabbelt.com, pasha.tatashin@soleen.com, paul.walmsley@sifive.com, paulus@samba.org, peterz@infradead.org, robin.murphy@arm.com, rppt@linux.ibm.com, steve.capper@arm.com, tglx@linutronix.de, tony.luck@intel.com, torvalds@linux-foundation.org, will@kernel.org, willy@infradead.org, yuzhao@google.com Subject: [patch 111/163] arm64/mm: enable vmem_altmap support for vmemmap mappings Message-ID: <20200807062329.wUIkG4Im_%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 08E9018011E4B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Anshuman Khandual Subject: arm64/mm: enable vmem_altmap support for vmemmap mappings Device memory ranges when getting hot added into ZONE_DEVICE, might require their vmemmap mapping's backing memory to be allocated from their own range instead of consuming system memory. This prevents large system memory usage for potentially large device memory ranges. Device driver communicates this request via vmem_altmap structure. Architecture needs to take this request into account while creating and tearing down vemmmap mappings. This enables vmem_altmap support in vmemmap_populate() and vmemmap_free() which includes vmemmap_populate_basepages() used for ARM64_16K_PAGES and ARM64_64K_PAGES configs. Link: http://lkml.kernel.org/r/1594004178-8861-4-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual Reviewed-by: Catalin Marinas Tested-by: Jia He Cc: Will Deacon Cc: Mark Rutland Cc: Steve Capper Cc: David Hildenbrand Cc: Yu Zhao Cc: Hsin-Yi Wang Cc: Thomas Gleixner Cc: Andy Lutomirski Cc: Benjamin Herrenschmidt Cc: Borislav Petkov Cc: Dan Williams Cc: Dave Hansen Cc: Fenghua Yu Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Jonathan Corbet Cc: "Kirill A. Shutemov" Cc: "Matthew Wilcox (Oracle)" Cc: Michael Ellerman Cc: Michal Hocko Cc: Mike Rapoport Cc: Palmer Dabbelt Cc: Paul Mackerras Cc: Paul Walmsley Cc: Pavel Tatashin Cc: Peter Zijlstra Cc: Robin Murphy Cc: Tony Luck Signed-off-by: Andrew Morton --- arch/arm64/mm/mmu.c | 58 +++++++++++++++++++++++++++--------------- 1 file changed, 38 insertions(+), 20 deletions(-) --- a/arch/arm64/mm/mmu.c~arm64-mm-enable-vmem_altmap-support-for-vmemmap-mappings +++ a/arch/arm64/mm/mmu.c @@ -761,15 +761,20 @@ int kern_addr_valid(unsigned long addr) } #ifdef CONFIG_MEMORY_HOTPLUG -static void free_hotplug_page_range(struct page *page, size_t size) +static void free_hotplug_page_range(struct page *page, size_t size, + struct vmem_altmap *altmap) { - WARN_ON(PageReserved(page)); - free_pages((unsigned long)page_address(page), get_order(size)); + if (altmap) { + vmem_altmap_free(altmap, size >> PAGE_SHIFT); + } else { + WARN_ON(PageReserved(page)); + free_pages((unsigned long)page_address(page), get_order(size)); + } } static void free_hotplug_pgtable_page(struct page *page) { - free_hotplug_page_range(page, PAGE_SIZE); + free_hotplug_page_range(page, PAGE_SIZE, NULL); } static bool pgtable_range_aligned(unsigned long start, unsigned long end, @@ -792,7 +797,8 @@ static bool pgtable_range_aligned(unsign } static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, - unsigned long end, bool free_mapped) + unsigned long end, bool free_mapped, + struct vmem_altmap *altmap) { pte_t *ptep, pte; @@ -806,12 +812,14 @@ static void unmap_hotplug_pte_range(pmd_ pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); if (free_mapped) - free_hotplug_page_range(pte_page(pte), PAGE_SIZE); + free_hotplug_page_range(pte_page(pte), + PAGE_SIZE, altmap); } while (addr += PAGE_SIZE, addr < end); } static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr, - unsigned long end, bool free_mapped) + unsigned long end, bool free_mapped, + struct vmem_altmap *altmap) { unsigned long next; pmd_t *pmdp, pmd; @@ -834,16 +842,17 @@ static void unmap_hotplug_pmd_range(pud_ flush_tlb_kernel_range(addr, addr + PAGE_SIZE); if (free_mapped) free_hotplug_page_range(pmd_page(pmd), - PMD_SIZE); + PMD_SIZE, altmap); continue; } WARN_ON(!pmd_table(pmd)); - unmap_hotplug_pte_range(pmdp, addr, next, free_mapped); + unmap_hotplug_pte_range(pmdp, addr, next, free_mapped, altmap); } while (addr = next, addr < end); } static void unmap_hotplug_pud_range(p4d_t *p4dp, unsigned long addr, - unsigned long end, bool free_mapped) + unsigned long end, bool free_mapped, + struct vmem_altmap *altmap) { unsigned long next; pud_t *pudp, pud; @@ -866,16 +875,17 @@ static void unmap_hotplug_pud_range(p4d_ flush_tlb_kernel_range(addr, addr + PAGE_SIZE); if (free_mapped) free_hotplug_page_range(pud_page(pud), - PUD_SIZE); + PUD_SIZE, altmap); continue; } WARN_ON(!pud_table(pud)); - unmap_hotplug_pmd_range(pudp, addr, next, free_mapped); + unmap_hotplug_pmd_range(pudp, addr, next, free_mapped, altmap); } while (addr = next, addr < end); } static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsigned long addr, - unsigned long end, bool free_mapped) + unsigned long end, bool free_mapped, + struct vmem_altmap *altmap) { unsigned long next; p4d_t *p4dp, p4d; @@ -888,16 +898,24 @@ static void unmap_hotplug_p4d_range(pgd_ continue; WARN_ON(!p4d_present(p4d)); - unmap_hotplug_pud_range(p4dp, addr, next, free_mapped); + unmap_hotplug_pud_range(p4dp, addr, next, free_mapped, altmap); } while (addr = next, addr < end); } static void unmap_hotplug_range(unsigned long addr, unsigned long end, - bool free_mapped) + bool free_mapped, struct vmem_altmap *altmap) { unsigned long next; pgd_t *pgdp, pgd; + /* + * altmap can only be used as vmemmap mapping backing memory. + * In case the backing memory itself is not being freed, then + * altmap is irrelevant. Warn about this inconsistency when + * encountered. + */ + WARN_ON(!free_mapped && altmap); + do { next = pgd_addr_end(addr, end); pgdp = pgd_offset_k(addr); @@ -906,7 +924,7 @@ static void unmap_hotplug_range(unsigned continue; WARN_ON(!pgd_present(pgd)); - unmap_hotplug_p4d_range(pgdp, addr, next, free_mapped); + unmap_hotplug_p4d_range(pgdp, addr, next, free_mapped, altmap); } while (addr = next, addr < end); } @@ -1070,7 +1088,7 @@ static void free_empty_tables(unsigned l int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { - return vmemmap_populate_basepages(start, end, node, NULL); + return vmemmap_populate_basepages(start, end, node, altmap); } #else /* !ARM64_SWAPPER_USES_SECTION_MAPS */ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, @@ -1102,7 +1120,7 @@ int __meminit vmemmap_populate(unsigned if (pmd_none(READ_ONCE(*pmdp))) { void *p = NULL; - p = vmemmap_alloc_block_buf(PMD_SIZE, node, NULL); + p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); if (!p) return -ENOMEM; @@ -1120,7 +1138,7 @@ void vmemmap_free(unsigned long start, u #ifdef CONFIG_MEMORY_HOTPLUG WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END)); - unmap_hotplug_range(start, end, true); + unmap_hotplug_range(start, end, true, altmap); free_empty_tables(start, end, VMEMMAP_START, VMEMMAP_END); #endif } @@ -1411,7 +1429,7 @@ static void __remove_pgd_mapping(pgd_t * WARN_ON(pgdir != init_mm.pgd); WARN_ON((start < PAGE_OFFSET) || (end > PAGE_END)); - unmap_hotplug_range(start, end, false); + unmap_hotplug_range(start, end, false, NULL); free_empty_tables(start, end, PAGE_OFFSET, PAGE_END); } From patchwork Fri Aug 7 06:23:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704895 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1C2FC138C for ; Fri, 7 Aug 2020 06:23:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DEA9B221E5 for ; Fri, 7 Aug 2020 06:23:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="RUmvebJM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DEA9B221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6927B8D0079; Fri, 7 Aug 2020 02:23:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6429F8D0026; Fri, 7 Aug 2020 02:23:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52E1A8D0079; Fri, 7 Aug 2020 02:23:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0227.hostedemail.com [216.40.44.227]) by kanga.kvack.org (Postfix) with ESMTP id 3E5488D0026 for ; Fri, 7 Aug 2020 02:23:36 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0947F180AD804 for ; Fri, 7 Aug 2020 06:23:36 +0000 (UTC) X-FDA: 77122781232.27.smile48_490aed926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id D85313D663 for ; Fri, 7 Aug 2020 06:23:35 +0000 (UTC) X-Spam-Summary: 1,0,0,80d6801fb6653aec,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2196:2199:2393:2525:2553:2559:2563:2682:2685:2859:2895:2902:2914:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6119:6261:6653:7576:7903:7974:9025:9391:9545:10004:10913:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13229:13846:14096:14181:14721:14849:21080:21212:21324:21325:21433:21451:21627:21796:21819:21939:30006:30036:30054:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yg4qiz57as9yntaurewbww6yxbzypoe5m7ftq7qdi5p88bjaryjspb4xrc4ca.auezruadpfbofq7ndrhfy1uu89og9hito45nk5yifz789zdck5ez9jegapzibw3.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Ba yesian:0 X-HE-Tag: smile48_490aed926fbe X-Filterd-Recvd-Size: 3728 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:35 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8240F22CB3; Fri, 7 Aug 2020 06:23:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781414; bh=7M0jg7yholAju8SOMhmmiV8eGyURQU75l//ZPH4NJrE=; h=Date:From:To:Subject:In-Reply-To:From; b=RUmvebJM2Y1REtpnY/HVM2ZsMsg3zYJjBo7SDDw4eVr8vTrzVUxQ1Sa4WaT9gLg2E XT3ERrhYapRzTXH2XRT2cSprPzFG5m5SDQ8IWoZtcYEHo6tVkSKG3nIzbodQj5up5j wI11fUgO9TimBDCfDogA3d3Fubk9PCBJruoPXPp4= Date: Thu, 06 Aug 2020 23:23:34 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linmiaohe@huawei.com, linux-mm@kvack.org, louhongxiang@huawei.com, mm-commits@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 112/163] mm: mmap: merge vma after call_mmap() if possible Message-ID: <20200807062334.ItecIkHkO%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: D85313D663 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Miaohe Lin Subject: mm: mmap: merge vma after call_mmap() if possible The vm_flags may be changed after call_mmap() because drivers may set some flags for their own purpose. As a result, we failed to merge the adjacent vma due to the different vm_flags as userspace can't pass in the same one. Try to merge vma after call_mmap() to fix this issue. Link: http://lkml.kernel.org/r/1594954065-23733-1-git-send-email-linmiaohe@huawei.com Signed-off-by: Hongxiang Lou Signed-off-by: Miaohe Lin Reviewed-by: Andrew Morton Signed-off-by: Andrew Morton --- mm/mmap.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) --- a/mm/mmap.c~mm-mmap-merge-vma-after-call_mmap-if-possible +++ a/mm/mmap.c @@ -1690,7 +1690,7 @@ unsigned long mmap_region(struct file *f struct list_head *uf) { struct mm_struct *mm = current->mm; - struct vm_area_struct *vma, *prev; + struct vm_area_struct *vma, *prev, *merge; int error; struct rb_node **rb_link, *rb_parent; unsigned long charged = 0; @@ -1774,6 +1774,25 @@ unsigned long mmap_region(struct file *f if (error) goto unmap_and_free_vma; + /* If vm_flags changed after call_mmap(), we should try merge vma again + * as we may succeed this time. + */ + if (unlikely(vm_flags != vma->vm_flags && prev)) { + merge = vma_merge(mm, prev, vma->vm_start, vma->vm_end, vma->vm_flags, + NULL, vma->vm_file, vma->vm_pgoff, NULL, NULL_VM_UFFD_CTX); + if (merge) { + fput(file); + vm_area_free(vma); + vma = merge; + /* Update vm_flags and possible addr to pick up the change. We don't + * warn here if addr changed as the vma is not linked by vma_link(). + */ + addr = vma->vm_start; + vm_flags = vma->vm_flags; + goto unmap_writable; + } + } + /* Can addr have changed?? * * Answer: Yes, several device drivers can do it in their @@ -1796,6 +1815,7 @@ unsigned long mmap_region(struct file *f vma_link(mm, vma, prev, rb_link, rb_parent); /* Once vma denies write, undo our temporary denial count */ if (file) { +unmap_writable: if (vm_flags & VM_SHARED) mapping_unmap_writable(file->f_mapping); if (vm_flags & VM_DENYWRITE) From patchwork Fri Aug 7 06:23:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704897 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C33914B7 for ; Fri, 7 Aug 2020 06:23:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40B0B2177B for ; Fri, 7 Aug 2020 06:23:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="kehVFLz7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40B0B2177B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D94F98D007A; Fri, 7 Aug 2020 02:23:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D6B338D0026; Fri, 7 Aug 2020 02:23:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA8D08D007A; Fri, 7 Aug 2020 02:23:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0175.hostedemail.com [216.40.44.175]) by kanga.kvack.org (Postfix) with ESMTP id B449C8D0026 for ; Fri, 7 Aug 2020 02:23:39 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 78D2B1F0A for ; Fri, 7 Aug 2020 06:23:39 +0000 (UTC) X-FDA: 77122781358.19.spot29_5a11f4e26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 4EE0A1AD1B5 for ; Fri, 7 Aug 2020 06:23:39 +0000 (UTC) X-Spam-Summary: 1,0,0,8fb1576475526668,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:2:41:69:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:2198:2199:2393:2525:2553:2559:2563:2682:2685:2693:2859:2901:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4052:4250:4321:4423:4605:5007:6119:6261:6653:7576:7875:7903:8599:8603:8660:9025:9545:9592:10004:10913:11026:11233:11473:11658:11914:12043:12048:12295:12296:12297:12438:12517:12519:12555:12679:12783:12986:13148:13161:13229:13230:13846:21080:21451:21627:21939:21990:30003:30012:30051:30054:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygas6qpc7r3qq48b5357soxsjziocc6ba59jzaftutk8jh98a57a181dsk6tw.3dksqerys6gqicg77s9j18rs69yuwnzaop67ktq9rk68zbkuyotm3qxn91zq1zq.r-lbl8.mailshell.net-223.238.255.100,CacheIP: none,Bay X-HE-Tag: spot29_5a11f4e26fbe X-Filterd-Recvd-Size: 10874 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:38 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A1EAE22D3E; Fri, 7 Aug 2020 06:23:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781418; bh=J3ne5N9GlgjmKN8hmoTrnuGRHod6Y/jhhnSOVJ2HH1M=; h=Date:From:To:Subject:In-Reply-To:From; b=kehVFLz7nzsfE6zz/PWM0rWPMHSFO2/vjSzN5dmtDHbwFhB4DRxaMCCyBaScAVrfN zUCyDVA0ZMh8WBvP/v7TO22aWQi7zuMG1v8kSf/ViRyr4ScBlxwEoY868kiGhrEXyN 7RV3pkh+9+Nt7ojlzsKqxBYFdLjguA/FNOfWjfvo= Date: Thu, 06 Aug 2020 23:23:37 -0700 From: Andrew Morton To: akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, pcc@google.com, torvalds@linux-foundation.org Subject: [patch 113/163] mm: remove unnecessary wrapper function do_mmap_pgoff() Message-ID: <20200807062337.sbenOoxBu%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 4EE0A1AD1B5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Peter Collingbourne Subject: mm: remove unnecessary wrapper function do_mmap_pgoff() The current split between do_mmap() and do_mmap_pgoff() was introduced in commit 1fcfd8db7f82 ("mm, mpx: add "vm_flags_t vm_flags" arg to do_mmap_pgoff()") to support MPX. The wrapper function do_mmap_pgoff() always passed 0 as the value of the vm_flags argument to do_mmap(). However, MPX support has subsequently been removed from the kernel and there were no more direct callers of do_mmap(); all calls were going via do_mmap_pgoff(). Simplify the code by removing do_mmap_pgoff() and changing all callers to directly call do_mmap(), which now no longer takes a vm_flags argument. Link: http://lkml.kernel.org/r/20200727194109.1371462-1-pcc@google.com Signed-off-by: Peter Collingbourne Reviewed-by: Andrew Morton Reviewed-by: David Hildenbrand Signed-off-by: Andrew Morton --- fs/aio.c | 6 +++--- fs/hugetlbfs/inode.c | 2 +- include/linux/fs.h | 2 +- include/linux/mm.h | 12 +----------- ipc/shm.c | 2 +- mm/mmap.c | 16 ++++++++-------- mm/nommu.c | 6 +++--- mm/shmem.c | 2 +- mm/util.c | 4 ++-- 9 files changed, 21 insertions(+), 31 deletions(-) --- a/fs/aio.c~mm-remove-unnecessary-wrapper-function-do_mmap_pgoff +++ a/fs/aio.c @@ -525,9 +525,9 @@ static int aio_setup_ring(struct kioctx return -EINTR; } - ctx->mmap_base = do_mmap_pgoff(ctx->aio_ring_file, 0, ctx->mmap_size, - PROT_READ | PROT_WRITE, - MAP_SHARED, 0, &unused, NULL); + ctx->mmap_base = do_mmap(ctx->aio_ring_file, 0, ctx->mmap_size, + PROT_READ | PROT_WRITE, + MAP_SHARED, 0, &unused, NULL); mmap_write_unlock(mm); if (IS_ERR((void *)ctx->mmap_base)) { ctx->mmap_size = 0; --- a/fs/hugetlbfs/inode.c~mm-remove-unnecessary-wrapper-function-do_mmap_pgoff +++ a/fs/hugetlbfs/inode.c @@ -140,7 +140,7 @@ static int hugetlbfs_file_mmap(struct fi * already been checked by prepare_hugepage_range. If you add * any error returns here, do so after setting VM_HUGETLB, so * is_vm_hugetlb_page tests below unmap_region go the right - * way when do_mmap_pgoff unwinds (may be important on powerpc + * way when do_mmap unwinds (may be important on powerpc * and ia64). */ vma->vm_flags |= VM_HUGETLB | VM_DONTEXPAND; --- a/include/linux/fs.h~mm-remove-unnecessary-wrapper-function-do_mmap_pgoff +++ a/include/linux/fs.h @@ -528,7 +528,7 @@ static inline int mapping_mapped(struct /* * Might pages of this file have been modified in userspace? - * Note that i_mmap_writable counts all VM_SHARED vmas: do_mmap_pgoff + * Note that i_mmap_writable counts all VM_SHARED vmas: do_mmap * marks vma as VM_SHARED if it is shared, and the file was opened for * writing i.e. vma may be mprotected writable even if now readonly. * --- a/include/linux/mm.h~mm-remove-unnecessary-wrapper-function-do_mmap_pgoff +++ a/include/linux/mm.h @@ -2546,23 +2546,13 @@ extern unsigned long mmap_region(struct struct list_head *uf); extern unsigned long do_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, unsigned long flags, - vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, - struct list_head *uf); + unsigned long pgoff, unsigned long *populate, struct list_head *uf); extern int __do_munmap(struct mm_struct *, unsigned long, size_t, struct list_head *uf, bool downgrade); extern int do_munmap(struct mm_struct *, unsigned long, size_t, struct list_head *uf); extern int do_madvise(unsigned long start, size_t len_in, int behavior); -static inline unsigned long -do_mmap_pgoff(struct file *file, unsigned long addr, - unsigned long len, unsigned long prot, unsigned long flags, - unsigned long pgoff, unsigned long *populate, - struct list_head *uf) -{ - return do_mmap(file, addr, len, prot, flags, 0, pgoff, populate, uf); -} - #ifdef CONFIG_MMU extern int __mm_populate(unsigned long addr, unsigned long len, int ignore_errors); --- a/ipc/shm.c~mm-remove-unnecessary-wrapper-function-do_mmap_pgoff +++ a/ipc/shm.c @@ -1558,7 +1558,7 @@ long do_shmat(int shmid, char __user *sh goto invalid; } - addr = do_mmap_pgoff(file, addr, size, prot, flags, 0, &populate, NULL); + addr = do_mmap(file, addr, size, prot, flags, 0, &populate, NULL); *raddr = addr; err = 0; if (IS_ERR_VALUE(addr)) --- a/mm/mmap.c~mm-remove-unnecessary-wrapper-function-do_mmap_pgoff +++ a/mm/mmap.c @@ -1030,7 +1030,7 @@ static inline int is_mergeable_anon_vma( * anon_vmas, nor if same anon_vma is assigned but offsets incompatible. * * We don't check here for the merged mmap wrapping around the end of pagecache - * indices (16TB on ia32) because do_mmap_pgoff() does not permit mmap's which + * indices (16TB on ia32) because do_mmap() does not permit mmap's which * wrap, nor mmaps which cover the final page at index -1UL. */ static int @@ -1365,11 +1365,11 @@ static inline bool file_mmap_ok(struct f */ unsigned long do_mmap(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, - unsigned long flags, vm_flags_t vm_flags, - unsigned long pgoff, unsigned long *populate, - struct list_head *uf) + unsigned long flags, unsigned long pgoff, + unsigned long *populate, struct list_head *uf) { struct mm_struct *mm = current->mm; + vm_flags_t vm_flags; int pkey = 0; *populate = 0; @@ -1431,7 +1431,7 @@ unsigned long do_mmap(struct file *file, * to. we assume access permissions have been handled by the open * of the memory object, so we don't do any here. */ - vm_flags |= calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) | + vm_flags = calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) | mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC; if (flags & MAP_LOCKED) @@ -2230,7 +2230,7 @@ get_unmapped_area(struct file *file, uns /* * mmap_region() will call shmem_zero_setup() to create a file, * so use shmem's get_unmapped_area in case it can be huge. - * do_mmap_pgoff() will clear pgoff, so match alignment. + * do_mmap() will clear pgoff, so match alignment. */ pgoff = 0; get_area = shmem_get_unmapped_area; @@ -3003,7 +3003,7 @@ SYSCALL_DEFINE5(remap_file_pages, unsign } file = get_file(vma->vm_file); - ret = do_mmap_pgoff(vma->vm_file, start, size, + ret = do_mmap(vma->vm_file, start, size, prot, flags, pgoff, &populate, NULL); fput(file); out: @@ -3223,7 +3223,7 @@ int insert_vm_struct(struct mm_struct *m * By setting it to reflect the virtual start address of the * vma, merges and splits can happen in a seamless way, just * using the existing file pgoff checks and manipulations. - * Similarly in do_mmap_pgoff and in do_brk. + * Similarly in do_mmap and in do_brk. */ if (vma_is_anonymous(vma)) { BUG_ON(vma->anon_vma); --- a/mm/nommu.c~mm-remove-unnecessary-wrapper-function-do_mmap_pgoff +++ a/mm/nommu.c @@ -1078,7 +1078,6 @@ unsigned long do_mmap(struct file *file, unsigned long len, unsigned long prot, unsigned long flags, - vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, struct list_head *uf) @@ -1086,6 +1085,7 @@ unsigned long do_mmap(struct file *file, struct vm_area_struct *vma; struct vm_region *region; struct rb_node *rb; + vm_flags_t vm_flags; unsigned long capabilities, result; int ret; @@ -1104,7 +1104,7 @@ unsigned long do_mmap(struct file *file, /* we've determined that we can make the mapping, now translate what we * now know into VMA flags */ - vm_flags |= determine_vm_flags(file, prot, flags, capabilities); + vm_flags = determine_vm_flags(file, prot, flags, capabilities); /* we're going to need to record the mapping */ region = kmem_cache_zalloc(vm_region_jar, GFP_KERNEL); @@ -1763,7 +1763,7 @@ EXPORT_SYMBOL_GPL(access_process_vm); * * Check the shared mappings on an inode on behalf of a shrinking truncate to * make sure that that any outstanding VMAs aren't broken and then shrink the - * vm_regions that extend that beyond so that do_mmap_pgoff() doesn't + * vm_regions that extend that beyond so that do_mmap() doesn't * automatically grant mappings that are too large. */ int nommu_shrink_inode_mappings(struct inode *inode, size_t size, --- a/mm/shmem.c~mm-remove-unnecessary-wrapper-function-do_mmap_pgoff +++ a/mm/shmem.c @@ -4245,7 +4245,7 @@ EXPORT_SYMBOL_GPL(shmem_file_setup_with_ /** * shmem_zero_setup - setup a shared anonymous mapping - * @vma: the vma to be mmapped is prepared by do_mmap_pgoff + * @vma: the vma to be mmapped is prepared by do_mmap */ int shmem_zero_setup(struct vm_area_struct *vma) { --- a/mm/util.c~mm-remove-unnecessary-wrapper-function-do_mmap_pgoff +++ a/mm/util.c @@ -503,8 +503,8 @@ unsigned long vm_mmap_pgoff(struct file if (!ret) { if (mmap_write_lock_killable(mm)) return -EINTR; - ret = do_mmap_pgoff(file, addr, len, prot, flag, pgoff, - &populate, &uf); + ret = do_mmap(file, addr, len, prot, flag, pgoff, &populate, + &uf); mmap_write_unlock(mm); userfaultfd_unmap_complete(mm, &uf); if (populate) From patchwork Fri Aug 7 06:23:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704899 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 18B3F14B7 for ; Fri, 7 Aug 2020 06:23:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D9C1322D03 for ; Fri, 7 Aug 2020 06:23:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="AKEk6zTG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D9C1322D03 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8DA2E8D007B; Fri, 7 Aug 2020 02:23:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 889898D0026; Fri, 7 Aug 2020 02:23:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79F9C8D007B; Fri, 7 Aug 2020 02:23:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0039.hostedemail.com [216.40.44.39]) by kanga.kvack.org (Postfix) with ESMTP id 652FE8D0026 for ; Fri, 7 Aug 2020 02:23:43 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2A76D180AD804 for ; Fri, 7 Aug 2020 06:23:43 +0000 (UTC) X-FDA: 77122781526.10.bears48_2d07fdc26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id F22B316A0BE for ; Fri, 7 Aug 2020 06:23:42 +0000 (UTC) X-Spam-Summary: 1,0,0,2de5873820c53fa7,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1544:1711:1730:1747:1777:1792:1963:2194:2198:2199:2200:2393:2525:2559:2566:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3355:3503:3504:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4117:4250:4321:4605:5007:6119:6261:6653:6737:7514:7576:7807:7903:8599:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13229:13255:14181:14721:14849:21060:21080:21450:21451:21627:21939:21990:30012:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yg6knyw5x7aq8f7a6tmnsea7z6hopeota6zrwqytrw5bkca4zgnjbrurctrbu.8y1yghyu4tu6d7efnfyar9q3t8zqfx7i8z8hdj8qodewob3y3be37tg8yj394k3.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesia n:0.5,0. X-HE-Tag: bears48_2d07fdc26fbe X-Filterd-Recvd-Size: 6625 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:42 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0B714221E5; Fri, 7 Aug 2020 06:23:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781421; bh=pKp42157LVeOknNBd7LiaFPgBvnjnDzy1vxKfs6KgOI=; h=Date:From:To:Subject:In-Reply-To:From; b=AKEk6zTGJY6nCeZ95yvOiLyta2cG6YTWT7oPin0yY9Eo5/ToDzsoF1jiaFpCRn7YJ Hri51jXjWPFKneplf3XkLt/aBP30ZgzJ4zbRE0RxwCzhxF+V4VxPgxjqrN8ZIlmYRt lq1QDT+XZiaFvj1U2RSCGPnaPy5iaMSZXDM4XABY= Date: Thu, 06 Aug 2020 23:23:40 -0700 From: Andrew Morton To: akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com, anshuman.khandual@arm.com, digetx@gmail.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, peterx@redhat.com, richard.weiyang@linux.alibaba.com, sean.j.christopherson@intel.com, thellstrom@vmware.com, thomas_os@shipmail.org, torvalds@linux-foundation.org, vbabka@suse.cz, willy@infradead.org, yang.shi@linux.alibaba.com Subject: [patch 114/163] mm/mremap: it is sure to have enough space when extent meets requirement Message-ID: <20200807062340.n7YybPMLI%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: F22B316A0BE X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Wei Yang Subject: mm/mremap: it is sure to have enough space when extent meets requirement Patch series "mm/mremap: cleanup move_page_tables() a little", v5. move_page_tables() tries to move page table by PMD or PTE. The root reason is if it tries to move PMD, both old and new range should be PMD aligned. But current code calculate old range and new range separately. This leads to some redundant check and calculation. This cleanup tries to consolidate the range check in one place to reduce some extra range handling. This patch (of 3): old_end is passed to these two functions to check whether there is enough space to do the move, while this check is done before invoking these functions. These two functions only would be invoked when extent meets the requirement and there is one check before invoking these functions: if (extent > old_end - old_addr) extent = old_end - old_addr; This implies (old_end - old_addr) won't fail the check in these two functions. Link: http://lkml.kernel.org/r/20200710092835.56368-1-richard.weiyang@linux.alibaba.com Link: http://lkml.kernel.org/r/20200710092835.56368-2-richard.weiyang@linux.alibaba.com Link: http://lkml.kernel.org/r/20200708095028.41706-1-richard.weiyang@linux.alibaba.com Link: http://lkml.kernel.org/r/20200708095028.41706-2-richard.weiyang@linux.alibaba.com Signed-off-by: Wei Yang Tested-by: Dmitry Osipenko Acked-by: Kirill A. Shutemov Cc: Vlastimil Babka Cc: Yang Shi Cc: Thomas Hellstrom (VMware) Cc: Anshuman Khandual Cc: Sean Christopherson Cc: Wei Yang Cc: Peter Xu Cc: Aneesh Kumar K.V Cc: Matthew Wilcox Cc: Thomas Hellstrom Signed-off-by: Andrew Morton --- include/linux/huge_mm.h | 2 +- mm/huge_memory.c | 7 ++----- mm/mremap.c | 10 ++++------ 3 files changed, 7 insertions(+), 12 deletions(-) --- a/include/linux/huge_mm.h~mm-mremap-it-is-sure-to-have-enough-space-when-extent-meets-requirement +++ a/include/linux/huge_mm.h @@ -42,7 +42,7 @@ extern int mincore_huge_pmd(struct vm_ar unsigned long addr, unsigned long end, unsigned char *vec); extern bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, - unsigned long new_addr, unsigned long old_end, + unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd); extern int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, pgprot_t newprot, --- a/mm/huge_memory.c~mm-mremap-it-is-sure-to-have-enough-space-when-extent-meets-requirement +++ a/mm/huge_memory.c @@ -1722,17 +1722,14 @@ static pmd_t move_soft_dirty_pmd(pmd_t p } bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, - unsigned long new_addr, unsigned long old_end, - pmd_t *old_pmd, pmd_t *new_pmd) + unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd) { spinlock_t *old_ptl, *new_ptl; pmd_t pmd; struct mm_struct *mm = vma->vm_mm; bool force_flush = false; - if ((old_addr & ~HPAGE_PMD_MASK) || - (new_addr & ~HPAGE_PMD_MASK) || - old_end - old_addr < HPAGE_PMD_SIZE) + if ((old_addr & ~HPAGE_PMD_MASK) || (new_addr & ~HPAGE_PMD_MASK)) return false; /* --- a/mm/mremap.c~mm-mremap-it-is-sure-to-have-enough-space-when-extent-meets-requirement +++ a/mm/mremap.c @@ -193,15 +193,13 @@ static void move_ptes(struct vm_area_str #ifdef CONFIG_HAVE_MOVE_PMD static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, - unsigned long new_addr, unsigned long old_end, - pmd_t *old_pmd, pmd_t *new_pmd) + unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd) { spinlock_t *old_ptl, *new_ptl; struct mm_struct *mm = vma->vm_mm; pmd_t pmd; - if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) - || old_end - old_addr < PMD_SIZE) + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)) return false; /* @@ -292,7 +290,7 @@ unsigned long move_page_tables(struct vm if (need_rmap_locks) take_rmap_locks(vma); moved = move_huge_pmd(vma, old_addr, new_addr, - old_end, old_pmd, new_pmd); + old_pmd, new_pmd); if (need_rmap_locks) drop_rmap_locks(vma); if (moved) @@ -312,7 +310,7 @@ unsigned long move_page_tables(struct vm if (need_rmap_locks) take_rmap_locks(vma); moved = move_normal_pmd(vma, old_addr, new_addr, - old_end, old_pmd, new_pmd); + old_pmd, new_pmd); if (need_rmap_locks) drop_rmap_locks(vma); if (moved) From patchwork Fri Aug 7 06:23:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704901 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B562A14B7 for ; Fri, 7 Aug 2020 06:23:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8548822C9F for ; Fri, 7 Aug 2020 06:23:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="xpJXDGYJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8548822C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 41E708D007C; Fri, 7 Aug 2020 02:23:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3CE768D0026; Fri, 7 Aug 2020 02:23:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E55F8D007C; Fri, 7 Aug 2020 02:23:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0032.hostedemail.com [216.40.44.32]) by kanga.kvack.org (Postfix) with ESMTP id 1A53B8D0026 for ; Fri, 7 Aug 2020 02:23:47 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E0A58181AC212 for ; Fri, 7 Aug 2020 06:23:46 +0000 (UTC) X-FDA: 77122781652.20.smash87_170ccd926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id BB6BA180C07A3 for ; Fri, 7 Aug 2020 06:23:46 +0000 (UTC) X-Spam-Summary: 1,0,0,5b65a6c88a4777fe,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:1963:2194:2198:2199:2200:2393:2525:2559:2563:2682:2685:2859:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3503:3504:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6119:6261:6653:6737:7514:7576:7903:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12296:12297:12517:12519:12555:12679:12783:12986:13161:13229:14096:14181:14721:14849:21060:21080:21450:21451:21627:21740:21939:21990:30012:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y89z3joncknhn9qt43mxkxcaknpycin7zwa8mbr7sr4y8qs9zceofb7nukra5.55oyj7ij113um8ki9b9qj7etndy8d18ub5bdre6hftccazk8uykp1aujsjqynyw.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netchec k:none,D X-HE-Tag: smash87_170ccd926fbe X-Filterd-Recvd-Size: 3818 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:46 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C115F2177B; Fri, 7 Aug 2020 06:23:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781425; bh=qOMzU/b2o0LxbCa6/jIzP2n5jQBT0Us4j4G+LRsBMJY=; h=Date:From:To:Subject:In-Reply-To:From; b=xpJXDGYJclN/gTi7cQ/r3+RXiiQfB8J/dFvkBFnV/WzG93drDmPfGUhtcV0irf+gS vDJAuS/DXA2Z6HZnWybEUAXwxdAfwRYWsjaggVVbxmSFhuZee3FD139+TADi661Cqh 7ullftIekTrfJwcOVdaJ4PT90Vc/kBRd9DETfJmo= Date: Thu, 06 Aug 2020 23:23:44 -0700 From: Andrew Morton To: akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com, anshuman.khandual@arm.com, digetx@gmail.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, peterx@redhat.com, richard.weiyang@linux.alibaba.com, sean.j.christopherson@intel.com, thellstrom@vmware.com, thomas_os@shipmail.org, torvalds@linux-foundation.org, vbabka@suse.cz, willy@infradead.org, yang.shi@linux.alibaba.com Subject: [patch 115/163] mm/mremap: calculate extent in one place Message-ID: <20200807062344.aiW9-gNx4%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: BB6BA180C07A3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Wei Yang Subject: mm/mremap: calculate extent in one place Page tables is moved on the base of PMD. This requires both source and destination range should meet the requirement. Current code works well since move_huge_pmd() and move_normal_pmd() would check old_addr and new_addr again. And then return to move_ptes() if the either of them is not aligned. Instead of calculating the extent separately, it is better to calculate in one place, so we know it is not necessary to try move pmd. By doing so, the logic seems a little clear. Link: http://lkml.kernel.org/r/20200708095028.41706-3-richard.weiyang@linux.alibaba.com Signed-off-by: Wei Yang Tested-by: Dmitry Osipenko Acked-by: Kirill A. Shutemov Cc: Aneesh Kumar K.V Cc: Anshuman Khandual Cc: Matthew Wilcox Cc: Peter Xu Cc: Sean Christopherson Cc: Thomas Hellstrom Cc: Thomas Hellstrom (VMware) Cc: Vlastimil Babka Cc: Yang Shi Signed-off-by: Andrew Morton --- mm/mremap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --- a/mm/mremap.c~mm-mremap-calculate-extent-in-one-place +++ a/mm/mremap.c @@ -277,6 +277,9 @@ unsigned long move_page_tables(struct vm extent = next - old_addr; if (extent > old_end - old_addr) extent = old_end - old_addr; + next = (new_addr + PMD_SIZE) & PMD_MASK; + if (extent > next - new_addr) + extent = next - new_addr; old_pmd = get_old_pmd(vma->vm_mm, old_addr); if (!old_pmd) continue; @@ -320,9 +323,6 @@ unsigned long move_page_tables(struct vm if (pte_alloc(new_vma->vm_mm, new_pmd)) break; - next = (new_addr + PMD_SIZE) & PMD_MASK; - if (extent > next - new_addr) - extent = next - new_addr; move_ptes(vma, old_pmd, old_addr, old_addr + extent, new_vma, new_pmd, new_addr, need_rmap_locks); } From patchwork Fri Aug 7 06:23:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704903 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F8F714B7 for ; Fri, 7 Aug 2020 06:23:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 42D7322D03 for ; Fri, 7 Aug 2020 06:23:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="DbFe08ux" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 42D7322D03 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 112938D007D; Fri, 7 Aug 2020 02:23:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 09BB58D0026; Fri, 7 Aug 2020 02:23:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F19268D007D; Fri, 7 Aug 2020 02:23:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id DDCA88D0026 for ; Fri, 7 Aug 2020 02:23:50 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B4D731730840 for ; Fri, 7 Aug 2020 06:23:50 +0000 (UTC) X-FDA: 77122781820.28.cars82_05095ff26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 888786D78 for ; Fri, 7 Aug 2020 06:23:50 +0000 (UTC) X-Spam-Summary: 1,0,0,a72c0ad1a20a03bd,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:1963:2194:2196:2198:2199:2200:2201:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3503:3504:3865:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:4605:5007:6119:6261:6653:6737:7514:7576:7903:8599:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12517:12519:12555:12679:12783:12986:14181:14721:14849:21060:21080:21450:21451:21627:21939:21987:21990:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygbiemq99uxp5x55ir5p8untodjopfeyk36oz3epcjd636b1wswpdfqqjwkse.84bitzkbceebq1frsp4rjk8whn1fqn9ucxf3q6jg8xnbgz9x8cwmocyqs1m9hqn.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Ne tcheck:n X-HE-Tag: cars82_05095ff26fbe X-Filterd-Recvd-Size: 3683 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:50 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8F2F222D07; Fri, 7 Aug 2020 06:23:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781429; bh=AO7ZsYlf8yuzOsLkjD0XaffZ817+RBXIjfbT/CH1b7o=; h=Date:From:To:Subject:In-Reply-To:From; b=DbFe08ux3jzzWbqZtZF/qY0hLREvFjayawE1EGJXTsJKNUNzUuPbIzwQXfnRpHPiF +/1s+7JrotbpG6vFP/nOvzG109rZJXFBPmPkYEQUdqfSNgYG7QJx3/h5q5/7bMHluw jcVuGrfwrS2C4+bIwIbT3PIAIfsIVlX7KuQOIWKc= Date: Thu, 06 Aug 2020 23:23:48 -0700 From: Andrew Morton To: akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com, anshuman.khandual@arm.com, digetx@gmail.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, peterx@redhat.com, richard.weiyang@linux.alibaba.com, sean.j.christopherson@intel.com, thellstrom@vmware.com, thomas_os@shipmail.org, torvalds@linux-foundation.org, vbabka@suse.cz, willy@infradead.org, yang.shi@linux.alibaba.com Subject: [patch 116/163] mm/mremap: start addresses are properly aligned Message-ID: <20200807062348.GokC3GbG3%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 888786D78 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Wei Yang Subject: mm/mremap: start addresses are properly aligned After previous cleanup, extent is the minimal step for both source and destination. This means when extent is HPAGE_PMD_SIZE or PMD_SIZE, old_addr and new_addr are properly aligned too. Since these two functions are only invoked in move_page_tables, it is safe to remove the check now. Link: http://lkml.kernel.org/r/20200708095028.41706-4-richard.weiyang@linux.alibaba.com Signed-off-by: Wei Yang Tested-by: Dmitry Osipenko Acked-by: Kirill A. Shutemov Cc: Aneesh Kumar K.V Cc: Anshuman Khandual Cc: Matthew Wilcox Cc: Peter Xu Cc: Sean Christopherson Cc: Thomas Hellstrom Cc: Thomas Hellstrom (VMware) Cc: Vlastimil Babka Cc: Yang Shi Signed-off-by: Andrew Morton --- mm/huge_memory.c | 3 --- mm/mremap.c | 3 --- 2 files changed, 6 deletions(-) --- a/mm/huge_memory.c~mm-mremap-start-addresses-are-properly-aligned +++ a/mm/huge_memory.c @@ -1729,9 +1729,6 @@ bool move_huge_pmd(struct vm_area_struct struct mm_struct *mm = vma->vm_mm; bool force_flush = false; - if ((old_addr & ~HPAGE_PMD_MASK) || (new_addr & ~HPAGE_PMD_MASK)) - return false; - /* * The destination pmd shouldn't be established, free_pgtables() * should have release it. --- a/mm/mremap.c~mm-mremap-start-addresses-are-properly-aligned +++ a/mm/mremap.c @@ -199,9 +199,6 @@ static bool move_normal_pmd(struct vm_ar struct mm_struct *mm = vma->vm_mm; pmd_t pmd; - if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK)) - return false; - /* * The destination pmd shouldn't be established, free_pgtables() * should have released it. From patchwork Fri Aug 7 06:23:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704905 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0934D16B1 for ; Fri, 7 Aug 2020 06:23:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BBA5A22C9F for ; Fri, 7 Aug 2020 06:23:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="inbMnpls" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BBA5A22C9F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 640F88D007E; Fri, 7 Aug 2020 02:23:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5F2218D0026; Fri, 7 Aug 2020 02:23:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 507F28D007E; Fri, 7 Aug 2020 02:23:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0204.hostedemail.com [216.40.44.204]) by kanga.kvack.org (Postfix) with ESMTP id 3B6048D0026 for ; Fri, 7 Aug 2020 02:23:54 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0372C181AC212 for ; Fri, 7 Aug 2020 06:23:54 +0000 (UTC) X-FDA: 77122781988.17.nerve88_240d36a26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id CF91B180D0180 for ; Fri, 7 Aug 2020 06:23:53 +0000 (UTC) X-Spam-Summary: 1,0,0,2bfdecac13f50e2f,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:1:41:152:355:379:800:960:966:967:973:988:989:1260:1263:1277:1311:1313:1314:1345:1359:1381:1431:1437:1513:1515:1516:1518:1521:1593:1594:1605:1730:1747:1777:1792:2196:2199:2393:2525:2538:2559:2563:2636:2682:2685:2689:2693:2859:2902:2914:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4384:4385:4395:4605:5007:6119:6261:6653:6690:7576:7875:7903:8599:8603:8957:9025:9545:10004:10913:10946:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13846:21080:21451:21627:21795:21939:21990:30003:30034:30051:30054:30056:30067:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y85icgkf36kjbc1jey3ye55up86yp3yifk5r43euwoqzprncpsn77q6j878jd.71ano4k4zpse4y4gf1qt978m76stweqqebhp9zqmeqzzhboixbm7gufa1bwrdr1.w-lb l8.mails X-HE-Tag: nerve88_240d36a26fbe X-Filterd-Recvd-Size: 12249 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:53 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3F900221E5; Fri, 7 Aug 2020 06:23:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781432; bh=tv3W76p30fBb41KESHP6CKwJpQen4Xdf/DadVUwvzh8=; h=Date:From:To:Subject:In-Reply-To:From; b=inbMnplsYP6FcaJ+aeOyi+Wap0leQthgGGgKuQn0hf3QS/Gr37MgEZIaLy9bwR6pY fGQieF1u1EdGtT21HXqR30HkFpjw9FivqqWbpUfn88O36uvTeCODd1k0hVyAYRu3AD DMzhxftrTwskCMDlQEg6a8zzgjdgU7/P4qoAN/SQ= Date: Thu, 06 Aug 2020 23:23:51 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, ricardo.canuelo@collabora.com, torvalds@linux-foundation.org Subject: [patch 117/163] selftests: add mincore() tests Message-ID: <20200807062351.IGy0ZUxvY%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 MIME-Version: 1.0 X-Rspamd-Queue-Id: CF91B180D0180 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Ricardo Cañuelo Subject: selftests: add mincore() tests Add a test suite for the mincore() syscall. It tests most of its use cases as well as its interface. Tests implemented: - basic interface test - behavior on anonymous mappings - behavior on anonymous mappings with huge tlb pages - file-backed mapping with a regular file - file-backed mapping with a tmpfs file Link: http://lkml.kernel.org/r/20200728100450.4065-1-ricardo.canuelo@collabora.com Signed-off-by: Ricardo Cañuelo Signed-off-by: Andrew Morton --- tools/testing/selftests/Makefile | 1 tools/testing/selftests/mincore/.gitignore | 2 tools/testing/selftests/mincore/Makefile | 6 tools/testing/selftests/mincore/mincore_selftest.c | 361 +++++++++++ 4 files changed, 370 insertions(+) --- a/tools/testing/selftests/Makefile~selftests-add-mincore-tests +++ a/tools/testing/selftests/Makefile @@ -32,6 +32,7 @@ TARGETS += lkdtm TARGETS += membarrier TARGETS += memfd TARGETS += memory-hotplug +TARGETS += mincore TARGETS += mount TARGETS += mqueue TARGETS += net --- /dev/null +++ a/tools/testing/selftests/mincore/.gitignore @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0+ +mincore_selftest --- /dev/null +++ a/tools/testing/selftests/mincore/Makefile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0+ + +CFLAGS += -Wall + +TEST_GEN_PROGS := mincore_selftest +include ../lib.mk --- /dev/null +++ a/tools/testing/selftests/mincore/mincore_selftest.c @@ -0,0 +1,361 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * kselftest suite for mincore(). + * + * Copyright (C) 2020 Collabora, Ltd. + */ + +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" +#include "../kselftest_harness.h" + +/* Default test file size: 4MB */ +#define MB (1UL << 20) +#define FILE_SIZE (4 * MB) + + +/* + * Tests the user interface. This test triggers most of the documented + * error conditions in mincore(). + */ +TEST(basic_interface) +{ + int retval; + int page_size; + unsigned char vec[1]; + char *addr; + + page_size = sysconf(_SC_PAGESIZE); + + /* Query a 0 byte sized range */ + retval = mincore(0, 0, vec); + EXPECT_EQ(0, retval); + + /* Addresses in the specified range are invalid or unmapped */ + errno = 0; + retval = mincore(NULL, page_size, vec); + EXPECT_EQ(-1, retval); + EXPECT_EQ(ENOMEM, errno); + + errno = 0; + addr = mmap(NULL, page_size, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_ANONYMOUS, -1, 0); + ASSERT_NE(MAP_FAILED, addr) { + TH_LOG("mmap error: %s", strerror(errno)); + } + + /* argument is not page-aligned */ + errno = 0; + retval = mincore(addr + 1, page_size, vec); + EXPECT_EQ(-1, retval); + EXPECT_EQ(EINVAL, errno); + + /* argument is too large */ + errno = 0; + retval = mincore(addr, -1, vec); + EXPECT_EQ(-1, retval); + EXPECT_EQ(ENOMEM, errno); + + /* argument points to an illegal address */ + errno = 0; + retval = mincore(addr, page_size, NULL); + EXPECT_EQ(-1, retval); + EXPECT_EQ(EFAULT, errno); + munmap(addr, page_size); +} + + +/* + * Test mincore() behavior on a private anonymous page mapping. + * Check that the page is not loaded into memory right after the mapping + * but after accessing it (on-demand allocation). + * Then free the page and check that it's not memory-resident. + */ +TEST(check_anonymous_locked_pages) +{ + unsigned char vec[1]; + char *addr; + int retval; + int page_size; + + page_size = sysconf(_SC_PAGESIZE); + + /* Map one page and check it's not memory-resident */ + errno = 0; + addr = mmap(NULL, page_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + ASSERT_NE(MAP_FAILED, addr) { + TH_LOG("mmap error: %s", strerror(errno)); + } + retval = mincore(addr, page_size, vec); + ASSERT_EQ(0, retval); + ASSERT_EQ(0, vec[0]) { + TH_LOG("Page found in memory before use"); + } + + /* Touch the page and check again. It should now be in memory */ + addr[0] = 1; + mlock(addr, page_size); + retval = mincore(addr, page_size, vec); + ASSERT_EQ(0, retval); + ASSERT_EQ(1, vec[0]) { + TH_LOG("Page not found in memory after use"); + } + + /* + * It shouldn't be memory-resident after unlocking it and + * marking it as unneeded. + */ + munlock(addr, page_size); + madvise(addr, page_size, MADV_DONTNEED); + retval = mincore(addr, page_size, vec); + ASSERT_EQ(0, retval); + ASSERT_EQ(0, vec[0]) { + TH_LOG("Page in memory after being zapped"); + } + munmap(addr, page_size); +} + + +/* + * Check mincore() behavior on huge pages. + * This test will be skipped if the mapping fails (ie. if there are no + * huge pages available). + * + * Make sure the system has at least one free huge page, check + * "HugePages_Free" in /proc/meminfo. + * Increment /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages if + * needed. + */ +TEST(check_huge_pages) +{ + unsigned char vec[1]; + char *addr; + int retval; + int page_size; + + page_size = sysconf(_SC_PAGESIZE); + + errno = 0; + addr = mmap(NULL, page_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, + -1, 0); + if (addr == MAP_FAILED) { + if (errno == ENOMEM) + SKIP(return, "No huge pages available."); + else + TH_LOG("mmap error: %s", strerror(errno)); + } + retval = mincore(addr, page_size, vec); + ASSERT_EQ(0, retval); + ASSERT_EQ(0, vec[0]) { + TH_LOG("Page found in memory before use"); + } + + addr[0] = 1; + mlock(addr, page_size); + retval = mincore(addr, page_size, vec); + ASSERT_EQ(0, retval); + ASSERT_EQ(1, vec[0]) { + TH_LOG("Page not found in memory after use"); + } + + munlock(addr, page_size); + munmap(addr, page_size); +} + + +/* + * Test mincore() behavior on a file-backed page. + * No pages should be loaded into memory right after the mapping. Then, + * accessing any address in the mapping range should load the page + * containing the address and a number of subsequent pages (readahead). + * + * The actual readahead settings depend on the test environment, so we + * can't make a lot of assumptions about that. This test covers the most + * general cases. + */ +TEST(check_file_mmap) +{ + unsigned char *vec; + int vec_size; + char *addr; + int retval; + int page_size; + int fd; + int i; + int ra_pages = 0; + + page_size = sysconf(_SC_PAGESIZE); + vec_size = FILE_SIZE / page_size; + if (FILE_SIZE % page_size) + vec_size++; + + vec = calloc(vec_size, sizeof(unsigned char)); + ASSERT_NE(NULL, vec) { + TH_LOG("Can't allocate array"); + } + + errno = 0; + fd = open(".", O_TMPFILE | O_RDWR, 0600); + ASSERT_NE(-1, fd) { + TH_LOG("Can't create temporary file: %s", + strerror(errno)); + } + errno = 0; + retval = fallocate(fd, 0, 0, FILE_SIZE); + ASSERT_EQ(0, retval) { + TH_LOG("Error allocating space for the temporary file: %s", + strerror(errno)); + } + + /* + * Map the whole file, the pages shouldn't be fetched yet. + */ + errno = 0; + addr = mmap(NULL, FILE_SIZE, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + ASSERT_NE(MAP_FAILED, addr) { + TH_LOG("mmap error: %s", strerror(errno)); + } + retval = mincore(addr, FILE_SIZE, vec); + ASSERT_EQ(0, retval); + for (i = 0; i < vec_size; i++) { + ASSERT_EQ(0, vec[i]) { + TH_LOG("Unexpected page in memory"); + } + } + + /* + * Touch a page in the middle of the mapping. We expect the next + * few pages (the readahead window) to be populated too. + */ + addr[FILE_SIZE / 2] = 1; + retval = mincore(addr, FILE_SIZE, vec); + ASSERT_EQ(0, retval); + ASSERT_EQ(1, vec[FILE_SIZE / 2 / page_size]) { + TH_LOG("Page not found in memory after use"); + } + + i = FILE_SIZE / 2 / page_size + 1; + while (i < vec_size && vec[i]) { + ra_pages++; + i++; + } + EXPECT_GT(ra_pages, 0) { + TH_LOG("No read-ahead pages found in memory"); + } + + EXPECT_LT(i, vec_size) { + TH_LOG("Read-ahead pages reached the end of the file"); + } + /* + * End of the readahead window. The rest of the pages shouldn't + * be in memory. + */ + if (i < vec_size) { + while (i < vec_size && !vec[i]) + i++; + EXPECT_EQ(vec_size, i) { + TH_LOG("Unexpected page in memory beyond readahead window"); + } + } + + munmap(addr, FILE_SIZE); + close(fd); + free(vec); +} + + +/* + * Test mincore() behavior on a page backed by a tmpfs file. This test + * performs the same steps as the previous one. However, we don't expect + * any readahead in this case. + */ +TEST(check_tmpfs_mmap) +{ + unsigned char *vec; + int vec_size; + char *addr; + int retval; + int page_size; + int fd; + int i; + int ra_pages = 0; + + page_size = sysconf(_SC_PAGESIZE); + vec_size = FILE_SIZE / page_size; + if (FILE_SIZE % page_size) + vec_size++; + + vec = calloc(vec_size, sizeof(unsigned char)); + ASSERT_NE(NULL, vec) { + TH_LOG("Can't allocate array"); + } + + errno = 0; + fd = open("/dev/shm", O_TMPFILE | O_RDWR, 0600); + ASSERT_NE(-1, fd) { + TH_LOG("Can't create temporary file: %s", + strerror(errno)); + } + errno = 0; + retval = fallocate(fd, 0, 0, FILE_SIZE); + ASSERT_EQ(0, retval) { + TH_LOG("Error allocating space for the temporary file: %s", + strerror(errno)); + } + + /* + * Map the whole file, the pages shouldn't be fetched yet. + */ + errno = 0; + addr = mmap(NULL, FILE_SIZE, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + ASSERT_NE(MAP_FAILED, addr) { + TH_LOG("mmap error: %s", strerror(errno)); + } + retval = mincore(addr, FILE_SIZE, vec); + ASSERT_EQ(0, retval); + for (i = 0; i < vec_size; i++) { + ASSERT_EQ(0, vec[i]) { + TH_LOG("Unexpected page in memory"); + } + } + + /* + * Touch a page in the middle of the mapping. We expect only + * that page to be fetched into memory. + */ + addr[FILE_SIZE / 2] = 1; + retval = mincore(addr, FILE_SIZE, vec); + ASSERT_EQ(0, retval); + ASSERT_EQ(1, vec[FILE_SIZE / 2 / page_size]) { + TH_LOG("Page not found in memory after use"); + } + + i = FILE_SIZE / 2 / page_size + 1; + while (i < vec_size && vec[i]) { + ra_pages++; + i++; + } + ASSERT_EQ(ra_pages, 0) { + TH_LOG("Read-ahead pages found in memory"); + } + + munmap(addr, FILE_SIZE); + close(fd); + free(vec); +} + +TEST_HARNESS_MAIN From patchwork Fri Aug 7 06:23:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704907 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 27C4214B7 for ; Fri, 7 Aug 2020 06:24:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E92D522D0A for ; Fri, 7 Aug 2020 06:23:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="gWMpAblf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E92D522D0A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B4B678D007F; Fri, 7 Aug 2020 02:23:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B22E38D0026; Fri, 7 Aug 2020 02:23:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A37AD8D007F; Fri, 7 Aug 2020 02:23:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0019.hostedemail.com [216.40.44.19]) by kanga.kvack.org (Postfix) with ESMTP id 8F9578D0026 for ; Fri, 7 Aug 2020 02:23:58 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5EA0D3628 for ; Fri, 7 Aug 2020 06:23:58 +0000 (UTC) X-FDA: 77122782156.05.top41_2d184fa26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 3804E180137B7 for ; Fri, 7 Aug 2020 06:23:58 +0000 (UTC) X-Spam-Summary: 1,0,0,e53cb0319cb36b1f,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1606:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2693:2859:2901:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3873:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4117:4250:4321:4385:4605:5007:6119:6238:6261:6653:7558:7576:7903:7904:8599:8603:8660:8784:8957:9025:9163:9165:9545:9901:10004:10226:10913:11026:11233:11473:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12740:12783:12895:12986:13053:13148:13161:13229:13230:13870:21063:21080:21451:21524:21611:21622:21789:21939:21966:30003:30012:30029:30054:30056:30064:30070:30080,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04yfp79zi5xynjo1iz8tttkr3u14soc4uuosmq5kdndehx5k4r7be9i1n7th8bm.ajqbrqggdwbb84eb6 7ttq849u X-HE-Tag: top41_2d184fa26fbe X-Filterd-Recvd-Size: 6876 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:23:57 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 719BB221E5; Fri, 7 Aug 2020 06:23:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781436; bh=rnbhHXv2YQS+aGl/5sOHyn+tm1ghIhz2DAN8yFPs5Cg=; h=Date:From:To:Subject:In-Reply-To:From; b=gWMpAblfCgRwvFhgx1768b3iOpMtyymh2uGrQooeZrlIRpYUvYBWht5Fs4JsS6N8v r8o2A3+Qcf/oX+NxhtWFHoKYHDgcPpHaWBViEfdgcxI4BxASu3kF3O/4oYwHKjb/z+ 7Jo7H0/hfou7SI11eFQrqmtHVlPscukE+2zcqHpY= Date: Thu, 06 Aug 2020 23:23:55 -0700 From: Andrew Morton To: akpm@linux-foundation.org, dan.j.williams@intel.com, david@redhat.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, osalvador@suse.de, richard.weiyang@linux.alibaba.com, torvalds@linux-foundation.org Subject: [patch 118/163] mm/sparse: never partially remove memmap for early section Message-ID: <20200807062355.jiySDgHZL%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 3804E180137B7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Wei Yang Subject: mm/sparse: never partially remove memmap for early section For early sections, its memmap is handled specially even sub-section is enabled. The memmap could only be populated as a whole. Quoted from the comment of section_activate(): * The early init code does not consider partially populated * initial sections, it simply assumes that memory will never be * referenced. If we hot-add memory into such a section then we * do not need to populate the memmap and can simply reuse what * is already there. While current section_deactivate() breaks this rule. When hot-remove a sub-section, section_deactivate() would depopulate its memmap. The consequence is if we hot-add this subsection again, its memmap never get proper populated. We can reproduce the case by following steps: 1. Hacking qemu to allow sub-section early section : diff --git a/hw/i386/pc.c b/hw/i386/pc.c : index 51b3050d01..c6a78d83c0 100644 : --- a/hw/i386/pc.c : +++ b/hw/i386/pc.c : @@ -1010,7 +1010,7 @@ void pc_memory_init(PCMachineState *pcms, : } : : machine->device_memory->base = : - ROUND_UP(0x100000000ULL + x86ms->above_4g_mem_size, 1 * GiB); : + 0x100000000ULL + x86ms->above_4g_mem_size; : : if (pcmc->enforce_aligned_dimm) { : /* size device region assuming 1G page max alignment per slot */ 2. Bootup qemu with PSE disabled and a sub-section aligned memory size Part of the qemu command would look like this: sudo x86_64-softmmu/qemu-system-x86_64 \ --enable-kvm -cpu host,pse=off \ -m 4160M,maxmem=20G,slots=1 \ -smp sockets=2,cores=16 \ -numa node,nodeid=0,cpus=0-1 -numa node,nodeid=1,cpus=2-3 \ -machine pc,nvdimm \ -nographic \ -object memory-backend-ram,id=mem0,size=8G \ -device nvdimm,id=vm0,memdev=mem0,node=0,addr=0x144000000,label-size=128k 3. Re-config a pmem device with sub-section size in guest ndctl create-namespace --force --reconfig=namespace0.0 --mode=devdax --size=16M Then you would see the following call trace: pmem0: detected capacity change from 0 to 16777216 BUG: unable to handle page fault for address: ffffec73c51000b4 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 81ff8067 P4D 81ff8067 PUD 81ff7067 PMD 1437cb067 PTE 0 Oops: 0002 [#1] SMP NOPTI CPU: 16 PID: 1348 Comm: ndctl Kdump: loaded Tainted: G W 5.8.0-rc2+ #24 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.4 RIP: 0010:memmap_init_zone+0x154/0x1c2 Code: 77 16 f6 40 10 02 74 10 48 03 48 08 48 89 cb 48 c1 eb 0c e9 3a ff ff ff 48 89 df 48 c1 e7 06 48f RSP: 0018:ffffbdc7011a39b0 EFLAGS: 00010282 RAX: ffffec73c5100088 RBX: 0000000000144002 RCX: 0000000000144000 RDX: 0000000000000004 RSI: 007ffe0000000000 RDI: ffffec73c5100080 RBP: 027ffe0000000000 R08: 0000000000000001 R09: ffff9f8d38f6d708 R10: ffffec73c0000000 R11: 0000000000000000 R12: 0000000000000004 R13: 0000000000000001 R14: 0000000000144200 R15: 0000000000000000 FS: 00007efe6b65d780(0000) GS:ffff9f8d3f780000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffec73c51000b4 CR3: 000000007d718000 CR4: 0000000000340ee0 Call Trace: move_pfn_range_to_zone+0x128/0x150 memremap_pages+0x4e4/0x5a0 devm_memremap_pages+0x1e/0x60 dev_dax_probe+0x69/0x160 [device_dax] really_probe+0x298/0x3c0 driver_probe_device+0xe1/0x150 ? driver_allows_async_probing+0x50/0x50 bus_for_each_drv+0x7e/0xc0 __device_attach+0xdf/0x160 bus_probe_device+0x8e/0xa0 device_add+0x3b9/0x740 __devm_create_dev_dax+0x127/0x1c0 __dax_pmem_probe+0x1f2/0x219 [dax_pmem_core] dax_pmem_probe+0xc/0x1b [dax_pmem] nvdimm_bus_probe+0x69/0x1c0 [libnvdimm] really_probe+0x147/0x3c0 driver_probe_device+0xe1/0x150 device_driver_attach+0x53/0x60 bind_store+0xd1/0x110 kernfs_fop_write+0xce/0x1b0 vfs_write+0xb6/0x1a0 ksys_write+0x5f/0xe0 do_syscall_64+0x4d/0x90 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Link: http://lkml.kernel.org/r/20200625223534.18024-1-richard.weiyang@linux.alibaba.com Fixes: ba72b4c8cf60 ("mm/sparsemem: support sub-section hotplug") Signed-off-by: Wei Yang Acked-by: David Hildenbrand Cc: Oscar Salvador Cc: Dan Williams Signed-off-by: Andrew Morton --- mm/sparse.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) --- a/mm/sparse.c~mm-sparse-never-partially-remove-memmap-for-early-section +++ a/mm/sparse.c @@ -824,10 +824,14 @@ static void section_deactivate(unsigned ms->section_mem_map &= ~SECTION_HAS_MEM_MAP; } - if (section_is_early && memmap) - free_map_bootmem(memmap); - else + /* + * The memmap of early sections is always fully populated. See + * section_activate() and pfn_valid() . + */ + if (!section_is_early) depopulate_section_memmap(pfn, nr_pages, altmap); + else if (memmap) + free_map_bootmem(memmap); if (empty) ms->section_mem_map = (unsigned long)NULL; From patchwork Fri Aug 7 06:23:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704909 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F12314B7 for ; Fri, 7 Aug 2020 06:24:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4EF60221E5 for ; Fri, 7 Aug 2020 06:24:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="f8N6pgOa" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4EF60221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E7F5E8D0080; Fri, 7 Aug 2020 02:24:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E57238D0026; Fri, 7 Aug 2020 02:24:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D924C8D0080; Fri, 7 Aug 2020 02:24:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0152.hostedemail.com [216.40.44.152]) by kanga.kvack.org (Postfix) with ESMTP id C47178D0026 for ; Fri, 7 Aug 2020 02:24:01 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 8AF3F2489 for ; Fri, 7 Aug 2020 06:24:01 +0000 (UTC) X-FDA: 77122782282.15.blood63_0b0838326fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 5E3931814B0C1 for ; Fri, 7 Aug 2020 06:24:01 +0000 (UTC) X-Spam-Summary: 1,0,0,86b55235e7706dcd,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2393:2525:2559:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6119:6261:6653:7576:7903:8599:9025:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13229:14096:14181:14721:14849:21080:21451:21627:21796:21939:21990:30012:30036:30054:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8ays1t5otjju1figeqfyhd4p17ypht1u4x58zdge4kjpz56k17zbmqzhhx5w.hzmtcnm5ozreo3cc51h1jybwmf1w7qfs5ffrpd97z3oks3nhm9sxcoutusghtc6.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF: fp,MSBL: X-HE-Tag: blood63_0b0838326fbe X-Filterd-Recvd-Size: 3513 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:24:00 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id CC87222D0B; Fri, 7 Aug 2020 06:23:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781440; bh=cZLwUj13ml3wJX9dyoNRSrj8qUJ1+Vkfmrc5G7DJAWk=; h=Date:From:To:Subject:In-Reply-To:From; b=f8N6pgOaRG6TPxELiQcCFIkS/kG7XrZYBYqEhX12Itv7IyaSh7vpgA/Tc3B8IMQOW 94u2H6sgT111em/4SJ+YfTSh1lo3NDm7ZHJcFU0LYLOimPm7ST8PdhC/Dyo00aygAX OK1yNAOesxgcRiYsX0wRp32MtIlfyffRwjZVggN0= Date: Thu, 06 Aug 2020 23:23:59 -0700 From: Andrew Morton To: akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, richard.weiyang@linux.alibaba.com, torvalds@linux-foundation.org Subject: [patch 119/163] mm/sparse: only sub-section aligned range would be populated Message-ID: <20200807062359.Hd_vkkVYt%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 5E3931814B0C1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Wei Yang Subject: mm/sparse: only sub-section aligned range would be populated There are two code path which invoke __populate_section_memmap() * sparse_init_nid() * sparse_add_section() For both case, we are sure the memory range is sub-section aligned. * we pass PAGES_PER_SECTION to sparse_init_nid() * we check range by check_pfn_span() before calling sparse_add_section() Also, the counterpart of __populate_section_memmap(), we don't do such calculation and check since the range is checked by check_pfn_span() in __remove_pages(). Clear the calculation and check to keep it simple and comply with its counterpart. Link: http://lkml.kernel.org/r/20200703031828.14645-1-richard.weiyang@linux.alibaba.com Signed-off-by: Wei Yang Acked-by: David Hildenbrand Signed-off-by: Andrew Morton --- mm/sparse-vmemmap.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) --- a/mm/sparse-vmemmap.c~mm-sparse-only-sub-section-aligned-range-would-be-populated +++ a/mm/sparse-vmemmap.c @@ -251,20 +251,12 @@ int __meminit vmemmap_populate_basepages struct page * __meminit __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap) { - unsigned long start; - unsigned long end; + unsigned long start = (unsigned long) pfn_to_page(pfn); + unsigned long end = start + nr_pages * sizeof(struct page); - /* - * The minimum granularity of memmap extensions is - * PAGES_PER_SUBSECTION as allocations are tracked in the - * 'subsection_map' bitmap of the section. - */ - end = ALIGN(pfn + nr_pages, PAGES_PER_SUBSECTION); - pfn &= PAGE_SUBSECTION_MASK; - nr_pages = end - pfn; - - start = (unsigned long) pfn_to_page(pfn); - end = start + nr_pages * sizeof(struct page); + if (WARN_ON_ONCE(!IS_ALIGNED(pfn, PAGES_PER_SUBSECTION) || + !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) + return NULL; if (vmemmap_populate(start, end, nid, altmap)) return NULL; From patchwork Fri Aug 7 06:24:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704911 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0AD09138C for ; Fri, 7 Aug 2020 06:24:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AF228221E5 for ; Fri, 7 Aug 2020 06:24:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="1ndFTeAr" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AF228221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 15ED58D0081; Fri, 7 Aug 2020 02:24:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 10FF18D0026; Fri, 7 Aug 2020 02:24:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 04BFC8D0081; Fri, 7 Aug 2020 02:24:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0068.hostedemail.com [216.40.44.68]) by kanga.kvack.org (Postfix) with ESMTP id E3E378D0026 for ; Fri, 7 Aug 2020 02:24:04 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B6088824934B for ; Fri, 7 Aug 2020 06:24:04 +0000 (UTC) X-FDA: 77122782408.20.train78_020c05726fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 8BAFC180C07AB for ; Fri, 7 Aug 2020 06:24:04 +0000 (UTC) X-Spam-Summary: 1,0,0,d36d8dff9af914fe,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:4:41:69:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1605:1730:1747:1777:1792:1801:2194:2196:2198:2199:2200:2201:2393:2525:2553:2559:2563:2640:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3165:3308:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4605:5007:6119:6261:6653:7576:7875:7903:8599:8603:8660:9025:9036:9389:9545:9592:10004:10913:11026:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12783:12986:13141:13148:13230:13846:13972:14096:21063:21080:21324:21433:21451:21611:21627:21939:21990:30012:30034:30045:30054:30070:30090,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-62.2.0.100 64.100.201.201;04yrpwca1smfn6hy847ad7353w1bfypkscnmyidtur7a5tjzanmbs6ua3zd1udg.tqzpjdduqphe45z5b9sk6rafjsg1b69d6judytyynodpmpdtkie7xf qozqqgba X-HE-Tag: train78_020c05726fbe X-Filterd-Recvd-Size: 16247 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:24:03 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0171C22CAF; Fri, 7 Aug 2020 06:24:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781443; bh=L3xLOuHZeWx4tiuxNjnHRMHchQP0icitgOp8pj30imc=; h=Date:From:To:Subject:In-Reply-To:From; b=1ndFTeAroPX0zxYgELw33LLJWxA/CHaMZ2yUDaXGEWUlopHfwE912xUuNLOqeIDWd nBe9IeecKpKPZAuKdhsh13t7YDsJ8C7SOhCWdSv4ywrVTH1GbbM3hnvg1RZlqERCK7 YISiyYuCrTGZ30BD6mkVQntSup5Vh/U9RsNApd90= Date: Thu, 06 Aug 2020 23:24:02 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, rppt@linux.ibm.com, torvalds@linux-foundation.org Subject: [patch 120/163] mm/sparse: cleanup the code surrounding memory_present() Message-ID: <20200807062402.6iY5419QD%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 8BAFC180C07AB X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport Subject: mm/sparse: cleanup the code surrounding memory_present() After removal of CONFIG_HAVE_MEMBLOCK_NODE_MAP we have two equivalent functions that call memory_present() for each region in memblock.memory: sparse_memory_present_with_active_regions() and membocks_present(). Moreover, all architectures have a call to either of these functions preceding the call to sparse_init() and in the most cases they are called one after the other. Mark the regions from memblock.memory as present during sparce_init() by making sparse_init() call memblocks_present(), make memblocks_present() and memory_present() functions static and remove redundant sparse_memory_present_with_active_regions() function. Also remove no longer required HAVE_MEMORY_PRESENT configuration option. Link: http://lkml.kernel.org/r/20200712083130.22919-1-rppt@kernel.org Signed-off-by: Mike Rapoport Signed-off-by: Andrew Morton --- Documentation/vm/memory-model.rst | 7 ++----- arch/arm/mm/init.c | 9 ++------- arch/arm64/mm/init.c | 6 ++---- arch/ia64/mm/discontig.c | 1 - arch/microblaze/mm/init.c | 3 --- arch/mips/kernel/setup.c | 8 -------- arch/mips/loongson64/numa.c | 1 - arch/mips/sgi-ip27/ip27-memory.c | 2 -- arch/parisc/mm/init.c | 5 ----- arch/powerpc/mm/mem.c | 2 -- arch/powerpc/mm/numa.c | 1 - arch/riscv/mm/init.c | 1 - arch/s390/mm/init.c | 1 - arch/sh/mm/init.c | 6 ------ arch/sh/mm/numa.c | 3 --- arch/sparc/mm/init_64.c | 1 - arch/x86/mm/init_32.c | 2 -- arch/x86/mm/init_64.c | 1 - include/linux/mm.h | 4 ---- include/linux/mmzone.h | 14 -------------- mm/Kconfig | 6 +----- mm/page_alloc.c | 16 ---------------- mm/sparse.c | 20 ++++++++++++-------- 23 files changed, 19 insertions(+), 101 deletions(-) --- a/arch/arm64/mm/init.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/arm64/mm/init.c @@ -430,11 +430,9 @@ void __init bootmem_init(void) #endif /* - * Sparsemem tries to allocate bootmem in memory_present(), so must be - * done after the fixed reservations. + * sparse_init() tries to allocate memory from memblock, so must be + * done after the fixed reservations */ - memblocks_present(); - sparse_init(); zone_sizes_init(min, max); --- a/arch/arm/mm/init.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/arm/mm/init.c @@ -243,13 +243,8 @@ void __init bootmem_init(void) (phys_addr_t)max_low_pfn << PAGE_SHIFT); /* - * Sparsemem tries to allocate bootmem in memory_present(), - * so must be done after the fixed reservations - */ - memblocks_present(); - - /* - * sparse_init() needs the bootmem allocator up and running. + * sparse_init() tries to allocate memory from memblock, so must be + * done after the fixed reservations */ sparse_init(); --- a/arch/ia64/mm/discontig.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/ia64/mm/discontig.c @@ -600,7 +600,6 @@ void __init paging_init(void) max_dma = virt_to_phys((void *) MAX_DMA_ADDRESS) >> PAGE_SHIFT; - sparse_memory_present_with_active_regions(MAX_NUMNODES); sparse_init(); #ifdef CONFIG_VIRTUAL_MEM_MAP --- a/arch/microblaze/mm/init.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/microblaze/mm/init.c @@ -172,9 +172,6 @@ void __init setup_memory(void) &memblock.memory, 0); } - /* XXX need to clip this if using highmem? */ - sparse_memory_present_with_active_regions(0); - paging_init(); } --- a/arch/mips/kernel/setup.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/mips/kernel/setup.c @@ -371,14 +371,6 @@ static void __init bootmem_init(void) #endif } - - /* - * In any case the added to the memblock memory regions - * (highmem/lowmem, available/reserved, etc) are considered - * as present, so inform sparsemem about them. - */ - memblocks_present(); - /* * Reserve initrd memory if needed. */ --- a/arch/mips/loongson64/numa.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/mips/loongson64/numa.c @@ -220,7 +220,6 @@ static __init void prom_meminit(void) cpumask_clear(&__node_cpumask[node]); } } - memblocks_present(); max_low_pfn = PHYS_PFN(memblock_end_of_DRAM()); for (cpu = 0; cpu < loongson_sysconf.nr_cpus; cpu++) { --- a/arch/mips/sgi-ip27/ip27-memory.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/mips/sgi-ip27/ip27-memory.c @@ -402,8 +402,6 @@ void __init prom_meminit(void) } __node_data[node] = &null_node; } - - memblocks_present(); } void __init prom_free_prom_memory(void) --- a/arch/parisc/mm/init.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/parisc/mm/init.c @@ -689,11 +689,6 @@ void __init paging_init(void) flush_cache_all_local(); /* start with known state */ flush_tlb_all_local(NULL); - /* - * Mark all memblocks as present for sparsemem using - * memory_present() and then initialize sparsemem. - */ - memblocks_present(); sparse_init(); parisc_bootmem_free(); } --- a/arch/powerpc/mm/mem.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/powerpc/mm/mem.c @@ -183,8 +183,6 @@ void __init mem_topology_setup(void) void __init initmem_init(void) { - /* XXX need to clip this if using highmem? */ - sparse_memory_present_with_active_regions(0); sparse_init(); } --- a/arch/powerpc/mm/numa.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/powerpc/mm/numa.c @@ -949,7 +949,6 @@ void __init initmem_init(void) get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); setup_node_data(nid, start_pfn, end_pfn); - sparse_memory_present_with_active_regions(nid); } sparse_init(); --- a/arch/riscv/mm/init.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/riscv/mm/init.c @@ -544,7 +544,6 @@ void mark_rodata_ro(void) void __init paging_init(void) { setup_vm_final(); - memblocks_present(); sparse_init(); setup_zero_page(); zone_sizes_init(); --- a/arch/s390/mm/init.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/s390/mm/init.c @@ -115,7 +115,6 @@ void __init paging_init(void) __load_psw_mask(psw.mask); kasan_free_early_identity(); - sparse_memory_present_with_active_regions(MAX_NUMNODES); sparse_init(); zone_dma_bits = 31; memset(max_zone_pfns, 0, sizeof(max_zone_pfns)); --- a/arch/sh/mm/init.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/sh/mm/init.c @@ -241,12 +241,6 @@ static void __init do_init_bootmem(void) plat_mem_setup(); - for_each_memblock(memory, reg) { - int nid = memblock_get_region_node(reg); - - memory_present(nid, memblock_region_memory_base_pfn(reg), - memblock_region_memory_end_pfn(reg)); - } sparse_init(); } --- a/arch/sh/mm/numa.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/sh/mm/numa.c @@ -53,7 +53,4 @@ void __init setup_bootmem_node(int nid, /* It's up */ node_set_online(nid); - - /* Kick sparsemem */ - sparse_memory_present_with_active_regions(nid); } --- a/arch/sparc/mm/init_64.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/sparc/mm/init_64.c @@ -1610,7 +1610,6 @@ static unsigned long __init bootmem_init /* XXX cpu notifier XXX */ - sparse_memory_present_with_active_regions(MAX_NUMNODES); sparse_init(); return end_pfn; --- a/arch/x86/mm/init_32.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/x86/mm/init_32.c @@ -678,7 +678,6 @@ void __init initmem_init(void) #endif memblock_set_node(0, PHYS_ADDR_MAX, &memblock.memory, 0); - sparse_memory_present_with_active_regions(0); #ifdef CONFIG_FLATMEM max_mapnr = IS_ENABLED(CONFIG_HIGHMEM) ? highend_pfn : max_low_pfn; @@ -718,7 +717,6 @@ void __init paging_init(void) * NOTE: at this point the bootmem allocator is fully available. */ olpc_dt_build_devicetree(); - sparse_memory_present_with_active_regions(MAX_NUMNODES); sparse_init(); zone_sizes_init(); } --- a/arch/x86/mm/init_64.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/arch/x86/mm/init_64.c @@ -817,7 +817,6 @@ void __init initmem_init(void) void __init paging_init(void) { - sparse_memory_present_with_active_regions(MAX_NUMNODES); sparse_init(); /* --- a/Documentation/vm/memory-model.rst~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/Documentation/vm/memory-model.rst @@ -141,11 +141,8 @@ sections: `mem_section` objects and the number of rows is calculated to fit all the memory sections. -The architecture setup code should call :c:func:`memory_present` for -each active memory range or use :c:func:`memblocks_present` or -:c:func:`sparse_memory_present_with_active_regions` wrappers to -initialize the memory sections. Next, the actual memory maps should be -set up using :c:func:`sparse_init`. +The architecture setup code should call sparse_init() to +initialize the memory sections and the memory maps. With SPARSEMEM there are two possible ways to convert a PFN to the corresponding `struct page` - a "classic sparse" and "sparse --- a/include/linux/mm.h~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/include/linux/mm.h @@ -2382,9 +2382,6 @@ static inline unsigned long get_num_phys * for_each_valid_physical_page_range() * memblock_add_node(base, size, nid) * free_area_init(max_zone_pfns); - * - * sparse_memory_present_with_active_regions() calls memory_present() for - * each range when SPARSEMEM is enabled. */ void free_area_init(unsigned long *max_zone_pfn); unsigned long node_map_pfn_alignment(void); @@ -2395,7 +2392,6 @@ extern unsigned long absent_pages_in_ran extern void get_pfn_range_for_nid(unsigned int nid, unsigned long *start_pfn, unsigned long *end_pfn); extern unsigned long find_min_pfn_with_active_regions(void); -extern void sparse_memory_present_with_active_regions(int nid); #ifndef CONFIG_NEED_MULTIPLE_NODES static inline int early_pfn_to_nid(unsigned long pfn) --- a/include/linux/mmzone.h~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/include/linux/mmzone.h @@ -839,18 +839,6 @@ static inline struct pglist_data *lruvec extern unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone_idx); -#ifdef CONFIG_HAVE_MEMORY_PRESENT -void memory_present(int nid, unsigned long start, unsigned long end); -#else -static inline void memory_present(int nid, unsigned long start, unsigned long end) {} -#endif - -#if defined(CONFIG_SPARSEMEM) -void memblocks_present(void); -#else -static inline void memblocks_present(void) {} -#endif - #ifdef CONFIG_HAVE_MEMORYLESS_NODES int local_memory_node(int node_id); #else @@ -1407,8 +1395,6 @@ struct mminit_pfnnid_cache { #define early_pfn_valid(pfn) (1) #endif -void memory_present(int nid, unsigned long start, unsigned long end); - /* * If it is possible to have holes within a MAX_ORDER_NR_PAGES, then we * need to check pfn validity within that MAX_ORDER_NR_PAGES block. --- a/mm/Kconfig~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/mm/Kconfig @@ -88,13 +88,9 @@ config NEED_MULTIPLE_NODES def_bool y depends on DISCONTIGMEM || NUMA -config HAVE_MEMORY_PRESENT - def_bool y - depends on ARCH_HAVE_MEMORY_PRESENT || SPARSEMEM - # # SPARSEMEM_EXTREME (which is the default) does some bootmem -# allocations when memory_present() is called. If this cannot +# allocations when sparse_init() is called. If this cannot # be done on your architecture, select this option. However, # statically allocating the mem_section[] array can potentially # consume vast quantities of .bss, so be careful. --- a/mm/page_alloc.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/mm/page_alloc.c @@ -6325,22 +6325,6 @@ void __meminit init_currently_empty_zone } /** - * sparse_memory_present_with_active_regions - Call memory_present for each active range - * @nid: The node to call memory_present for. If MAX_NUMNODES, all nodes will be used. - * - * If an architecture guarantees that all ranges registered contain no holes and may - * be freed, this function may be used instead of calling memory_present() manually. - */ -void __init sparse_memory_present_with_active_regions(int nid) -{ - unsigned long start_pfn, end_pfn; - int i, this_nid; - - for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, &this_nid) - memory_present(this_nid, start_pfn, end_pfn); -} - -/** * get_pfn_range_for_nid - Return the start and end page frames for a node * @nid: The nid to return the range for. If MAX_NUMNODES, the min and max PFN are returned. * @start_pfn: Passed by reference. On return, it will have the node start_pfn. --- a/mm/sparse.c~mm-sparse-cleanup-the-code-surrounding-memory_present +++ a/mm/sparse.c @@ -249,7 +249,7 @@ void __init subsection_map_init(unsigned #endif /* Record a memory area against a node. */ -void __init memory_present(int nid, unsigned long start, unsigned long end) +static void __init memory_present(int nid, unsigned long start, unsigned long end) { unsigned long pfn; @@ -285,11 +285,11 @@ void __init memory_present(int nid, unsi } /* - * Mark all memblocks as present using memory_present(). This is a - * convenience function that is useful for a number of arches - * to mark all of the systems memory as present during initialization. + * Mark all memblocks as present using memory_present(). + * This is a convenience function that is useful to mark all of the systems + * memory as present during initialization. */ -void __init memblocks_present(void) +static void __init memblocks_present(void) { struct memblock_region *reg; @@ -574,9 +574,13 @@ failed: */ void __init sparse_init(void) { - unsigned long pnum_begin = first_present_section_nr(); - int nid_begin = sparse_early_nid(__nr_to_section(pnum_begin)); - unsigned long pnum_end, map_count = 1; + unsigned long pnum_end, pnum_begin, map_count = 1; + int nid_begin; + + memblocks_present(); + + pnum_begin = first_present_section_nr(); + nid_begin = sparse_early_nid(__nr_to_section(pnum_begin)); /* Setup pageblock_order for HUGETLB_PAGE_SIZE_VARIABLE */ set_pageblock_order(); From patchwork Fri Aug 7 06:24:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704913 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E231138C for ; Fri, 7 Aug 2020 06:24:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EEFFC221E5 for ; Fri, 7 Aug 2020 06:24:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="ceD+U38X" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EEFFC221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 691A68D0082; Fri, 7 Aug 2020 02:24:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6663B8D0026; Fri, 7 Aug 2020 02:24:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52DD88D0082; Fri, 7 Aug 2020 02:24:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0198.hostedemail.com [216.40.44.198]) by kanga.kvack.org (Postfix) with ESMTP id 3B5BC8D0026 for ; Fri, 7 Aug 2020 02:24:08 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0E42C181AEF1A for ; Fri, 7 Aug 2020 06:24:08 +0000 (UTC) X-FDA: 77122782576.14.scent81_391765926fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id D733718229818 for ; Fri, 7 Aug 2020 06:24:07 +0000 (UTC) X-Spam-Summary: 1,0,0,e797797de4f85818,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2731:2734:2736:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:5007:6119:6261:6653:7576:7808:7903:8599:8603:9010:9025:9036:9545:9592:10004:10913:11026:11232:11473:11658:11914:12043:12048:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13229:13846:14096:14181:14721:14849:21080:21324:21433:21451:21627:21795:21939:21990:30003:30051:30054:30064:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8z16ptwm8ynxb88b137qwei68nyputb7wmsden6u7nwcptc7b4x9umajreux.fj69xfzuu1oi3cmb95sphw847sh3suizxonjnix379yz84o48oook6157hx71od.g-lbl8.mailshell.net -223.238 X-HE-Tag: scent81_391765926fbe X-Filterd-Recvd-Size: 4979 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:24:07 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3FBAF22C9F; Fri, 7 Aug 2020 06:24:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781446; bh=j60hILS+04q1X42aFjjEsmWXOjtlkE+HTO7X96Y6vtE=; h=Date:From:To:Subject:In-Reply-To:From; b=ceD+U38X51DHvCQCAg9NMfyAwn2XQBiAfrbG/Lv2/kcUuIuXXQGZW+U5d+08ClH6Q l9qvBMatxf5txBXEiQJQ1meQ0ItnuBXGX+S6PcaTBAgFPHH6kFYs3aRH5gZUb/B70r oVwGyZ4kgrATsWEtTNqCZ38AnRD+NcCR/fLDHLrc= Date: Thu, 06 Aug 2020 23:24:05 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, torvalds@linux-foundation.org, william.kucharski@oracle.com, willy@infradead.org Subject: [patch 121/163] vmalloc: convert to XArray Message-ID: <20200807062405.J29JzBXLB%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: D733718229818 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Subject: vmalloc: convert to XArray The radix tree of vmap blocks is simpler to express as an XArray. Reduces both the text and data sizes of the object file and eliminates a user of the radix tree preload API. Link: http://lkml.kernel.org/r/20200603171448.5894-1-willy@infradead.org Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: William Kucharski Signed-off-by: Andrew Morton --- mm/vmalloc.c | 40 +++++++++++----------------------------- 1 file changed, 11 insertions(+), 29 deletions(-) --- a/mm/vmalloc.c~vmalloc-convert-to-xarray +++ a/mm/vmalloc.c @@ -25,7 +25,7 @@ #include #include #include -#include +#include #include #include #include @@ -1514,12 +1514,11 @@ struct vmap_block { static DEFINE_PER_CPU(struct vmap_block_queue, vmap_block_queue); /* - * Radix tree of vmap blocks, indexed by address, to quickly find a vmap block + * XArray of vmap blocks, indexed by address, to quickly find a vmap block * in the free path. Could get rid of this if we change the API to return a * "cookie" from alloc, to be passed to free. But no big deal yet. */ -static DEFINE_SPINLOCK(vmap_block_tree_lock); -static RADIX_TREE(vmap_block_tree, GFP_ATOMIC); +static DEFINE_XARRAY(vmap_blocks); /* * We should probably have a fallback mechanism to allocate virtual memory @@ -1576,13 +1575,6 @@ static void *new_vmap_block(unsigned int return ERR_CAST(va); } - err = radix_tree_preload(gfp_mask); - if (unlikely(err)) { - kfree(vb); - free_vmap_area(va); - return ERR_PTR(err); - } - vaddr = vmap_block_vaddr(va->va_start, 0); spin_lock_init(&vb->lock); vb->va = va; @@ -1595,11 +1587,12 @@ static void *new_vmap_block(unsigned int INIT_LIST_HEAD(&vb->free_list); vb_idx = addr_to_vb_idx(va->va_start); - spin_lock(&vmap_block_tree_lock); - err = radix_tree_insert(&vmap_block_tree, vb_idx, vb); - spin_unlock(&vmap_block_tree_lock); - BUG_ON(err); - radix_tree_preload_end(); + err = xa_insert(&vmap_blocks, vb_idx, vb, gfp_mask); + if (err) { + kfree(vb); + free_vmap_area(va); + return ERR_PTR(err); + } vbq = &get_cpu_var(vmap_block_queue); spin_lock(&vbq->lock); @@ -1613,12 +1606,8 @@ static void *new_vmap_block(unsigned int static void free_vmap_block(struct vmap_block *vb) { struct vmap_block *tmp; - unsigned long vb_idx; - vb_idx = addr_to_vb_idx(vb->va->va_start); - spin_lock(&vmap_block_tree_lock); - tmp = radix_tree_delete(&vmap_block_tree, vb_idx); - spin_unlock(&vmap_block_tree_lock); + tmp = xa_erase(&vmap_blocks, addr_to_vb_idx(vb->va->va_start)); BUG_ON(tmp != vb); free_vmap_area_noflush(vb->va); @@ -1724,7 +1713,6 @@ static void *vb_alloc(unsigned long size static void vb_free(unsigned long addr, unsigned long size) { unsigned long offset; - unsigned long vb_idx; unsigned int order; struct vmap_block *vb; @@ -1734,14 +1722,8 @@ static void vb_free(unsigned long addr, flush_cache_vunmap(addr, addr + size); order = get_order(size); - offset = (addr & (VMAP_BLOCK_SIZE - 1)) >> PAGE_SHIFT; - - vb_idx = addr_to_vb_idx(addr); - rcu_read_lock(); - vb = radix_tree_lookup(&vmap_block_tree, vb_idx); - rcu_read_unlock(); - BUG_ON(!vb); + vb = xa_load(&vmap_blocks, addr_to_vb_idx(addr)); unmap_kernel_range_noflush(addr, size); From patchwork Fri Aug 7 06:24:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704915 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79FCB138C for ; Fri, 7 Aug 2020 06:24:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4DE5C221E5 for ; Fri, 7 Aug 2020 06:24:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="QtHat6zk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4DE5C221E5 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 976E48D0083; Fri, 7 Aug 2020 02:24:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 926BC8D0026; Fri, 7 Aug 2020 02:24:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 863078D0083; Fri, 7 Aug 2020 02:24:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0168.hostedemail.com [216.40.44.168]) by kanga.kvack.org (Postfix) with ESMTP id 70F5E8D0026 for ; Fri, 7 Aug 2020 02:24:11 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3F6A03637 for ; Fri, 7 Aug 2020 06:24:11 +0000 (UTC) X-FDA: 77122782702.14.sense33_2611e7726fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 0A3EB18229835 for ; Fri, 7 Aug 2020 06:24:11 +0000 (UTC) X-Spam-Summary: 1,0,0,8ad1955d439d1dde,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2559:2563:2682:2685:2859:2895:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:5007:6261:6653:7514:7576:7903:8599:9010:9025:9545:9592:10004:10913:11026:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12986:13161:13229:13846:14096:14181:14721:14849:21080:21451:21627:21939:21990:30054:30070,0,RBL:198.145.29.99:@linux-foundation.org:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04y8tgdfcy4e39x6o77e8o37dbmjhycyrwza51qkr838m6yy5s1ct84ytq7fxsf.hnu956kyerqrh4bydjxbozwggm7p8qyt5uj34g1rfsu38u5bspwqg436pas7xhi.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,Dom ainCache X-HE-Tag: sense33_2611e7726fbe X-Filterd-Recvd-Size: 3698 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:24:10 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9AE0F22C9F; Fri, 7 Aug 2020 06:24:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781449; bh=bKRS2E2NSJm5l9hlRu7PtmqXf8IADNxbLGGc4LTR+OM=; h=Date:From:To:Subject:In-Reply-To:From; b=QtHat6zkUiYVHiLelPEn94+T4a8fgas/JOXEHoHCU3Ta77D6t8UtYWAyu/w+GrleG hRV9PF9vVsMkz3MRmwn8vqMLB7rXH1EXj71/zGj4acygDHS/P+dkz9iCFmUFYESozm Z9jQNeQ5Fn0zx1STo+5CvEaKHt326jER6RF1wF7w= Date: Thu, 06 Aug 2020 23:24:09 -0700 From: Andrew Morton To: akpm@linux-foundation.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, torvalds@linux-foundation.org, urezki@gmail.com Subject: [patch 122/163] mm/vmalloc: simplify merge_or_add_vmap_area() Message-ID: <20200807062409.LB5mTxlwD%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 0A3EB18229835 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Uladzislau Rezki (Sony)" Subject: mm/vmalloc: simplify merge_or_add_vmap_area() Currently when a VA is deallocated and is about to be placed back to the tree, it can be either: merged with next/prev neighbors or inserted if not coalesced. On those steps the tree can be populated several times. For example when both neighbors are merged. It can be avoided and simplified in fact. Therefore do it only once when VA points to final merged area, after all manipulations: merging/removing/inserting. Link: http://lkml.kernel.org/r/20200527205054.1696-1-urezki@gmail.com Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Andrew Morton --- mm/vmalloc.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) --- a/mm/vmalloc.c~mm-vmalloc-simplify-merge_or_add_vmap_area-func +++ a/mm/vmalloc.c @@ -797,9 +797,6 @@ merge_or_add_vmap_area(struct vmap_area if (sibling->va_start == va->va_end) { sibling->va_start = va->va_start; - /* Check and update the tree if needed. */ - augment_tree_propagate_from(sibling); - /* Free vmap_area object. */ kmem_cache_free(vmap_area_cachep, va); @@ -819,14 +816,18 @@ merge_or_add_vmap_area(struct vmap_area if (next->prev != head) { sibling = list_entry(next->prev, struct vmap_area, list); if (sibling->va_end == va->va_start) { - sibling->va_end = va->va_end; - - /* Check and update the tree if needed. */ - augment_tree_propagate_from(sibling); - + /* + * If both neighbors are coalesced, it is important + * to unlink the "next" node first, followed by merging + * with "previous" one. Otherwise the tree might not be + * fully populated if a sibling's augmented value is + * "normalized" because of rotation operations. + */ if (merged) unlink_va(va, root); + sibling->va_end = va->va_end; + /* Free vmap_area object. */ kmem_cache_free(vmap_area_cachep, va); @@ -837,11 +838,13 @@ merge_or_add_vmap_area(struct vmap_area } insert: - if (!merged) { + if (!merged) link_va(va, root, parent, link, head); - augment_tree_propagate_from(va); - } + /* + * Last step is to check and update the tree. + */ + augment_tree_propagate_from(va); return va; } From patchwork Fri Aug 7 06:25:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704935 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DB3DA14B7 for ; Fri, 7 Aug 2020 06:26:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A693E22D0A for ; Fri, 7 Aug 2020 06:26:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="o0rRsH6K" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A693E22D0A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 77F8C8D0085; Fri, 7 Aug 2020 02:26:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 68E688D0026; Fri, 7 Aug 2020 02:26:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E4718D0085; Fri, 7 Aug 2020 02:26:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0180.hostedemail.com [216.40.44.180]) by kanga.kvack.org (Postfix) with ESMTP id 39C528D0026 for ; Fri, 7 Aug 2020 02:26:12 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3BCAC181AEF1E for ; Fri, 7 Aug 2020 06:26:03 +0000 (UTC) X-FDA: 77122787406.30.need70_4817dc826fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 11976180B3C83 for ; Fri, 7 Aug 2020 06:26:03 +0000 (UTC) X-Spam-Summary: 50,0,0,38065c4e3017a764,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:981:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2560:2563:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3871:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6261:6653:7576:8599:9025:9545:10004:10913:11658:11914:12043:12048:12297:12517:12519:12555:12679:12783:12986:13069:13311:13357:14181:14384:14721:14849:21080:21451:21627:21939:30054,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:6,LUA_SUMMARY:none X-HE-Tag: need70_4817dc826fbe X-Filterd-Recvd-Size: 2608 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:02 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 92E2E22CAE; Fri, 7 Aug 2020 06:25:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781558; bh=B4u3/7m1qpu/zy/YSXhr99cRUqeBVHYzNXa4WKkkhVc=; h=Date:From:To:Subject:In-Reply-To:From; b=o0rRsH6KomA914+cYqJPMMOWJfmLNTZvMKHQ3c4KeInU3m8FU+/aI4uDxG1kVMsDN l/hug2uGm7mobwBYR/82MshhjR5c+5+rLU5LGCFjL7nVtOwew24G8Y/SkDQur6ldlU 34smeS6DFOgKTFkFzfO7/wXyQ3bSGtdoMVlj8zY0= Date: Thu, 06 Aug 2020 23:25:58 -0700 From: Andrew Morton To: akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, richard.weiyang@linux.alibaba.com, torvalds@linux-foundation.org Subject: [patch 153/163] mm/page_alloc: fallbacks at most has 3 elements Message-ID: <20200807062558.ui4uGHaCT%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 11976180B3C83 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Wei Yang Subject: mm/page_alloc: fallbacks at most has 3 elements MIGRAGE_TYPES is used to be the mark of end and there are at most 3 elements for the one dimension array. Reduce to 3 to save little memory. Link: http://lkml.kernel.org/r/20200625231022.18784-1-richard.weiyang@linux.alibaba.com Signed-off-by: Wei Yang Reviewed-by: David Hildenbrand Signed-off-by: Andrew Morton --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/mm/page_alloc.c~mm-page_alloc-fallbacks-at-most-has-3-elements +++ a/mm/page_alloc.c @@ -2268,7 +2268,7 @@ struct page *__rmqueue_smallest(struct z * This array describes the order lists are fallen back to when * the free lists for the desirable migrate type are depleted */ -static int fallbacks[MIGRATE_TYPES][4] = { +static int fallbacks[MIGRATE_TYPES][3] = { [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES }, [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES }, From patchwork Fri Aug 7 06:26:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704937 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 235DE138C for ; Fri, 7 Aug 2020 06:26:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E2A0F22CF6 for ; Fri, 7 Aug 2020 06:26:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="y4RP1TvT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E2A0F22CF6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2C26B8D0086; Fri, 7 Aug 2020 02:26:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 226858D0026; Fri, 7 Aug 2020 02:26:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 077D08D0086; Fri, 7 Aug 2020 02:26:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0087.hostedemail.com [216.40.44.87]) by kanga.kvack.org (Postfix) with ESMTP id E644C8D0026 for ; Fri, 7 Aug 2020 02:26:12 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8DFC482499B9 for ; Fri, 7 Aug 2020 06:26:03 +0000 (UTC) X-FDA: 77122787406.09.metal25_1000b5d26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 6C433180AD801 for ; Fri, 7 Aug 2020 06:26:03 +0000 (UTC) X-Spam-Summary: 50,0,0,b73b1f948d783909,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2393:2525:2560:2563:2682:2685:2859:2902:2904:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7576:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12783:12986:13069:13161:13229:13311:13357:13846:14181:14384:14721:14849:21080:21433:21451:21627:21939:30054,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: metal25_1000b5d26fbe X-Filterd-Recvd-Size: 2947 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:03 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E450322CF6; Fri, 7 Aug 2020 06:26:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781562; bh=uPxPl7fj33NhHRitqMtGFKS3Mnq8K9kSTW0t/ChhI14=; h=Date:From:To:Subject:In-Reply-To:From; b=y4RP1TvTtg/yPdNs0LItx5l6U3pKW+NFeEM/9YDekHY/dK3BeeDA7HHI39lJENkhT sCaYGtgyKxsokC9uBkPm9v2Pye8v3jWBCknmgBLyIGuh8yxcxaxGpyvPWwnSxOv77M RYW5bLTRzMyjXrRpsyT4+rBuTEjc9+2IcMnz2NrQ= Date: Thu, 06 Aug 2020 23:26:01 -0700 From: Andrew Morton To: akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, songmuchun@bytedance.com, torvalds@linux-foundation.org Subject: [patch 154/163] mm/page_alloc.c: skip setting nodemask when we are in interrupt Message-ID: <20200807062601.7kNJ1Pi0Y%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 6C433180AD801 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Muchun Song Subject: mm/page_alloc.c: skip setting nodemask when we are in interrupt When we are in the interrupt context, it is irrelevant to the current task context. If we use current task's mems_allowed, we can be fair to alloc pages in the fast path and fall back to slow path memory allocation when the current node(which is the current task mems_allowed) does not have enough memory to allocate. In this case, it slows down the memory allocation speed of interrupt context. So we can skip setting the nodemask to allow any node to allocate memory, so that fast path allocation can success. Link: http://lkml.kernel.org/r/20200706025921.53683-1-songmuchun@bytedance.com Signed-off-by: Muchun Song Reviewed-by: Pekka Enberg Cc: David Hildenbrand Signed-off-by: Andrew Morton --- mm/page_alloc.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) --- a/mm/page_alloc.c~mm-page_alloc-skip-setting-nodemask-when-we-are-in-interrupt +++ a/mm/page_alloc.c @@ -4788,7 +4788,11 @@ static inline bool prepare_alloc_pages(g if (cpusets_enabled()) { *alloc_mask |= __GFP_HARDWALL; - if (!ac->nodemask) + /* + * When we are in the interrupt context, it is irrelevant + * to the current task context. It means that any node ok. + */ + if (!in_interrupt() && !ac->nodemask) ac->nodemask = &cpuset_current_mems_allowed; else *alloc_flags |= ALLOC_CPUSET; From patchwork Fri Aug 7 06:26:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704933 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A426514B7 for ; Fri, 7 Aug 2020 06:26:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 747D922D37 for ; Fri, 7 Aug 2020 06:26:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="gHT7nTvb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 747D922D37 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8B2DF8D0084; Fri, 7 Aug 2020 02:26:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 839DE8D0026; Fri, 7 Aug 2020 02:26:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 728E88D0084; Fri, 7 Aug 2020 02:26:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id 62E388D0026 for ; Fri, 7 Aug 2020 02:26:10 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 3B28D8248D52 for ; Fri, 7 Aug 2020 06:26:10 +0000 (UTC) X-FDA: 77122787700.29.swing20_19020a426fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 1B082180868E6 for ; Fri, 7 Aug 2020 06:26:10 +0000 (UTC) X-Spam-Summary: 50,0,0,0e57e2c9124c13c6,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:2:41:69:355:379:800:960:966:967:973:981:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1605:1606:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2560:2563:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4117:4250:4321:4385:4605:5007:6119:6261:6653:6737:7576:7903:7904:9025:9391:9545:10004:10913:11026:11473:11658:11914:12043:12048:12114:12291:12296:12297:12438:12517:12519:12555:12679:12683:12783:12986:13161:13215:13229:13846:14096:14130:21060:21080:21220:21433:21451:21627:21740:21795:21819:21939:21990:30051:30054:30064:30069:30070,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: swing20_19020a426fbe X-Filterd-Recvd-Size: 6984 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:09 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5C9F422CAE; Fri, 7 Aug 2020 06:26:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781565; bh=UEfTgQjDYcxWaAF+cf2UHfau4lgVzD7tN9C9HPC+RG4=; h=Date:From:To:Subject:In-Reply-To:From; b=gHT7nTvbuaIMeu2kL65c6PmX2fEdqqvDxh45dCOwrj32xlpt2Du5t4aFzBOFe4Hyu pfUBuYupuKWec8SeSk4Ma5mP5zafRuHvL/O6rgUQKzxHAwViY0tvKqyWcjt0J/SpVI aNngaPNc0NiGLYYu06UburE0FFjMfzXH+zJeXsUk= Date: Thu, 06 Aug 2020 23:26:04 -0700 From: Andrew Morton To: akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com, guro@fb.com, hch@infradead.org, iamjoonsoo.kim@lge.com, linux-mm@kvack.org, mhocko@suse.com, mike.kravetz@oracle.com, mm-commits@vger.kernel.org, n-horiguchi@ah.jp.nec.com, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 155/163] mm/page_alloc: fix memalloc_nocma_{save/restore} APIs Message-ID: <20200807062604.b4fQe0zYx%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 1B082180868E6 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joonsoo Kim Subject: mm/page_alloc: fix memalloc_nocma_{save/restore} APIs Currently, memalloc_nocma_{save/restore} API that prevents CMA area in page allocation is implemented by using current_gfp_context(). However, there are two problems of this implementation. First, this doesn't work for allocation fastpath. In the fastpath, original gfp_mask is used since current_gfp_context() is introduced in order to control reclaim and it is on slowpath. So, CMA area can be allocated through the allocation fastpath even if memalloc_nocma_{save/restore} APIs are used. Currently, there is just one user for these APIs and it has a fallback method to prevent actual problem. Second, clearing __GFP_MOVABLE in current_gfp_context() has a side effect to exclude the memory on the ZONE_MOVABLE for allocation target. To fix these problems, this patch changes the implementation to exclude CMA area in page allocation. Main point of this change is using the alloc_flags. alloc_flags is mainly used to control allocation so it fits for excluding CMA area in allocation. Link: http://lkml.kernel.org/r/1595468942-29687-1-git-send-email-iamjoonsoo.kim@lge.com Fixes: d7fefcc8de91 (mm/cma: add PF flag to force non cma alloc) Signed-off-by: Joonsoo Kim Reviewed-by: Vlastimil Babka Cc: Christoph Hellwig Cc: Roman Gushchin Cc: Mike Kravetz Cc: Naoya Horiguchi Cc: Michal Hocko Cc: "Aneesh Kumar K . V" Signed-off-by: Andrew Morton --- include/linux/sched/mm.h | 8 +------- mm/page_alloc.c | 31 +++++++++++++++++++++---------- 2 files changed, 22 insertions(+), 17 deletions(-) --- a/include/linux/sched/mm.h~mm-page_alloc-fix-memalloc_nocma_save-restore-apis +++ a/include/linux/sched/mm.h @@ -175,12 +175,10 @@ static inline bool in_vfork(struct task_ * Applies per-task gfp context to the given allocation flags. * PF_MEMALLOC_NOIO implies GFP_NOIO * PF_MEMALLOC_NOFS implies GFP_NOFS - * PF_MEMALLOC_NOCMA implies no allocation from CMA region. */ static inline gfp_t current_gfp_context(gfp_t flags) { - if (unlikely(current->flags & - (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS | PF_MEMALLOC_NOCMA))) { + if (unlikely(current->flags & (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS))) { /* * NOIO implies both NOIO and NOFS and it is a weaker context * so always make sure it makes precedence @@ -189,10 +187,6 @@ static inline gfp_t current_gfp_context( flags &= ~(__GFP_IO | __GFP_FS); else if (current->flags & PF_MEMALLOC_NOFS) flags &= ~__GFP_FS; -#ifdef CONFIG_CMA - if (current->flags & PF_MEMALLOC_NOCMA) - flags &= ~__GFP_MOVABLE; -#endif } return flags; } --- a/mm/page_alloc.c~mm-page_alloc-fix-memalloc_nocma_save-restore-apis +++ a/mm/page_alloc.c @@ -2785,7 +2785,7 @@ __rmqueue(struct zone *zone, unsigned in * allocating from CMA when over half of the zone's free memory * is in the CMA area. */ - if (migratetype == MIGRATE_MOVABLE && + if (alloc_flags & ALLOC_CMA && zone_page_state(zone, NR_FREE_CMA_PAGES) > zone_page_state(zone, NR_FREE_PAGES) / 2) { page = __rmqueue_cma_fallback(zone, order); @@ -2796,7 +2796,7 @@ __rmqueue(struct zone *zone, unsigned in retry: page = __rmqueue_smallest(zone, order, migratetype); if (unlikely(!page)) { - if (migratetype == MIGRATE_MOVABLE) + if (alloc_flags & ALLOC_CMA) page = __rmqueue_cma_fallback(zone, order); if (!page && __rmqueue_fallback(zone, order, migratetype, @@ -3687,6 +3687,20 @@ alloc_flags_nofragment(struct zone *zone return alloc_flags; } +static inline unsigned int current_alloc_flags(gfp_t gfp_mask, + unsigned int alloc_flags) +{ +#ifdef CONFIG_CMA + unsigned int pflags = current->flags; + + if (!(pflags & PF_MEMALLOC_NOCMA) && + gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + alloc_flags |= ALLOC_CMA; + +#endif + return alloc_flags; +} + /* * get_page_from_freelist goes through the zonelist trying to allocate * a page. @@ -4333,10 +4347,8 @@ gfp_to_alloc_flags(gfp_t gfp_mask) } else if (unlikely(rt_task(current)) && !in_interrupt()) alloc_flags |= ALLOC_HARDER; -#ifdef CONFIG_CMA - if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) - alloc_flags |= ALLOC_CMA; -#endif + alloc_flags = current_alloc_flags(gfp_mask, alloc_flags); + return alloc_flags; } @@ -4637,7 +4649,7 @@ retry: reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); if (reserve_flags) - alloc_flags = reserve_flags; + alloc_flags = current_alloc_flags(gfp_mask, reserve_flags); /* * Reset the nodemask and zonelist iterators if memory policies can be @@ -4714,7 +4726,7 @@ retry: /* Avoid allocations with no watermarks from looping endlessly */ if (tsk_is_oom_victim(current) && - (alloc_flags == ALLOC_OOM || + (alloc_flags & ALLOC_OOM || (gfp_mask & __GFP_NOMEMALLOC))) goto nopage; @@ -4806,8 +4818,7 @@ static inline bool prepare_alloc_pages(g if (should_fail_alloc_page(gfp_mask, order)) return false; - if (IS_ENABLED(CONFIG_CMA) && ac->migratetype == MIGRATE_MOVABLE) - *alloc_flags |= ALLOC_CMA; + *alloc_flags = current_alloc_flags(gfp_mask, *alloc_flags); return true; } From patchwork Fri Aug 7 06:26:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704939 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 91C4B14B7 for ; Fri, 7 Aug 2020 06:26:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 54BF822D2A for ; Fri, 7 Aug 2020 06:26:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="TRzxUwsU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 54BF822D2A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B4CBF8D0087; Fri, 7 Aug 2020 02:26:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AD9168D0026; Fri, 7 Aug 2020 02:26:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 973FF8D0087; Fri, 7 Aug 2020 02:26:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0203.hostedemail.com [216.40.44.203]) by kanga.kvack.org (Postfix) with ESMTP id 888318D0026 for ; Fri, 7 Aug 2020 02:26:16 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 11C09180AD804 for ; Fri, 7 Aug 2020 06:26:15 +0000 (UTC) X-FDA: 77122787910.01.copy05_160d01226fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id C20B510050E28 for ; Fri, 7 Aug 2020 06:26:11 +0000 (UTC) X-Spam-Summary: 50,0,0,918f0b9b093c76ea,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:12:41:355:379:800:901:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2525:2565:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4605:5007:6117:6119:6261:6653:7576:7901:8599:8603:8814:8957:9025:9121:9388:9545:10004:10030:10913:11026:11256:11257:11320:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12783:12986:13069:13311:13357:13846:14094:14096:14107:14181:14384:14721:14849:21080:21451:21627:21939:21990:30054:30055:30064,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: copy05_160d01226fbe X-Filterd-Recvd-Size: 3260 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:11 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id CF4BA22CF6; Fri, 7 Aug 2020 06:26:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781569; bh=2xJbUln9aDC0YWWrHAT4MCCjeEmJ50MCGNZYEQbYMRo=; h=Date:From:To:Subject:In-Reply-To:From; b=TRzxUwsUSsVrC+ZpoYFDMttKhbSoKX5fzak+9d0g0C+9qAkwuP/tzGDGOr66N34aC ByOXxptRA5E7qqKboQSW0epY6bDFcoUTaTovU41DZq4qKO/8xemZmUdTkjvwbP+qJx +iu850KsdnCDlWxzwMHE8l3cYkMqNTHUqZHs1vZ0= Date: Thu, 06 Aug 2020 23:26:08 -0700 From: Andrew Morton To: akpm@linux-foundation.org, grandmaster@al2klimov.de, linux-mm@kvack.org, mm-commits@vger.kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz Subject: [patch 156/163] mm: thp: replace HTTP links with HTTPS ones Message-ID: <20200807062608.WadMSEwki%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: C20B510050E28 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Alexander A. Klimov" Subject: mm: thp: replace HTTP links with HTTPS ones Rationale: Reduces attack surface on kernel devs opening the links for MITM as HTTPS traffic is much harder to manipulate. Deterministic algorithm: For each file: If not .svg: For each line: If doesn't contain `xmlns`: For each link, `http://[^# ]*(?:\w|/)`: If neither `gnu\.org/license`, nor `mozilla\.org/MPL`: If both the HTTP and HTTPS versions return 200 OK and serve the same content: Replace HTTP with HTTPS. [akpm@linux-foundation.org: fix amd.com URL, per Vlastimil] Link: http://lkml.kernel.org/r/20200713164345.36088-1-grandmaster@al2klimov.de Signed-off-by: Alexander A. Klimov Reviewed-by: Andrew Morton Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/huge_memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --- a/mm/huge_memory.c~mm-thp-replace-http-links-with-https-ones +++ a/mm/huge_memory.c @@ -2063,8 +2063,8 @@ static void __split_huge_pmd_locked(stru * free), userland could trigger a small page size TLB miss on the * small sized TLB while the hugepage TLB entry is still established in * the huge TLB. Some CPU doesn't like that. - * See http://support.amd.com/us/Processor_TechDocs/41322.pdf, Erratum - * 383 on page 93. Intel should be safe but is also warns that it's + * See http://support.amd.com/TechDocs/41322_10h_Rev_Gd.pdf, Erratum + * 383 on page 105. Intel should be safe but is also warns that it's * only safe if the permission and cache attributes of the two entries * loaded in the two TLB is identical (which should be the case here). * But it is generally safer to never allow small and huge TLB entries From patchwork Fri Aug 7 06:26:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704941 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFD30138C for ; Fri, 7 Aug 2020 06:26:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C004422D6D for ; Fri, 7 Aug 2020 06:26:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="a4g+Y6A0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C004422D6D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7533F8D0088; Fri, 7 Aug 2020 02:26:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6DA9D8D0026; Fri, 7 Aug 2020 02:26:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F1118D0088; Fri, 7 Aug 2020 02:26:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id 4FE448D0026 for ; Fri, 7 Aug 2020 02:26:20 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 29FD6282D for ; Fri, 7 Aug 2020 06:26:20 +0000 (UTC) X-FDA: 77122788120.29.ants21_030c07b26fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 35B07180868E8 for ; Fri, 7 Aug 2020 06:26:19 +0000 (UTC) X-Spam-Summary: 50,0,0,e8fe13374dace2a6,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2194:2199:2393:2525:2560:2563:2682:2685:2693:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3354:3865:3866:3867:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:5007:6119:6261:6653:7576:7903:8599:9025:9545:10004:10913:11026:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12663:12679:12783:12986:13141:13153:13161:13228:13229:13230:13255:13846:14093:14096:14181:14721:14849:21080:21451:21627:21740:21939:21990:30012:30054:30064,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: ants21_030c07b26fbe X-Filterd-Recvd-Size: 4372 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:14 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7140222D07; Fri, 7 Aug 2020 06:26:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781572; bh=9vvKDgJBhmzpnVGlr68RQj6uv0/viRH/J2s/tIT/nEI=; h=Date:From:To:Subject:In-Reply-To:From; b=a4g+Y6A011ftX0aX5yfLd9TjTdvHz+k/6/3a6E9wVVQetmWPC0EPetpepwDCy5teH Qdxj1pN60obk5rDaWXKGd+XrtH1PKVNXdjkXcS2WCuGorPzG3ZaINPzSyjGO8ZW84j QAIlx5noJnrZTuBsd5qLw+GWRW/eCyMOl/may8IY= Date: Thu, 06 Aug 2020 23:26:11 -0700 From: Andrew Morton To: aarcange@redhat.com, akpm@linux-foundation.org, linux-mm@kvack.org, mike.kravetz@oracle.com, mm-commits@vger.kernel.org, peterx@redhat.com, stable@vger.kernel.org, torvalds@linux-foundation.org, willy@infradead.org Subject: [patch 157/163] mm/hugetlb: fix calculation of adjust_range_if_pmd_sharing_possible Message-ID: <20200807062611.RGbHdqUiZ%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 35B07180868E8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Peter Xu Subject: mm/hugetlb: fix calculation of adjust_range_if_pmd_sharing_possible This is found by code observation only. Firstly, the worst case scenario should assume the whole range was covered by pmd sharing. The old algorithm might not work as expected for ranges like (1g-2m, 1g+2m), where the adjusted range should be (0, 1g+2m) but the expected range should be (0, 2g). Since at it, remove the loop since it should not be required. With that, the new code should be faster too when the invalidating range is huge. Mike said: : With range (1g-2m, 1g+2m) within a vma (0, 2g) the existing code will only : adjust to (0, 1g+2m) which is incorrect. : : We should cc stable. The original reason for adjusting the range was to : prevent data corruption (getting wrong page). Since the range is not : always adjusted correctly, the potential for corruption still exists. : : However, I am fairly confident that adjust_range_if_pmd_sharing_possible : is only gong to be called in two cases: : : 1) for a single page : 2) for range == entire vma : : In those cases, the current code should produce the correct results. : : To be safe, let's just cc stable. Link: http://lkml.kernel.org/r/20200730201636.74778-1-peterx@redhat.com Fixes: 017b1660df89 ("mm: migration: fix migration of huge PMD shared pages") Signed-off-by: Peter Xu Reviewed-by: Mike Kravetz Cc: Andrea Arcangeli Cc: Matthew Wilcox Cc: Signed-off-by: Andrew Morton --- mm/hugetlb.c | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) --- a/mm/hugetlb.c~mm-hugetlb-fix-calculation-of-adjust_range_if_pmd_sharing_possible +++ a/mm/hugetlb.c @@ -5314,25 +5314,21 @@ static bool vma_shareable(struct vm_area void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, unsigned long *start, unsigned long *end) { - unsigned long check_addr; + unsigned long a_start, a_end; if (!(vma->vm_flags & VM_MAYSHARE)) return; - for (check_addr = *start; check_addr < *end; check_addr += PUD_SIZE) { - unsigned long a_start = check_addr & PUD_MASK; - unsigned long a_end = a_start + PUD_SIZE; + /* Extend the range to be PUD aligned for a worst case scenario */ + a_start = ALIGN_DOWN(*start, PUD_SIZE); + a_end = ALIGN(*end, PUD_SIZE); - /* - * If sharing is possible, adjust start/end if necessary. - */ - if (range_in_vma(vma, a_start, a_end)) { - if (a_start < *start) - *start = a_start; - if (a_end > *end) - *end = a_end; - } - } + /* + * Intersect the range with the vma range, since pmd sharing won't be + * across vma after all + */ + *start = max(vma->vm_start, a_start); + *end = min(vma->vm_end, a_end); } /* From patchwork Fri Aug 7 06:26:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704943 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 84F42138C for ; Fri, 7 Aug 2020 06:26:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4CDAF22CF6 for ; Fri, 7 Aug 2020 06:26:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="LmJFtiTL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4CDAF22CF6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B5ACD8D0089; Fri, 7 Aug 2020 02:26:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B4A868D008A; Fri, 7 Aug 2020 02:26:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 99C718D0089; Fri, 7 Aug 2020 02:26:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0210.hostedemail.com [216.40.44.210]) by kanga.kvack.org (Postfix) with ESMTP id 805A28D0026 for ; Fri, 7 Aug 2020 02:26:22 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A540F2C32 for ; Fri, 7 Aug 2020 06:26:20 +0000 (UTC) X-FDA: 77122788120.01.start98_59018e526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 3798310050E20 for ; Fri, 7 Aug 2020 06:26:19 +0000 (UTC) X-Spam-Summary: 50,0,0,5312a5f686e7fda3,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2525:2560:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6119:6261:6653:7576:7903:8599:9025:9545:10004:11026:11658:11914:12043:12048:12114:12297:12438:12517:12519:12555:12679:12783:12895:12986:13069:13161:13229:13311:13357:14181:14384:14721:14849:21080:21451:21611:21627:21939:30003:30054:30064:30091,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: start98_59018e526fbe X-Filterd-Recvd-Size: 3333 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:18 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0018222D2C; Fri, 7 Aug 2020 06:26:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781576; bh=R/J9wYhkC2ROECBjlhXoxiqOfcsT0hjnDSt3+QsEc3Q=; h=Date:From:To:Subject:In-Reply-To:From; b=LmJFtiTLFGpWtnjMDkY/FcGBBIZC0Vm+T6XLdRC7IYBoTGQjANHx+Ag4Q0mGYRX59 2LEUWapAdYZpVkFgFNoI9Z+OWIdT9IZwg6NOylgC/YXbOu8D6CY69GGOS77LEcVop2 qw7oOgdBHXCMULZBYx5QyZWnnuyTVJLudlCEwP4A= Date: Thu, 06 Aug 2020 23:26:15 -0700 From: Andrew Morton To: aarcange@redhat.com, akpm@linux-foundation.org, hughd@google.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, mike.kravetz@oracle.com, mm-commits@vger.kernel.org, songliubraving@fb.com, stable@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 158/163] khugepaged: collapse_pte_mapped_thp() flush the right range Message-ID: <20200807062615.qiADx5DeQ%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 3798310050E20 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Hugh Dickins Subject: khugepaged: collapse_pte_mapped_thp() flush the right range pmdp_collapse_flush() should be given the start address at which the huge page is mapped, haddr: it was given addr, which at that point has been used as a local variable, incremented to the end address of the extent. Found by source inspection while chasing a hugepage locking bug, which I then could not explain by this. At first I thought this was very bad; then saw that all of the page translations that were not flushed would actually still point to the right pages afterwards, so harmless; then realized that I know nothing of how different architectures and models cache intermediate paging structures, so maybe it matters after all - particularly since the page table concerned is immediately freed. Much easier to fix than to think about. Link: http://lkml.kernel.org/r/alpine.LSU.2.11.2008021204390.27773@eggly.anvils Fixes: 27e1f8273113 ("khugepaged: enable collapse pmd for pte-mapped THP") Signed-off-by: Hugh Dickins Acked-by: Kirill A. Shutemov Cc: Andrea Arcangeli Cc: Mike Kravetz Cc: Song Liu Cc: [5.4+] Signed-off-by: Andrew Morton --- mm/khugepaged.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/mm/khugepaged.c~khugepaged-collapse_pte_mapped_thp-flush-the-right-range +++ a/mm/khugepaged.c @@ -1502,7 +1502,7 @@ void collapse_pte_mapped_thp(struct mm_s /* step 4: collapse pmd */ ptl = pmd_lock(vma->vm_mm, pmd); - _pmd = pmdp_collapse_flush(vma, addr, pmd); + _pmd = pmdp_collapse_flush(vma, haddr, pmd); spin_unlock(ptl); mm_dec_nr_ptes(mm); pte_free(mm, pmd_pgtable(_pmd)); From patchwork Fri Aug 7 06:26:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704945 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED56A138C for ; Fri, 7 Aug 2020 06:26:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B994222D0A for ; Fri, 7 Aug 2020 06:26:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="O2f/2sbE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B994222D0A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 008978D008A; Fri, 7 Aug 2020 02:26:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E099A8D0026; Fri, 7 Aug 2020 02:26:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC06C8D008B; Fri, 7 Aug 2020 02:26:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id AE3698D0026 for ; Fri, 7 Aug 2020 02:26:22 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B7D3A181AEF1D for ; Fri, 7 Aug 2020 06:26:21 +0000 (UTC) X-FDA: 77122788162.12.fear82_3717e6126fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 925A81800ACBF for ; Fri, 7 Aug 2020 06:26:21 +0000 (UTC) X-Spam-Summary: 50,0,0,7b9501741f3a46e5,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:966:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1544:1605:1711:1730:1747:1777:1792:2196:2199:2393:2525:2553:2560:2563:2682:2685:2693:2859:2897:2902:2909:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:4385:4605:5007:6119:6120:6261:6653:7576:7901:7903:8568:8599:8957:9025:9545:9592:10004:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12683:12783:12895:12986:13161:13229:14096:14181:14721:14849:21080:21324:21451:21627:21796:21939:21990:30016:30036:30054:30064:30070:30090,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: fear82_3717e6126fbe X-Filterd-Recvd-Size: 5703 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:20 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7E40F22CAE; Fri, 7 Aug 2020 06:26:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781580; bh=1tjUPsuMrDoDaN0OwDKkWfmJhb52c1RdcfD5pTRT4QU=; h=Date:From:To:Subject:In-Reply-To:From; b=O2f/2sbEfPz4vd9g5pVlXVwpavgqGFOdQ9j5CQcBggCAPy6GobgyTZnKeBk4d2WnA I/6G4XCc2VCJGNtrATT9jiOTEIhEF1Dr4v9cztOmvOUtcq7WyuAlHPdhfkeK917Tdo zSpgLQayffDSGApyVMOw0zZzwDyzun0iBUUljtrs= Date: Thu, 06 Aug 2020 23:26:18 -0700 From: Andrew Morton To: aarcange@redhat.com, akpm@linux-foundation.org, hughd@google.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, mike.kravetz@oracle.com, mm-commits@vger.kernel.org, songliubraving@fb.com, stable@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 159/163] khugepaged: collapse_pte_mapped_thp() protect the pmd lock Message-ID: <20200807062618.riSUc8ZXB%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 925A81800ACBF X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Hugh Dickins Subject: khugepaged: collapse_pte_mapped_thp() protect the pmd lock When retract_page_tables() removes a page table to make way for a huge pmd, it holds huge page lock, i_mmap_lock_write, mmap_write_trylock and pmd lock; but when collapse_pte_mapped_thp() does the same (to handle the case when the original mmap_write_trylock had failed), only mmap_write_trylock and pmd lock are held. That's not enough. One machine has twice crashed under load, with "BUG: spinlock bad magic" and GPF on 6b6b6b6b6b6b6b6b. Examining the second crash, page_vma_mapped_walk_done()'s spin_unlock of pvmw->ptl (serving page_referenced() on a file THP, that had found a page table at *pmd) discovers that the page table page and its lock have already been freed by the time it comes to unlock. Follow the example of retract_page_tables(), but we only need one of huge page lock or i_mmap_lock_write to secure against this: because it's the narrower lock, and because it simplifies collapse_pte_mapped_thp() to know the hpage earlier, choose to rely on huge page lock here. Link: http://lkml.kernel.org/r/alpine.LSU.2.11.2008021213070.27773@eggly.anvils Fixes: 27e1f8273113 ("khugepaged: enable collapse pmd for pte-mapped THP") Signed-off-by: Hugh Dickins Acked-by: Kirill A. Shutemov Cc: Andrea Arcangeli Cc: Mike Kravetz Cc: Song Liu Cc: [5.4+] Signed-off-by: Andrew Morton --- mm/khugepaged.c | 44 +++++++++++++++++++------------------------- 1 file changed, 19 insertions(+), 25 deletions(-) --- a/mm/khugepaged.c~khugepaged-collapse_pte_mapped_thp-protect-the-pmd-lock +++ a/mm/khugepaged.c @@ -1412,7 +1412,7 @@ void collapse_pte_mapped_thp(struct mm_s { unsigned long haddr = addr & HPAGE_PMD_MASK; struct vm_area_struct *vma = find_vma(mm, haddr); - struct page *hpage = NULL; + struct page *hpage; pte_t *start_pte, *pte; pmd_t *pmd, _pmd; spinlock_t *ptl; @@ -1432,9 +1432,17 @@ void collapse_pte_mapped_thp(struct mm_s if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE)) return; + hpage = find_lock_page(vma->vm_file->f_mapping, + linear_page_index(vma, haddr)); + if (!hpage) + return; + + if (!PageHead(hpage)) + goto drop_hpage; + pmd = mm_find_pmd(mm, haddr); if (!pmd) - return; + goto drop_hpage; start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); @@ -1453,30 +1461,11 @@ void collapse_pte_mapped_thp(struct mm_s page = vm_normal_page(vma, addr, *pte); - if (!page || !PageCompound(page)) - goto abort; - - if (!hpage) { - hpage = compound_head(page); - /* - * The mapping of the THP should not change. - * - * Note that uprobe, debugger, or MAP_PRIVATE may - * change the page table, but the new page will - * not pass PageCompound() check. - */ - if (WARN_ON(hpage->mapping != vma->vm_file->f_mapping)) - goto abort; - } - /* - * Confirm the page maps to the correct subpage. - * - * Note that uprobe, debugger, or MAP_PRIVATE may change - * the page table, but the new page will not pass - * PageCompound() check. + * Note that uprobe, debugger, or MAP_PRIVATE may change the + * page table, but the new page will not be a subpage of hpage. */ - if (WARN_ON(hpage + i != page)) + if (hpage + i != page) goto abort; count++; } @@ -1495,7 +1484,7 @@ void collapse_pte_mapped_thp(struct mm_s pte_unmap_unlock(start_pte, ptl); /* step 3: set proper refcount and mm_counters. */ - if (hpage) { + if (count) { page_ref_sub(hpage, count); add_mm_counter(vma->vm_mm, mm_counter_file(hpage), -count); } @@ -1506,10 +1495,15 @@ void collapse_pte_mapped_thp(struct mm_s spin_unlock(ptl); mm_dec_nr_ptes(mm); pte_free(mm, pmd_pgtable(_pmd)); + +drop_hpage: + unlock_page(hpage); + put_page(hpage); return; abort: pte_unmap_unlock(start_pte, ptl); + goto drop_hpage; } static int khugepaged_collapse_pte_mapped_thps(struct mm_slot *mm_slot) From patchwork Fri Aug 7 06:26:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704947 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 383B914B7 for ; Fri, 7 Aug 2020 06:26:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0521B22D70 for ; Fri, 7 Aug 2020 06:26:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="CnzQKDTO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0521B22D70 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 425348D008B; Fri, 7 Aug 2020 02:26:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 383308D0026; Fri, 7 Aug 2020 02:26:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 221DF8D008B; Fri, 7 Aug 2020 02:26:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0056.hostedemail.com [216.40.44.56]) by kanga.kvack.org (Postfix) with ESMTP id 0DE7B8D0026 for ; Fri, 7 Aug 2020 02:26:25 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D9796180AD801 for ; Fri, 7 Aug 2020 06:26:24 +0000 (UTC) X-FDA: 77122788288.01.place70_2e0d4f726fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id AF02710050E20 for ; Fri, 7 Aug 2020 06:26:24 +0000 (UTC) X-Spam-Summary: 50,0,0,ba8c09caaf43f16e,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:69:355:379:800:960:966:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2525:2560:2563:2682:2685:2693:2731:2859:2898:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:4385:5007:6119:6261:6653:7576:7903:8599:9025:9545:10004:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12783:12895:12986:13161:13229:13870:14096:14181:14721:14849:21067:21080:21433:21451:21627:21939:21972:30054:30064:30070,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: place70_2e0d4f726fbe X-Filterd-Recvd-Size: 5206 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:24 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2EABA22CAE; Fri, 7 Aug 2020 06:26:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781583; bh=+H9h3gbZQV/nqeTyWDou5z8jxCRM8miFBCjcPPPwNUE=; h=Date:From:To:Subject:In-Reply-To:From; b=CnzQKDTO/GhwIWZl+lmurwcUcn9cgAM7w/bTtkuKYNLmooK4exhSIJgt/KwHs7zdl exq6ia374gVa1Ux3ey3bZ6+s2BOj7+X+cK+4lZafT1Z7RsdOFbIWUV7+AuOcQ+Xnr2 6ajky4vnFJLz+0lHTv3/Ucj2AHSLWfmGq9PGF9gc= Date: Thu, 06 Aug 2020 23:26:22 -0700 From: Andrew Morton To: aarcange@redhat.com, akpm@linux-foundation.org, hughd@google.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, mike.kravetz@oracle.com, mm-commits@vger.kernel.org, songliubraving@fb.com, stable@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 160/163] khugepaged: retract_page_tables() remember to test exit Message-ID: <20200807062622.MKPXQwQaY%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: AF02710050E20 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Hugh Dickins Subject: khugepaged: retract_page_tables() remember to test exit Only once have I seen this scenario (and forgot even to notice what forced the eventual crash): a sequence of "BUG: Bad page map" alerts from vm_normal_page(), from zap_pte_range() servicing exit_mmap(); pmd:00000000, pte values corresponding to data in physical page 0. The pte mappings being zapped in this case were supposed to be from a huge page of ext4 text (but could as well have been shmem): my belief is that it was racing with collapse_file()'s retract_page_tables(), found *pmd pointing to a page table, locked it, but *pmd had become 0 by the time start_pte was decided. In most cases, that possibility is excluded by holding mmap lock; but exit_mmap() proceeds without mmap lock. Most of what's run by khugepaged checks khugepaged_test_exit() after acquiring mmap lock: khugepaged_collapse_pte_mapped_thps() and hugepage_vma_revalidate() do so, for example. But retract_page_tables() did not: fix that. The fix is for retract_page_tables() to check khugepaged_test_exit(), after acquiring mmap lock, before doing anything to the page table. Getting the mmap lock serializes with __mmput(), which briefly takes and drops it in __khugepaged_exit(); then the khugepaged_test_exit() check on mm_users makes sure we don't touch the page table once exit_mmap() might reach it, since exit_mmap() will be proceeding without mmap lock, not expecting anyone to be racing with it. Link: http://lkml.kernel.org/r/alpine.LSU.2.11.2008021215400.27773@eggly.anvils Fixes: f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages") Signed-off-by: Hugh Dickins Acked-by: Kirill A. Shutemov Cc: Andrea Arcangeli Cc: Mike Kravetz Cc: Song Liu Cc: [4.8+] Signed-off-by: Andrew Morton --- mm/khugepaged.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) --- a/mm/khugepaged.c~khugepaged-retract_page_tables-remember-to-test-exit +++ a/mm/khugepaged.c @@ -1532,6 +1532,7 @@ out: static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) { struct vm_area_struct *vma; + struct mm_struct *mm; unsigned long addr; pmd_t *pmd, _pmd; @@ -1560,7 +1561,8 @@ static void retract_page_tables(struct a continue; if (vma->vm_end < addr + HPAGE_PMD_SIZE) continue; - pmd = mm_find_pmd(vma->vm_mm, addr); + mm = vma->vm_mm; + pmd = mm_find_pmd(mm, addr); if (!pmd) continue; /* @@ -1570,17 +1572,19 @@ static void retract_page_tables(struct a * mmap_lock while holding page lock. Fault path does it in * reverse order. Trylock is a way to avoid deadlock. */ - if (mmap_write_trylock(vma->vm_mm)) { - spinlock_t *ptl = pmd_lock(vma->vm_mm, pmd); - /* assume page table is clear */ - _pmd = pmdp_collapse_flush(vma, addr, pmd); - spin_unlock(ptl); - mmap_write_unlock(vma->vm_mm); - mm_dec_nr_ptes(vma->vm_mm); - pte_free(vma->vm_mm, pmd_pgtable(_pmd)); + if (mmap_write_trylock(mm)) { + if (!khugepaged_test_exit(mm)) { + spinlock_t *ptl = pmd_lock(mm, pmd); + /* assume page table is clear */ + _pmd = pmdp_collapse_flush(vma, addr, pmd); + spin_unlock(ptl); + mm_dec_nr_ptes(mm); + pte_free(mm, pmd_pgtable(_pmd)); + } + mmap_write_unlock(mm); } else { /* Try again later */ - khugepaged_add_pte_mapped_thp(vma->vm_mm, addr); + khugepaged_add_pte_mapped_thp(mm, addr); } } i_mmap_unlock_write(mapping); From patchwork Fri Aug 7 06:26:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704949 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78937138C for ; Fri, 7 Aug 2020 06:26:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 440AA22CAE for ; Fri, 7 Aug 2020 06:26:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="19rWJdeF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 440AA22CAE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D1FB28D008C; Fri, 7 Aug 2020 02:26:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CAAA38D0026; Fri, 7 Aug 2020 02:26:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAE598D008C; Fri, 7 Aug 2020 02:26:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0068.hostedemail.com [216.40.44.68]) by kanga.kvack.org (Postfix) with ESMTP id 91B7C8D0026 for ; Fri, 7 Aug 2020 02:26:28 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 714AE824934B for ; Fri, 7 Aug 2020 06:26:28 +0000 (UTC) X-FDA: 77122788456.23.low41_0b03aa126fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 4997137606 for ; Fri, 7 Aug 2020 06:26:28 +0000 (UTC) X-Spam-Summary: 50,0,0,ccd6d6b6ea21fd38,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1542:1711:1730:1747:1777:1792:2393:2525:2560:2563:2682:2685:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4250:4321:5007:6119:6261:6653:7576:7682:7903:8599:9025:9545:10004:11026:11473:11658:11914:12043:12048:12052:12114:12296:12297:12438:12517:12519:12555:12679:12783:12895:12986:14181:14721:14849:21080:21433:21451:21627:21939:21990:30003:30054:30064:30070,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: low41_0b03aa126fbe X-Filterd-Recvd-Size: 3511 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:27 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8AB2D22CF6; Fri, 7 Aug 2020 06:26:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781587; bh=L411IMx8s6AO3vyH70GRl4hazc8nPOEqZzjjm7QWvWI=; h=Date:From:To:Subject:In-Reply-To:From; b=19rWJdeFJOoS8bkEAzfeHxevXyDePWDfnMGT++D+/Po2gRjxccU7K6+25SKlCt8jW 7wnInz2k4y+cisU+64RtQ0P7tEOiENsLIk/pxGyxuB9rlj3EFoCOsoVuDkxzqMNbg0 tlp6D3/8vLFoQlK8Ksbu2qOjTWFsUCNNTWKbGiX0= Date: Thu, 06 Aug 2020 23:26:25 -0700 From: Andrew Morton To: aarcange@redhat.com, akpm@linux-foundation.org, hughd@google.com, kirill.shutemov@linux.intel.com, linux-mm@kvack.org, mike.kravetz@oracle.com, mm-commits@vger.kernel.org, songliubraving@fb.com, stable@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 161/163] khugepaged: khugepaged_test_exit() check mmget_still_valid() Message-ID: <20200807062625.sPpz4GT5t%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 4997137606 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Hugh Dickins Subject: khugepaged: khugepaged_test_exit() check mmget_still_valid() Move collapse_huge_page()'s mmget_still_valid() check into khugepaged_test_exit() itself. collapse_huge_page() is used for anon THP only, and earned its mmget_still_valid() check because it inserts a huge pmd entry in place of the page table's pmd entry; whereas collapse_file()'s retract_page_tables() or collapse_pte_mapped_thp() merely clears the page table's pmd entry. But core dumping without mmap lock must have been as open to mistaking a racily cleared pmd entry for a page table at physical page 0, as exit_mmap() was. And we certainly have no interest in mapping as a THP once dumping core. Link: http://lkml.kernel.org/r/alpine.LSU.2.11.2008021217020.27773@eggly.anvils Fixes: 59ea6d06cfa9 ("coredump: fix race condition between collapse_huge_page() and core dumping") Signed-off-by: Hugh Dickins Cc: Andrea Arcangeli Cc: Song Liu Cc: Mike Kravetz Cc: Kirill A. Shutemov Cc: [4.8+] Signed-off-by: Andrew Morton --- mm/khugepaged.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) --- a/mm/khugepaged.c~khugepaged-khugepaged_test_exit-check-mmget_still_valid +++ a/mm/khugepaged.c @@ -431,7 +431,7 @@ static void insert_to_mm_slots_hash(stru static inline int khugepaged_test_exit(struct mm_struct *mm) { - return atomic_read(&mm->mm_users) == 0; + return atomic_read(&mm->mm_users) == 0 || !mmget_still_valid(mm); } static bool hugepage_vma_check(struct vm_area_struct *vma, @@ -1100,9 +1100,6 @@ static void collapse_huge_page(struct mm * handled by the anon_vma lock + PG_lock. */ mmap_write_lock(mm); - result = SCAN_ANY_PROCESS; - if (!mmget_still_valid(mm)) - goto out; result = hugepage_vma_revalidate(mm, address, &vma); if (result) goto out; From patchwork Fri Aug 7 06:26:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704951 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0209F138C for ; Fri, 7 Aug 2020 06:26:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C37B022CAE for ; Fri, 7 Aug 2020 06:26:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="I76bF/Lx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C37B022CAE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3376A8D008D; Fri, 7 Aug 2020 02:26:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 247B18D0026; Fri, 7 Aug 2020 02:26:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 111798D008D; Fri, 7 Aug 2020 02:26:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0062.hostedemail.com [216.40.44.62]) by kanga.kvack.org (Postfix) with ESMTP id F109D8D0026 for ; Fri, 7 Aug 2020 02:26:31 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C7779180AD801 for ; Fri, 7 Aug 2020 06:26:31 +0000 (UTC) X-FDA: 77122788582.15.ocean03_091102226fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id A0B5C1814B0C1 for ; Fri, 7 Aug 2020 06:26:31 +0000 (UTC) X-Spam-Summary: 50,0,0,b6caa23967d34917,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:968:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1540:1711:1730:1747:1777:1792:2198:2199:2393:2525:2560:2563:2682:2685:2731:2859:2902:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3867:3870:3871:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7514:7576:8599:9025:9545:10004:10913:11658:11914:12043:12048:12297:12517:12519:12555:12679:12783:12986:13069:13311:13357:13846:14181:14384:14721:14849:21080:21451:21627:21809:21939:30003:30054:30070,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: ocean03_091102226fbe X-Filterd-Recvd-Size: 2270 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:31 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 170DC22CF6; Fri, 7 Aug 2020 06:26:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781590; bh=ZZ9s+7THUYsNALh7tX+k1Qcg5zL3aj8zuj4I56CUPYw=; h=Date:From:To:Subject:In-Reply-To:From; b=I76bF/LxO04T9JSZM+8ruIJ2WKUx3wlxKv619X/UEfFr+Q19Km2jR6eBiVyGf6H2i FnRH/mpIgCWVRCDxuCnfLYgZo0h7wF0BG0bh1527kYd8VNrZXzuVpaU989eesC6i3l jlanj0aeQEY/K4lckLKoTb4TAlC9gwQjzBb1WVqM= Date: Thu, 06 Aug 2020 23:26:29 -0700 From: Andrew Morton To: akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, spacct.spacct@gmail.com, torvalds@linux-foundation.org Subject: [patch 162/163] mm/vmscan.c: fix typo Message-ID: <20200807062629.4TnTBWcyR%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: A0B5C1814B0C1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: dylan-meiners Subject: mm/vmscan.c: fix typo Change "optizimation" to "optimization". Link: http://lkml.kernel.org/r/20200609185144.10049-1-spacct.spacct@gmail.com Signed-off-by: dylan-meiners Reviewed-by: David Hildenbrand Signed-off-by: Andrew Morton --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/mm/vmscan.c~mm-vmscanc-fixed-typo +++ a/mm/vmscan.c @@ -910,7 +910,7 @@ static int __remove_mapping(struct addre * order to detect refaults, thus thrashing, later on. * * But don't store shadows in an address space that is - * already exiting. This is not just an optizimation, + * already exiting. This is not just an optimization, * inode reclaim needs to empty out the radix tree or * the nodes are lost. Don't plant shadows behind its * back. From patchwork Fri Aug 7 06:26:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 11704953 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 03AA3138C for ; Fri, 7 Aug 2020 06:26:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C8CEE22CAE for ; Fri, 7 Aug 2020 06:26:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="0wmJPurs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C8CEE22CAE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B236C8D008E; Fri, 7 Aug 2020 02:26:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AD2338D0026; Fri, 7 Aug 2020 02:26:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E7E98D008E; Fri, 7 Aug 2020 02:26:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0244.hostedemail.com [216.40.44.244]) by kanga.kvack.org (Postfix) with ESMTP id 8F16F8D0026 for ; Fri, 7 Aug 2020 02:26:43 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 37F31364D for ; Fri, 7 Aug 2020 06:26:35 +0000 (UTC) X-FDA: 77122788750.30.vein14_341616526fbe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 0B355180B3C83 for ; Fri, 7 Aug 2020 06:26:35 +0000 (UTC) X-Spam-Summary: 50,0,0,261fe9922ad2c0b9,d41d8cd98f00b204,akpm@linux-foundation.org,,RULES_HIT:41:355:379:800:960:967:973:988:989:1260:1263:1345:1359:1381:1431:1437:1534:1541:1711:1730:1747:1777:1792:2195:2199:2393:2525:2560:2563:2682:2685:2859:2895:2902:2912:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:3872:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4321:5007:6261:6653:7514:7576:8599:9025:9545:10004:10913:11026:11473:11658:11914:12043:12048:12297:12438:12517:12519:12555:12679:12783:12986:13069:13161:13229:13311:13357:13846:14181:14384:14721:14849:21080:21433:21451:21627:21939:30054:30064:30070,0,RBL:none,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: vein14_341616526fbe X-Filterd-Recvd-Size: 2954 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Aug 2020 06:26:34 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7A56422CF6; Fri, 7 Aug 2020 06:26:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1596781593; bh=DRFCcEScmQDmGhb5Lm/X4w55dtL6vDILWoa6JBc/yEw=; h=Date:From:To:Subject:In-Reply-To:From; b=0wmJPurs5alYMTmXme/aBdS7VerirRzLboitYXUK3sPBVs9GQRGOsiQFko8hYSu0O xJz3U0NtPajJ1CFoJy8qdXfdNTvYUvrC8lrrie3Anmrf7b179UrzXwuIEedlV4Pl3y 3AP5dsDD9Tz3GGqP1bIKXaXmfpQchPTfuJhsTsXo= Date: Thu, 06 Aug 2020 23:26:32 -0700 From: Andrew Morton To: akpm@linux-foundation.org, chris@chrisdown.name, guro@fb.com, hannes@cmpxchg.org, laoar.shao@gmail.com, linux-mm@kvack.org, mhocko@kernel.org, mm-commits@vger.kernel.org, shakeelb@google.com, torvalds@linux-foundation.org Subject: [patch 163/163] mm: vmscan: consistent update to pgrefill Message-ID: <20200807062632.VjEnptD9e%akpm@linux-foundation.org> In-Reply-To: <20200806231643.a2711a608dd0f18bff2caf2b@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 0B355180B3C83 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000050, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Shakeel Butt Subject: mm: vmscan: consistent update to pgrefill The vmstat pgrefill is useful together with pgscan and pgsteal stats to measure the reclaim efficiency. However vmstat's pgrefill is not updated consistently at system level. It gets updated for both global and memcg reclaim however pgscan and pgsteal are updated for only global reclaim. So, update pgrefill only for global reclaim. If someone is interested in the stats representing both system level as well as memcg level reclaim, then consult the root memcg's memory.stat instead of /proc/vmstat. Link: http://lkml.kernel.org/r/20200711011459.1159929-1-shakeelb@google.com Signed-off-by: Shakeel Butt Acked-by: Yafang Shao Acked-by: Roman Gushchin Acked-by: Chris Down Cc: Johannes Weiner Cc: Michal Hocko Signed-off-by: Andrew Morton --- mm/vmscan.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --- a/mm/vmscan.c~mm-vmscan-consistent-update-to-pgrefill +++ a/mm/vmscan.c @@ -2030,7 +2030,8 @@ static void shrink_active_list(unsigned __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken); - __count_vm_events(PGREFILL, nr_scanned); + if (!cgroup_reclaim(sc)) + __count_vm_events(PGREFILL, nr_scanned); __count_memcg_events(lruvec_memcg(lruvec), PGREFILL, nr_scanned); spin_unlock_irq(&pgdat->lru_lock);