From patchwork Wed Feb 23 05:21:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Junaid Shahid X-Patchwork-Id: 12756379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A911FC433F5 for ; Wed, 23 Feb 2022 05:24:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BCA538D0015; Wed, 23 Feb 2022 00:24:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B7AD58D0001; Wed, 23 Feb 2022 00:24:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9CB788D0015; Wed, 23 Feb 2022 00:24:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0182.hostedemail.com [216.40.44.182]) by kanga.kvack.org (Postfix) with ESMTP id 8A6058D0001 for ; Wed, 23 Feb 2022 00:24:29 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4812D181AF5D0 for ; Wed, 23 Feb 2022 05:24:29 +0000 (UTC) X-FDA: 79172904258.18.8FF179E Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf03.hostedemail.com (Postfix) with ESMTP id CE02A20002 for ; Wed, 23 Feb 2022 05:24:28 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id i6-20020a255406000000b006241e97e420so18864029ybb.5 for ; Tue, 22 Feb 2022 21:24:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0QeFswLcX9iEWT+ZD7rOizRcb8mHIR1VarnP+zim9mU=; b=AoOb1g/bwih9VgHUpDsuRj472kE7mvs4Hd5rKJ629FXGqay4OsONzwnNGCn4KSszBQ 7tR0Ehd/9kdVBMulmZqqn0EKhuRWwo5FmNiVUwzvcobRe45iRRTjqmq29bq6BO+aH2aK iVFm5Q63DJUj0wcMPLNxFHfvSAy/DcNXa1BAm45xpqSab7/o/iDvyz2xnot3Gux2i/bz ep5/x08uO4oi4Oa/VRppWnEJLR30fLBFaY+9aHNaO68XzDzENmX6oojUDxShaZpJxCzn AzyiCxL/E+FUObYZMNwPUDfM+bxPv2Rn6QD88aXQUJCPLfukjGOAKhVZbWGz7HVG4WwU EzCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0QeFswLcX9iEWT+ZD7rOizRcb8mHIR1VarnP+zim9mU=; b=j0FhkAYBYF8b58ZT8v3ispYsoFjTaSCQQ+hdlv82NGUhkN4S7RAxQC08cAPb596r3I xrs+O85L6GbcLbcuukNq/eIkjgJSgdpENQrrxI4jTgD/rcu1WFNELJ6IEGV3KbCCJ8Gd 2+67UaY2myNwzCYu5Lqv8YyXktpFsI1ZOcLaLlV2aWOgYXe/f7R+yTRka2xSL5T96cFh BDuBn87mJ2oAjUKMowetS8dZiIeGx9qE3yV3CBaTux+HB6B08m31onoz1JLnOhF04fN5 0BbKunzam6+8D9Ca+LASiI1cOVVbNuFzesi0fFCpuhlQCe3tAuZt+RLBsfwVx7i1lnr5 duJg== X-Gm-Message-State: AOAM531f8CDGubbZ76QQcGITCDWMG7T+WVv35Xg0J1zLoA81W47HrmBB kfVIgbWXEdWTj+1I5jocf2wFxpBlGBWH X-Google-Smtp-Source: ABdhPJx4QwVPvcY0cnBZCAO3p1suuFvqTv6OhknMLXEOsTBctmZQz9YZ5chFPkEZtJ5h60746WaCODirXSsF X-Received: from js-desktop.svl.corp.google.com ([2620:15c:2cd:202:ccbe:5d15:e2e6:322]) (user=junaids job=sendgmr) by 2002:a25:6fc1:0:b0:624:43a0:c16c with SMTP id k184-20020a256fc1000000b0062443a0c16cmr21681170ybc.222.1645593868088; Tue, 22 Feb 2022 21:24:28 -0800 (PST) Date: Tue, 22 Feb 2022 21:21:55 -0800 In-Reply-To: <20220223052223.1202152-1-junaids@google.com> Message-Id: <20220223052223.1202152-20-junaids@google.com> Mime-Version: 1.0 References: <20220223052223.1202152-1-junaids@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [RFC PATCH 19/47] mm: asi: Support for locally nonsensitive page allocations From: Junaid Shahid To: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, pjt@google.com, oweisse@google.com, alexandre.chartre@oracle.com, rppt@linux.ibm.com, dave.hansen@linux.intel.com, peterz@infradead.org, tglx@linutronix.de, luto@kernel.org, linux-mm@kvack.org X-Rspamd-Queue-Id: CE02A20002 X-Stat-Signature: x6n5aub3uzh4ge3eunu8qr1sk6ayqw6x Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="AoOb1g/b"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3DMUVYgcKCAUozsfnixlttlqj.htrqnsz2-rrp0fhp.twl@flex--junaids.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3DMUVYgcKCAUozsfnixlttlqj.htrqnsz2-rrp0fhp.twl@flex--junaids.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1645593868-23008 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A new GFP flag, __GFP_LOCAL_NONSENSITIVE, is added to allocate pages that are considered non-sensitive within the context of the current process, but sensitive in the context of other processes. For these allocations, page->asi_mm is set to the current mm during allocation. It must be set to the same value when the page is freed. Though it can potentially be overwritten and used for some other purpose in the meantime, as long as it is restored before freeing. Signed-off-by: Junaid Shahid --- include/linux/gfp.h | 5 +++- include/linux/mm_types.h | 17 ++++++++++-- include/trace/events/mmflags.h | 1 + mm/page_alloc.c | 47 ++++++++++++++++++++++++++++------ tools/perf/builtin-kmem.c | 1 + 5 files changed, 60 insertions(+), 11 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 07a99a463a34..2ab394adbda3 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -62,8 +62,10 @@ struct vm_area_struct; #endif #ifdef CONFIG_ADDRESS_SPACE_ISOLATION #define ___GFP_GLOBAL_NONSENSITIVE 0x4000000u +#define ___GFP_LOCAL_NONSENSITIVE 0x8000000u #else #define ___GFP_GLOBAL_NONSENSITIVE 0 +#define ___GFP_LOCAL_NONSENSITIVE 0 #endif /* If the above are modified, __GFP_BITS_SHIFT may need updating */ @@ -255,9 +257,10 @@ struct vm_area_struct; /* Allocate non-sensitive memory */ #define __GFP_GLOBAL_NONSENSITIVE ((__force gfp_t)___GFP_GLOBAL_NONSENSITIVE) +#define __GFP_LOCAL_NONSENSITIVE ((__force gfp_t)___GFP_LOCAL_NONSENSITIVE) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT 27 +#define __GFP_BITS_SHIFT 28 #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8624d2783661..f9702d070975 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -193,8 +193,21 @@ struct page { struct rcu_head rcu_head; #ifdef CONFIG_ADDRESS_SPACE_ISOLATION - /* Links the pages_to_free_async list */ - struct llist_node async_free_node; + struct { + /* Links the pages_to_free_async list */ + struct llist_node async_free_node; + + unsigned long _asi_pad_1; + unsigned long _asi_pad_2; + + /* + * Upon allocation of a locally non-sensitive page, set + * to the allocating mm. Must be set to the same mm when + * the page is freed. May potentially be overwritten in + * the meantime, as long as it is restored before free. + */ + struct mm_struct *asi_mm; + }; #endif }; diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 96e61d838bec..c00b8a4e1968 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -51,6 +51,7 @@ {(unsigned long)__GFP_KSWAPD_RECLAIM, "__GFP_KSWAPD_RECLAIM"},\ {(unsigned long)__GFP_ZEROTAGS, "__GFP_ZEROTAGS"}, \ {(unsigned long)__GFP_SKIP_KASAN_POISON,"__GFP_SKIP_KASAN_POISON"},\ + {(unsigned long)__GFP_LOCAL_NONSENSITIVE, "__GFP_LOCAL_NONSENSITIVE"},\ {(unsigned long)__GFP_GLOBAL_NONSENSITIVE, "__GFP_GLOBAL_NONSENSITIVE"}\ #define show_gfp_flags(flags) \ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a4048fa1868a..01784bff2a80 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5231,19 +5231,33 @@ early_initcall(asi_page_alloc_init); static int asi_map_alloced_pages(struct page *page, uint order, gfp_t gfp_mask) { uint i; + struct asi *asi; + + VM_BUG_ON((gfp_mask & (__GFP_GLOBAL_NONSENSITIVE | + __GFP_LOCAL_NONSENSITIVE)) == + (__GFP_GLOBAL_NONSENSITIVE | __GFP_LOCAL_NONSENSITIVE)); if (!static_asi_enabled()) return 0; + if (!(gfp_mask & (__GFP_GLOBAL_NONSENSITIVE | + __GFP_LOCAL_NONSENSITIVE))) + return 0; + if (gfp_mask & __GFP_GLOBAL_NONSENSITIVE) { + asi = ASI_GLOBAL_NONSENSITIVE; for (i = 0; i < (1 << order); i++) __SetPageGlobalNonSensitive(page + i); - - return asi_map_gfp(ASI_GLOBAL_NONSENSITIVE, page_to_virt(page), - PAGE_SIZE * (1 << order), gfp_mask); + } else { + asi = ASI_LOCAL_NONSENSITIVE; + for (i = 0; i < (1 << order); i++) { + __SetPageLocalNonSensitive(page + i); + page[i].asi_mm = current->mm; + } } - return 0; + return asi_map_gfp(asi, page_to_virt(page), + PAGE_SIZE * (1 << order), gfp_mask); } static bool asi_unmap_freed_pages(struct page *page, unsigned int order) @@ -5251,18 +5265,28 @@ static bool asi_unmap_freed_pages(struct page *page, unsigned int order) void *va; size_t len; bool async_flush_needed; + struct asi *asi; + + VM_BUG_ON(PageGlobalNonSensitive(page) && PageLocalNonSensitive(page)); if (!static_asi_enabled()) return true; - if (!PageGlobalNonSensitive(page)) + if (PageGlobalNonSensitive(page)) + asi = ASI_GLOBAL_NONSENSITIVE; + else if (PageLocalNonSensitive(page)) + asi = &page->asi_mm->asi[0]; + else return true; + /* Heuristic to check that page->asi_mm is actually an mm_struct */ + VM_BUG_ON(PageLocalNonSensitive(page) && asi->mm != page->asi_mm); + va = page_to_virt(page); len = PAGE_SIZE * (1 << order); async_flush_needed = irqs_disabled() || in_interrupt(); - asi_unmap(ASI_GLOBAL_NONSENSITIVE, va, len, !async_flush_needed); + asi_unmap(asi, va, len, !async_flush_needed); if (!async_flush_needed) return true; @@ -5476,8 +5500,15 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, return NULL; } - if (static_asi_enabled() && (gfp & __GFP_GLOBAL_NONSENSITIVE)) - gfp |= __GFP_ZERO; + if (static_asi_enabled()) { + if ((gfp & __GFP_LOCAL_NONSENSITIVE) && + !mm_asi_enabled(current->mm)) + gfp &= ~__GFP_LOCAL_NONSENSITIVE; + + if (gfp & (__GFP_GLOBAL_NONSENSITIVE | + __GFP_LOCAL_NONSENSITIVE)) + gfp |= __GFP_ZERO; + } gfp &= gfp_allowed_mask; /* diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c index 5857953cd5c1..a2337fc3404f 100644 --- a/tools/perf/builtin-kmem.c +++ b/tools/perf/builtin-kmem.c @@ -661,6 +661,7 @@ static const struct { { "__GFP_DIRECT_RECLAIM", "DR" }, { "__GFP_KSWAPD_RECLAIM", "KR" }, { "__GFP_GLOBAL_NONSENSITIVE", "GNS" }, + { "__GFP_LOCAL_NONSENSITIVE", "LNS" }, }; static size_t max_gfp_len;