From patchwork Thu Mar 13 18:11:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 14015742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98B18C282DE for ; Thu, 13 Mar 2025 18:12:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5344B28001A; Thu, 13 Mar 2025 14:12:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49638280010; Thu, 13 Mar 2025 14:12:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33ACA28001A; Thu, 13 Mar 2025 14:12:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 11951280010 for ; Thu, 13 Mar 2025 14:12:06 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A5FA552A70 for ; Thu, 13 Mar 2025 18:12:07 +0000 (UTC) X-FDA: 83217321894.19.A05692D Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf06.hostedemail.com (Postfix) with ESMTP id 47A43180002 for ; Thu, 13 Mar 2025 18:12:04 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YDi4XS6Z; spf=pass (imf06.hostedemail.com: domain of 38x_TZwgKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=38x_TZwgKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889525; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jZCBSIxQMZuxgPH5nxXznTpvfPN1Wm6wtktBoLoOnVE=; b=O5b1QZG6hyFjiQrTtA5rhD/QWKE9dTRBhh6fICvxhPMlDzMuEVlo985c/BN5pPp+aV0yLl 40DgLedk18AZkaiBxKvreSeNh3yko4XnuAj55Byg1o9unWKL5udrc8OFghfVOpzN+fTQ9t T1Hcro7bTVf0G/7ZSGOFiXKpgMD8N7w= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YDi4XS6Z; spf=pass (imf06.hostedemail.com: domain of 38x_TZwgKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=38x_TZwgKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889525; a=rsa-sha256; cv=none; b=5E40XVyngn5PAkKt/xwKSUG+X0XKZ/HIlseTvO+gUhGpcdO+gA1CVfsMs9xuMcbteKyHUY JIXPEZ47a918Y6Or4LarpYxo8wES7lclhazLteImCcWp9PCmtEIPP5NSG0qOcwAM6APA6N bQNsdZXPqkF7iRIntWSfrW5LEw/0aX4= Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43ceeaf1524so6822145e9.1 for ; Thu, 13 Mar 2025 11:12:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889523; x=1742494323; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jZCBSIxQMZuxgPH5nxXznTpvfPN1Wm6wtktBoLoOnVE=; b=YDi4XS6ZV67BTzUJR8FQcP7aBnCeqZ7YoMpOIiwhl8MwNsh94Oss62eNhJvRcxAeYw OA1Q0wMkmdqGVC8z2WdHYHmY9Strddruxw2+gPIdYypxvVAeOAOG8zSio//dc5KLcJNo sD5W9qLNNs0Le0/PdMZ8rH51XobPInJiJpeNxNUbwHTs4yzNNgoP+Nyma/m1pTii1d4S DhyIdF4nN6rG80k9UofJGzcNLYaHIDS8ReSugmC0Tj1m4qi2cJJx0b7TYPy0NH8Q5EUu FTMVNs5//CrtdLzY9TVrPszl7A9qDQ4RXeDiVf//GbGLbTUhEnwi0Gg/iB3FFYwD7XGP 7l2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889523; x=1742494323; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jZCBSIxQMZuxgPH5nxXznTpvfPN1Wm6wtktBoLoOnVE=; b=KqGn1Iuql4dKtSL6cXwyxr9SiO/x0xzYswwjvLFhhkl7jO9GI+elqGdWli9e6geK2s BPtU/cioJDrNuc+JES431QzGFpyjkap9kim7prPXxhvEzq9wBVREM/FvQ1KQbzh6vOBP +wfbYTgwUt8+qzW6bI6+a6H2sDCImWGuXxEMytUtjMCQyEkGa5HZnfAfV6bucFNBSq1b SQzYVhjI0ig1M5zfWmYjMSc1e+uBzHhntLXrSmJdXU2dzWICY2NwpDvT1MtmCmRapQ5O 6FbGYBY+AN8NtKFYqe+Eja8Q3kTd8OEOiAuNkZzmrXOzTB8JAcfffITjefuExqg7cxl0 YBgQ== X-Forwarded-Encrypted: i=1; AJvYcCWjQ2WjKf/UCTSyVkvqz0HXHAaJP6Trw8WW69CgN0IXPwpmSXH36PIf55xBzfENac1TO4sc7X6Sxg==@kvack.org X-Gm-Message-State: AOJu0YxkaPUJ9Aa2VWPeb00w5AGzbZg5R/ZzWWyyjshm04hWuif3jlXP njk6jw3WnOwMsgqCXKCUDBv8WHPruozJQne2FH82C6bmzT1uPGP60JLBgyVopoBpidpxjyBM5hL hPY9Oe0fj/A== X-Google-Smtp-Source: AGHT+IFbs6k3vnl5QQ+C0J0hTTMoncea9PLFMkDkNtbfor/v1Jhe1WlJqAJFv/1WYWGnmKJfkaAhbBEhyKJgWQ== X-Received: from wmrn39.prod.google.com ([2002:a05:600c:5027:b0:43c:f03d:18aa]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3143:b0:439:4c1e:d810 with SMTP id 5b1f17b1804b1-43d1808593dmr31746155e9.9.1741889523325; Thu, 13 Mar 2025 11:12:03 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:30 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-11-04972e046cea@google.com> Subject: [PATCH RFC 11/11] mm/page_alloc: Add support for ASI-unmapping pages From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed X-Rspam-User: X-Rspamd-Queue-Id: 47A43180002 X-Rspamd-Server: rspam08 X-Stat-Signature: zz195gxmfa1tdso4xdsqbexygeeamwh9 X-HE-Tag: 1741889524-975425 X-HE-Meta: U2FsdGVkX1/BE1Cc2ZVM9QzsAKaClfk6rSHlhvb9hNTps/PSgsA4KWp6Jvyth69d69zrg+gUx/ED0A9Y6f7rCAof71Thx10Uo4idgqj+pIwUsB2P6CXc7gsVWc4FK8HwdNhG4ZK8y0ImNNZmDoyFkWCibYNY6Tv0g6UZitEVB6PvkKlXGP6OsQc/N7crvdvmGQZKNPf6xcsRq5Zv1OKdleEvz5PzKXs9qFU/RvfyGZ04LTV5WucYVZtemEymwEETo3meSBrH0HEutG7DvF0TjwffbpBbxW7oLwMidOx33uzq50n2ottLkR4dpmV9jraqWjNkUiot08EfAjpUFTkpDkV3uldP3oPMP6mDQMMOSGClcjQMPTi9j/115TmlyBGi+UvBGzvHeOUuBz+YgDxc3rQvsMU35xNdPAYc/s0eE4yXhdp6bU66/Ypi82KpOg6CJYLRJcA5mlT568nr2dx48A24XWC4TeYZeyX1sJPLq/maDDYCm8KgNaAzFPOj5gyZCdQuSZtKMSGwu0h3iUBCDiaVinOfBWOUgt2xdWkR4VXQ8pQlA0aaSkR4iZ2PqtV4PHwdNyAypGzCapMqVSA+ywJYeKe3ZLv1XAov3a9wqzw5q5tQU+3Ser9S0Icoo/ar/6wrXd2AgG4I2+T6o67dE+ruwN3uXGxzyoOVMrhdGUWGtrw0ZSAASk9MeJDJWOwAEtl5IehcgIFNtFbgpIwDFnOk/erd9ulNaAn/dlgteErS2JXcH8Dw9FUDB04ptypb+8CyZrOvsp45m/FLgj2Kzwu+R/LlefyFV0Puw0N8FjugsaTydr8y60TSw+mCfOPODJSiGew5LdMJHZkzD+UbFsPKAaIyTpe4+O0gDkwUTPK9kW9q/UZ20t+T68lX+FhrXC229/kXR5czNpUNML3lORcqMlbRcHhxRrIa28DErK3ySEPsrn6nmJ3yAdo7Ap8HvuQHeJecwPiMfCGtZJX WruXYgp/ s8G3Mkkwo6/Q5Uiw6hjncqCcpBzX3j7+sOtrSK/FX0BngSm92rsB1oVyJYyr1ZlW7lGIFVRb86jwNP7KZeCu/I0yn6VXRt3uJZqlEhnUc3mLrYjbBqw19yD4eY4wC5tFjduX9p8Du3iEv1RHC57BY7GmO9EVSQ5QC6+BoOSgaxXUrVtSLAmJ/68wQEZS4aSj8wqgLd2ACRWnMjF0e9QMsHlDmA1+TWg19hFdQou8HG9VYlAa2kwr2Ol7U/dfSSnIj0nNMESBEnKaakRkKVRErMwAU6+OgPH+KGCpiCEdNa8qAa7brw4d+Aw8jPYLedNDp23z/velKV8MKxWR+CHux3G/G56DBEfMBa4f/+wgM58+y8EsPhsx3W7xUi57EIqCNRqyVa7ZE+AlNO6DvoSe0oZ6nsV7gnpnZHyvZnW1FoIsoIRGPrQAfIbhzNp8D7HghOjN8sv9HPVvR1jdgvMGvkaTJVZK3Dydl8X8YRECy7j0ZhN3B6koWxoCWb4qumjO4Yw2rlCxtUSKCQquyfgpQ3rBny2pyW2u/JjX5BPrWF5wBkW/iB6LEuBSaQ21G+pSQ56O+F7PlIGG57M2HHyFxuJ07AmDi4M2nMZfY5+NNuAs6cbM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: While calling asi_map() is pretty easy, to unmap pages we need to ensure a TLB shootdown is complete before we allow them to be allocated. Therefore, treat this as a special case of buddy allocation. Allocate an entire block, release the zone lock and enable interrupts, then do the unmap and TLB shootdown. Once that's complete, return any unwanted pages within the block to the freelists. Signed-off-by: Brendan Jackman --- mm/internal.h | 5 ++++ mm/page_alloc.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 86 insertions(+), 7 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index b82ab345fb994b7c4971e550556e24bb68f315f6..7904be86fa2c7fded62c100d84fe572c15407ccf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1189,6 +1189,11 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define ALLOC_ASI_UNMAP 0x1000 /* allow asi_unmap(), requiring TLB shootdown. */ +#else +#define ALLOC_ASI_UNMAP 0x0 +#endif /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0d8bbad8675c99282f308c4a4122d5d9c4b14dae..9ac883d7a71387d291bc04bad675e2545dd7ba0f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1627,6 +1627,7 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, * The other migratetypes do not have fallbacks. * * Note there are no fallbacks from sensitive to nonsensitive migratetypes. + * That's instead handled as a totally special case in __rmqueue_asi_unmap(). */ #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION #define TERMINATE_FALLBACK -1 @@ -2790,7 +2791,77 @@ static inline void zone_statistics(struct zone *preferred_zone, struct zone *z, #endif } -static __always_inline +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +/* + * Allocate a page by converting some memory to sensitive, by doing an ASI + * unmap. This can't be done via __rmqueue_fallback because that unmap requires + * a TLB flush which can only be done with interrupts on. + */ +static inline +struct page *__rmqueue_asi_unmap(struct zone *zone, unsigned int request_order, + unsigned int alloc_flags, int migratetype) +{ + struct page *page; + int alloc_order; + int i; + + lockdep_assert_irqs_enabled(); + + if (!(alloc_flags & ALLOC_ASI_UNMAP)) + return NULL; + + VM_WARN_ON_ONCE(migratetype == MIGRATE_UNMOVABLE_NONSENSITIVE); + + /* + * Need to unmap a whole pageblock (otherwise it might require + * allocating pagetables). Can't do that with the zone lock held, but we + * also can't flip the block's migratetype until the flush is complete, + * otherwise any concurrent sensitive allocations could momentarily leak + * data into the restricted address space. As a simple workaround, + * "allocate" at least the whole block, unmap it (with IRQs enabled), + * then free any remainder of the block again. + * + * An alternative to this could be to synchronize an intermediate state + * on the pageblock (since since this code can't be called in an IRQ, + * this shouldn't be too bad - it's likely OK to just busy-wait until + * any conurrent TLB flush completes.). + */ + + alloc_order = max(request_order, pageblock_order); + spin_lock_irq(&zone->lock); + page = __rmqueue_smallest(zone, alloc_order, MIGRATE_UNMOVABLE_NONSENSITIVE); + spin_unlock_irq(&zone->lock); + if (!page) + return NULL; + + asi_unmap(page, 1 << alloc_order); + + change_pageblock_range(page, alloc_order, migratetype); + + if (request_order >= alloc_order) + return page; + + spin_lock_irq(&zone->lock); + for (i = request_order; i < alloc_order; i++) { + struct page *page_to_free = page + (1 << i); + + __free_one_page(page_to_free, page_to_pfn(page_to_free), zone, i, + migratetype, FPI_SKIP_REPORT_NOTIFY); + } + spin_unlock_irq(&zone->lock); + + return page; +} +#else +static inline +struct page *__rmqueue_asi_unmap(struct zone *zone, unsigned int order, + unsigned int alloc_flags, int migratetype) +{ + return NULL; +} +#endif + +static noinline struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, unsigned int order, unsigned int alloc_flags, int migratetype) @@ -2814,13 +2885,14 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, */ if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK))) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); - - if (!page) { - spin_unlock_irqrestore(&zone->lock, flags); - return NULL; - } } spin_unlock_irqrestore(&zone->lock, flags); + + if (!page) + page = __rmqueue_asi_unmap(zone, order, alloc_flags, migratetype); + + if (!page) + return NULL; } while (check_new_pages(page, order)); __count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order); @@ -3356,6 +3428,8 @@ static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask, if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; #endif + if (gfp_mask & __GFP_SENSITIVE && gfpflags_allow_blocking(gfp_mask)) + alloc_flags |= ALLOC_ASI_UNMAP; return alloc_flags; } @@ -4382,7 +4456,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); if (reserve_flags) alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, reserve_flags) | - (alloc_flags & ALLOC_KSWAPD); + (alloc_flags & (ALLOC_KSWAPD | ALLOC_ASI_UNMAP)); /* * Reset the nodemask and zonelist iterators if memory policies can be