From patchwork Fri Jul 12 17:00:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD906C2BD09 for ; Fri, 12 Jul 2024 17:01:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F8216B00A3; Fri, 12 Jul 2024 13:01:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 282516B00A4; Fri, 12 Jul 2024 13:01:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 086566B00A5; Fri, 12 Jul 2024 13:01:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DB9846B00A3 for ; Fri, 12 Jul 2024 13:01:27 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3A36C160C58 for ; Fri, 12 Jul 2024 17:01:27 +0000 (UTC) X-FDA: 82331716614.08.6E54B5B Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf04.hostedemail.com (Postfix) with ESMTP id 4069740012 for ; Fri, 12 Jul 2024 17:01:24 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ROohRSID; spf=pass (imf04.hostedemail.com: domain of 3Y2GRZggKCKYPGIQSGTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3Y2GRZggKCKYPGIQSGTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803669; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tzyFc6toxGAQrtxbLGqZIE4RB+JOOXOhclq2H8+llZE=; b=IhYU5mLtgQxizFsp/huhhNgwQh4hgVnKp1XYaIo2VklO9S0my3kvB05Qs/sqpzv0XEJR/K TNNf/Rqc7PjAVylryzsr2k6AWWxwGAp/CMqmXjPGnJBTC8gQ3bMHKlNF2IziGn/jI+Fu9h BLxMkt1WcHLc7MXzQTCjEdcIBEMMwW4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ROohRSID; spf=pass (imf04.hostedemail.com: domain of 3Y2GRZggKCKYPGIQSGTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3Y2GRZggKCKYPGIQSGTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803669; a=rsa-sha256; cv=none; b=gpbwvGOrbPSTqcIL7AO6EEBSjlRjWyRwwgzj/XzQUVs5PG4eN8VPeLMs3rfnLBKCXHJ0rm kvN9rDykp4tHfmu3oACb/bUhdDykm5jEUP8HBYEUWH9Kc6kQryvWMCBBm6Y8YNX61Um+L9 k1yTr8UhQv+m3U9rAPBwNx1smuaW8ZE= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4266fbae4c6so15467995e9.0 for ; Fri, 12 Jul 2024 10:01:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803684; x=1721408484; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tzyFc6toxGAQrtxbLGqZIE4RB+JOOXOhclq2H8+llZE=; b=ROohRSIDsX0lUX6mhLpijgiXanWO/6e3v2FVn9RZisArO+vpP7dLXy3HWsahUSwA5J halAz2WsFU6k3LdoOlxgdPW7esN0PAqYUGid4Gup8CDjQVAv8gboJpBg7GdIegrGBoiG +mAQI9k3AkvsTvYi4wC1eFx67RTdWPpx/EcRl7m7khDowEO2LwMUoBLmIbR4bwqxu0MA DlRtlr/41+n20C8cN82YvOcMVs+wMDMKvpeXOsyTzKWRKEYhYJGir45ZVlZgfOfY1+D8 WPKyR/8M1Hevokvs85v6MSpxwVXVueOc6L1D8sTFnjG8j9OkUv2JeCRgmWDeyjTP+I1c MtIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803684; x=1721408484; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tzyFc6toxGAQrtxbLGqZIE4RB+JOOXOhclq2H8+llZE=; b=VMpfWkAfyPQs0Apv4f2AA6Sol0UNHCilpd9n19XdzO5h1wC5JYImpZfXPR4POkj1Sx 8KkzXqrO/HSD2vl1LifPYvdKt1/CgI3wiaMYqQIL4vILMHKZnSakZ3Udk27GBnED+NhB mtTnwxJKSMEJ8rrD48KvGxosL9BNAQypeobVuiPUv/y1JlKVKUYKeiZtJek+4dWF8ly6 WXnLj1Yfl6nqbhNYn16lhpafum0+hPXv0018abcN3w6egHIejbZUpNK8FuBay7s+jQgK a+et8TyHX+axD/EKOvUk+m9lUP5LiKS41Io/2I/XTL5/BBqbydImLaFQcMDZF6F8yO3O ClfA== X-Forwarded-Encrypted: i=1; AJvYcCUkyKu+k61vJUa/0J9gxQ7OBTfJED8GXnoMgh3U82cOsuTaKpNz0E4oR2NUmZM6dHoWoFBL7fEzxpd27cwM9bo9T5c= X-Gm-Message-State: AOJu0YzfvsAs/DNdQS8x27m+zhKZAmq0XDme4VuwFSbtkbTlqAg/TJVC NAPuIA26u2PB/K6kwdinfvmzIMkukzWbIEPhALAOy+Us3uRYgbm97W8jhqNJqrIiSUxaFJMYZ7t ETfbUctN4/w== X-Google-Smtp-Source: AGHT+IEU6XhUTfh8cq6gCvMggnXLFkSt7XlYIzIUi6HAiuQhdPaLLmoiwQpzZ2rTTEjQ37T3mDQb8/UfRpK6jw== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:4f0e:b0:426:6a73:fb5f with SMTP id 5b1f17b1804b1-426708f9ab5mr1904875e9.7.1720803683507; Fri, 12 Jul 2024 10:01:23 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:29 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-11-144b319a40d8@google.com> Subject: [PATCH 11/26] mm: asi: ASI page table allocation functions From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 4069740012 X-Stat-Signature: ctks44strrwncq8s6kcq46aix5mcr11u X-HE-Tag: 1720803684-748232 X-HE-Meta: U2FsdGVkX1/lQBXojyAdueW/AqWyKEnp5SDIGEoUocd+VUlV2maBBww5AQqtj+oiiQjKhePIcAhlM+hxmL66fReyOE3eF0+JksglrUSuSPXEwbpl2hpDwdsziZSminJdh2OMt2DmeXhIlJWSrA7eq7lp3QuKGB+EAyr/C6Xm9la8f8aZbdOL8EH7DU6R4GZPyEmncn1+APL0yyB4BwizHHj9/IlsCdZTZ+7++a4gd7kt1ggFM9DYHBIddWmMDmGTcd0QO6//NRs8W2cfHN5LJBsC08Bz2Vt8WSqTY5aVfx/+84cti6zlQiigr7/ienXJeombnOrUYyMAg//ipysZ7iRnmdxIHRZRwuaaSmblNuH2XfzM1vAaNvRERh7lkim4nWE+OOtoJGjAn8hq5r50Xh/2RmfgEnKMmpvd5x23XC5gPp/6n2D3R+7CPlwfb8GDJluHRxuq+hReMeGM+Enf+E+Z+IiqoYoGuG4iLjTYDATDYIJddpIqTgdTE11q+LYJw4ioMlRCrnTFsdgFRlYW81BL5nDvP5RUGa1ntlqPsDDRrdJXu0JYlWxHJaaqtwJR+2/7I3zTiGJBeRK+Lvoxv4k87y9jkZZZckU11J5OLSsb3/sKHODIpinPNfW0iq1JqznaQfvaizGrK5DjrgxqwWJ/FOfrrr2V8w7Og4I9UnL2U3+KfeO6yIBqqXky/HEPk+WZEoZapRvTmYs1sP1uOGc+rpb1AZ699ATJFaIeJCvG+f4h0CSaGF3oLKhX4PzhpppSUo0NI7WiJP+aOX/ze9/UWoNpz4npvDZwWP+rl5p5OTs4Z865/nGRAb15UqgbKP+0TXyEahJXlUFHXHLu2iirKz2jqiUhzrzT515Pk/v4E/xflwQIS+61/vITllmSyVUSzmLq4kzAdo8kb8zf4e6xsWo8P5Z5amNi86875BtcilyKMv+gpCO7sZ36ImTxdiGce5HaGkqRynrr5ns iPxYM6wX P7IXgVH4k9s/KajSMr0tJMo5B2jKUj+LPdoYuieV6cG3C7hR04xBw7LhIZgWElfLMQZ3Cc+7IRQclvluNoAy0XBCFEJ4pdELElzSkd5B0R6KVhT/PnZ1bHCVuAC2Qs8TeL0rhAzLEBi1bwJjuaVkqaqNZlq9qFAJ7OeURjLRShR/vK7seawKZsVQthj4ilZvJ6YrsBLQ/WTgwBFGaLg1g7Mj7OkTC1mdakly82r9ul78ZBUdy95m9IfL8K7OFOZNwhRIwoNeN9NEMqfl0qZU/M04e6VQkDmcttX701wAby62IDlrQfKI5qtcCE4PxDFz7462RP+ZBygNlMmDiJK+X3h9hwE+dGB7Xf1cb5QORflOmF7TNdBFhFSVzV8lVwSfPdrYm8aQZS1whP9ZJiI5JNA93QnzVz84rb/kehjY52FM8Uy1+fyKNoxMznpXmF3oHdk+5H4zvNcGHkYlrwFz+3hGkadpEewEv1WUrgoiNHU62u7Ywb/Ill04hpZKwDglSn+A7g1Qe22neWbJBZXWLLfLuTtfNjBT0DjFG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid This adds custom allocation and free functions for ASI page tables. The alloc functions support allocating memory using different GFP reclaim flags, in order to be able to support non-sensitive allocations from both standard and atomic contexts. They also install the page tables locklessly, which makes it slightly simpler to handle non-sensitive allocations from interrupts/exceptions. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/mm/asi.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 0ba156f879d3..8798aab66748 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -71,6 +71,65 @@ void asi_unregister_class(int index) } EXPORT_SYMBOL_GPL(asi_unregister_class); +#ifndef mm_inc_nr_p4ds +#define mm_inc_nr_p4ds(mm) do {} while (false) +#endif + +#ifndef mm_dec_nr_p4ds +#define mm_dec_nr_p4ds(mm) do {} while (false) +#endif + +#define pte_offset pte_offset_kernel + +/* + * asi_p4d_alloc, asi_pud_alloc, asi_pmd_alloc, asi_pte_alloc. + * + * These are like the normal xxx_alloc functions, but: + * + * - They use atomic operations instead of taking a spinlock; this allows them + * to be used from interrupts. This is necessary because we use the page + * allocator from interrupts and the page allocator ultimately calls this + * code. + * - They support customizing the allocation flags. + * + * On the other hand, they do not use the normal page allocation infrastructure, + * that means that PTE pages do not have the PageTable type nor the PagePgtable + * flag and we don't increment the meminfo stat (NR_PAGETABLE) as they do. + */ +static_assert(!IS_ENABLED(CONFIG_PARAVIRT)); +#define DEFINE_ASI_PGTBL_ALLOC(base, level) \ +__maybe_unused \ +static level##_t * asi_##level##_alloc(struct asi *asi, \ + base##_t *base, ulong addr, \ + gfp_t flags) \ +{ \ + if (unlikely(base##_none(*base))) { \ + ulong pgtbl = get_zeroed_page(flags); \ + phys_addr_t pgtbl_pa; \ + \ + if (!pgtbl) \ + return NULL; \ + \ + pgtbl_pa = __pa(pgtbl); \ + \ + if (cmpxchg((ulong *)base, 0, \ + pgtbl_pa | _PAGE_TABLE) != 0) { \ + free_page(pgtbl); \ + goto out; \ + } \ + \ + mm_inc_nr_##level##s(asi->mm); \ + } \ +out: \ + VM_BUG_ON(base##_leaf(*base)); \ + return level##_offset(base, addr); \ +} + +DEFINE_ASI_PGTBL_ALLOC(pgd, p4d) +DEFINE_ASI_PGTBL_ALLOC(p4d, pud) +DEFINE_ASI_PGTBL_ALLOC(pud, pmd) +DEFINE_ASI_PGTBL_ALLOC(pmd, pte) + void __init asi_check_boottime_disable(void) { bool enabled = IS_ENABLED(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION_DEFAULT_ON);