From patchwork Wed Apr 17 19:26:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13633769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B07CC4345F for ; Wed, 17 Apr 2024 19:26:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78F126B0092; Wed, 17 Apr 2024 15:26:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 73F3E6B0093; Wed, 17 Apr 2024 15:26:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6069D6B0095; Wed, 17 Apr 2024 15:26:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 424946B0092 for ; Wed, 17 Apr 2024 15:26:52 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DFC8A140F6D for ; Wed, 17 Apr 2024 19:26:51 +0000 (UTC) X-FDA: 82020006222.06.7F05C74 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id DC3DA14000D for ; Wed, 17 Apr 2024 19:26:48 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="hlfI6/PY"; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713382009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Ian3ROIXX/Wi51a3xW5sv9PCZzatN1IXxVDjvA9s1M0=; b=KhWYjR6Q5cG1z6jpMnF/6Fo4El14gTw+90JIa6XYD5ikvaS05YASoshsrvQQ/PNO1omuy8 D1dfyb5MXRDQsk6PSs8GaZ3qW9fSsvK7lFk7dso6qGPG9MTXXagSJQ4S1lpUCrl6yqPAJc Ql0zaQdIlz48CoHAc3Y8nkWoejQBFXI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="hlfI6/PY"; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713382009; a=rsa-sha256; cv=none; b=A+VbKyURwuaMTZjpzn4jpYOvrbywMfCjrTTGz78bEKApw4bcXVsqnJXv/6v27DqEN2u012 DqJcqbHICI1FwJ7IwbAnUQkQlKcWq3S8zlZs/Z4RArDtPHwSqfn7fyXwhI2AREqgAlSmFA IeP8EEHml0ZoRZ0XeoADLJVs3IcDBIA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1713382008; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Ian3ROIXX/Wi51a3xW5sv9PCZzatN1IXxVDjvA9s1M0=; b=hlfI6/PYI42gZbtgQVaCLAT4IWWn7bUZV3WpDgXEoNF2Y/2nXFx/YeIlXSHDZa09t2yCHS QjpFld2Lcn0N8YlX+FX5JA5xLj2MsmHGEWcJe252F57CNaFPSL30Jj2cJyaJeM8DvHkeCj zcwrnIX3crMRHbdshKcE2ILhywb7GSQ= Received: from mail-oo1-f71.google.com (mail-oo1-f71.google.com [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-512-egG7dHhlPxqkD6exv3uxTg-1; Wed, 17 Apr 2024 15:26:46 -0400 X-MC-Unique: egG7dHhlPxqkD6exv3uxTg-1 Received: by mail-oo1-f71.google.com with SMTP id 006d021491bc7-5acdbfb5b36so21960eaf.2 for ; Wed, 17 Apr 2024 12:26:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713382006; x=1713986806; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ian3ROIXX/Wi51a3xW5sv9PCZzatN1IXxVDjvA9s1M0=; b=Y9Q/dCYVH8EI0FpflqlSS3//kKqIzTTSrHvzMyR6c5x9Qu7G3Qz8d/wenYj5xjFdoy AIyPfES1vlmMSGD4Fqxi00sCpUkqHcJLIue5kGkY1w27wbJ2k/xhZQjFbaqGMoPbcHZm pbdWSy24NxrCdl/AVlj7V8M/gL2RUfniWyCxonxpBG/EaUmxvo9X2DIUHXPheDFGjC6F 4E5XWUhAVJ493YwqGOh8R3DHmeXrb75s67KoJwYOlE2Zu51HBydr7mLSK1XGbvxCyR75 KQ9jtageX6moosC5Jm/6L9rRmI25aD7X3MOF2AAWWOpllAI1WyH1ILN9bxg2CJiiiDn7 tLPw== X-Forwarded-Encrypted: i=1; AJvYcCX2LyDy6X9l3gt+6I37cjOBZ+XmQh8pTL2DefovvEn/cME9Qf0YWuWoC4zVbMu2Xo4U77s6mfcZ9ji1RfgS4AvVTMU= X-Gm-Message-State: AOJu0YwKikha44UF8HrKaFEdg7R5jQGi+3dz4VTVN8ePuGc1yJ5Ku7uS EfwwYDT7xD8DumqarnPLZSf+TdVPpdpuQrcvIGfAESnsUiZsuhzlpI/bSMtTwAoLxrf6DpgWffb Jh5DOKX3y0Rtt63pjHbXLp2TsBUQXXrQFXK/Mg3uRN47avbwd X-Received: by 2002:a05:6359:4589:b0:186:54b2:4141 with SMTP id no9-20020a056359458900b0018654b24141mr367566rwb.1.1713382005390; Wed, 17 Apr 2024 12:26:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHnVmD3TlN/+ZoWfExYdyVW0wph5CE17OJJDEMLdhCW9wdwYKiYfienQBNmah1jUXCkrGsFhw== X-Received: by 2002:a05:6359:4589:b0:186:54b2:4141 with SMTP id no9-20020a056359458900b0018654b24141mr367537rwb.1.1713382004772; Wed, 17 Apr 2024 12:26:44 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id kd9-20020a056214400900b0069b42d51d3bsm8465915qvb.103.2024.04.17.12.26.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Apr 2024 12:26:44 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , David Hildenbrand , Nadav Amit , Axel Rasmussen , peterx@redhat.com, Pasha Tatashin Subject: [PATCH v3] mm/page_table_check: Support userfault wr-protect entries Date: Wed, 17 Apr 2024 15:26:43 -0400 Message-ID: <20240417192643.2671335-1-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: 98cc1uusm3dgbifnwkba9yg3mp95fxib X-Rspamd-Queue-Id: DC3DA14000D X-Rspamd-Server: rspam02 X-Rspam-User: X-HE-Tag: 1713382008-155783 X-HE-Meta: U2FsdGVkX1/20KawLAAZw9wGJ9xgD9yijHQMJYESzDhdvhKzpox77+aTIpyW1mB/8fFAQ7r4tvae8O7yLfZ8BTGEfBzQsRdioyY3ivLjC4w+fkx8NhZexogrpoBO4Qk56qdtg8VOeCxwPrAr/ar+KbkdvO+1Tyv11bojVD20XXd6dnvb2S0itUNs1GoELhXOYjuUvYLRpdcKs35IwOpukt7UjcYJ06zRRPf4iTAiwr5vuukksy5rLYD8GwQvK99nmDCNvhdEMfZ3JuEbokkffraSyaNGi7B4UKqWvpxuiirud7Ar7ENnXv478DTqAw6CG22Z9iZAfs1PAzAAvjPzZxfwmshiXyHlXdMcEOonyCf4xBa5HIT3lFWVh67DqoE3jgfrTXrC9xgBcpNkfC3JTOwoLoHhI6N3cWMed/K0QUCoybjxUcOfoixNyG76hCN+MIZSu8hCOPVwoOjb9HGgatpQRULmBbuC7cG8DdMPp7Qk0GibBoHXzMiQFhX+gWkm8W4OG2KhydGVh6tZ8ihF4b9bFZgJoVNlI+zB0qlMSiH14XnyXj1i9ZxENHMWg3aaihQ78zQZ+1UL9v2i8xozaZ8tki06WNHdJrYEyYRbhqLDiJ2Ok3QpAw+vcs0DjBq+k/hlIf8WElxRM82h8su8hc/Asn97yjcz2LgoR+wuP0aT1cqeJngEE0uNEZ6vSpcEakX1m0ju8qLh2qk0dTRG8uc/A6RiVkbpiYcShfpESlDNDg75Rb1iIEMxQRQ4qQMv6D1ON8B/OJIB2Lrfchvz4H3N4wiL8lKi39cKi4+PwOc08rLx7QvVcp5WFCpuaIYBsqhEtIrZSvruOewv0GqGWQHsMLEr3LcKjTpNiza+Ig3x6hdqhG//l4apSnsfOQmlRBdB89xYSBnHij7SAjxFwFFGaxRp/kwk9LT5irYdRFZ0GaVGBYI0TFUOMkhmLwXzYwMuboiX5sE4FP0xCP6 SO/Q9Sfq Qt+xgn6pioxAtPf/XtsJID7f16ysCwmbzv6TaNqC90OqVzsvso64r7/ZFkC/0AAjVLdkYmfud9QgJ3WwbVaLoogAyILY4DciceCo2lLSjiJlxz8la2mAl+6qvmD/SXqnqQV2MdMlkl4kyilC987cSEVjSzWeuJL4rm+0qWTkEex9Q37pFV9qtlg4NDfrYjiIYuo8/tjpo2kP5pP6bXcPiOWJy1iPWstzPhNde4Srj+Gap9WFKOGQE59Bjm7RwGkp3kHpWBfYlVEBrirfQtUazNOM2Uywoy3XTsmlySz9JkyMbTokC9V12g3elzxziaQRw9TgharyxWukYDlSPnGjdVWLHejKJoUXx4HaeEX7Gvnn1DEeDCHwFILcv7UK3YMu/bspQ7jiKQr1Va27O/Y9BMaW4Q7cIoXWMtX4NOi1xrgzWuaTOH57lNVjBDcakPevaH9luMk0SrvDxBBrjrWilef3LBSj4wg1Q3m2cz5YCDwo6ezJx+gWMmSufmcCDM3EQMJrCO/1SM6cH8Br1qRkTzEN0oUx4QGqICq7Eqd4ydLY8g8wob1I5c0mmrTOQCh/gAtz0JRR27yH7oL92eMl/kqXpWIxrUU59TAjAum9hMWlayMIWU6r6fEhMfnztWsRA2epggmqB4ojAKaY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Allow page_table_check hooks to check over userfaultfd wr-protect criteria upon pgtable updates. The rule is no co-existance allowed for any writable flag against userfault wr-protect flag. This should be better than c2da319c2e, where we used to only sanitize such issues during a pgtable walk, but when hitting such issue we don't have a good chance to know where does that writable bit came from [1], so that even the pgtable walk exposes a kernel bug (which is still helpful on triaging) but not easy to track and debug. Now we switch to track the source. It's much easier too with the recent introduction of page table check. There are some limitations with using the page table check here for userfaultfd wr-protect purpose: - It is only enabled with explicit enablement of page table check configs and/or boot parameters, but should be good enough to track at least syzbot issues, as syzbot should enable PAGE_TABLE_CHECK[_ENFORCED] for x86 [1]. We used to have DEBUG_VM but it's now off for most distros, while distros also normally not enable PAGE_TABLE_CHECK[_ENFORCED], which is similar. - It conditionally works with the ptep_modify_prot API. It will be bypassed when e.g. XEN PV is enabled, however still work for most of the rest scenarios, which should be the common cases so should be good enough. - Hugetlb check is a bit hairy, as the page table check cannot identify hugetlb pte or normal pte via trapping at set_pte_at(), because of the current design where hugetlb maps every layers to pte_t... For example, the default set_huge_pte_at() can invoke set_pte_at() directly and lose the hugetlb context, treating it the same as a normal pte_t. So far it's fine because we have huge_pte_uffd_wp() always equals to pte_uffd_wp() as long as supported (x86 only). It'll be a bigger problem when we'll define _PAGE_UFFD_WP differently at various pgtable levels, because then one huge_pte_uffd_wp() per-arch will stop making sense first.. as of now we can leave this for later too. This patch also removes commit c2da319c2e altogether, as we have something better now. [1] https://lore.kernel.org/all/000000000000dce0530615c89210@google.com/ Cc: Pasha Tatashin Signed-off-by: Peter Xu --- v2: - Rename __page_table_check_pxx() to page_table_check_pxx_flags(), meanwhile move the pte check out of the loop [Pasha] - Fix build issues reported from the bot, also added SWP_DEVICE_WRITE which was overlooked before v3: - Add missing doc update [Pasha] --- Documentation/mm/page_table_check.rst | 9 ++++++- arch/x86/include/asm/pgtable.h | 18 +------------ mm/page_table_check.c | 39 +++++++++++++++++++++++++++ 3 files changed, 48 insertions(+), 18 deletions(-) diff --git a/Documentation/mm/page_table_check.rst b/Documentation/mm/page_table_check.rst index c12838ce6b8d..5bd1d987d76d 100644 --- a/Documentation/mm/page_table_check.rst +++ b/Documentation/mm/page_table_check.rst @@ -14,7 +14,7 @@ Page table check performs extra verifications at the time when new pages become accessible from the userspace by getting their page table entries (PTEs PMDs etc.) added into the table. -In case of detected corruption, the kernel is crashed. There is a small +In case of most detected corruption, the kernel is crashed. There is a small performance and memory overhead associated with the page table check. Therefore, it is disabled by default, but can be optionally enabled on systems where the extra hardening outweighs the performance costs. Also, because page table check @@ -22,6 +22,13 @@ is synchronous, it can help with debugging double map memory corruption issues, by crashing kernel at the time wrong mapping occurs instead of later which is often the case with memory corruptions bugs. +It can also be used to do page table entry checks over various flags, dump +warnings when illegal combinations of entry flags are detected. Currently, +userfaultfd is the only user of such to sanity check wr-protect bit against +any writable flags. Illegal flag combinations will not directly cause data +corruption in this case immediately, but that will cause read-only data to +be writable, causing data corrupt when the page content is later modified. + Double mapping detection logic ============================== diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 273f7557218c..65b8e5bb902c 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -388,23 +388,7 @@ static inline pte_t pte_wrprotect(pte_t pte) #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pte_uffd_wp(pte_t pte) { - bool wp = pte_flags(pte) & _PAGE_UFFD_WP; - -#ifdef CONFIG_DEBUG_VM - /* - * Having write bit for wr-protect-marked present ptes is fatal, - * because it means the uffd-wp bit will be ignored and write will - * just go through. - * - * Use any chance of pgtable walking to verify this (e.g., when - * page swapped out or being migrated for all purposes). It means - * something is already wrong. Tell the admin even before the - * process crashes. We also nail it with wrong pgtable setup. - */ - WARN_ON_ONCE(wp && pte_write(pte)); -#endif - - return wp; + return pte_flags(pte) & _PAGE_UFFD_WP; } static inline pte_t pte_mkuffd_wp(pte_t pte) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index af69c3c8f7c2..388bcf60d8b5 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -7,6 +7,8 @@ #include #include #include +#include +#include #undef pr_fmt #define pr_fmt(fmt) "page_table_check: " fmt @@ -182,6 +184,31 @@ void __page_table_check_pud_clear(struct mm_struct *mm, pud_t pud) } EXPORT_SYMBOL(__page_table_check_pud_clear); +/* Whether the swap entry cached writable information */ +static inline bool swap_cached_writable(swp_entry_t entry) +{ + unsigned type = swp_type(entry); + +#ifdef CONFIG_DEVICE_PRIVATE + if (type == SWP_DEVICE_EXCLUSIVE_WRITE || type == SWP_DEVICE_WRITE) + return true; +#endif +#ifdef CONFIG_MIGRATION + if (type == SWP_MIGRATION_WRITE) + return true; +#endif + + return false; +} + +static inline void page_table_check_pte_flags(pte_t pte) +{ + if (pte_present(pte) && pte_uffd_wp(pte)) + WARN_ON_ONCE(pte_write(pte)); + else if (is_swap_pte(pte) && pte_swp_uffd_wp(pte)) + WARN_ON_ONCE(swap_cached_writable(pte_to_swp_entry(pte))); +} + void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte, unsigned int nr) { @@ -190,6 +217,8 @@ void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte, if (&init_mm == mm) return; + page_table_check_pte_flags(pte); + for (i = 0; i < nr; i++) __page_table_check_pte_clear(mm, ptep_get(ptep + i)); if (pte_user_accessible_page(pte)) @@ -197,11 +226,21 @@ void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte, } EXPORT_SYMBOL(__page_table_check_ptes_set); +static inline void page_table_check_pmd_flags(pmd_t pmd) +{ + if (pmd_present(pmd) && pmd_uffd_wp(pmd)) + WARN_ON_ONCE(pmd_write(pmd)); + else if (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd)) + WARN_ON_ONCE(swap_cached_writable(pmd_to_swp_entry(pmd))); +} + void __page_table_check_pmd_set(struct mm_struct *mm, pmd_t *pmdp, pmd_t pmd) { if (&init_mm == mm) return; + page_table_check_pmd_flags(pmd); + __page_table_check_pmd_clear(mm, *pmdp); if (pmd_user_accessible_page(pmd)) { page_table_check_set(pmd_pfn(pmd), PMD_SIZE >> PAGE_SHIFT,