From patchwork Tue Aug 9 22:00:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12939963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 173F8C25B0D for ; Tue, 9 Aug 2022 22:01:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDC028E0003; Tue, 9 Aug 2022 18:01:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D63AD8E0001; Tue, 9 Aug 2022 18:01:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB4878E0003; Tue, 9 Aug 2022 18:01:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AD6B58E0001 for ; Tue, 9 Aug 2022 18:01:07 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 84F648124C for ; Tue, 9 Aug 2022 22:01:07 +0000 (UTC) X-FDA: 79781425374.16.7352007 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id ADAA140168 for ; Tue, 9 Aug 2022 22:01:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660082466; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ViWF5oZvZDnfPWYKxgTTuTS7ZUI2Vs+P8BNbSIRNDkc=; b=Wtx9ntMVoao6aSfVQwtkLU6f3kgNof32gE01lYNzjtbXaad5b65BSH0SQc9lObdlVr9ErG QyNdDX0H6a9FJwg3A25sIXJH1/HjzyeYhN7t6Q1bsWrqcFWYcYjsw8aOE1h/dmTEpRS95l w3R0PyIGHncqnJrsT3iqo1hi4geM6Bk= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-282-YMwYEx8bNF6HobipVXYWKA-1; Tue, 09 Aug 2022 18:01:05 -0400 X-MC-Unique: YMwYEx8bNF6HobipVXYWKA-1 Received: by mail-qk1-f197.google.com with SMTP id bj26-20020a05620a191a00b006b5c4e2dc77so11196382qkb.16 for ; Tue, 09 Aug 2022 15:01:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=ViWF5oZvZDnfPWYKxgTTuTS7ZUI2Vs+P8BNbSIRNDkc=; b=LZ16Nh5jLZVFhFzjUbY2W1L4iEpfxuQSJPO0yBB20QtCtGuBPPhVVzPg8Y3zMGbEu8 aUSt8sORl8HVuTGmStMfj+x55H/TfMRDU0nbSGJBo+tLGUwBhgDvu4r6iBFmvkwhmLsU RKGgvLbaCXg6kx0PpGOrQF44jSiJpxAxzLYbtzar38vsksBowkKODb79MFFOkYaickP3 iy0ubm8iFxg4KSpw0OW11PRqhGnj+KvpJYRUOQuwjfWPyxetH0nRVNM0xxaiA4L7TdMt 8MAdh3FE5tt+2cN543benpKfFgkY19f8BQPF7kOXUtWq6e0x/erL8WVif69k5+UEbLOG Ui3g== X-Gm-Message-State: ACgBeo0SqnfBDdsanPOUKVdm5eSiwCkTtqPzniP0agwDr86eHTc+v2HC xL8NyHdbraC6QouFN84u/eNa34l2pJLmgAfWTdO42LtWctxtLjTzQLdOoKUriV/oJaHDeU/QVeX EV0db8WV85QNIVU0HbPxsWw7uGQf+t05pX2MHlxvy2kfSK/J6LN8vXzmHFVzW X-Received: by 2002:a05:620a:2552:b0:6b4:8dbf:8992 with SMTP id s18-20020a05620a255200b006b48dbf8992mr18881470qko.109.1660082464726; Tue, 09 Aug 2022 15:01:04 -0700 (PDT) X-Google-Smtp-Source: AA6agR6dluwUjmka3W+U32hW/DI5Wis2lr4liBhxv8a6zycyIUy5OEofCGKK4slj2jFFitTD8LOwnA== X-Received: by 2002:a05:620a:2552:b0:6b4:8dbf:8992 with SMTP id s18-20020a05620a255200b006b48dbf8992mr18881433qko.109.1660082464471; Tue, 09 Aug 2022 15:01:04 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id b16-20020a05622a021000b0031eddc83560sm10563383qtx.90.2022.08.09.15.01.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 09 Aug 2022 15:01:04 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Minchan Kim , David Hildenbrand , Nadav Amit , Andrew Morton , Hugh Dickins , Vlastimil Babka , Andrea Arcangeli , Andi Kleen , Huang Ying , "Kirill A . Shutemov" Subject: [PATCH v3 1/7] mm/x86: Use SWP_TYPE_BITS in 3-level swap macros Date: Tue, 9 Aug 2022 18:00:54 -0400 Message-Id: <20220809220100.20033-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220809220100.20033-1-peterx@redhat.com> References: <20220809220100.20033-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660082466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ViWF5oZvZDnfPWYKxgTTuTS7ZUI2Vs+P8BNbSIRNDkc=; b=5QD7day2/h7w7BvP5YgyRmrghsAPktMBHdjUFhv1+3i6DyFpPy5KbIsz5axZo6FysUe3G3 PurzfyCqvSwS5HjZsT5sT4RUY/m9I8W08BxSr9CUSB/Ikuczx2vAsyyx2pUH5+LIOCSBMj 2WED/Cm6fcscKD58Z8Eoyt8jkIDKJK8= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Wtx9ntMV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660082466; a=rsa-sha256; cv=none; b=Xv3NouXySUCur8bYLLwa/3lf7/OOhFxwPeL/I2lkQDjlaD34+EqWypoi33o7lQHs/jRZ9N ooz3sg0S7MQcFzVIYvTWYK4cJLa7adtQsT9gAYxhGDILTC80zfZ1xwhW1ZODopVppZ3NWJ O0Btd0wRZT7RG2SMoR6Ma0zenb8ni1E= Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Wtx9ntMV; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam02 X-Stat-Signature: h5i31qsr5eab4g46jybaz4bx7chxkuug X-Rspamd-Queue-Id: ADAA140168 X-Rspam-User: X-HE-Tag: 1660082466-248628 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace all the magic "5" with the macro. Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu Reviewed-by: "Huang, Ying" --- arch/x86/include/asm/pgtable-3level.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/pgtable-3level.h b/arch/x86/include/asm/pgtable-3level.h index e896ebef8c24..28421a887209 100644 --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -256,10 +256,10 @@ static inline pud_t native_pudp_get_and_clear(pud_t *pudp) /* We always extract/encode the offset by shifting it all the way up, and then down again */ #define SWP_OFFSET_SHIFT (SWP_OFFSET_FIRST_BIT + SWP_TYPE_BITS) -#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > 5) -#define __swp_type(x) (((x).val) & 0x1f) -#define __swp_offset(x) ((x).val >> 5) -#define __swp_entry(type, offset) ((swp_entry_t){(type) | (offset) << 5}) +#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_BITS) +#define __swp_type(x) (((x).val) & ((1UL << SWP_TYPE_BITS) - 1)) +#define __swp_offset(x) ((x).val >> SWP_TYPE_BITS) +#define __swp_entry(type, offset) ((swp_entry_t){(type) | (offset) << SWP_TYPE_BITS}) /* * Normally, __swp_entry() converts from arch-independent swp_entry_t to From patchwork Tue Aug 9 22:00:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12939966 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5B48C25B08 for ; Tue, 9 Aug 2022 22:01:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74AFC8E0006; Tue, 9 Aug 2022 18:01:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D5738E0001; Tue, 9 Aug 2022 18:01:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41AB18E0007; Tue, 9 Aug 2022 18:01:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 188FF8E0006 for ; Tue, 9 Aug 2022 18:01:11 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E0852121249 for ; Tue, 9 Aug 2022 22:01:10 +0000 (UTC) X-FDA: 79781425500.01.BCA25DB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 2DE11100166 for ; Tue, 9 Aug 2022 22:01:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660082467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KJCqPuuo957wqhSKd7LcKTP/opuSiey74Ok7HcmJ/F0=; b=NoLT7q6oCXqtVwFy7b8KU5j5J5Bf1jrwczrbWLOy0pO6ke3GjCeith3sxsgjjvtLzo0Aj0 NFlWsYyhkOxZ4zVchYaJk51A5V2TRRAd1YuWgjOqA7/gxAeQrDw84VWxzAaKH+UB2qZcly Gip6xEla+YxzhJIa5IVG0MvLOikhU44= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-433-NWgKkz39N42paqziDuhEXw-1; Tue, 09 Aug 2022 18:01:06 -0400 X-MC-Unique: NWgKkz39N42paqziDuhEXw-1 Received: by mail-qv1-f71.google.com with SMTP id f6-20020a0cc306000000b004768e692cddso6870199qvi.8 for ; Tue, 09 Aug 2022 15:01:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=KJCqPuuo957wqhSKd7LcKTP/opuSiey74Ok7HcmJ/F0=; b=HzN1XlPeg2SJfidBsEH83S0G6w5njxcOAS9ZSUOIuSabs36ddA80tp5PUtL7cPUxGy 0iQgq8N5CDlCbPhxf0VHloci9xyI349Y4wxVHnjn5sKjP1+FWLy990sjXLePFGnS3+ZA VF16QzG6jPF6Ua/vIufqN1lhd4eW60dz/y1T0CVT0fXXFACF+/Uim4yKCnO1gye5Bo3W jEJ5iO79f4ZRn0NqdAgFcJpQueGpWujkM4oO9e4LMOAcePqdkLfymS6r8QvieLirs0VF ThdZwKhRN+TMaXh0qUWaqB//P7pWN6VJrUjEIs3NY1Hf7QyOrcINaxNvUp20l2whpBmh oLlg== X-Gm-Message-State: ACgBeo0pHK7Lj2DoZcMEKPrbARAn8HjU0RQ/DhAfuBxcxwLY/EybsrW3 zpqTcFxTPFHynFkSlTzHFkaYj2Xs0c7O77hm7XuNxA8zqOxjD4S3O1+IDGWAen7Hr60rWPfQBgp ge+K9UINKP0HR/U++/orXBhLkRV7RMXjRW5RLZs/Zbj5NpD+OVjSRVy46/2u7 X-Received: by 2002:a05:622a:100d:b0:31f:25e3:7a45 with SMTP id d13-20020a05622a100d00b0031f25e37a45mr21707236qte.365.1660082466048; Tue, 09 Aug 2022 15:01:06 -0700 (PDT) X-Google-Smtp-Source: AA6agR6NiOVfmdP51FutgyMM1ys4/L82L2FgMCUHx+9da7UIEtdC/XWDWNPnpXmCoiTlsJ6ASHYdnw== X-Received: by 2002:a05:622a:100d:b0:31f:25e3:7a45 with SMTP id d13-20020a05622a100d00b0031f25e37a45mr21707192qte.365.1660082465744; Tue, 09 Aug 2022 15:01:05 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id b16-20020a05622a021000b0031eddc83560sm10563383qtx.90.2022.08.09.15.01.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 09 Aug 2022 15:01:05 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Minchan Kim , David Hildenbrand , Nadav Amit , Andrew Morton , Hugh Dickins , Vlastimil Babka , Andrea Arcangeli , Andi Kleen , Huang Ying , "Kirill A . Shutemov" Subject: [PATCH v3 2/7] mm/swap: Comment all the ifdef in swapops.h Date: Tue, 9 Aug 2022 18:00:55 -0400 Message-Id: <20220809220100.20033-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220809220100.20033-1-peterx@redhat.com> References: <20220809220100.20033-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NoLT7q6o; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660082470; a=rsa-sha256; cv=none; b=dYXqM+Cz8sBvFZXpGwdtW4kZtJZZG/+7aSXWgYwL06RdskATZfeaNMCo+779zEnDn1yK3U rGYSOr7s2KDUhW8b7dOIccwWx9v/ce3+4aKFwozVKjPBoIUfdQGbYAqFTKBVwZS+Z6So6L 1FbFmrPM9qYmZEk1DP650WOQl2xIgH8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660082470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KJCqPuuo957wqhSKd7LcKTP/opuSiey74Ok7HcmJ/F0=; b=KfpngE7VC64269/DLray0FM6yNlMQolqqIz4He7+yr02gi7DLQhlV8sFFhlVyBVR/HbOqe nQP9TEdI4tQMhPeYS5yXc+1T4zfHcbNU299zpsM9PyL90SGPvXEre1dl4azqwFIoXtgyqP 7VB6tILPNwcuP7hFZ2pT+8lUUbg25Rc= X-Stat-Signature: owgeekqrdoh5hrxzha88ji6mi6ji6tj3 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2DE11100166 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NoLT7q6o; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1660082468-277239 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: swapops.h contains quite a few layers of ifdef, some of the "else" and "endif" doesn't get proper comment on the macro so it's hard to follow on what are they referring to. Add the comments. Suggested-by: Nadav Amit Signed-off-by: Peter Xu Reviewed-by: "Huang, Ying" --- include/linux/swapops.h | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index a3d435bf9f97..3a2901ff4f1e 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -247,8 +247,8 @@ extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, #ifdef CONFIG_HUGETLB_PAGE extern void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte); -#endif -#else +#endif /* CONFIG_HUGETLB_PAGE */ +#else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry(pgoff_t offset) { return swp_entry(0, 0); @@ -276,7 +276,7 @@ static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, #ifdef CONFIG_HUGETLB_PAGE static inline void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl) { } static inline void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { } -#endif +#endif /* CONFIG_HUGETLB_PAGE */ static inline int is_writable_migration_entry(swp_entry_t entry) { return 0; @@ -286,7 +286,7 @@ static inline int is_readable_migration_entry(swp_entry_t entry) return 0; } -#endif +#endif /* CONFIG_MIGRATION */ typedef unsigned long pte_marker; @@ -426,7 +426,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) { return is_swap_pmd(pmd) && is_migration_entry(pmd_to_swp_entry(pmd)); } -#else +#else /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ static inline int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, struct page *page) { @@ -455,7 +455,7 @@ static inline int is_pmd_migration_entry(pmd_t pmd) { return 0; } -#endif +#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */ #ifdef CONFIG_MEMORY_FAILURE @@ -495,7 +495,7 @@ static inline void num_poisoned_pages_sub(long i) atomic_long_sub(i, &num_poisoned_pages); } -#else +#else /* CONFIG_MEMORY_FAILURE */ static inline swp_entry_t make_hwpoison_entry(struct page *page) { @@ -514,7 +514,7 @@ static inline void num_poisoned_pages_inc(void) static inline void num_poisoned_pages_sub(long i) { } -#endif +#endif /* CONFIG_MEMORY_FAILURE */ static inline int non_swap_entry(swp_entry_t entry) { From patchwork Tue Aug 9 22:00:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12939965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 921C0C19F2D for ; Tue, 9 Aug 2022 22:01:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15B928E0005; Tue, 9 Aug 2022 18:01:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0987F8E0001; Tue, 9 Aug 2022 18:01:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E003B8E0005; Tue, 9 Aug 2022 18:01:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CD43A8E0001 for ; Tue, 9 Aug 2022 18:01:10 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9F299141207 for ; Tue, 9 Aug 2022 22:01:10 +0000 (UTC) X-FDA: 79781425500.03.4D9E298 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id A84C6140186 for ; Tue, 9 Aug 2022 22:01:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660082469; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xPnoJX1BpuaXYPt2aIEkvm8MVF1Jb7/8BoIuTWGPV+k=; b=ijjhNIm4zfMVU/kyrYGzvt1ADLCcXvEcG/TB3JNDrI/4C/URV6rUTZhFOLTE+sQA1/YSpA ujs80jBmT7d26TO778tABT3vLZK+aMfyIkgBUQyyJ1iJAE49kUeZ9o9EzQcuo/ddN6wh/x cEfZDy/0TGnc8CvyoOLF63qzzQhOUvY= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-594-6dyA7MlaO2OwQe1aYxH3xQ-1; Tue, 09 Aug 2022 18:01:08 -0400 X-MC-Unique: 6dyA7MlaO2OwQe1aYxH3xQ-1 Received: by mail-qk1-f199.google.com with SMTP id q20-20020a05620a0d9400b006b6540e8d79so11242059qkl.14 for ; Tue, 09 Aug 2022 15:01:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=xPnoJX1BpuaXYPt2aIEkvm8MVF1Jb7/8BoIuTWGPV+k=; b=N/mM0A3A6FYWMVe+lOV0Mymgp4vVgcsOoLSL/GI9j9jPJzYdTaGnBGXmsYfqya6VCE tGuQXJd0GH48NxEp5PKsLWB+LHuWoeKt/HtqClX7zj/zjhijJcjrURKV2KvLXb+OojHU 73g/IdhjTTe5EQ7OQ8pDuKvtabfBg5BBEt6DpqiyVhcu2wd0pgTMdj/nGHF+qcr1iVz1 tXsiecb/N75WFnweayiTIzPnrBTOBAsUeQKyySeFrQP9LJ8yLSvM+lzNPz03Wrnts+EB rLzpjN76LXbyxFb8THBDI6JVUXoe/d2PKNe4D2Rq+qK4KEwKmJOjnDlFgSlEthlbq7cI XiWw== X-Gm-Message-State: ACgBeo2ihr52uDBnIV4m/sI3RI081cPrKzk4c8UiggsU7bkeaE8Vo6py AvK8dXp/rnS7JNHzQoU03SWf+WvKZJDW2rngzhxNrmee5N+XiuSPQbH8K/aPhbIugBOSkmQde2R IxUbPPmPGlh4cANhkaKTiw2E4dUPAtaGMcAEpd6lCrgA4am8LgYTbIYplmxo2 X-Received: by 2002:ac8:5ad0:0:b0:342:fc73:2aa4 with SMTP id d16-20020ac85ad0000000b00342fc732aa4mr8242517qtd.207.1660082467560; Tue, 09 Aug 2022 15:01:07 -0700 (PDT) X-Google-Smtp-Source: AA6agR7OkYIB1b3FqUM56/3uXh2sqRYh6V2WKfpCSssEbSfHcPbhC6taRal2aT6/tKNRyYns5FB3Og== X-Received: by 2002:ac8:5ad0:0:b0:342:fc73:2aa4 with SMTP id d16-20020ac85ad0000000b00342fc732aa4mr8242456qtd.207.1660082467089; Tue, 09 Aug 2022 15:01:07 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id b16-20020a05622a021000b0031eddc83560sm10563383qtx.90.2022.08.09.15.01.05 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 09 Aug 2022 15:01:06 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Minchan Kim , David Hildenbrand , Nadav Amit , Andrew Morton , Hugh Dickins , Vlastimil Babka , Andrea Arcangeli , Andi Kleen , Huang Ying , "Kirill A . Shutemov" Subject: [PATCH v3 3/7] mm/swap: Add swp_offset_pfn() to fetch PFN from swap entry Date: Tue, 9 Aug 2022 18:00:56 -0400 Message-Id: <20220809220100.20033-4-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220809220100.20033-1-peterx@redhat.com> References: <20220809220100.20033-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660082470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xPnoJX1BpuaXYPt2aIEkvm8MVF1Jb7/8BoIuTWGPV+k=; b=cXmOThOc4wT2p/ZLhSMxPrrTQybURLITlqc4hjhf0MUlb8CvcXFdG0L3j8qtImNtcuZTfC s6UeXwNLwZETaRPhmFf9gv8GMk7eqK/aiJzndzHkKGUC0gkkvCEQwlpOVgYwPx2Z3hK8ss 9u74QFKKEb3Zs0m13gfyKo5YeXUuwXI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ijjhNIm4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660082470; a=rsa-sha256; cv=none; b=ejc2w1++4qYX3t2TIHuhhWKCkueLvuaQCuVJ/IwcY5Ae7bVIoQa/bviISMEDY2uf70M8TV +TYPvv7f7lu2HS0I4wrUp/rZVU2XrbsPs+0kbQzT7xmuj4f0rAVJbE+pCcNMNoRfYsNVjn L0RF1h5IY/YO8DQ8PCsbe4PBwoEGvoM= Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ijjhNIm4; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam02 X-Stat-Signature: abihehki85gsjdy76yatae3xpgrnjyup X-Rspamd-Queue-Id: A84C6140186 X-Rspam-User: X-HE-Tag: 1660082469-281269 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We've got a bunch of special swap entries that stores PFN inside the swap offset fields. To fetch the PFN, normally the user just calls swp_offset() assuming that'll be the PFN. Add a helper swp_offset_pfn() to fetch the PFN instead, fetching only the max possible length of a PFN on the host, meanwhile doing proper check with MAX_PHYSMEM_BITS to make sure the swap offsets can actually store the PFNs properly always using the BUILD_BUG_ON() in is_pfn_swap_entry(). One reason to do so is we never tried to sanitize whether swap offset can really fit for storing PFN. At the meantime, this patch also prepares us with the future possibility to store more information inside the swp offset field, so assuming "swp_offset(entry)" to be the PFN will not stand any more very soon. Replace many of the swp_offset() callers to use swp_offset_pfn() where proper. Note that many of the existing users are not candidates for the replacement, e.g.: (1) When the swap entry is not a pfn swap entry at all, or, (2) when we wanna keep the whole swp_offset but only change the swp type. For the latter, it can happen when fork() triggered on a write-migration swap entry pte, we may want to only change the migration type from write->read but keep the rest, so it's not "fetching PFN" but "changing swap type only". They're left aside so that when there're more information within the swp offset they'll be carried over naturally in those cases. Since at it, dropping hwpoison_entry_to_pfn() because that's exactly what the new swp_offset_pfn() is about. Signed-off-by: Peter Xu --- arch/arm64/mm/hugetlbpage.c | 2 +- include/linux/swapops.h | 35 +++++++++++++++++++++++++++++------ mm/hmm.c | 2 +- mm/memory-failure.c | 2 +- mm/page_vma_mapped.c | 6 +++--- 5 files changed, 35 insertions(+), 12 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 0795028f017c..35e9a468d13e 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -245,7 +245,7 @@ static inline struct folio *hugetlb_swap_entry_to_folio(swp_entry_t entry) { VM_BUG_ON(!is_migration_entry(entry) && !is_hwpoison_entry(entry)); - return page_folio(pfn_to_page(swp_offset(entry))); + return page_folio(pfn_to_page(swp_offset_pfn(entry))); } void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 3a2901ff4f1e..e1accbcd1136 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -23,6 +23,20 @@ #define SWP_TYPE_SHIFT (BITS_PER_XA_VALUE - MAX_SWAPFILES_SHIFT) #define SWP_OFFSET_MASK ((1UL << SWP_TYPE_SHIFT) - 1) +/* + * Definitions only for PFN swap entries (see is_pfn_swap_entry()). To + * store PFN, we only need SWP_PFN_BITS bits. Each of the pfn swap entries + * can use the extra bits to store other information besides PFN. + */ +#ifdef MAX_PHYSMEM_BITS +#define SWP_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT) +#else /* MAX_PHYSMEM_BITS */ +#define SWP_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT) +#endif /* MAX_PHYSMEM_BITS */ +#define SWP_PFN_MASK ((1UL << SWP_PFN_BITS) - 1) + +static inline bool is_pfn_swap_entry(swp_entry_t entry); + /* Clear all flags but only keep swp_entry_t related information */ static inline pte_t pte_swp_clear_flags(pte_t pte) { @@ -64,6 +78,17 @@ static inline pgoff_t swp_offset(swp_entry_t entry) return entry.val & SWP_OFFSET_MASK; } +/* + * This should only be called upon a pfn swap entry to get the PFN stored + * in the swap entry. Please refers to is_pfn_swap_entry() for definition + * of pfn swap entry. + */ +static inline unsigned long swp_offset_pfn(swp_entry_t entry) +{ + VM_BUG_ON(!is_pfn_swap_entry(entry)); + return swp_offset(entry) & SWP_PFN_MASK; +} + /* check whether a pte points to a swap entry */ static inline int is_swap_pte(pte_t pte) { @@ -369,7 +394,7 @@ static inline int pte_none_mostly(pte_t pte) static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry) { - struct page *p = pfn_to_page(swp_offset(entry)); + struct page *p = pfn_to_page(swp_offset_pfn(entry)); /* * Any use of migration entries may only occur while the @@ -387,6 +412,9 @@ static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry) */ static inline bool is_pfn_swap_entry(swp_entry_t entry) { + /* Make sure the swp offset can always store the needed fields */ + BUILD_BUG_ON(SWP_TYPE_SHIFT < SWP_PFN_BITS); + return is_migration_entry(entry) || is_device_private_entry(entry) || is_device_exclusive_entry(entry); } @@ -475,11 +503,6 @@ static inline int is_hwpoison_entry(swp_entry_t entry) return swp_type(entry) == SWP_HWPOISON; } -static inline unsigned long hwpoison_entry_to_pfn(swp_entry_t entry) -{ - return swp_offset(entry); -} - static inline void num_poisoned_pages_inc(void) { atomic_long_inc(&num_poisoned_pages); diff --git a/mm/hmm.c b/mm/hmm.c index f2aa63b94d9b..3850fb625dda 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -253,7 +253,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, cpu_flags = HMM_PFN_VALID; if (is_writable_device_private_entry(entry)) cpu_flags |= HMM_PFN_WRITE; - *hmm_pfn = swp_offset(entry) | cpu_flags; + *hmm_pfn = swp_offset_pfn(entry) | cpu_flags; return 0; } diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 14439806b5ef..c8a7d38c1da4 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -632,7 +632,7 @@ static int check_hwpoisoned_entry(pte_t pte, unsigned long addr, short shift, swp_entry_t swp = pte_to_swp_entry(pte); if (is_hwpoison_entry(swp)) - pfn = hwpoison_entry_to_pfn(swp); + pfn = swp_offset_pfn(swp); } if (!pfn || pfn != poisoned_pfn) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 8e9e574d535a..93e13fc17d3c 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -86,7 +86,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) !is_device_exclusive_entry(entry)) return false; - pfn = swp_offset(entry); + pfn = swp_offset_pfn(entry); } else if (is_swap_pte(*pvmw->pte)) { swp_entry_t entry; @@ -96,7 +96,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) !is_device_exclusive_entry(entry)) return false; - pfn = swp_offset(entry); + pfn = swp_offset_pfn(entry); } else { if (!pte_present(*pvmw->pte)) return false; @@ -221,7 +221,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) return not_found(pvmw); entry = pmd_to_swp_entry(pmde); if (!is_migration_entry(entry) || - !check_pmd(swp_offset(entry), pvmw)) + !check_pmd(swp_offset_pfn(entry), pvmw)) return not_found(pvmw); return true; } From patchwork Tue Aug 9 22:00:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12939967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 029D7C19F2D for ; Tue, 9 Aug 2022 22:01:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0E058E0008; Tue, 9 Aug 2022 18:01:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A094B8E0007; Tue, 9 Aug 2022 18:01:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80D738E0008; Tue, 9 Aug 2022 18:01:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 603C28E0001 for ; Tue, 9 Aug 2022 18:01:13 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1C326C12B1 for ; Tue, 9 Aug 2022 22:01:13 +0000 (UTC) X-FDA: 79781425626.20.17C2564 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id B244B20184 for ; Tue, 9 Aug 2022 22:01:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660082470; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KrXV1mhD8NFSPZwAf9c4FL2hL9qvLlZq2jcNtWuZzdA=; b=YWQlnY9GGa44oSDEKV+26Vjp8Ia44VGC1qVFzT6lr+WhlET05t0YRfya4RNVcyjEkSOXju bI3FY4OiuoGNUquJ13Yz/SDv2WDzL2hX0fQw/eYNcuPGnSdJMYd/XaYbYgc2M93Gc2l507 LZ9jrkmiOXE2diKDFvwZSuSDGD8FUAE= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-191-pFteTs01MJqzzbBycbxJtg-1; Tue, 09 Aug 2022 18:01:09 -0400 X-MC-Unique: pFteTs01MJqzzbBycbxJtg-1 Received: by mail-qv1-f70.google.com with SMTP id u9-20020a0cec89000000b0047498457e00so6903852qvo.3 for ; Tue, 09 Aug 2022 15:01:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=KrXV1mhD8NFSPZwAf9c4FL2hL9qvLlZq2jcNtWuZzdA=; b=0XyJxXuLxtvwGZYfmZJyNilH/RfpiwPUpQHS5aXdFS/mpcY9ZJdGc6eajpK0sYA5uo x3dh/2X40FFMsC3P4lRYyBgsKCuuwcfm3sMFPvFAQPwjV1LkFDl/ZQmc1+qQk8iybECx /3SPkmb/gk14/KV4NxHxavTFzV8e7sBxufbmdH7Y/mgm9p33uhpHXvsPJAgo5NW4LJ9I lcgDf9e1U4qzuyOJ44qLuuigiEZEpAcuW1wueTEqRRfc2kNhJZANq45tdId8+7kzujzn A6A4ptyvix7SVaoPDVztqEXQM8wqeFgDOgSuXQ2l9+fBV7TGkKSFzhvcZiaYd6nXx/b/ V67Q== X-Gm-Message-State: ACgBeo35dRFERT7FdY6HhEXTD7FFwKIR8/OWapzj45AcjMo0+0ilTGdr H9SdZMoznZq4EUtXNdorKLbxof4yZrdJL4X9hjJ0hzQUw/FxQ+SM097knMy17eETDy8DNSBMQig MVBRxoaVpvZNlUp+F+P9zka9fcqGVUejs02GcTFVIMJ5UT7FQ2lhALdvrh83e X-Received: by 2002:a05:6214:f24:b0:476:72bb:2473 with SMTP id iw4-20020a0562140f2400b0047672bb2473mr21833898qvb.82.1660082468671; Tue, 09 Aug 2022 15:01:08 -0700 (PDT) X-Google-Smtp-Source: AA6agR53TiwBc9GpPqRdwAZDQe2UaVr8TnIHRxStNg8j6qBByy+9hjyDW7m38tabiWtpuwTjDL75Bw== X-Received: by 2002:a05:6214:f24:b0:476:72bb:2473 with SMTP id iw4-20020a0562140f2400b0047672bb2473mr21833862qvb.82.1660082468439; Tue, 09 Aug 2022 15:01:08 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id b16-20020a05622a021000b0031eddc83560sm10563383qtx.90.2022.08.09.15.01.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 09 Aug 2022 15:01:08 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Minchan Kim , David Hildenbrand , Nadav Amit , Andrew Morton , Hugh Dickins , Vlastimil Babka , Andrea Arcangeli , Andi Kleen , Huang Ying , "Kirill A . Shutemov" Subject: [PATCH v3 4/7] mm/thp: Carry over dirty bit when thp splits on pmd Date: Tue, 9 Aug 2022 18:00:57 -0400 Message-Id: <20220809220100.20033-5-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220809220100.20033-1-peterx@redhat.com> References: <20220809220100.20033-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660082472; a=rsa-sha256; cv=none; b=3rzBzjAfxnLoZRJaliRK/70BKN9gl9HFbtsGFCNMZIq8gopFEiq/KAH31he1HrJQjZ7zpj UxBk4CtGuf78P7PlS3OmGtffjxLNT1eNOmeq/5w1kDg2a/T+/uvptVgp7DDoQV7UmUbU/h Z2cOhGrqAhtOQhCcViymM8BqJYS1K/o= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YWQlnY9G; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660082472; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KrXV1mhD8NFSPZwAf9c4FL2hL9qvLlZq2jcNtWuZzdA=; b=eRUqXF539fcacGcBGSKDVg5BJUNgWr2zZhBpeQRcdxRiyDk2mtQDc6kOpaDZ71vwS7ZRSr o5//rVfFL8Z8KZamamLguUoT3d6qZwIM5uuk4OgOXn7yebr1hqwHZ89/T0SiDUvJokvvr4 ngLZow0lcj2/Z2Snu6A1IrC5NvSTF68= X-Rspamd-Queue-Id: B244B20184 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YWQlnY9G; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: eeux6b4bms3pqogm5hs3a36hby1rn5an X-HE-Tag: 1660082470-465105 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Carry over the dirty bit from pmd to pte when a huge pmd splits. It shouldn't be a correctness issue since when pmd_dirty() we'll have the page marked dirty anyway, however having dirty bit carried over helps the next initial writes of split ptes on some archs like x86. Signed-off-by: Peter Xu Reviewed-by: "Huang, Ying" --- mm/huge_memory.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0611b2fd145a..e8e78d1bac5f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2005,7 +2005,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pgtable_t pgtable; pmd_t old_pmd, _pmd; bool young, write, soft_dirty, pmd_migration = false, uffd_wp = false; - bool anon_exclusive = false; + bool anon_exclusive = false, dirty = false; unsigned long addr; int i; @@ -2098,6 +2098,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, SetPageDirty(page); write = pmd_write(old_pmd); young = pmd_young(old_pmd); + dirty = pmd_dirty(old_pmd); soft_dirty = pmd_soft_dirty(old_pmd); uffd_wp = pmd_uffd_wp(old_pmd); @@ -2161,6 +2162,9 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_wrprotect(entry); if (!young) entry = pte_mkold(entry); + /* NOTE: this may set soft-dirty too on some archs */ + if (dirty) + entry = pte_mkdirty(entry); if (soft_dirty) entry = pte_mksoft_dirty(entry); if (uffd_wp) From patchwork Tue Aug 9 22:00:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12939968 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6315AC25B08 for ; Tue, 9 Aug 2022 22:01:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DAA138E0001; Tue, 9 Aug 2022 18:01:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE6F18E0009; Tue, 9 Aug 2022 18:01:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 946658E0001; Tue, 9 Aug 2022 18:01:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 657098E0007 for ; Tue, 9 Aug 2022 18:01:13 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2493B8034C for ; Tue, 9 Aug 2022 22:01:13 +0000 (UTC) X-FDA: 79781425626.10.85E5454 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 9A1D34005C for ; Tue, 9 Aug 2022 22:01:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660082471; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1RwmKRYJuvTG/btGRX62JpRnXDc5C3jxVCpX+grs9Ro=; b=KIPM2jJQZaYSTPaCLsWHotXRAjDekjJACRB/oocteg3EL9KBUgHICq1VZOOji2MeVsumSC EA+LjnuEYbvhLlXO8TZkcOxrkUC2FHZE6q6Ah5dF3atTJ8rYGVKCD6QmVv7NPS+mb4alot T7QCEI7B4I8jbMqPRj0//fjUwRngrJ4= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-284-UfqGy2LLPaOxzES9M9edXw-1; Tue, 09 Aug 2022 18:01:10 -0400 X-MC-Unique: UfqGy2LLPaOxzES9M9edXw-1 Received: by mail-qv1-f72.google.com with SMTP id a11-20020ad45c4b000000b004747a998b9eso6881997qva.9 for ; Tue, 09 Aug 2022 15:01:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=1RwmKRYJuvTG/btGRX62JpRnXDc5C3jxVCpX+grs9Ro=; b=yXjaTQ3Si127vwGzzlgmn42hMA+pa/1GaaJAsmOAznKiVLS/3iEW5MvrT4g/aucPc6 n8x87HRMGzpXnKOTksUWp65Uo0scYCjj8VweYb5AzgAbTI1e7MTsPp5uRhDvR7P4dq/D EcAintH0/WG/6UAw+PPBUKgDwXPSfiVjv0i/X19Lvv87sbHps7+ZB7HfWW0pOGCx+iH0 74FKF0rnxSrPQZZrgEKvgNpe4ACVhPzxWpwbPji0RwXgbdSrMq9PvTjSceJ8TCttZhbz NBy78MQaVbpBdS7FA99RPPh82NXUYUpfP7sjirHxbwrdoNIB4zALo+yx9DBFy9gffEif bYOA== X-Gm-Message-State: ACgBeo0fhZaYFef9Hh7wISXtcF+2iEOsegdv/nrBBNNQOawd7jBP+st2 tMn5c8BVYlijbq6g0d9tCWWtOm6HvpOETGN7q9Z9uhgGhHzhmpo10foJkMHnRdLlnRaUGs1cab/ TtIS62WzKNa7c3+kjo67aeP/GAHKqWwQYLycyeoEOWsm3CFFtfP3qumPWfPBx X-Received: by 2002:ac8:5a51:0:b0:342:e917:f25b with SMTP id o17-20020ac85a51000000b00342e917f25bmr16799595qta.284.1660082470174; Tue, 09 Aug 2022 15:01:10 -0700 (PDT) X-Google-Smtp-Source: AA6agR7F8awg0xQpCbkyT+nfZ6Y4i6eUOSmTv7BxMbnhF6Tnq2rHRG8SF8KC3nPcS8vkXeC9hjxxmA== X-Received: by 2002:ac8:5a51:0:b0:342:e917:f25b with SMTP id o17-20020ac85a51000000b00342e917f25bmr16799543qta.284.1660082469795; Tue, 09 Aug 2022 15:01:09 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id b16-20020a05622a021000b0031eddc83560sm10563383qtx.90.2022.08.09.15.01.08 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 09 Aug 2022 15:01:09 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Minchan Kim , David Hildenbrand , Nadav Amit , Andrew Morton , Hugh Dickins , Vlastimil Babka , Andrea Arcangeli , Andi Kleen , Huang Ying , "Kirill A . Shutemov" Subject: [PATCH v3 5/7] mm: Remember young/dirty bit for page migrations Date: Tue, 9 Aug 2022 18:00:58 -0400 Message-Id: <20220809220100.20033-6-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220809220100.20033-1-peterx@redhat.com> References: <20220809220100.20033-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660082472; a=rsa-sha256; cv=none; b=8cl/o5tnzccwPijWE8HVSLpbXkePTXhQJgzNeDiNyjWSzFSNlZ8llKhSvBRdXejDTjuff3 n1W6IqObIJVshlQ6ZkOYQuYtqiLAOZevwVFNby58/iZFEE09hAy2Pt2gjx2TB5Ut+kas6S M0YVRPvEIjuNQ3OgQ+XoPi1r++wSu8o= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KIPM2jJQ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660082472; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1RwmKRYJuvTG/btGRX62JpRnXDc5C3jxVCpX+grs9Ro=; b=b9Hiasbkn5GHOgioep/5I5GqmjbW0zq/HUrOTD+LHcJGXoMRSuUxBl0dP/9LWvVnJ5Uk8E lJ6m4AIdOUixOQoSqReub56sypbg3IT3Pifhku6LXz0p7x12VMEEpTxqt3L4iNNfug7rUc TM8BwbjNq1iYABRoJ3I7z8KtScwux+c= X-Rspamd-Server: rspam10 X-Stat-Signature: 4ininqbmax114hzf563ezmagfa1t86fz Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KIPM2jJQ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com X-Rspam-User: X-Rspamd-Queue-Id: 9A1D34005C X-HE-Tag: 1660082472-977724 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When page migration happens, we always ignore the young/dirty bit settings in the old pgtable, and marking the page as old in the new page table using either pte_mkold() or pmd_mkold(), and keeping the pte clean. That's fine from functional-wise, but that's not friendly to page reclaim because the moving page can be actively accessed within the procedure. Not to mention hardware setting the young bit can bring quite some overhead on some systems, e.g. x86_64 needs a few hundreds nanoseconds to set the bit. The same slowdown problem to dirty bits when the memory is first written after page migration happened. Actually we can easily remember the A/D bit configuration and recover the information after the page is migrated. To achieve it, define a new set of bits in the migration swap offset field to cache the A/D bits for old pte. Then when removing/recovering the migration entry, we can recover the A/D bits even if the page changed. One thing to mention is that here we used max_swapfile_size() to detect how many swp offset bits we have, and we'll only enable this feature if we know the swp offset can be big enough to store both the PFN value and the young bit. Otherwise the A/D bits are dropped like before. Signed-off-by: Peter Xu --- include/linux/swapops.h | 99 +++++++++++++++++++++++++++++++++++++++++ mm/huge_memory.c | 18 +++++++- mm/migrate.c | 6 ++- mm/migrate_device.c | 4 ++ mm/rmap.c | 5 ++- 5 files changed, 128 insertions(+), 4 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index e1accbcd1136..0e9579b90659 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -8,6 +8,10 @@ #ifdef CONFIG_MMU +#ifdef CONFIG_SWAP +#include +#endif /* CONFIG_SWAP */ + /* * swapcache pages are stored in the swapper_space radix tree. We want to * get good packing density in that tree, so the index should be dense in @@ -35,6 +39,31 @@ #endif /* MAX_PHYSMEM_BITS */ #define SWP_PFN_MASK ((1UL << SWP_PFN_BITS) - 1) +/** + * Migration swap entry specific bitfield definitions. Layout: + * + * |----------+--------------------| + * | swp_type | swp_offset | + * |----------+--------+-+-+-------| + * | | resv |D|A| PFN | + * |----------+--------+-+-+-------| + * + * @SWP_MIG_YOUNG_BIT: Whether the page used to have young bit set (bit A) + * @SWP_MIG_DIRTY_BIT: Whether the page used to have dirty bit set (bit D) + * + * Note: A/D bits will be stored in migration entries iff there're enough + * free bits in arch specific swp offset. By default we'll ignore A/D bits + * when migrating a page. Please refer to migration_entry_supports_ad() + * for more information. If there're more bits besides PFN and A/D bits, + * they should be reserved and always be zeros. + */ +#define SWP_MIG_YOUNG_BIT (SWP_PFN_BITS) +#define SWP_MIG_DIRTY_BIT (SWP_PFN_BITS + 1) +#define SWP_MIG_TOTAL_BITS (SWP_PFN_BITS + 2) + +#define SWP_MIG_YOUNG BIT(SWP_MIG_YOUNG_BIT) +#define SWP_MIG_DIRTY BIT(SWP_MIG_DIRTY_BIT) + static inline bool is_pfn_swap_entry(swp_entry_t entry); /* Clear all flags but only keep swp_entry_t related information */ @@ -265,6 +294,57 @@ static inline swp_entry_t make_writable_migration_entry(pgoff_t offset) return swp_entry(SWP_MIGRATION_WRITE, offset); } +/* + * Returns whether the host has large enough swap offset field to support + * carrying over pgtable A/D bits for page migrations. The result is + * pretty much arch specific. + */ +static inline bool migration_entry_supports_ad(void) +{ + /* + * max_swapfile_size() returns the max supported swp-offset plus 1. + * We can support the migration A/D bits iff the pfn swap entry has + * the offset large enough to cover all of them (PFN, A & D bits). + */ +#ifdef CONFIG_SWAP + return max_swapfile_size() >= (1UL << SWP_MIG_TOTAL_BITS); +#else /* CONFIG_SWAP */ + return false; +#endif /* CONFIG_SWAP */ +} + +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_entry(swp_type(entry), + swp_offset(entry) | SWP_MIG_YOUNG); + return entry; +} + +static inline bool is_migration_entry_young(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_offset(entry) & SWP_MIG_YOUNG; + /* Keep the old behavior of aging page after migration */ + return false; +} + +static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_entry(swp_type(entry), + swp_offset(entry) | SWP_MIG_DIRTY); + return entry; +} + +static inline bool is_migration_entry_dirty(swp_entry_t entry) +{ + if (migration_entry_supports_ad()) + return swp_offset(entry) & SWP_MIG_DIRTY; + /* Keep the old behavior of clean page after migration */ + return false; +} + extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, @@ -311,6 +391,25 @@ static inline int is_readable_migration_entry(swp_entry_t entry) return 0; } +static inline swp_entry_t make_migration_entry_young(swp_entry_t entry) +{ + return entry; +} + +static inline bool is_migration_entry_young(swp_entry_t entry) +{ + return false; +} + +static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry) +{ + return entry; +} + +static inline bool is_migration_entry_dirty(swp_entry_t entry) +{ + return false; +} #endif /* CONFIG_MIGRATION */ typedef unsigned long pte_marker; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e8e78d1bac5f..1644e9f59d73 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2089,7 +2089,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, write = is_writable_migration_entry(entry); if (PageAnon(page)) anon_exclusive = is_readable_exclusive_migration_entry(entry); - young = false; + young = is_migration_entry_young(entry); + dirty = is_migration_entry_dirty(entry); soft_dirty = pmd_swp_soft_dirty(old_pmd); uffd_wp = pmd_swp_uffd_wp(old_pmd); } else { @@ -2148,6 +2149,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, else swp_entry = make_readable_migration_entry( page_to_pfn(page + i)); + if (young) + swp_entry = make_migration_entry_young(swp_entry); + if (dirty) + swp_entry = make_migration_entry_dirty(swp_entry); entry = swp_entry_to_pte(swp_entry); if (soft_dirty) entry = pte_swp_mksoft_dirty(entry); @@ -3157,6 +3162,10 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, entry = make_readable_exclusive_migration_entry(page_to_pfn(page)); else entry = make_readable_migration_entry(page_to_pfn(page)); + if (pmd_young(pmdval)) + entry = make_migration_entry_young(entry); + if (pmd_dirty(pmdval)) + entry = make_migration_entry_dirty(entry); pmdswp = swp_entry_to_pmd(entry); if (pmd_soft_dirty(pmdval)) pmdswp = pmd_swp_mksoft_dirty(pmdswp); @@ -3182,13 +3191,18 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) entry = pmd_to_swp_entry(*pvmw->pmd); get_page(new); - pmde = pmd_mkold(mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot))); + pmde = mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot)); if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde = pmd_mksoft_dirty(pmde); if (is_writable_migration_entry(entry)) pmde = maybe_pmd_mkwrite(pmde, vma); if (pmd_swp_uffd_wp(*pvmw->pmd)) pmde = pmd_wrprotect(pmd_mkuffd_wp(pmde)); + if (!is_migration_entry_young(entry)) + pmde = pmd_mkold(pmde); + /* NOTE: this may contain setting soft-dirty on some archs */ + if (PageDirty(new) && is_migration_entry_dirty(entry)) + pmde = pmd_mkdirty(pmde); if (PageAnon(new)) { rmap_t rmap_flags = RMAP_COMPOUND; diff --git a/mm/migrate.c b/mm/migrate.c index 6a1597c92261..0433a71d2bee 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -198,7 +198,7 @@ static bool remove_migration_pte(struct folio *folio, #endif folio_get(folio); - pte = pte_mkold(mk_pte(new, READ_ONCE(vma->vm_page_prot))); + pte = mk_pte(new, READ_ONCE(vma->vm_page_prot)); if (pte_swp_soft_dirty(*pvmw.pte)) pte = pte_mksoft_dirty(pte); @@ -206,6 +206,10 @@ static bool remove_migration_pte(struct folio *folio, * Recheck VMA as permissions can change since migration started */ entry = pte_to_swp_entry(*pvmw.pte); + if (!is_migration_entry_young(entry)) + pte = pte_mkold(pte); + if (folio_test_dirty(folio) && is_migration_entry_dirty(entry)) + pte = pte_mkdirty(pte); if (is_writable_migration_entry(entry)) pte = maybe_mkwrite(pte, vma); else if (pte_swp_uffd_wp(*pvmw.pte)) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 27fb37d65476..699f821b8443 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -221,6 +221,10 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, else entry = make_readable_migration_entry( page_to_pfn(page)); + if (pte_young(pte)) + entry = make_migration_entry_young(entry); + if (pte_dirty(pte)) + entry = make_migration_entry_dirty(entry); swp_pte = swp_entry_to_pte(entry); if (pte_present(pte)) { if (pte_soft_dirty(pte)) diff --git a/mm/rmap.c b/mm/rmap.c index af775855e58f..28aef434ea41 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2065,7 +2065,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, else entry = make_readable_migration_entry( page_to_pfn(subpage)); - + if (pte_young(pteval)) + entry = make_migration_entry_young(entry); + if (pte_dirty(pteval)) + entry = make_migration_entry_dirty(entry); swp_pte = swp_entry_to_pte(entry); if (pte_soft_dirty(pteval)) swp_pte = pte_swp_mksoft_dirty(swp_pte); From patchwork Tue Aug 9 22:00:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12939969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53D2EC19F2D for ; Tue, 9 Aug 2022 22:01:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59F7A8E0009; Tue, 9 Aug 2022 18:01:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4FF328E0007; Tue, 9 Aug 2022 18:01:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3516D8E0009; Tue, 9 Aug 2022 18:01:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 181638E0007 for ; Tue, 9 Aug 2022 18:01:14 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E2DF1141212 for ; Tue, 9 Aug 2022 22:01:13 +0000 (UTC) X-FDA: 79781425626.30.06CBCA5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 67B38180166 for ; Tue, 9 Aug 2022 22:01:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660082472; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QP8/u5AChrIXiwIzqFnG/IHURDZ5n0i12x48YvKEviU=; b=h1CbR7yHaD22366O//ft4s+Xv1/ab9TFFcnCwrzihvwvFPSdxd5agLClY2CyQKAkrEsPVZ K9WM6ouhbn0dS3pfEvNVPDdxyPMuNTTvtUUMTA4hsONx2K0VsEJq+kkKfcTQw49oPRs9z/ aT+o6H7apbMG6qhMiLQTIOMAemDOTdg= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-640-Yb5E4k9LOeWtGEm_ScSKIw-1; Tue, 09 Aug 2022 18:01:12 -0400 X-MC-Unique: Yb5E4k9LOeWtGEm_ScSKIw-1 Received: by mail-qk1-f199.google.com with SMTP id m17-20020a05620a291100b006b6590830feso11236738qkp.13 for ; Tue, 09 Aug 2022 15:01:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=QP8/u5AChrIXiwIzqFnG/IHURDZ5n0i12x48YvKEviU=; b=gC2ofxMIrrBUK5OeCRDYMLyF0wgaLuPxd5qc72M0SG2anKRnhvh4v1T1G4hHVawMTM LoJ5Ij2GQ7J3bQ74QWu9JuKuTcHVDY5WobDbRMLwgIyQQqKfreOqjJAn623GrHgXY+nF Im7B5SW9tGdDavoBQ7nUWIqKKAHflMoJTItPlY8l9JKdkbba3EN4iH5n9Dd+cKI2FLWf irzCOxbWbxkFHXVmoQslHfuPkk8ISaY/LSVgTa4RmcjoWd/EMIsSW9OM4ABt3iMwlInz 7zNKhkQcjeor1oK+wz2AZhDmzlEhMqO1e8xtho/xbNphirVPONtqWdOZl3Cg2O9TuH4q bjkQ== X-Gm-Message-State: ACgBeo1j1N1UbSurojqWdhMeqyBUFMKYDzHzZ8x7oVCdXTt/Jdm0PSrd qbhRtXfSCyZ8guhBVuzKPNFYOVJVWPXzucYOmjTJ598K9dDHNWqSdY59bNgyYQy/LK3OuJSJWx0 9zpXnrYjo1hjEBhUWVQB2R1Ldp9npgE6aFDX1DE5+sya6/ygoirtWVsVx99dF X-Received: by 2002:a05:622a:184:b0:31e:d9a2:1e7 with SMTP id s4-20020a05622a018400b0031ed9a201e7mr21769890qtw.600.1660082471440; Tue, 09 Aug 2022 15:01:11 -0700 (PDT) X-Google-Smtp-Source: AA6agR6rCoAdukBJ919s0iooFl+Su6+6WLl8fX28qz4Ljb6jLzuDkULdmT6xOGx8775yUr2wJWOu3Q== X-Received: by 2002:a05:622a:184:b0:31e:d9a2:1e7 with SMTP id s4-20020a05622a018400b0031ed9a201e7mr21769841qtw.600.1660082471059; Tue, 09 Aug 2022 15:01:11 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id b16-20020a05622a021000b0031eddc83560sm10563383qtx.90.2022.08.09.15.01.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 09 Aug 2022 15:01:10 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Minchan Kim , David Hildenbrand , Nadav Amit , Andrew Morton , Hugh Dickins , Vlastimil Babka , Andrea Arcangeli , Andi Kleen , Huang Ying , "Kirill A . Shutemov" Subject: [PATCH v3 6/7] mm/swap: Cache maximum swapfile size when init swap Date: Tue, 9 Aug 2022 18:00:59 -0400 Message-Id: <20220809220100.20033-7-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220809220100.20033-1-peterx@redhat.com> References: <20220809220100.20033-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660082473; a=rsa-sha256; cv=none; b=cwhi737gHnOHWFEq0QrmyOtNziY/VP482eRSgthaBnh0oi13oUNx/T7GGwWEInEdna/q2S KpcyUjKeXhCZ029ZCcPejCOinZJk2Rvy5iLnjbZAKSo6fS0+AbPvH2M9zK6wAPbn8hqMCQ CIgIvqeCQjIb2peXomR7KrVodVW5VWs= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h1CbR7yH; spf=pass (imf06.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660082473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QP8/u5AChrIXiwIzqFnG/IHURDZ5n0i12x48YvKEviU=; b=BtbEwlcpCd3Iqafq78Ub+7r5tPvyRxw7YLaAc2+iHb2r7sf2jnPuVcp3d2y1QW3x79XK6n s4BHO+Ic6Z7jJ0TO6c3F40DPszPJNOZtCGXQEnHj1dNZsvsrt2OSgT+BhkECgXcZB/B5hr EXa3ULWjL0hGvSJeIMOslrPPfSR1g9I= X-Stat-Signature: oisjn19ns8bwzb471mmrnzghnwhpjzpw X-Rspamd-Queue-Id: 67B38180166 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=h1CbR7yH; spf=pass (imf06.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1660082473-395599 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We used to have swapfile_maximum_size() fetching a maximum value of swapfile size per-arch. As the caller of max_swapfile_size() grows, this patch introduce a variable "swapfile_maximum_size" and cache the value of old max_swapfile_size(), so that we don't need to calculate the value every time. Caching the value in swapfile_init() is safe because when reaching the phase we should have initialized all the relevant information. Here the major arch to look after is x86, which defines the max size based on L1TF mitigation. Here both X86_BUG_L1TF or l1tf_mitigation should have been setup properly when reaching swapfile_init(). As a reference, the code path looks like this for x86: - start_kernel - setup_arch - early_cpu_init - early_identify_cpu --> setup X86_BUG_L1TF - parse_early_param - l1tf_cmdline --> set l1tf_mitigation - check_bugs - l1tf_select_mitigation --> set l1tf_mitigation - arch_call_rest_init - rest_init - kernel_init - kernel_init_freeable - do_basic_setup - do_initcalls --> calls swapfile_init() (initcall level 4) The swapfile size only depends on swp pte format on non-x86 archs, so caching it is safe too. Since at it, rename max_swapfile_size() to arch_max_swapfile_size() because arch can define its own function, so it's more straightforward to have "arch_" as its prefix. At the meantime, keep the swapfile_maximum_size() function to fetch the value from the cache initialized in swapfile_init(). Signed-off-by: Peter Xu --- arch/x86/mm/init.c | 2 +- mm/swapfile.c | 10 +++++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 82a042c03824..9121bc1b9453 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -1054,7 +1054,7 @@ void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache) } #ifdef CONFIG_SWAP -unsigned long max_swapfile_size(void) +unsigned long arch_max_swapfile_size(void) { unsigned long pages; diff --git a/mm/swapfile.c b/mm/swapfile.c index 1fdccd2f1422..794fa37bd0c3 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -63,6 +63,7 @@ EXPORT_SYMBOL_GPL(nr_swap_pages); /* protected with swap_lock. reading in vm_swap_full() doesn't need lock */ long total_swap_pages; static int least_priority = -1; +static unsigned long swapfile_maximum_size; static const char Bad_file[] = "Bad swap file entry "; static const char Unused_file[] = "Unused swap file entry "; @@ -2816,11 +2817,16 @@ unsigned long generic_max_swapfile_size(void) } /* Can be overridden by an architecture for additional checks. */ -__weak unsigned long max_swapfile_size(void) +__weak unsigned long arch_max_swapfile_size(void) { return generic_max_swapfile_size(); } +unsigned long max_swapfile_size(void) +{ + return swapfile_maximum_size; +} + static unsigned long read_swap_header(struct swap_info_struct *p, union swap_header *swap_header, struct inode *inode) @@ -3677,6 +3683,8 @@ static int __init swapfile_init(void) for_each_node(nid) plist_head_init(&swap_avail_heads[nid]); + swapfile_maximum_size = arch_max_swapfile_size(); + return 0; } subsys_initcall(swapfile_init); From patchwork Tue Aug 9 22:01:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12939970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 373E7C25B06 for ; Tue, 9 Aug 2022 22:01:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDF1F8E000A; Tue, 9 Aug 2022 18:01:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B43378E0007; Tue, 9 Aug 2022 18:01:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 995218E000A; Tue, 9 Aug 2022 18:01:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 655B78E0007 for ; Tue, 9 Aug 2022 18:01:15 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4996581146 for ; Tue, 9 Aug 2022 22:01:15 +0000 (UTC) X-FDA: 79781425710.09.DAB8F90 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id DB85FA0174 for ; Tue, 9 Aug 2022 22:01:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660082474; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8oTmLlquT/JmXxP88HUbcjeMqANq5xLJcNEKCKe+fKE=; b=S69FzfOWOVq7JyjaS/1d3hshovEm2x5unC1AWQ3zrK02vGC1wDgkd/NW2qMYCGJ8QHfuqJ +bf9aMoUtk21uhmD/QK6tvXkIEo6jJfMcNGaiBUx/hDkPqKuvb9hwrN/605NnzgNevRBBj gQikm7ehjiQLwgyjpJylN+gf90rHmdM= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-536-iur_ptNwNkSblXjoGAPuvQ-1; Tue, 09 Aug 2022 18:01:13 -0400 X-MC-Unique: iur_ptNwNkSblXjoGAPuvQ-1 Received: by mail-qk1-f199.google.com with SMTP id i15-20020a05620a404f00b006b55998179bso11148065qko.4 for ; Tue, 09 Aug 2022 15:01:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=8oTmLlquT/JmXxP88HUbcjeMqANq5xLJcNEKCKe+fKE=; b=hOfe4SIEiKZAAiab2DUtkIMXuXlaCeao0raVEjNXqL1tyHbfgSvlh1ES8ajMeNdCMO cNqqGReV27X3SFNhds5q326BIUjk2AeHQY0dDiH0nyE13pJo0sMwtcTwh9AOCXfr5TbX 5cMadvo9hyC8/PSKH87PK5CIY6GE1gnta61D4+LysMDoZaCMyoDcBdclbDNj/hxYWOiu 2+1Qi/8eeYijMLkiydMQcMpI5i4RR+yAy2j2uQ+K2q9R7HAWLL6A7gbG4NGhfAHK6q4d 0O8Ca4hAW7ZtYbkxYdyZ0bnuicQ1GjTEJkI0mQy3/6cyBNplcwVGopVL3JMUkpo6PwfG uDJg== X-Gm-Message-State: ACgBeo2Mx1tIK6ll6e7l1i8I75ZM/1Z9nxpPMpAaJa8gp0LkzVBCcjuo qP0brEVpmclclD7bF5tF3C6qBol7UCROPSAbDbSnfeofKHijbF0niqsJYt1+mG7uTSriDiai0P3 oUdkfx1VSUpVXOdW5CIrL7fQeUNGJO/GoP8k0E1Rwu5FYUY5UcgInre/sLfH6 X-Received: by 2002:ad4:5745:0:b0:474:73b3:8e6b with SMTP id q5-20020ad45745000000b0047473b38e6bmr21943608qvx.30.1660082472729; Tue, 09 Aug 2022 15:01:12 -0700 (PDT) X-Google-Smtp-Source: AA6agR7zy2JuWvJjf9RyNp22pt6EAQPYAGfW6c3P8B5Al6S1qhuoVayGnuKePcXRi5AkcG0+83tbPw== X-Received: by 2002:ad4:5745:0:b0:474:73b3:8e6b with SMTP id q5-20020ad45745000000b0047473b38e6bmr21943562qvx.30.1660082472384; Tue, 09 Aug 2022 15:01:12 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id b16-20020a05622a021000b0031eddc83560sm10563383qtx.90.2022.08.09.15.01.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 09 Aug 2022 15:01:12 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Minchan Kim , David Hildenbrand , Nadav Amit , Andrew Morton , Hugh Dickins , Vlastimil Babka , Andrea Arcangeli , Andi Kleen , Huang Ying , "Kirill A . Shutemov" Subject: [PATCH v3 7/7] mm/swap: Cache swap migration A/D bits support Date: Tue, 9 Aug 2022 18:01:00 -0400 Message-Id: <20220809220100.20033-8-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220809220100.20033-1-peterx@redhat.com> References: <20220809220100.20033-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660082475; a=rsa-sha256; cv=none; b=HAqgreu6Qvq5LlVHFk5C6vOnQE6gl0QUPjLuKGujz0CcmnmuTVsVfuyVa0PrqfSX7YGFby gfU0gvS4X9EVeq2fmfU0xDg6kjik/dlGJPB7n3om7dBFSWb4lYAt9Ok2QQzziitk1iroaQ 64hzghIfwEDTfQrz/spqJPtukjtkfKk= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=S69FzfOW; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660082475; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8oTmLlquT/JmXxP88HUbcjeMqANq5xLJcNEKCKe+fKE=; b=psJU17Tud7aYtFiJUv78fyf7bmo3SR2ZgewKU9h7GKXEjyEno0McjzHMv9rc+wIXjJzb+v DGlU/LfSS1e7csIMzeLC/QAoX9yfkqn5l58Ac+DZ0UZTiGH7hGWGAirjtPLX3eaD05um5W aWs1xi3qufNpQ5GYAxdKy4TR8lZPehc= X-Rspamd-Queue-Id: DB85FA0174 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=S69FzfOW; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: zxzzm93niirk3brw11u3e435jtxq9zsu X-HE-Tag: 1660082474-848686 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a variable swap_migration_ad_supported to cache whether the arch supports swap migration A/D bits. Here one thing to mention is that SWP_MIG_TOTAL_BITS will internally reference the other macro MAX_PHYSMEM_BITS, which is a function call on x86 (constant on all the rest of archs). It's safe to reference it in swapfile_init() because when reaching here we're already during initcalls level 4 so we must have initialized 5-level pgtable for x86_64 (right after early_identify_cpu() finishes). - start_kernel - setup_arch - early_cpu_init - get_cpu_cap --> fetch from CPUID (including X86_FEATURE_LA57) - early_identify_cpu --> clear X86_FEATURE_LA57 (if early lvl5 not enabled (USE_EARLY_PGTABLE_L5)) - arch_call_rest_init - rest_init - kernel_init - kernel_init_freeable - do_basic_setup - do_initcalls --> calls swapfile_init() (initcall level 4) This should slightly speed up the migration swap entry handlings. Signed-off-by: Peter Xu --- include/linux/swapfile.h | 1 + include/linux/swapops.h | 7 +------ mm/swapfile.c | 8 ++++++++ 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/include/linux/swapfile.h b/include/linux/swapfile.h index 54078542134c..87ec5e2cdb02 100644 --- a/include/linux/swapfile.h +++ b/include/linux/swapfile.h @@ -9,5 +9,6 @@ extern struct swap_info_struct *swap_info[]; extern unsigned long generic_max_swapfile_size(void); extern unsigned long max_swapfile_size(void); +extern bool swap_migration_ad_supported; #endif /* _LINUX_SWAPFILE_H */ diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 0e9579b90659..e6afc77c51ad 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -301,13 +301,8 @@ static inline swp_entry_t make_writable_migration_entry(pgoff_t offset) */ static inline bool migration_entry_supports_ad(void) { - /* - * max_swapfile_size() returns the max supported swp-offset plus 1. - * We can support the migration A/D bits iff the pfn swap entry has - * the offset large enough to cover all of them (PFN, A & D bits). - */ #ifdef CONFIG_SWAP - return max_swapfile_size() >= (1UL << SWP_MIG_TOTAL_BITS); + return swap_migration_ad_supported; #else /* CONFIG_SWAP */ return false; #endif /* CONFIG_SWAP */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 794fa37bd0c3..c49cf25f0d08 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -64,6 +64,9 @@ EXPORT_SYMBOL_GPL(nr_swap_pages); long total_swap_pages; static int least_priority = -1; static unsigned long swapfile_maximum_size; +#ifdef CONFIG_MIGRATION +bool swap_migration_ad_supported; +#endif /* CONFIG_MIGRATION */ static const char Bad_file[] = "Bad swap file entry "; static const char Unused_file[] = "Unused swap file entry "; @@ -3685,6 +3688,11 @@ static int __init swapfile_init(void) swapfile_maximum_size = arch_max_swapfile_size(); +#ifdef CONFIG_MIGRATION + if (swapfile_maximum_size >= (1UL << SWP_MIG_TOTAL_BITS)) + swap_migration_ad_supported = true; +#endif /* CONFIG_MIGRATION */ + return 0; } subsys_initcall(swapfile_init);