From patchwork Thu Aug 25 10:10:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12954467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43DEEC28D13 for ; Thu, 25 Aug 2022 10:11:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B7836940009; Thu, 25 Aug 2022 06:11:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B27C7940007; Thu, 25 Aug 2022 06:11:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A25C940009; Thu, 25 Aug 2022 06:11:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 89067940007 for ; Thu, 25 Aug 2022 06:11:37 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 113C7C1A0C for ; Thu, 25 Aug 2022 10:11:37 +0000 (UTC) X-FDA: 79837698234.05.11E66DC Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf29.hostedemail.com (Postfix) with ESMTP id BA0F5120002 for ; Thu, 25 Aug 2022 10:11:36 +0000 (UTC) Received: by mail-pf1-f171.google.com with SMTP id w29so13687718pfj.3 for ; Thu, 25 Aug 2022 03:11:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=NKL5sVtokizfWAI3mo2jb9YTpdgBm9GDnhWm26Gpw5s=; b=Gy1ISArxPeO5t0wyjubpA3M0UwCqZnmexobJgdIoq7tqU9HMbig6Y+7mGWKf54ECC8 DqQndeO6hmcTQOcP0wGzgvV+4sXY89dYzb7/lyOeA8+OpBHUu8O3rCfoFzhAot3HDtVY Ce7vtNvX7ItLbUkB2MFUwWbkWA/FzUlnDgneBXcMJqO+AhBTgppDm+wxJv3E3ETTr4QA lFF/lELPdk9gCZYniMOeQsec8xH89bdgarwyYZHV7mZzwiPpvZzq/IjP1XQeYBvqQs7V 2fftWuXXkPuQNSe0e7Y7HoDIW6QdtAB1sOCVQ3StK0qJInlmGTpxoS+XYQRxpKtzCGj8 gCaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=NKL5sVtokizfWAI3mo2jb9YTpdgBm9GDnhWm26Gpw5s=; b=f0hTl641z9xQaRoDHsxngEVzT2qkfdzoZl8sotA16O0JPPGNsTlFMprBE3P9L/3/F5 Am6W2nOGC83Krfe9v5XoHp4htAhXROwrQyAdmeN7F+lBv2FHAiVpvliYWP7kUjEV3CJ1 tlkZUy/WF7g8vagXnmIyH8f6IsTW1vlguHAmKjE+CqrSghs0+PiLZXByW13+/un7JIbh NR+j0DJNHf1hJMiHN/kG9Bv+6xqxCFHvhFJI/7LIDcTnL5gsab4Z4jxTg2ZaHIxYFBwE ZP6EL8CnZWAb53LZYjDajgPW1dpxka0h52e4i2qIpHUe/StY2PQibekyzYfiuCxpZaDm BQqQ== X-Gm-Message-State: ACgBeo3tIHlrKYtqsEW859Ekn1a96OjzOn0SFBtQ8kMzjg1G3HjQr1ql iZ6XoH59K0FDthdg0vzhXfy73g== X-Google-Smtp-Source: AA6agR5IcwxykGGpHLVe3CX1H3gs3cvYf5zfNNUsTN/8NtEiGwMN/bJ7KCESq8870YRROxxIqH28lw== X-Received: by 2002:a05:6a00:e8a:b0:535:cc5c:3d87 with SMTP id bo10-20020a056a000e8a00b00535cc5c3d87mr3497198pfb.24.1661422295360; Thu, 25 Aug 2022 03:11:35 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.234]) by smtp.gmail.com with ESMTPSA id b10-20020a170903228a00b001714c36a6e7sm8477581plh.284.2022.08.25.03.11.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 03:11:34 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@redhat.com, kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com, jgg@nvidia.com, tglx@linutronix.de, willy@infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, muchun.song@linux.dev, Qi Zheng Subject: [RFC PATCH 1/7] mm: use ptep_clear() in non-present cases Date: Thu, 25 Aug 2022 18:10:31 +0800 Message-Id: <20220825101037.96517-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20220825101037.96517-1-zhengqi.arch@bytedance.com> References: <20220825101037.96517-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661422296; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NKL5sVtokizfWAI3mo2jb9YTpdgBm9GDnhWm26Gpw5s=; b=wKoc5yWcT9tKIg6Yr9GOx9Tr6jfnFcFNs7bjfqP2T1hpdJAqaQ35kgsCNxmG3MEeIKcLMW XeFNELV8iettrJgBybkcvT+xkKxf1Yomyzql6O2Pk+jaxCS/8F6uL2TaC4kcVn/rEp+4tO HKf0lKW8Y6QrjAtMD8dzp0sOgQn5o10= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=Gy1ISArx; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661422296; a=rsa-sha256; cv=none; b=1WWl09+VNbEQfX9Vf5GctxrDXukFtZf6pukrEVIQ295CXrU+uJ7WGcekSRPU6bmB3ewEhL HU3dAK5PwEUQ0J7H8xK79i6OaQKaU72+aRdfVmQFFYgc+/Z69UYUTjRtgvavnqLVmFiGLp l/lGes7S0mS+Ac2odLHU8kGP8CPZPhA= Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=Gy1ISArx; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BA0F5120002 X-Stat-Signature: bswgp9nxdb9rwn6pawujre3p6kzwqc3j X-Rspam-User: X-HE-Tag: 1661422296-588769 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After commit 08d5b29eac7d ("mm: ptep_clear() page table helper"), the ptep_clear() can be used to track the clearing of PTE entries, but it skips some places since the page table check does not care about non-present PTE entries. Subsequent patches need to use ptep_clear() to track all clearing PTE entries, so this patch makes ptep_clear() used for all cases including clearing non-present PTE entries. Signed-off-by: Qi Zheng --- include/linux/pgtable.h | 2 +- mm/memory.c | 2 +- mm/mprotect.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 3cdc16cfd867..9745684b0cdb 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -428,7 +428,7 @@ static inline void pte_clear_not_present_full(struct mm_struct *mm, pte_t *ptep, int full) { - pte_clear(mm, address, ptep); + ptep_clear(mm, address, ptep); } #endif diff --git a/mm/memory.c b/mm/memory.c index 1c6027adc542..207e0ee657e9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3655,7 +3655,7 @@ static vm_fault_t pte_marker_clear(struct vm_fault *vmf) * none pte. Otherwise it means the pte could have changed, so retry. */ if (is_pte_marker(*vmf->pte)) - pte_clear(vmf->vma->vm_mm, vmf->address, vmf->pte); + ptep_clear(vmf->vma->vm_mm, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); return 0; } diff --git a/mm/mprotect.c b/mm/mprotect.c index ba5592655ee3..1a01bd22a4ed 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -201,7 +201,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb, * fault will trigger without uffd trapping. */ if (uffd_wp_resolve) { - pte_clear(vma->vm_mm, addr, pte); + ptep_clear(vma->vm_mm, addr, pte); pages++; } continue; From patchwork Thu Aug 25 10:10:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12954468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00BC7C04AA5 for ; Thu, 25 Aug 2022 10:11:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 75A5494000B; Thu, 25 Aug 2022 06:11:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E1E4940007; Thu, 25 Aug 2022 06:11:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55BE094000B; Thu, 25 Aug 2022 06:11:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 43EFA940007 for ; Thu, 25 Aug 2022 06:11:43 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 197B3C19F6 for ; Thu, 25 Aug 2022 10:11:43 +0000 (UTC) X-FDA: 79837698486.30.E8550EF Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf19.hostedemail.com (Postfix) with ESMTP id ABE991A0007 for ; Thu, 25 Aug 2022 10:11:42 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id x63-20020a17090a6c4500b001fabbf8debfso4383693pjj.4 for ; Thu, 25 Aug 2022 03:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=SFdsSjWAQONAyj0B4BGWWGxwj5pOAu9AOlwRQ0TgC5A=; b=TctdZfIB9N0zQ09kkMlDWP+Kr7s/oP2QOFrEA8cG2q9SH3cDPtW2Xm2qtLCu0ZF1Ie 1cUD++3IFC9j8HSTyFV/pRd4U1vDay6piC5UoSjKLxTr6kc80Lf5dVy8/65A5EzAXWiF YMBmiu4+p+0A/8rJdD5MLJdpOay+sCeIwrpr6v7Su66828TBlx27Q2fM06Ep3JN4IfU9 0hmRqRattiBx+fp4aSQX6w7soFQ4ZKtwWmtlgH2Thaf0C9I0j2gDoTqFdYSpCFOPl9oE 6RdNvpO9I+k0yShSBe6PWhEeJBrKGzdMUel3hRhbBF7mg/X5dUX8vUVwYjWo8giRUEAl xdWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=SFdsSjWAQONAyj0B4BGWWGxwj5pOAu9AOlwRQ0TgC5A=; b=fu2iv5fdX8zt9d1hHu9qYSbcOjSMLuUwWNC4O9TCEr15q3/ZWpRO8A1YXO4kPxMcjI elJyY5R9FqDozrutOkKKprt9o8fX0X0upvgHd9LOPdngTdz+JColYWz4F6hCf9oivzHi 0tDHaa5slTqBkRyW6qOOxSrY9zfkh3irTjKNVoHHHUilPdn5nwfa4/kOC2CqQYh7B7Ru QRGDU+TvaFL2vxwNL7plc8F6c8iB/513YAWh9lrDoQNl9rPU0cFEscBUf+qw8x/5xPcp utPRc71rJVa6gh46AxupyYossZ3hULGSZPA9RNsOGPK1z9kKxfcXJ4yzUHLZo+4++4bv w7fg== X-Gm-Message-State: ACgBeo1ZGqiO50o/i8cLVO0laO3riZ6eQmg66y8B5FLRtL5sZebrhwf2 uQ8cBRnFrJ/pZIy5KtYkSyTcsA== X-Google-Smtp-Source: AA6agR4nupyddDVBBxbVHoklijRwE4BfvVlD41bJo7HEk5T2gjgiaqdFrpkQmsyV98KPMM8cpv+8MQ== X-Received: by 2002:a17:902:f641:b0:172:9642:1bf1 with SMTP id m1-20020a170902f64100b0017296421bf1mr3175044plg.36.1661422301791; Thu, 25 Aug 2022 03:11:41 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.234]) by smtp.gmail.com with ESMTPSA id b10-20020a170903228a00b001714c36a6e7sm8477581plh.284.2022.08.25.03.11.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 03:11:40 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@redhat.com, kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com, jgg@nvidia.com, tglx@linutronix.de, willy@infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, muchun.song@linux.dev, Qi Zheng Subject: [RFC PATCH 2/7] mm: introduce CONFIG_FREE_USER_PTE Date: Thu, 25 Aug 2022 18:10:32 +0800 Message-Id: <20220825101037.96517-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20220825101037.96517-1-zhengqi.arch@bytedance.com> References: <20220825101037.96517-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=TctdZfIB; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661422302; a=rsa-sha256; cv=none; b=q7AYHFi2mQHMl2uBL6NC/yfbnEeKFFzmRqFibZaoAIfpOxQxPny/wC12EXU9hTeIGZMtX5 3TVNkIEAcjrEz0uJisJN0reXXa8yFaRupmBtdpqII9MVYPRAVRTY0t80UBdslfMT5DxH1B ExRLUvBHG1cvqIjVHmRhtyMM0ZomDEo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661422302; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SFdsSjWAQONAyj0B4BGWWGxwj5pOAu9AOlwRQ0TgC5A=; b=2J1/r3fVNr9aGIbTRKEKL6/NmTYth270QoIct7z2HU8R0tb5/pNidoaSStOR7KbVyxtMhD HCE1WoaRxfs1UoKrOFKYHyI2Tjhws2uQ05YdTcbbBW2PSemmEf62vOJPmcBxL6uN1C76vf kH/W2TqhpQxRoJpMKSkDC6nt7D87BCU= X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ABE991A0007 X-Rspam-User: X-Stat-Signature: kftkqkchhybwdqx1eaim9i6fuwtremfd Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=TctdZfIB; spf=pass (imf19.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1661422302-881373 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This configuration variable will be used to build the code needed to free user PTE page table pages. The PTE page table setting and clearing functions(such as set_pte_at()) are in the architecture's files, and these functions will be hooked to implement FREE_USER_PTE, so the architecture support is needed. Signed-off-by: Qi Zheng --- mm/Kconfig | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 169e64192e48..d2a5a24cee2d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1130,6 +1130,17 @@ config PTE_MARKER_UFFD_WP purposes. It is required to enable userfaultfd write protection on file-backed memory types like shmem and hugetlbfs. +config ARCH_SUPPORTS_FREE_USER_PTE + def_bool n + +config FREE_USER_PTE + bool "Free user PTE page table pages" + default y + depends on ARCH_SUPPORTS_FREE_USER_PTE && MMU && SMP + help + Try to free user PTE page table page when its all entries are none or + mapped shared zero page. + source "mm/damon/Kconfig" endmenu From patchwork Thu Aug 25 10:10:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12954469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACC2AC3F6B0 for ; Thu, 25 Aug 2022 10:11:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C15E940008; Thu, 25 Aug 2022 06:11:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44998940007; Thu, 25 Aug 2022 06:11:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C3E2940008; Thu, 25 Aug 2022 06:11:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 19A64940007 for ; Thu, 25 Aug 2022 06:11:50 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E70EA1C733F for ; Thu, 25 Aug 2022 10:11:49 +0000 (UTC) X-FDA: 79837698738.05.26D9B91 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf02.hostedemail.com (Postfix) with ESMTP id AA03480007 for ; Thu, 25 Aug 2022 10:11:49 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id f21so19848109pjt.2 for ; Thu, 25 Aug 2022 03:11:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=5x32MEDCgaNmHcojCpTje6cvDrY80Yxg7sYtN2i0CQc=; b=Te/5IP11Lj2J2v23eu1UQFcoUwv4dykJGHh99WiFfsij/0Rki7IrrrZUo2uCyGFclm KhwIUXnn/hCKxxGZSDmvctNWutsnfG2jiFI6EDBMTtnQjsqUp8zty5btb0T2U1eMYaAw ZVGUNnbi1YLmhRT0hSrmnwApJbI2JEFFLm4Yz/v7Ft2oQvMGdrYASN7qLEC84irUpsWv 2BDeF6FZ7a0sXvUOV84T7gAfIJ4GiTQhwC0IvEgp5Rl+atGvYh3T4qDETv2PIwGprzon T94RtbPTcLwAJjd733YMYjkH9R1j/Sboj7Ln2UWI1Ee7eUSezRk7pfGK6VaMatgi1JCf DWLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=5x32MEDCgaNmHcojCpTje6cvDrY80Yxg7sYtN2i0CQc=; b=UG7/FJwELp5E/iNqr2ePnoGTpnJcFv48TGJ5ws1/Xj8vdJdgq3pe3axBun48pHunaL EZjJp7n5+BSwDcyGa5LtkpYcgzT68D9n8Y/Rn55tSRTneGZzJSlOTqc2nMj6vMw3/bZr l6yoRpOpXLcuajNcySxecyJbs9Hpku6wZt7eiWqFzr3OWDdnRx3sNE/4gsZZT+e7YuLE YOjX8Rvvbpuj7GXBC8bj4qS7RBwjHvKbS8C21vrtmG9/tjP8bDGSmfhBSeR10GHT2eh5 hp25HB7Aldy3DxZtqg0tZu/D3Vn3Htis7IgWRo1DYklP9ESdxeVO5nds+imdGabdLonE 0/xw== X-Gm-Message-State: ACgBeo3pBGQMUST29zSb9noaTLSvCRsHb5t60iCRz1cH580JnY7QwSMQ cU+PsdGDt4YPrYBNIE4Y0dv0VQ== X-Google-Smtp-Source: AA6agR7hp40eC3wRY8MGPaoNXwGLwbdIDjNQybY0ITEEfTLXjmMgba4xFQhcMKQZ1dlMtlSOnHbvOg== X-Received: by 2002:a17:902:d2c5:b0:172:8d5f:bf01 with SMTP id n5-20020a170902d2c500b001728d5fbf01mr3089264plc.119.1661422308812; Thu, 25 Aug 2022 03:11:48 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.234]) by smtp.gmail.com with ESMTPSA id b10-20020a170903228a00b001714c36a6e7sm8477581plh.284.2022.08.25.03.11.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 03:11:47 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@redhat.com, kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com, jgg@nvidia.com, tglx@linutronix.de, willy@infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, muchun.song@linux.dev, Qi Zheng Subject: [RFC PATCH 3/7] mm: add pte_to_page() helper Date: Thu, 25 Aug 2022 18:10:33 +0800 Message-Id: <20220825101037.96517-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20220825101037.96517-1-zhengqi.arch@bytedance.com> References: <20220825101037.96517-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="Te/5IP11"; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661422309; a=rsa-sha256; cv=none; b=nrryiY7rV6JaRz6YLRPBEoP4MpOJmpQrFFWeBeDJcWYzLSZ+T6eJvG28HzVnKwRCADd8y6 NKH81JNGIReDOsuuqi0yF4si3AGaPJ0/fcBtz+Ox1FtJezY8uMK0tUb1SLPS4UkOV5NGup fMnCZoFupYgCzo3ElU/YYBsTCr1k5GM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661422309; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5x32MEDCgaNmHcojCpTje6cvDrY80Yxg7sYtN2i0CQc=; b=0Q0eM4rdktBPfHPnVXUkg9B/fECazTGesxITZ3vXA+ZYf1C+CUB5exMmPYd6A/46GUS27B 52fERliFKnVnztzADO7hZAHo1Ybz8kJrLquJviteMJIwk3yINskoXm/E3fspki2aZOsGgg PUQmH8AntFQwAHm0mmzecPiVJy5jUkw= X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: AA03480007 X-Rspam-User: X-Stat-Signature: p1seekh4npfrncp9r8xqy8touiwu35o7 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="Te/5IP11"; spf=pass (imf02.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1661422309-723815 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add pte_to_page() helper similar to pmd_to_page(), which will be used to get the struct page of the PTE page table. Signed-off-by: Qi Zheng --- include/linux/pgtable.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 9745684b0cdb..c4a6bda6e965 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -86,6 +86,14 @@ static inline unsigned long pud_index(unsigned long address) #define pgd_index(a) (((a) >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1)) #endif +#ifdef CONFIG_FREE_USER_PTE +static inline struct page *pte_to_page(pte_t *pte) +{ + unsigned long mask = ~(PTRS_PER_PTE * sizeof(pte_t) - 1); + return virt_to_page((void *)((unsigned long) pte & mask)); +} +#endif + #ifndef pte_offset_kernel static inline pte_t *pte_offset_kernel(pmd_t *pmd, unsigned long address) { From patchwork Thu Aug 25 10:10:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12954470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2736C3F6B0 for ; Thu, 25 Aug 2022 10:11:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CF5A940009; Thu, 25 Aug 2022 06:11:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75735940007; Thu, 25 Aug 2022 06:11:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A9F1940009; Thu, 25 Aug 2022 06:11:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 447C1940007 for ; Thu, 25 Aug 2022 06:11:56 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1D0104199F for ; Thu, 25 Aug 2022 10:11:56 +0000 (UTC) X-FDA: 79837699032.14.6566412 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf28.hostedemail.com (Postfix) with ESMTP id B2E63C0005 for ; Thu, 25 Aug 2022 10:11:55 +0000 (UTC) Received: by mail-pf1-f172.google.com with SMTP id t129so538298pfb.6 for ; Thu, 25 Aug 2022 03:11:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=aSHAnJbXt+emb9j4QTyZL0s3Z7FutteIJpqm4S4DPUM=; b=RbsrREZnLtUzH8gMFFYqbdaGcqZdakHJIv4jEVHZfBUUk3tWVyBDnamcGPs1WsJ92r XJP90JRC4AVQlJZ50k+SQY6YhJNMEXbO+pw56gstS/zmz9D27iUjvirTz/+n1TCIeIqv Otm0knF+sTO56iugjSfAESn7+Zs+O3BSqFcgoaMGi/Y1s3ZE/gxsk61nXNShGQaAgkw6 T16FtOglAQKreSOxKFyKlyM69RkbmwiV69jFMM1ymowkokK5JY1s3UHXlEaOgDyC/wHz zi0Nrbo6R4Vx1XW5NoHASgr1T+r9Me0DG0TNIzP1G1c9yTqtjgzt3FeUuAV2M+8yAV6Y n8MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=aSHAnJbXt+emb9j4QTyZL0s3Z7FutteIJpqm4S4DPUM=; b=AUCrTeLSFeRgN9o41aGS6T8obH3CCjfewl/4dIdW3U+OqSZOFf7R0iMYFPEQ0cO/ts eLI2mjIUIsapjkwnej4aAVJaMvend7aKNPijAijVkGhsclAM9UVt3wIrwmwNsu0oMhfh HLa++9HvTTUYvQJ3OZyFwAKVDZDPNnigtNtkMLx2DFsFz6S4Abr1PtzeMTMtqOUKKhlv sfDL5iTxeqoIKm/UONt9Aj1MNOyzhFZt9eTzQ3WNiI9fPwM907e0Cz8H/r//xX4SYFBj qTfx2IKPewVjoSCx5fObP3dz8Mrv/rNSofRD9i1dYugI+Sx9WfNL+WP+5d62YkKHZqkC 1bNA== X-Gm-Message-State: ACgBeo3fIOxd4/Q6dBySqhBOhbBdVap8c+RQMKPBw/BEjeRqYp7ScG/+ 0NZm+4r2me/Njk73WqXKxOW2aELO7O5R6w== X-Google-Smtp-Source: AA6agR5AzI0RX0KK63LNqUrxwhd2qQ24X4dqPnkbn1vY7sUvc+FjOJ1dzhvq9jKl7nzUl8UApJFrjg== X-Received: by 2002:a05:6a00:1996:b0:52e:b0f7:8c83 with SMTP id d22-20020a056a00199600b0052eb0f78c83mr3235245pfl.59.1661422314575; Thu, 25 Aug 2022 03:11:54 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.234]) by smtp.gmail.com with ESMTPSA id b10-20020a170903228a00b001714c36a6e7sm8477581plh.284.2022.08.25.03.11.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 03:11:54 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@redhat.com, kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com, jgg@nvidia.com, tglx@linutronix.de, willy@infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, muchun.song@linux.dev, Qi Zheng Subject: [RFC PATCH 4/7] mm: introduce pte_refcount for user PTE page table page Date: Thu, 25 Aug 2022 18:10:34 +0800 Message-Id: <20220825101037.96517-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20220825101037.96517-1-zhengqi.arch@bytedance.com> References: <20220825101037.96517-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=RbsrREZn; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661422315; a=rsa-sha256; cv=none; b=lz7cf8wNIevM4kt+HtQZaVJ9hogGvODTEtD+uRVeNouR6VHhZjX3e5IeQSmmeDHukbrJVc l3pYQm6S9rNuwwsoYke8WiM6UIEaFCTLYC+qxVPfIajpQfCJpLyGwesoF1eMJimrluHQ+S 3QYfn0CZO1Y5C38gr2A8EqTdyMCOa8c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661422315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aSHAnJbXt+emb9j4QTyZL0s3Z7FutteIJpqm4S4DPUM=; b=h3NqlDLztLhjNWshxsyoYkingg3A/uPUNRTDFpyjb2xM7OrFDzoBxY8g/VbepQXTdFeLEw OzXmfegMiZ0FRA7OAa/sBFeZJxV+jXb1g8Bs5C08+6zV7Inwd0sqB93jrJfpm+QLun4snN ggmWTNNH224EKujBpE+SwHmSC4eW/oE= X-Rspam-User: X-Rspamd-Queue-Id: B2E63C0005 X-Stat-Signature: nycodawttkbypqgbzta4k9befqcwmfzb Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=RbsrREZn; spf=pass (imf28.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam01 X-HE-Tag: 1661422315-611136 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The following is the largest user PTE page table memory that can be allocated by a single user process in a 32-bit and a 64-bit system (assuming 4K page size). +---------------------------+--------+---------+ | | 32-bit | 64-bit | +===========================+========+=========+ | user PTE page table pages | 3 MiB | 512 GiB | +---------------------------+--------+---------+ | user PMD page table pages | 3 KiB | 1 GiB | +---------------------------+--------+---------+ (for 32-bit, take 3G user address space as an example; for 64-bit, take 48-bit address width as an example.) Today, 64-bit servers generally have only a few terabytes of physical memory, and mapping these memory does not require as many PTE page tables as above, but in some of the following scenarios, it is still possible to cause huge page table memory usage. 1. In order to pursue high performance, applications mostly use some high-performance user-mode memory allocators, such as jemalloc or tcmalloc. These memory allocators use madvise(MADV_DONTNEED or MADV_FREE) to release physical memory, but neither MADV_DONTNEED nor MADV_FREE will release page table memory, which may cause huge page table memory as follows: VIRT: 55t RES: 590g VmPTE: 110g In this case, most of the page table entries are empty. For such a PTE page where all entries are empty, we call it empty PTE page. 2. The shared zero page scenario mentioned by David Hildenbrand: Especially the shared zeropage is nasty, because there are sane use cases that can trigger it. Assume you have a VM (e.g., QEMU) that inflated the balloon to return free memory to the hypervisor. Simply migrating that VM will populate the shared zeropage to all inflated pages, because migration code ends up reading all VM memory. Similarly, the guest can just read that memory as well, for example, when the guest issues kdump itself. In this case, most of the page table entries are mapped to the shared zero page. For such a PTE page where all page table entries are mapped to zero pages, we call it zero PTE page. The page table entries for both types of PTE pages do not record "meaningful" information, so we can try to free these PTE pages at some point (such as when memory pressure is high) to reclaim more memory. To quickly identify these two types of pages, we have introduced a pte_refcount for each PTE page. We put the mapped and zero PTE entry counter into the pte_refcount of the PTE page. The bitmask has the following meaning: - bits 0-9 are mapped PTE entry count - bits 10-19 are zero PTE entry count Because the mapping and unmapping of PTE entries are under pte_lock, there is no concurrent thread to modify pte_refcount, so pte_refcount can be a non-atomic variable with little performance overhead. Signed-off-by: Qi Zheng --- include/linux/mm.h | 2 ++ include/linux/mm_types.h | 1 + include/linux/pte_ref.h | 23 +++++++++++++ mm/Makefile | 2 +- mm/pte_ref.c | 72 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 99 insertions(+), 1 deletion(-) create mode 100644 include/linux/pte_ref.h create mode 100644 mm/pte_ref.c diff --git a/include/linux/mm.h b/include/linux/mm.h index 7898e29bcfb5..23e2f1e75b4b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -28,6 +28,7 @@ #include #include #include +#include struct mempolicy; struct anon_vma; @@ -2336,6 +2337,7 @@ static inline bool pgtable_pte_page_ctor(struct page *page) return false; __SetPageTable(page); inc_lruvec_page_state(page, NR_PAGETABLE); + pte_ref_init(page); return true; } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c29ab4c0cd5c..da2738f87737 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -153,6 +153,7 @@ struct page { union { struct mm_struct *pt_mm; /* x86 pgds only */ atomic_t pt_frag_refcount; /* powerpc */ + unsigned long pte_refcount; /* only for PTE page */ }; #if ALLOC_SPLIT_PTLOCKS spinlock_t *ptl; diff --git a/include/linux/pte_ref.h b/include/linux/pte_ref.h new file mode 100644 index 000000000000..db14e03e1dff --- /dev/null +++ b/include/linux/pte_ref.h @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2022, ByteDance. All rights reserved. + * + * Author: Qi Zheng + */ + +#ifndef _LINUX_PTE_REF_H +#define _LINUX_PTE_REF_H + +#ifdef CONFIG_FREE_USER_PTE + +void pte_ref_init(pgtable_t pte); + +#else /* !CONFIG_FREE_USER_PTE */ + +static inline void pte_ref_init(pgtable_t pte) +{ +} + +#endif /* CONFIG_FREE_USER_PTE */ + +#endif /* _LINUX_PTE_REF_H */ diff --git a/mm/Makefile b/mm/Makefile index 6f9ffa968a1a..f8fa5078a13d 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -54,7 +54,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ mm_init.o percpu.o slab_common.o \ compaction.o vmacache.o \ interval_tree.o list_lru.o workingset.o \ - debug.o gup.o mmap_lock.o $(mmu-y) + debug.o gup.o mmap_lock.o $(mmu-y) pte_ref.o # Give 'page_alloc' its own module-parameter namespace page-alloc-y := page_alloc.o diff --git a/mm/pte_ref.c b/mm/pte_ref.c new file mode 100644 index 000000000000..12b27646e88c --- /dev/null +++ b/mm/pte_ref.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2022, ByteDance. All rights reserved. + * + * Author: Qi Zheng + */ +#include +#include + +#ifdef CONFIG_FREE_USER_PTE + +/* + * For a PTE page where all entries are empty, we call it empty PTE page. For a + * PTE page where all page table entries are mapped to zero pages, we call it + * zero PTE page. + * + * The page table entries for both types of PTE pages do not record "meaningful" + * information, so we can try to free these PTE pages at some point (such as + * when memory pressure is high) to reclaim more memory. + * + * We put the mapped and zero PTE entry counter into the pte_refcount of the + * PTE page. The bitmask has the following meaning: + * + * - bits 0-9 are mapped PTE entry count + * - bits 10-19 are zero PTE entry count + * + * Because the mapping and unmapping of PTE entries are under pte_lock, there is + * no concurrent thread to modify pte_refcount, so pte_refcount can be a + * non-atomic variable with little performance overhead. + */ +#define PTE_MAPPED_BITS 10 +#define PTE_ZERO_BITS 10 + +#define PTE_MAPPED_SHIFT 0 +#define PTE_ZERO_SHIFT (PTE_MAPPED_SHIFT + PTE_MAPPED_BITS) + +#define __PTE_REF_MASK(x) ((1UL << (x))-1) + +#define PTE_MAPPED_MASK (__PTE_REF_MASK(PTE_MAPPED_BITS) << PTE_MAPPED_SHIFT) +#define PTE_ZERO_MASK (__PTE_REF_MASK(PTE_ZERO_BITS) << PTE_ZERO_SHIFT) + +#define PTE_MAPPED_OFFSET (1UL << PTE_MAPPED_SHIFT) +#define PTE_ZERO_OFFSET (1UL << PTE_ZERO_SHIFT) + +static inline unsigned long pte_refcount(pgtable_t pte) +{ + return pte->pte_refcount; +} + +#define pte_mapped_count(pte) \ + ((pte_refcount(pte) & PTE_MAPPED_MASK) >> PTE_MAPPED_SHIFT) +#define pte_zero_count(pte) \ + ((pte_refcount(pte) & PTE_ZERO_MASK) >> PTE_ZERO_SHIFT) + +static __always_inline void pte_refcount_add(struct mm_struct *mm, + pgtable_t pte, int val) +{ + pte->pte_refcount += val; +} + +static __always_inline void pte_refcount_sub(struct mm_struct *mm, + pgtable_t pte, int val) +{ + pte->pte_refcount -= val; +} + +void pte_ref_init(pgtable_t pte) +{ + pte->pte_refcount = 0; +} + +#endif /* CONFIG_FREE_USER_PTE */ From patchwork Thu Aug 25 10:10:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12954471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BDF0C28D13 for ; Thu, 25 Aug 2022 10:12:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F211294000C; Thu, 25 Aug 2022 06:12:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAABF940007; Thu, 25 Aug 2022 06:12:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D24C194000C; Thu, 25 Aug 2022 06:12:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BC8B8940007 for ; Thu, 25 Aug 2022 06:12:01 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 989D3C199D for ; Thu, 25 Aug 2022 10:12:01 +0000 (UTC) X-FDA: 79837699242.06.2050CA6 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by imf06.hostedemail.com (Postfix) with ESMTP id 53DE218000C for ; Thu, 25 Aug 2022 10:12:01 +0000 (UTC) Received: by mail-pg1-f179.google.com with SMTP id w13so12378058pgq.7 for ; Thu, 25 Aug 2022 03:12:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=88xUUyVXHxDMCZmOYvZcYjojlwDW8mxu4+gwmvT3Uwg=; b=gCJc7t/irCxh/NsMtY1y28pn7+kS/+rCgGiEHuh81J6ucr5EX2vhMnTKm6bgUjyY0A FSH1ktN+iUPsTs7yW0vMLvS8VlTwmoaCjW/xDnuFk754nlVaaV5Sknqz7pgkUiF2NyNw kFNyD315ZQdYYefBhtxUO3hwVbrJh72K9xlvUHcJJytF8Hm+UucnXUn/bJV6GXT3KUbO qC74CEILcwNgX2/ksJAFI1VKjObcAa5bb61duB1Unbfwa4/HOAZINmkbhLrg/n2UTAM9 Y1LJVRSnkqYB4JH0aUKiKOhE31G/bmczwx7n5ekrUGCJhmRUyWj6K+A5mw3aTIe+aRlY 6TNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=88xUUyVXHxDMCZmOYvZcYjojlwDW8mxu4+gwmvT3Uwg=; b=SQBXlchEF9erhupyJYUf80lpYpnDMAVSsSgD3tz1GN8lUnRh4hsSex5x0CYvnWcgbI MhjMUbXrreNpy9DNOinrYxiaifZo1qlGYVsJiXCj9cwr632MKgTaUXQ+1m/VhBY0qyIB aaPhhpHLfqo5LdgzPWLP/cgrmdD6Vd0o9A0dMN4XzFkDDU5GaVw8KEmozcFchi3/UEsk 1JC8Perlx/GSePIWgtncr2azL0AY2yo1HdhVc5zGIMu3EQOnzaPzE07J6JqfstDdJ3OG c2kwk0qxMWV/2z7/PBknRFvS+Xp0j9jEgZCbr0Z8ePlYf2MwHZJIOwXyv9J36vEM8KeA wXRg== X-Gm-Message-State: ACgBeo26gCRo/eTCC65GK5CpStjoI1M/SeuHBRtX35eUfj9IANfoUsB8 RpBIxx8J0RsvpKqX63ja83897g== X-Google-Smtp-Source: AA6agR7wR0oMHsFEiShRabwQxkM0xbplFH+VDCYoO8SCvwJvs/eBFuYQvXDb00xbfLmujhgctEuB0w== X-Received: by 2002:a05:6a00:8c8:b0:52c:887d:fa25 with SMTP id s8-20020a056a0008c800b0052c887dfa25mr3524872pfu.86.1661422320411; Thu, 25 Aug 2022 03:12:00 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.234]) by smtp.gmail.com with ESMTPSA id b10-20020a170903228a00b001714c36a6e7sm8477581plh.284.2022.08.25.03.11.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 03:11:59 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@redhat.com, kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com, jgg@nvidia.com, tglx@linutronix.de, willy@infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, muchun.song@linux.dev, Qi Zheng Subject: [RFC PATCH 5/7] pte_ref: add track_pte_{set, clear}() helper Date: Thu, 25 Aug 2022 18:10:35 +0800 Message-Id: <20220825101037.96517-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20220825101037.96517-1-zhengqi.arch@bytedance.com> References: <20220825101037.96517-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661422321; a=rsa-sha256; cv=none; b=XQKDU5LrGuGvE5Z3nrmalfGMLFEhKs8Q2CewxsX63l6B3PISTcJa4cf51x8p9kWLJl8gx+ 4HWfLuzYJdC9SAsrt62QBUaRLs9eMj23fJw5t5kNm57xuI13AYDWz8+gHi6DPkk7i8Vnie sYAJaQdn2gqDoh0DGejgjkOHBeKCyw0= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="gCJc7t/i"; spf=pass (imf06.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661422321; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=88xUUyVXHxDMCZmOYvZcYjojlwDW8mxu4+gwmvT3Uwg=; b=8ewYyNPDlh8/8XhX7xosDO9tXsUGlKm7RCR9fYdI3u/qofRB8y6zUqtZcWoWpBUau7545K 3/TnJmJQcQLr3Sv4e21pdM0uvrnBgWA8exfW7G7F0rK8JI66ORQB2qQMSCtadpibZBHbK+ RZPmVMyM7SK2yA+8hO3RoFWBZLb6lyo= X-Stat-Signature: w1ta635pat8hrj7mgkioehjhftteq5ea X-Rspamd-Queue-Id: 53DE218000C X-Rspam-User: X-Rspamd-Server: rspam06 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="gCJc7t/i"; spf=pass (imf06.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1661422321-586658 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The track_pte_set() is used to track the setting of the PTE page table entry, and the track_pte_clear() is used to track the clearing of the PTE page table entry, we update the pte_refcount of the PTE page in these two functions. In this way, the usage of the PTE page table page can be tracked by its pte_refcount. Signed-off-by: Qi Zheng --- include/linux/pte_ref.h | 13 +++++++++++++ mm/pte_ref.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/include/linux/pte_ref.h b/include/linux/pte_ref.h index db14e03e1dff..ab49c7fac120 100644 --- a/include/linux/pte_ref.h +++ b/include/linux/pte_ref.h @@ -12,12 +12,25 @@ void pte_ref_init(pgtable_t pte); +void track_pte_set(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + pte_t pte); +void track_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + pte_t pte); #else /* !CONFIG_FREE_USER_PTE */ static inline void pte_ref_init(pgtable_t pte) { } +static inline void track_pte_set(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ +} + +static inline void track_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ +} #endif /* CONFIG_FREE_USER_PTE */ #endif /* _LINUX_PTE_REF_H */ diff --git a/mm/pte_ref.c b/mm/pte_ref.c index 12b27646e88c..818821d068af 100644 --- a/mm/pte_ref.c +++ b/mm/pte_ref.c @@ -69,4 +69,40 @@ void pte_ref_init(pgtable_t pte) pte->pte_refcount = 0; } +void track_pte_set(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + pte_t pte) +{ + pgtable_t page; + + if (&init_mm == mm || pte_huge(pte)) + return; + + page = pte_to_page(ptep); + if (pte_none(*ptep) && !pte_none(pte)) { + pte_refcount_add(mm, page, PTE_MAPPED_OFFSET); + if (is_zero_pfn(pte_pfn(pte))) + pte_refcount_add(mm, page, PTE_ZERO_OFFSET); + } else if (is_zero_pfn(pte_pfn(*ptep)) && !is_zero_pfn(pte_pfn(pte))) { + pte_refcount_sub(mm, page, PTE_ZERO_OFFSET); + } +} +EXPORT_SYMBOL(track_pte_set); + +void track_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + pte_t pte) +{ + pgtable_t page; + + if (&init_mm == mm || pte_huge(pte)) + return; + + page = pte_to_page(ptep); + if (!pte_none(pte)) { + pte_refcount_sub(mm, page, PTE_MAPPED_OFFSET); + if (is_zero_pfn(pte_pfn(pte))) + pte_refcount_sub(mm, page, PTE_ZERO_OFFSET); + } +} +EXPORT_SYMBOL(track_pte_clear); + #endif /* CONFIG_FREE_USER_PTE */ From patchwork Thu Aug 25 10:10:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12954472 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34034C28D13 for ; Thu, 25 Aug 2022 10:12:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3F51940008; Thu, 25 Aug 2022 06:12:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C839940007; Thu, 25 Aug 2022 06:12:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84218940008; Thu, 25 Aug 2022 06:12:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6F4C5940007 for ; Thu, 25 Aug 2022 06:12:08 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 498B141AD3 for ; Thu, 25 Aug 2022 10:12:08 +0000 (UTC) X-FDA: 79837699536.27.0208895 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf26.hostedemail.com (Postfix) with ESMTP id C5A22140016 for ; Thu, 25 Aug 2022 10:12:07 +0000 (UTC) Received: by mail-pl1-f177.google.com with SMTP id x23so18110093pll.7 for ; Thu, 25 Aug 2022 03:12:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=N/hACB6DJOnbW9YdcxCL4bYdw5iH8tr2HnsmRUbA/2k=; b=QK9DEIvE1XdTmyK0lBvfluXy8RYM/LAWM6lwGwePhGiIuALjqlWe153rtG2hWbimgP tADUbPJWllZwAGBl6q1qMoG7GiVV4Zxg5mCWYr5gpaptx+0cYenTa8MYu8BxylD1BYIR MuGpFgTWwlSreVJQ1YFHSoT6iayMMrkysAJwtTyc5sV4ffpifFreiRkvBdACM872Q6tU dkOoQSesVJHmEDw19sbAyC6q21uVp3rx9VXqWv5ZjPzjubyss70MeyGWM9AFYgnuU659 NiJrSv8GF1blOGvs/AogG2NrOnNaFsFxF+R6PRJfQ+I+YGspLrVxAViyyr0Bm4Nd080+ d75g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=N/hACB6DJOnbW9YdcxCL4bYdw5iH8tr2HnsmRUbA/2k=; b=zkuigoXTHd/0SyDt5mkpSmaGFK3Zr6Ymj4GRNZD115TsoH2EE52RfW1LnoH0vwz+cQ aE6+MGLGJDGWwBPQtb80++i6aALr8BwOLELy/vhXROqwvehoA3JFzNFtzdDlQkVESRXG gAR/W9IjdLoGAjXrsG33J2mowZwi/vsfeImft485KZyP7u+HfzpK1EKcC/0Yr4ZjZx32 lSp0UdgPn2cspUT2FRpu7qx33MkRhZ19CZJbBFzuyk+1Ka6aAK17VZELrcgVGGIwtse4 8e9M2PuDV6+WfdMAqCeXTKYLL8UstyttR7pbKp0fVIG26VHXqMbAeRF5w1buDsZ0bBgC M0ww== X-Gm-Message-State: ACgBeo2BI1XHb6qxU3E44t1PYmRN4oWv8ENfNV2rMNKeEawDVyrlNBBa wYiUxjSYfHZrfl3WVo8VQSLvHA== X-Google-Smtp-Source: AA6agR5cOXU69J240S/HoYf6HtB5GQPmumANKU1IMKISK6IYS4mPO26cn0vvNtm0snnl5pnh+tc46Q== X-Received: by 2002:a17:90b:2496:b0:1ef:a94:7048 with SMTP id nt22-20020a17090b249600b001ef0a947048mr3877401pjb.244.1661422326951; Thu, 25 Aug 2022 03:12:06 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.234]) by smtp.gmail.com with ESMTPSA id b10-20020a170903228a00b001714c36a6e7sm8477581plh.284.2022.08.25.03.12.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 03:12:06 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@redhat.com, kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com, jgg@nvidia.com, tglx@linutronix.de, willy@infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, muchun.song@linux.dev, Qi Zheng Subject: [RFC PATCH 6/7] x86/mm: add x86_64 support for pte_ref Date: Thu, 25 Aug 2022 18:10:36 +0800 Message-Id: <20220825101037.96517-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20220825101037.96517-1-zhengqi.arch@bytedance.com> References: <20220825101037.96517-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661422327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=N/hACB6DJOnbW9YdcxCL4bYdw5iH8tr2HnsmRUbA/2k=; b=eZdAf+eeA1Bso7AQ2e6Rz5uqBupBk7azeOrUSGeq83OMUwlBa9IQ6z23NZjPv1XxvjJ9rp CYYEeVyTdH0UzeJlrB/fBoRiyoJCNGbkniHdsKIg9k6LsgT59pciUsTHEkQMc0SfZJAMcR 5hSM1dEh4aahlYFKHr7qU0wX4Qm2lXI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=QK9DEIvE; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661422327; a=rsa-sha256; cv=none; b=GURRpId1gIAKGCJMePah/9RCwIRF7xDrprvC4QQjMGpdlvfzHRsKrxyRF2rAOEWZlLwBx+ HtnTr+MQJwiaOz/zpYBviA3k+fmGtMj4WVuSOm0wg23nW8bykaqX+B4iAQvYXR42apWNss Wddk4bSZEDxG2D2CmZsMXMSf47pNBJ0= Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=QK9DEIvE; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com X-Rspam-User: X-Rspamd-Queue-Id: C5A22140016 X-Rspamd-Server: rspam10 X-Stat-Signature: 3fhh9gggtw9bciud7rhrxrg9u6r8zbxq X-HE-Tag: 1661422327-899982 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add pte_ref hooks into routines that modify user PTE page tables, and select ARCH_SUPPORTS_FREE_USER_PTE, so that the pte_ref code can be compiled and worked on this architecture. Signed-off-by: Qi Zheng --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 4 ++++ include/linux/pgtable.h | 1 + 3 files changed, 6 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 52a7f91527fe..50215b05723e 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -34,6 +34,7 @@ config X86_64 select SWIOTLB select ARCH_HAS_ELFCORE_COMPAT select ZONE_DMA32 + select ARCH_SUPPORTS_FREE_USER_PTE config FORCE_DYNAMIC_FTRACE def_bool y diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 44e2d6f1dbaa..cbfcfa497fb9 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -23,6 +23,7 @@ #include #include #include +#include extern pgd_t early_top_pgt[PTRS_PER_PGD]; bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd); @@ -1005,6 +1006,7 @@ static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { page_table_check_pte_set(mm, addr, ptep, pte); + track_pte_set(mm, addr, ptep, pte); set_pte(ptep, pte); } @@ -1050,6 +1052,7 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, { pte_t pte = native_ptep_get_and_clear(ptep); page_table_check_pte_clear(mm, addr, pte); + track_pte_clear(mm, addr, ptep, pte); return pte; } @@ -1066,6 +1069,7 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, */ pte = native_local_ptep_get_and_clear(ptep); page_table_check_pte_clear(mm, addr, pte); + track_pte_clear(mm, addr, ptep, pte); } else { pte = ptep_get_and_clear(mm, addr, ptep); } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index c4a6bda6e965..908636f48c95 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -276,6 +276,7 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, pte_t pte = *ptep; pte_clear(mm, address, ptep); page_table_check_pte_clear(mm, address, pte); + track_pte_clear(mm, address, ptep, pte); return pte; } #endif From patchwork Thu Aug 25 10:10:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12954473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23D9CC04AA5 for ; Thu, 25 Aug 2022 10:12:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B02D5940009; Thu, 25 Aug 2022 06:12:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8B95940007; Thu, 25 Aug 2022 06:12:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 906B6940009; Thu, 25 Aug 2022 06:12:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7BAAC940007 for ; Thu, 25 Aug 2022 06:12:13 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5FC8941AD3 for ; Thu, 25 Aug 2022 10:12:13 +0000 (UTC) X-FDA: 79837699746.23.15B1CEB Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by imf08.hostedemail.com (Postfix) with ESMTP id 0D54D160006 for ; Thu, 25 Aug 2022 10:12:12 +0000 (UTC) Received: by mail-pg1-f178.google.com with SMTP id s206so17470461pgs.3 for ; Thu, 25 Aug 2022 03:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=4+N7HYlfplazgpIqzrok71tJ5+HL03OgoPpZtaJJ/JA=; b=Bu7SsRr4s/1augCft2+vJK75ZrctEK6Iyfqu3Pqt47wpkqwjMjUrUC1JkgguNrwL13 2hwv5VLcCfH+1TdJctlmV4tEW0D5p2uuFLXfW83lWXUXrRqn7wVkKO6VfJ+TG0znQv03 iVFjsT+BlKjd/AFLzEoJHLCxK2LgbnhjLHjQ2I0kzLOzbt/5nN04MawaWeD4I0Qm64w2 E9MSbNuPJ3pLKOYKGmuvPUahHRz+8UnEMIUSOM24Pd3OQIg9OBXQ7WM9H2O2mD3CUq/L a6iPYt/Xvmok3ln2e30SBo3/3tTpT4vmSws68CmrOOrJk1agvp1haB8koBrCgA4j8OJl 16qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=4+N7HYlfplazgpIqzrok71tJ5+HL03OgoPpZtaJJ/JA=; b=0WHpr7DBlhhQwiMqbY05F39XWxuyr3QAnBsTWSkgfL5TqivjZ0tATyjy9I4vTTnk7H 9y0L42J4o13liwMThaiiLFZDHEGq6Y4OsXjyLyi+oWimHlR+2xrf4J1hGGMafYGM5Uhs Fjy2aIMLemzKn97KKSRnq7wUj5/YfDFsMtDmhfqwSkBs4todv82eKrucH5ZdaVppEssO g82puwDV0DwTNyUcqooU1HYWAI/ERqaTDhkpw6Ey+l+GxicrtwznBA0HMGPiM2lOL2+O Ph8HngnJnvM0I8PrMNGTgq1UqVEvhJgmvF5Z0F/RcJiDhhPe1AsyEEj/YGW+xzR+Mj2h u6dg== X-Gm-Message-State: ACgBeo3f91Y8Sn/PU/TBlmtK8snkyYhDMPTpU5VhVvZPKW+cQ9h3F+sU EFbzbqsafy9M2NerEb9xS8t0MA== X-Google-Smtp-Source: AA6agR4E8A9z3wJppoQNhY57GIv03JlBORZW3VAf7AyAGpWyEGHzTY8vAaMpe8BdYq7iY2OzpCD70Q== X-Received: by 2002:a63:e851:0:b0:42a:3bc0:9ad1 with SMTP id a17-20020a63e851000000b0042a3bc09ad1mr2612576pgk.543.1661422332147; Thu, 25 Aug 2022 03:12:12 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.234]) by smtp.gmail.com with ESMTPSA id b10-20020a170903228a00b001714c36a6e7sm8477581plh.284.2022.08.25.03.12.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Aug 2022 03:12:11 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, david@redhat.com, kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com, jgg@nvidia.com, tglx@linutronix.de, willy@infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, muchun.song@linux.dev, Qi Zheng Subject: [RFC PATCH 7/7] mm: add proc interface to free user PTE page table pages Date: Thu, 25 Aug 2022 18:10:37 +0800 Message-Id: <20220825101037.96517-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20220825101037.96517-1-zhengqi.arch@bytedance.com> References: <20220825101037.96517-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661422333; a=rsa-sha256; cv=none; b=nDk247WR8Iw6akIZvQvYnQFicHii/5jbtxRAxB1R52wWbnVp3bvZOpU2gCqJe3L/hRdaI/ 3qHJc/eCWtt+X3YoeYTXLyano/E2i3bm+v7bE9MzhgnFdvh7SePOVTSH729Tl01bD6jRG6 vr/jeaEgmEIaJnoLEaavrIuIYlTy7xk= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=Bu7SsRr4; spf=pass (imf08.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661422333; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4+N7HYlfplazgpIqzrok71tJ5+HL03OgoPpZtaJJ/JA=; b=DLFqjquVnu/pSfjRE7bm95+cAn7RIDHjf1rubCSXb+gb/ix0mqbFLmjoTSxy9aBSePG5T4 KI1VfIVeXCNlDjeijDEqedgJOUQ+StAsrYTut8dCeR4pDAKLI/cshrd3DKHLQdLKdABiYP TIP95OtMBSsQtDIhVxeF7ExTPHR4xCo= X-Stat-Signature: 74zanx11gidw8uwapbtqy4j8gooa81it X-Rspamd-Queue-Id: 0D54D160006 X-Rspam-User: X-Rspamd-Server: rspam06 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=Bu7SsRr4; spf=pass (imf08.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1661422332-908826 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add /proc/sys/vm/free_ptes file to procfs, when pid is written to the file, we will traverse its process address space, find and free empty PTE pages or zero PTE pages. Signed-off-by: Qi Zheng --- include/linux/pte_ref.h | 5 ++ kernel/sysctl.c | 12 ++++ mm/pte_ref.c | 126 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 143 insertions(+) diff --git a/include/linux/pte_ref.h b/include/linux/pte_ref.h index ab49c7fac120..f7e244129291 100644 --- a/include/linux/pte_ref.h +++ b/include/linux/pte_ref.h @@ -16,6 +16,11 @@ void track_pte_set(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); void track_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); + +int free_ptes_sysctl_handler(struct ctl_table *table, int write, + void *buffer, size_t *length, loff_t *ppos); +extern int sysctl_free_ptes_pid; + #else /* !CONFIG_FREE_USER_PTE */ static inline void pte_ref_init(pgtable_t pte) diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 35d034219513..14e1a9841cb8 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -64,6 +64,7 @@ #include #include #include +#include #include "../lib/kstrtox.h" @@ -2153,6 +2154,17 @@ static struct ctl_table vm_table[] = { .extra1 = SYSCTL_ONE, .extra2 = SYSCTL_FOUR, }, +#ifdef CONFIG_FREE_USER_PTE + { + .procname = "free_ptes", + .data = &sysctl_free_ptes_pid, + .maxlen = sizeof(int), + .mode = 0200, + .proc_handler = free_ptes_sysctl_handler, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_INT_MAX, + }, +#endif #ifdef CONFIG_COMPACTION { .procname = "compact_memory", diff --git a/mm/pte_ref.c b/mm/pte_ref.c index 818821d068af..e7080a3100a6 100644 --- a/mm/pte_ref.c +++ b/mm/pte_ref.c @@ -6,6 +6,14 @@ */ #include #include +#include +#include +#include +#include +#include +#include + +#include "internal.h" #ifdef CONFIG_FREE_USER_PTE @@ -105,4 +113,122 @@ void track_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, } EXPORT_SYMBOL(track_pte_clear); +#ifdef CONFIG_DEBUG_VM +void pte_free_debug(pmd_t pmd) +{ + pte_t *ptep = (pte_t *)pmd_page_vaddr(pmd); + int i = 0; + + for (i = 0; i < PTRS_PER_PTE; i++, ptep++) { + pte_t pte = *ptep; + BUG_ON(!(pte_none(pte) || is_zero_pfn(pte_pfn(pte)))); + } +} +#else +static inline void pte_free_debug(pmd_t pmd) +{ +} +#endif + + +static int kfreeptd_pmd_entry(pmd_t *pmd, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + pmd_t pmdval; + pgtable_t page; + struct mm_struct *mm = walk->mm; + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + spinlock_t *ptl; + bool free = false; + unsigned long haddr = addr & PMD_MASK; + + if (pmd_trans_unstable(pmd)) + goto out; + + mmap_read_unlock(mm); + mmap_write_lock(mm); + + if (mm_find_pmd(mm, addr) != pmd) + goto unlock_out; + + ptl = pmd_lock(mm, pmd); + pmdval = *pmd; + if (pmd_none(pmdval) || pmd_leaf(pmdval)) { + spin_unlock(ptl); + goto unlock_out; + } + page = pmd_pgtable(pmdval); + if (!pte_mapped_count(page) || pte_zero_count(page) == PTRS_PER_PTE) { + pmd_clear(pmd); + flush_tlb_range(&vma, haddr, haddr + PMD_SIZE); + free = true; + } + spin_unlock(ptl); + +unlock_out: + mmap_write_unlock(mm); + mmap_read_lock(mm); + + if (free) { + pte_free_debug(pmdval); + mm_dec_nr_ptes(mm); + pgtable_pte_page_dtor(page); + __free_page(page); + } + +out: + cond_resched(); + return 0; +} + +static const struct mm_walk_ops kfreeptd_walk_ops = { + .pmd_entry = kfreeptd_pmd_entry, +}; + +int sysctl_free_ptes_pid; +int free_ptes_sysctl_handler(struct ctl_table *table, int write, + void *buffer, size_t *length, loff_t *ppos) +{ + int ret; + + ret = proc_dointvec_minmax(table, write, buffer, length, ppos); + if (ret) + return ret; + if (write) { + struct task_struct *task; + struct mm_struct *mm; + + rcu_read_lock(); + task = find_task_by_vpid(sysctl_free_ptes_pid); + if (!task) { + rcu_read_unlock(); + return -ESRCH; + } + mm = get_task_mm(task); + rcu_read_unlock(); + + if (!mm) { + mmput(mm); + return -ESRCH; + } + + do { + ret = -EBUSY; + + if (mmap_read_trylock(mm)) { + ret = walk_page_range(mm, FIRST_USER_ADDRESS, + ULONG_MAX, + &kfreeptd_walk_ops, NULL); + + mmap_read_unlock(mm); + } + + cond_resched(); + } while (ret == -EAGAIN); + + mmput(mm); + } + return ret; +} + #endif /* CONFIG_FREE_USER_PTE */