From patchwork Mon Jan 13 03:38:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13936678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98296E77188 for ; Mon, 13 Jan 2025 03:39:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3377D6B0085; Sun, 12 Jan 2025 22:39:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BF7A6B0088; Sun, 12 Jan 2025 22:39:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1119B6B0089; Sun, 12 Jan 2025 22:39:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E46C26B0085 for ; Sun, 12 Jan 2025 22:39:28 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8B37181A85 for ; Mon, 13 Jan 2025 03:39:28 +0000 (UTC) X-FDA: 83001023616.13.1B14B2A Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf05.hostedemail.com (Postfix) with ESMTP id B02F3100010 for ; Mon, 13 Jan 2025 03:39:26 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=drXssHAZ; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736739566; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IPcqXqaTIBo3x10L2Zu0olCr3NMK/mAmrBeyqDgRKcc=; b=46D4oal8mFPlkoWVhiKKKEAsT5bOWVsfEegbHcgtxRtOvVU1pjqT4W8Z6cInzhtrBYTQFl 2QHKUhBn/V09Xc2na9nyBzyX/r3O9RWfjq2t197AmQDvxI2uDMUXQNZSRkhbax5mQJqGVk FyVbdrPjfa4JhSKTKyPfeuF0k1Vr9Z0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736739566; a=rsa-sha256; cv=none; b=rwnsts0PMKY1SFoW/uJelse9TiPAwq8J77qB1NNf+TP8ycPBG66B7+sbPFdQNxyNgEyqW5 KiZHEF7I9H3ZWdxi8sWM+dW5BlWVaOHrsKDRk6W0CwpHetqgvvisWFZCrdgAT2FJSsAjF9 n0suM7QdImN07D/Lg2aeYskWtLnvILI= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=drXssHAZ; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-21649a7bcdcso62730135ad.1 for ; Sun, 12 Jan 2025 19:39:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736739565; x=1737344365; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IPcqXqaTIBo3x10L2Zu0olCr3NMK/mAmrBeyqDgRKcc=; b=drXssHAZUruRYspEs09d+T/MqsaMWjxMdTNLICgvVUYoUNIsfri65JuF3kVg2odvPR lenbBTwsNNtuUckQVMgWWwOH/wo0y4VA/AsA1UAM3qGISsQ7EEi4CdsN2ob61lnn+bBP 6q0v98wtyaV2yd8FjSQ2YJyvC5d7ka+FsWMqriF6OgXeKSvsazbHMgvLuKgrekJV115S QdHiwNucRn4pQI7s+3/IdJNZzpTMyYcqwn60WtroIEii5JOaEiAHZTIxTCubfcWaM6sK u/UCZ5eG1RuaOrmkt2EKb8aJTBuoECoMOOcji6DMm13QlFfWprQuhLF6US+zAxIDrKZT JuRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736739565; x=1737344365; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IPcqXqaTIBo3x10L2Zu0olCr3NMK/mAmrBeyqDgRKcc=; b=Sbqmw9K5IIM2MbAQhDRZ6DSDjwHdP0/uBubczRiA1KR9t79bXOuVJGx1gX2bp7CIon HSF73ZC9tT8Hfgjx4JLpb7ues4wM8WvE3Xwm6BM++2XmEQHMvrbwLRgri9FXYQu7RcPX 6d/3gV7E9F1nqwgQolkd32KVQ6VHE2rgI5jfvDlgBucMrduEo0mOcJ9PmxXjauFbjUfu +IhjUR2INE0kUUuaahYS1VwBnTRi1xbjniPelWMboJf2Nh/B7asAIIbsKFPZvL9UTDsc p0cdtqa0jnxR0IEwlpDE6FHRv0SxFGJ/ML7Mo30iCL6X21DEWa8RhqtbhYEW3dgYfQgP SzpQ== X-Forwarded-Encrypted: i=1; AJvYcCXY2mkb0N9m4zPXVdkzZUz3vjyEYhMIQuUTIpKm5fUYFz68R86iUCjooDfSjOV/AgOsc4Iuleqr9Q==@kvack.org X-Gm-Message-State: AOJu0YyPYBAZUJ2Z0N34MC7XEmWdxz7YQbiarn4k9MaJuH28X5nAOsRL yeYv8AJPhAYR5ZLuvAR8sSmiUoe2UnHDPCZyW+qPq/LJxbeqT7fO X-Gm-Gg: ASbGncvO99bYdiI3S1MP2XsB7Fht4Te/I2/JdQr0QVThnaAaAWFxRTjJjSztx1xLIdN Nx70EXPmJXE5ObH8n7B/D4eOvnzP72Y0Ay96M1Hr3T9FdK+OLzz9Ypj4QnWlS4tFrN9NmgkQ+xM 06REL65gKRba5ibSGxZBoAj3inEukhqPbyrXbvRXxvdMzvNDLgwR0Bb+gNDiaq0cYw80Bx5hUqA CSgUAcgf4tmbjrAonH/YCeaAUD+TpasOadk9WjPkEHrDOUsRjktTtl4Ey3/1pFYgYpaKdVgc53m Xoj7eqGK X-Google-Smtp-Source: AGHT+IHhm0qTbSZrWVTQXUqJ8k0+E+zfqHExh6ZNZDkVUuwdyBd3fwtM3FFGJhBiv45zTGk0mUeVsQ== X-Received: by 2002:a17:902:d2ca:b0:211:8404:a957 with SMTP id d9443c01a7336-21a83fc0619mr292344105ad.41.1736739565411; Sun, 12 Jan 2025 19:39:25 -0800 (PST) Received: from Barrys-MBP.hub ([2407:7000:af65:8200:39b5:3f0b:acf3:9158]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f25aabfsm44368405ad.246.2025.01.12.19.39.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 12 Jan 2025 19:39:25 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, ioworker0@gmail.com, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, v-songbaohua@oppo.com, x86@kernel.org, linux-riscv@lists.infradead.org, ying.huang@intel.com, zhengtangquan@oppo.com, lorenzo.stoakes@oracle.com Subject: [PATCH v2 1/4] mm: Set folio swapbacked iff folios are dirty in try_to_unmap_one Date: Mon, 13 Jan 2025 16:38:58 +1300 Message-Id: <20250113033901.68951-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250113033901.68951-1-21cnbao@gmail.com> References: <20250113033901.68951-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B02F3100010 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: jakf6wyo6rqwggi38rxu5yctnji5m3fh X-HE-Tag: 1736739566-644248 X-HE-Meta: U2FsdGVkX1+lUCNXDuARFIYSiOnMgUqPpkp/lmSEXQaB7KtzXG3OoMUBUn5BBPSjg1+xEw0tDfIo3GNC8249eq6NfZ/gkr/2WWkAn4n4+Up11WyHGwWQm3kTYdy5cSedB1eG/HwRE29mUiirBcvthNRbTeUfPyQXMQbgvGkLF+Qeo4Qkb6eeT4xHZRPqV8Qo7SqsdfhBFCh1cmvMLcChKIk0iw3mNintFjBx89dQ7Hn1kvgH4gzmQ0mb58xw1d+OPSAzNiD/QlU0pIoj6GCLfvbzuVOiyDJjJN3uLBT4M4ET/KpY5Cc9TOaGczRGawG1wrAU+SD3eNyKm+ZAm4ePurBRWRi1gV29XFZfnAS4lYzINSG68fr77i+DSkZI9aIgt93M97s3S+D5K3u8D4gRLHO5UWcIX+ubmcFzTR4KwTyR+wCfUlgExnQFtpfRBW0gA5GZgu6fQr/Bzl+H+7+LhV1a+lX4YxiA8INronrzCxHkPIMGjJ1nwjjRTfo2InqGAC4mkPyMyI/kQt6KYGS+G4/DmcRjLWdklEY9v4RwzhLss/4jsM0Sc/pgv77VCSq/V7KaR+UoYvMUIAGHYdRgEV7iG3G/r1LSmKchF44or3EF2kdiNzDztORIZbg1ycEJqzeo9NHJOqC+vjVSX15vCjVBi806xP7qTeO4HyFnf+Ktt/sGj4XkeMeGnuM1t3FufJUbXE0NjjYnhRxldd9nXMpzmvK+iQFaZUeJaHMepyVrz1tu1S/ilgOz0mWNoghcg1FqR7sbXe9if2JQk/s5brgNHNV8EcA58RmoqwPsO225D8dpy1rfa6YfCcOhx2Lme7wJWx2pPKrt6p4GY7p1IUZzoDBRL9fwu85avKZbBoXKgxnsrzvngu6EVPDHzqz6roUFLVAd5c1S3p9XWXYM1hdgA+3SQVAMu4t7wdV3lNMR/b7edXsbqWzZiPtqP2nvuJXiqDL2A+Dbt6HRDbJ Z2Au/atM wLqDyS70hCC7cCUUHqMNOgqcAP8IFuPlTr29nPFNWO1PFvreS0EaZ82FC5rTfLJKNbkBXVul3xVtLsXP6fqzBLw7qQ2P5z6s0VAIFOP5IHRspCA8CTv7QJGIE/vAFNchvjJOdF9Z2EkWSEVh8I68nwWohYMVC5caj5DmEOVsz1q5iZb9P2z4fc4ORlpeUaSa0UI9JG0re4f2aa0verpYC/IPpOZl61elFqxTeTH1b+NzwlSs/sFeIzqm+QhQrfgKw0UMYRAqNV0uUQ+LdIRJI3BLLx0K7qre9VpT1SPEMXp+c0AeT8gK9SE6LEZNGHTV9Ip2MUKMPbV0H+L8LTiHadwZdcG1NJriRfiV0ms+a1lSo7o9auSMT+qtqQV00o3O+Fe+r64qc2hWgxbnhIk/tYO+HwiKWmpK0Tn2/YrERAU2tRUFHxfytkjGxBPPKUroQ+729iMQD/SkIDNk2NLSIme+Li71quisI7q8Gm3WfJjv5f+M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song The refcount may be temporarily or long-term increased, but this does not change the fundamental nature of the folio already being lazy- freed. Therefore, we only reset 'swapbacked' when we are certain the folio is dirty and not droppable. Suggested-by: David Hildenbrand Signed-off-by: Barry Song --- mm/rmap.c | 49 ++++++++++++++++++++++--------------------------- 1 file changed, 22 insertions(+), 27 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index c6c4d4ea29a7..de6b8c34e98c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1868,34 +1868,29 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ smp_rmb(); - /* - * The only page refs must be one from isolation - * plus the rmap(s) (dropped by discard:). - */ - if (ref_count == 1 + map_count && - (!folio_test_dirty(folio) || - /* - * Unlike MADV_FREE mappings, VM_DROPPABLE - * ones can be dropped even if they've - * been dirtied. - */ - (vma->vm_flags & VM_DROPPABLE))) { - dec_mm_counter(mm, MM_ANONPAGES); - goto discard; - } - - /* - * If the folio was redirtied, it cannot be - * discarded. Remap the page to page table. - */ - set_pte_at(mm, address, pvmw.pte, pteval); - /* - * Unlike MADV_FREE mappings, VM_DROPPABLE ones - * never get swap backed on failure to drop. - */ - if (!(vma->vm_flags & VM_DROPPABLE)) + if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) { + /* + * redirtied either using the page table or a previously + * obtained GUP reference. + */ + set_pte_at(mm, address, pvmw.pte, pteval); folio_set_swapbacked(folio); - goto walk_abort; + goto walk_abort; + } else if (ref_count != 1 + map_count) { + /* + * Additional reference. Could be a GUP reference or any + * speculative reference. GUP users must mark the folio + * dirty if there was a modification. This folio cannot be + * reclaimed right now either way, so act just like nothing + * happened. + * We'll come back here later and detect if the folio was + * dirtied when the additional reference is gone. + */ + set_pte_at(mm, address, pvmw.pte, pteval); + goto walk_abort; + } + dec_mm_counter(mm, MM_ANONPAGES); + goto discard; } if (swap_duplicate(entry) < 0) { From patchwork Mon Jan 13 03:38:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13936679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D04F9E77188 for ; Mon, 13 Jan 2025 03:39:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 643C36B0089; Sun, 12 Jan 2025 22:39:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F29F6B008A; Sun, 12 Jan 2025 22:39:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41EE66B008C; Sun, 12 Jan 2025 22:39:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1D9DD6B0089 for ; Sun, 12 Jan 2025 22:39:42 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9BF9EA1B06 for ; Mon, 13 Jan 2025 03:39:41 +0000 (UTC) X-FDA: 83001024162.16.FC236A4 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf20.hostedemail.com (Postfix) with ESMTP id AECCA1C0007 for ; Mon, 13 Jan 2025 03:39:39 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=L6LkcFsO; spf=pass (imf20.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736739579; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eULRCDFJ0DJzA6lc7vvdVA8dxS/L+jVPc2aRWImXMe0=; b=paweJfwFXUxSg62cfi8WMDGrM1zBHj3vyfCCSYgdoWJwAEEKrMeRHUXFyhb9vGigNfVt86 w6RgLJRhWln1Cm0yAOh4ZtClo4MW69kiPUNlVgS+uwH25ra2LenidsLDPwJa+fBSmK+3fc UZ8hiXis2X5ax+hGjOFCCO41oq0288k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736739579; a=rsa-sha256; cv=none; b=vNLZVwKgZYCR8lgo5FYhI0pQyjhkfiljDew0UwdvytsF1L3DdqAYKLNk7aGZyniioiHyUw sGOzZMf4pmPchFpihsPJSYgVVy6TIcjgbqmgp8d4/gx6onMjt63u3DoyoNzw//12B583BV zrp686kr+t1+A7D5U3ZG3ePduhvtLYE= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=L6LkcFsO; spf=pass (imf20.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-219f8263ae0so60424195ad.0 for ; Sun, 12 Jan 2025 19:39:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736739578; x=1737344378; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eULRCDFJ0DJzA6lc7vvdVA8dxS/L+jVPc2aRWImXMe0=; b=L6LkcFsO5hCDRfzBoK8nxSUE+1dCXl4DLO5SpyjVdNDONurAc4IE8//ckYs5FgqiFf LWH0HNHnXNmvOUOuUeyM6KY1Y0oHbLq5xrtzEkZoi4jGj8r8OCYDs/o/yloeK5y2rplY n/62WUBBvp1m/ntPglnNgcgdM7OGjsjC0OOUvOJkttURhErPlej4tGg4dAzj0QhLXLVa 0gQB0BSAS7yHQWJkpQNampHWBRksC+g33s0DDA8zj0P00avaJ2aMetnxdujR4FyFQCJo JbbKOAeFUdd2PVLFN/GtqJ0LAw2wlEuT20agwOgIeFnlGgp6pNRdowGihWqN+bhWcJvJ L3QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736739578; x=1737344378; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eULRCDFJ0DJzA6lc7vvdVA8dxS/L+jVPc2aRWImXMe0=; b=G8ACp6zDmRrUukEqv6RnwGGUvgQYD9nEiqhow1p36uTCLorqqy3ViEGAqXNA/z++vY xE17EFQ78VXVvlthv7fDjZ0xTVIgH6ZHHhbxx8Zuo9hJNirT0Qlz32qCGoq/ugKNuIrc AlphinNTnHzSWhD7erhrWQXCXhrsSqrP2rqMOXY9OC9TBWYGgh2x6VFQ6Lgq0mah2C8a Pa8EDoRk1Ll9oXedGxGHP0wE4PKV23jmU6IRQs5+VzN1LgoladxlZSJUeNZ1odQgGl3r jBf/NLDUgh/FCyyAD0TE5K0I9BnEtfI+FoAwvkAmPQp4rV2ctqzMV0LZzKbYBr3cx8+q ZQxQ== X-Forwarded-Encrypted: i=1; AJvYcCWmniv4fG+u6uFpKhGiO1kn6pWBZFrskPDLdjnaj3UugRK0KNezbrwP1ovMj1BFS8JA1OlRLoQFUg==@kvack.org X-Gm-Message-State: AOJu0YzWYcW2errvxhI3sZ2fxkmkuHTTmMofz/ySgFbhDKReakbK9h+l /AckQDeMU3KkUcKPs2wHX+63we0M3iEY5Y1ofay3suNczwOJztql X-Gm-Gg: ASbGncvp7aUTx+YlMdIa6oVukw4RDeZCm7tzCdnmiJn3wbQJF28tKGJDiQT0jAkb7oj K77T1X3OP3urHw0gILoNk69mW49rD+4z1sW/Ps+LFjKsjxJYHDtn/V8lqwMHRg1Yq1i2pU8Ik+Z PENX+i0hARnK1vwlAS76y+Dp/TC6I86jctF1M87acIDzjqGx719Z0WWGawB9G1SaEzSfU6pLaN4 rJruVVRIaVKe/vMBLA4fw/wzHfc/xLOIW+51ib1uzWzmCSmxgDwgR888NKmsW/nY5e2r8nXBH8o k69H6eea X-Google-Smtp-Source: AGHT+IGyqwjrrsZeQxoPWWpl+Lu4WtU0rXovoTE/PMvKn9xWPcPyfiEtn9aeXvMxYF6gWs977A6wvQ== X-Received: by 2002:a17:902:c951:b0:215:bc30:c952 with SMTP id d9443c01a7336-21a83f4298fmr238697265ad.6.1736739578393; Sun, 12 Jan 2025 19:39:38 -0800 (PST) Received: from Barrys-MBP.hub ([2407:7000:af65:8200:39b5:3f0b:acf3:9158]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f25aabfsm44368405ad.246.2025.01.12.19.39.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 12 Jan 2025 19:39:38 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, ioworker0@gmail.com, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, v-songbaohua@oppo.com, x86@kernel.org, linux-riscv@lists.infradead.org, ying.huang@intel.com, zhengtangquan@oppo.com, lorenzo.stoakes@oracle.com, Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Anshuman Khandual , Shaoqin Huang , Gavin Shan , Kefeng Wang , Mark Rutland , "Kirill A. Shutemov" , Yosry Ahmed , Paul Walmsley , Palmer Dabbelt , Albert Ou , Yicong Yang Subject: [PATCH v2 2/4] mm: Support tlbbatch flush for a range of PTEs Date: Mon, 13 Jan 2025 16:38:59 +1300 Message-Id: <20250113033901.68951-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250113033901.68951-1-21cnbao@gmail.com> References: <20250113033901.68951-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: AECCA1C0007 X-Stat-Signature: h9sit7xwt46gm6nqu9xc7auek3mxgjdo X-Rspam-User: X-HE-Tag: 1736739579-131639 X-HE-Meta: U2FsdGVkX19S87SuM+eed8sem2IhbhPPpNUB6VlSWq+OD6zEq2aAFB6kaTy5LZkq/Ph7pEvxH0S5sxnAl/X7rYEg747ymfgOQQCX+hAWJnC0LQPvhL+JDBmfHvNZ74oebIMi1lu6x54KtMHt0xJ/JqwLXm5neTChWjJ9kvwZqcbIYXYr2+Ea9U7nPX72zfrexNVftsXvceS83PiQOLEYftqaQiGxBcV2Z92iHp9Fg44jC51MCpNcIs+BSPAej0iOB+KtF+1QVr1H7y90fIl1lm4dJzaTUreTP2P/rg4nCY7uSl8axCHuTpQhnjVr1RqPtCdLwz5J9dCT3GZ/moY6ZWhKev+RTmY+w4Vaw+sVlV5KA5H9b6ZEJUm+SEFHRAumx5YxtlBs7nXsV9VhZCqb3kh3givUEgkzbJiZtw6aIWEoKNemnIrME+XLnSYS60JwsrCmZC9eiDTbi4Dv7UVzw/k1n4/FpXA08fkgUmcWUZHU+L5Jg3Zq3X9LI1pGFLg6RyZ3ASnHQcvJf0TkMsHz6OJpIAxe5H/wo43/KtZ6yjp/2flt1972b9HebxdBhiLo8U0p9ASrPzhfploXE2RTIl6/wB0jCWgBWpB8Rvj6ZzudkVvxtMN5htkc3O8jMgowB/gQ0vip+gdBiHNqWXZ2ljC+iGUtRfQziD3cCrQk2AvMdUIb1TkMnRWnUgm8HipQXATDM3TF9/tb94YV9egMdyo6p8jyOVsW0UrnawDCcTvR84WyUAj+yZr4+laufM/38ysrdHq+5h4k4J+PZ7++cQLicH5zjx9VfWU7ym4MAkw1NrCp451V/oAeT/LnipfaVMsYuYPsNipvYHZuuU44hdNKxZHIyrcDXIEG51B3kCOj2swqFTnkML4YsIvopzoMYQ+672YeR8t8UTSC5NiAuldTmXJMkD8ZY6rRiRMxTkYTcU13c2ZlbLyE+xcxvuE90spxVQzJ8/E5W8hj3w+ m3miWQOc +cgUkd+IPpeyTaMMmtVYeWHHKawNyzerAUBVL/iTQmCMDciZ2t5GPAnL5Wh2OBkg1/IeO9/6isQ4SBkvTrMSjc0cG7aGNNQwbisJHsWtxN0PLvB+B+0YgS6rvvABHQQN44ztqtDuJP16+7e/YiDZAo+6tacwHUlVANruHFX/+dV/f4Qb6JCjH4XFymGejWvdTMXgyfAwYC2p/3As5CTnV+gXhBVmRU+djQPDWxHOOyUFhtTrwM0pJdGFGYtEDp+vH66WjyIsupOObBaIQ7evMB2IQnv89L4r4tM0RlGAZV1pjLTkVMmFTeS04yYR7fmaudv0aoIFYVVCn594P2DCRzv5Wphf/21q41qIRJoYXjXw+q0Xrf4MqiN014KipqB4rHfRZeLxDeDt1KPpeUbEFPvQbOVhdXTIzAG00lrUt7ArTfL+9r8KOKB7QKKBI0KzKXYm1gWqnjTqNwxa4mpYS1ilEEEY/O/Gpxvi8bAaF3pphRhOVjGDg8+blTgEq3lgzszb0eRczdhRddogWDCpljvfhu02RkzXnLeVoDa06yO26VnU3gWB9exutHl79HENKh7B80CdE0aBIgd9Y8oKRWX3eOTt9xGROBTleXkAqIV5Yek3Y/xZSoj/dpsBUew7lMKdaUpb5z4uZniKppb9DefN5zjBYUTqG6Anp6hwgODdAOkEmojgEJViCVS7zWhK/1cs1DIK2TefO93q9nJuqyRIgowWj8psSI1520ayx9+FJfzCf+zvzztCgj740juA+K8s9DAXHW2MdC03RX9TQElpPz5cCH69ARD5wyMTiN/Aey5SIojO/M9h+Semvr3MUOuwc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song This is a preparatory patch to support batch PTE unmapping in `try_to_unmap_one`. It first introduces range handling for `tlbbatch` flush. Currently, the range is always set to the size of PAGE_SIZE. Cc: Catalin Marinas Cc: Will Deacon Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Shaoqin Huang Cc: Gavin Shan Cc: Kefeng Wang Cc: Mark Rutland Cc: David Hildenbrand Cc: Lance Yang Cc: "Kirill A. Shutemov" Cc: Yosry Ahmed Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Yicong Yang Signed-off-by: Barry Song --- arch/arm64/include/asm/tlbflush.h | 26 ++++++++++++++------------ arch/arm64/mm/contpte.c | 2 +- arch/riscv/include/asm/tlbflush.h | 3 ++- arch/riscv/mm/tlbflush.c | 3 ++- arch/x86/include/asm/tlbflush.h | 3 ++- mm/rmap.c | 12 +++++++----- 6 files changed, 28 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bc94e036a26b..f34e4fab5aa2 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -322,13 +322,6 @@ static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm) return true; } -static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm, - unsigned long uaddr) -{ - __flush_tlb_page_nosync(mm, uaddr); -} - /* * If mprotect/munmap/etc occurs during TLB batched flushing, we need to * synchronise all the TLBI issued with a DSB to avoid the race mentioned in @@ -448,7 +441,7 @@ static inline bool __flush_tlb_range_limit_excess(unsigned long start, return false; } -static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, +static inline void __flush_tlb_range_nosync(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) @@ -460,12 +453,12 @@ static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, pages = (end - start) >> PAGE_SHIFT; if (__flush_tlb_range_limit_excess(start, end, pages, stride)) { - flush_tlb_mm(vma->vm_mm); + flush_tlb_mm(mm); return; } dsb(ishst); - asid = ASID(vma->vm_mm); + asid = ASID(mm); if (last_level) __flush_tlb_range_op(vale1is, start, pages, stride, asid, @@ -474,7 +467,7 @@ static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true, lpa2_is_enabled()); - mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } static inline void __flush_tlb_range(struct vm_area_struct *vma, @@ -482,7 +475,7 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long stride, bool last_level, int tlb_level) { - __flush_tlb_range_nosync(vma, start, end, stride, + __flush_tlb_range_nosync(vma->vm_mm, start, end, stride, last_level, tlb_level); dsb(ish); } @@ -533,6 +526,15 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr) dsb(ish); isb(); } + +static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, + unsigned long uaddr, + unsigned long size) +{ + __flush_tlb_range_nosync(mm, uaddr, uaddr + size, + PAGE_SIZE, true, 3); +} #endif #endif diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 55107d27d3f8..bcac4f55f9c1 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -335,7 +335,7 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, * eliding the trailing DSB applies here. */ addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - __flush_tlb_range_nosync(vma, addr, addr + CONT_PTE_SIZE, + __flush_tlb_range_nosync(vma->vm_mm, addr, addr + CONT_PTE_SIZE, PAGE_SIZE, true, 3); } diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 72e559934952..7f3ea687ce33 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -61,7 +61,8 @@ void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, bool arch_tlbbatch_should_defer(struct mm_struct *mm); void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, struct mm_struct *mm, - unsigned long uaddr); + unsigned long uaddr, + unsigned long size); void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 9b6e86ce3867..aeda64a36d50 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -187,7 +187,8 @@ bool arch_tlbbatch_should_defer(struct mm_struct *mm) void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, struct mm_struct *mm, - unsigned long uaddr) + unsigned long uaddr, + unsigned long size) { cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 69e79fff41b8..4b62a6329b8f 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -279,7 +279,8 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, struct mm_struct *mm, - unsigned long uaddr) + unsigned long uaddr, + unsigned long size) { inc_mm_tlb_gen(mm); cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); diff --git a/mm/rmap.c b/mm/rmap.c index de6b8c34e98c..365112af5291 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -672,7 +672,8 @@ void try_to_unmap_flush_dirty(void) (TLB_FLUSH_BATCH_PENDING_MASK / 2) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, - unsigned long uaddr) + unsigned long uaddr, + unsigned long size) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; int batch; @@ -681,7 +682,7 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, if (!pte_accessible(mm, pteval)) return; - arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr, size); tlb_ubc->flush_required = true; /* @@ -757,7 +758,8 @@ void flush_tlb_batched_pending(struct mm_struct *mm) } #else static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, - unsigned long uaddr) + unsigned long uaddr, + unsigned long size) { } @@ -1792,7 +1794,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pteval, address); + set_tlb_ubc_flush_pending(mm, pteval, address, PAGE_SIZE); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } @@ -2164,7 +2166,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pteval, address); + set_tlb_ubc_flush_pending(mm, pteval, address, PAGE_SIZE); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } From patchwork Mon Jan 13 03:39:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13936680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A5E7E7719E for ; Mon, 13 Jan 2025 03:39:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D6516B008C; Sun, 12 Jan 2025 22:39:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 160576B0092; Sun, 12 Jan 2025 22:39:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1B646B0093; Sun, 12 Jan 2025 22:39:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CA3266B008C for ; Sun, 12 Jan 2025 22:39:48 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7F8161A1A83 for ; Mon, 13 Jan 2025 03:39:48 +0000 (UTC) X-FDA: 83001024456.13.E46F148 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf23.hostedemail.com (Postfix) with ESMTP id 95192140009 for ; Mon, 13 Jan 2025 03:39:46 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Gymt825r; spf=pass (imf23.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736739586; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7ePF7LRaXT6WJxMxcE/kQzKdknlZvi8u0Wr7nYwGrGY=; b=D0H0IU/YpmYdJfKHPN76foqJFHLmN+YBl0J0hCrjILc5aOjekQn6V6kIBv5Hd56dA0wOxY 0RCAM8bxFqwKCa7EqNeyvDVRAarfjXuQllubQOnvLLglSkEfHGiPV0ddvoYppbrQ7Ae3sY Dl0PcC/3iW9mY+sliAb3Y+L9VJZqSQo= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Gymt825r; spf=pass (imf23.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736739586; a=rsa-sha256; cv=none; b=Vvy66ftWp/Xd3+uNaIrrmXGN0x0SmdhxonNwBOX2XN93omOLQZ9Z0gPQ7QXEQz6Xk9H3hT SHlCt3SQg1xmayprtpM9h5BSlCKB5qcCxOfYGeQA8G6F28DyXatJWghmdUDnUd5ufNFZZ3 1EdOL8g8i4CWF0dPlPTJAvrIQA6ydps= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2166022c5caso57669175ad.2 for ; Sun, 12 Jan 2025 19:39:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736739585; x=1737344385; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7ePF7LRaXT6WJxMxcE/kQzKdknlZvi8u0Wr7nYwGrGY=; b=Gymt825r7bA1WX6Qb48nToTN4El/vwq/2WwkGxSj/T1W7a1yDhyO4dg9g2mA3F+Q6U tbsjKMkhEvZEtXlzV+h39/hshbBlSjm3VSlq+nMS+SsoPx8xL5gQ5OkjM91gOH6m/s2p VIdrGdGHQXJckgHZVHDLjY1dX8pQa6WXSvNta1/r9tBclS7zyQd9OdmH+QJkMI8NR2gz do36YnMXbmKs6M1/w53mRFaDtTJWcMrnWsFa2hc325sysy8g90swmpFJybyzRnHYaKe+ trTbe9AnaYeTWFk7RCPH1bsMoskhxEKqwZQSxj2dsBqiyKkvk8nwJnD3JLMknLyjDpzU rj8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736739585; x=1737344385; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7ePF7LRaXT6WJxMxcE/kQzKdknlZvi8u0Wr7nYwGrGY=; b=p12U9u8vVt3M015pu56Z9uVJJknWxASQpUx1NLrKWgdqU2YuQG0tBrAwoBumF0ulp5 23/r/YPVqguxj1M+kCM/8sIdZkG3endOzWwOSXBiGr9zF/qLJTl0cpGO7Hhbjtq+DVLb G8RfL1s7SU5mhDUUnFVMLZHfwrLMyp2u6VO6Dxm7TSIrofBkmSVz7MBsJdHfKuDILH7h /1b5LlhVRJDO+n3IIj8zZHZyZnmG4Bse5WAuriQ586ifF+QxDU7+YT6RHG7A2NaQYmju 6/Y24p7Sw7Rimf2Qg5PO781H/OxEp11MQo2fqqZ7ZDPhm9QX3ivUMDx7DXjmODTSzc8R C60Q== X-Forwarded-Encrypted: i=1; AJvYcCUlm/4QM4HiDEkFYPWQydspDnsGeAHWScQjKo3sv4tklmrJ0z0sZRF1l5SVeH76ivki7iCfEBSWqg==@kvack.org X-Gm-Message-State: AOJu0Yzv7UgI84r1jezSGSWcrzCw/YzHHb7FjU7zmNCzSngIBgb9c4Vr 3r542teemBGnJ67ejOujnN1QXdcXRMz1jM70v9O9np1tEPcFgGXi X-Gm-Gg: ASbGncsPZx4VjiUvZo0FNzlfBYTbMnyxA1Wh0Y2jFToEx7AQXui6ttl5/7FSDCGWcIs NMY/h24BbY0p+pt4/3+LuypD7hn4ldvg/TxMUwW0+yKuH4zD9ZD0B6zQdDv2Y5H2SiEtWV443h2 4MTfaxb4LIF2lSZfMx5k8BiPoxXU2Ssy/VHl2HDswJv1PeJLzeX3BMhoOtYzX0Ag/Pg040tpUuW dqq/hLUAR99CnpIOtLszJAto/dUq6Ohu6HS55pqGXB6MJy+VwUpNgqxmV63DPzpNcAlgH76vyOw Htdwl6jU X-Google-Smtp-Source: AGHT+IEJyXTMuDf5NuY3j1SIPBbXyZqFASFj5tk1fMcghhN0EpO5xryOPxQNqGVnzZ7Hib6ZrPMFDA== X-Received: by 2002:a17:903:3385:b0:216:1cf8:8b8 with SMTP id d9443c01a7336-21a83f69643mr199408035ad.27.1736739585271; Sun, 12 Jan 2025 19:39:45 -0800 (PST) Received: from Barrys-MBP.hub ([2407:7000:af65:8200:39b5:3f0b:acf3:9158]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f25aabfsm44368405ad.246.2025.01.12.19.39.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 12 Jan 2025 19:39:44 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, ioworker0@gmail.com, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, v-songbaohua@oppo.com, x86@kernel.org, linux-riscv@lists.infradead.org, ying.huang@intel.com, zhengtangquan@oppo.com, lorenzo.stoakes@oracle.com Subject: [PATCH v2 3/4] mm: Support batched unmap for lazyfree large folios during reclamation Date: Mon, 13 Jan 2025 16:39:00 +1300 Message-Id: <20250113033901.68951-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250113033901.68951-1-21cnbao@gmail.com> References: <20250113033901.68951-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 95192140009 X-Rspam-User: X-Stat-Signature: egbrkpg1e7315zmnxmhsodrd9t5id8qg X-HE-Tag: 1736739586-985713 X-HE-Meta: U2FsdGVkX1+xhREVtnn63bAgwN9ujMFu4mK2Q1JVwiSXzbNzeCHtOKVdVxC7pNXF+Lhe5ilEERhW/X24sC0Z2KkifS4JIL3c2DWoVzQIX/pYUU8ZYNZqhKrtmsepnA1nvBlj4VmEr/u0bSPPl16LW+dStD5eznH33kxaaFg6XIgxaEjlmjEoBXFybc/TdLTFYpbg5ut4/N6wEsrjrxJvKAHvPu15hD5axXwjwvn1VPdoninmjldMkYvB+6YV229K1nXi9YKXBBSRtkRBdat2VxKcNEKdn4XhA3muOGPpaSookAOfFyhdS56K91JzB957v5qt9PY1mzmJb45o1fiIwU/q6g9jXb0/AFQ+qscacUfqwwj2FzI3kDGcCD9F78f0ZEdoPXfIH/dNJ1imYI+iYxAeFz/HHRU+5mVz0CHyemrtTpLI1/jo7+Q2Jrv8Yha8Wa5prJX3HF4RsGO9P2k3D2xOnnb4dz8sMH7xeH1lL5+16sMTYEEsrOIwS+uFhJ9KvC4gvGnPkYLc1YpG+WxeY46QBJ7C05m8QJtFLmtngJrTKYf2bfb5GSokGoAeqb0XdekXy9qRzJujcK2e5yO/HtDpKxmqvfpH07fZtEPfGvU7q5zK0+ZX396tl+cE3lo7KCCtNNTmtTH9ShH/m4nqoT/0dZDw9Up7l4uKodpowEn9qb+0YsqS8RKCZ3OdSyuEOnN+0oRTaJfBUI4Ai+Zr48CVzge0AJ6RNG3SPwjPuSpOu31ynmUFBJ4YdeGmRwdhchjVgLorPLQl+EBEKlJ2hNjEzbUbgd7D1e2tO7kFQQwejXRJm7r7PGIlcaN7BPQSKL5qothPfL1II316KmBHeKGmtczO51kJfiqGFxxwioPTfFGFoobS+e0o/Mu0oI6iPxKaoA0JK9TxamIQDqy7coDYLmlF6ou+0dmpLro7PNor7xgERj8Ln3LPShMlUMn77X8aAHDiFYqxMJT96YP TraVOsKk aXMnwLrHNpNjh1fRQgh/QONKFhL1/zo0MEln2I2rHaLmoVQcWIXVZSdF8Gcu1rjHCl8WAdG6JODFCOFWIFDFQ4OR3ahpiOYhhosvMORomJsv/N2euZRt6t4btx9FgBGhD8WvaINQ5N/kf4SUjmAudkMSfmVv/N1Rw1F+NiadPLpFuiXicddgJCFbbwZ+0ipnWTIqqBNll0rpZLwfI36vbqFwPRPzwPNl9rEMZhhBXAnSxOIyci1hVoX7bV+jBxmJ3Bt0r+kCi2ccYbVgsv91iRhL5VOyljeFSHVnTFSj5/+j1q6tulRrXHq8NZpvo18eTx5MAECu80F36me+QeD6W/+Hxp3cm2H8vGdCp49kaw/hnHRrPya6d+9oik/4WY0KWCfGYAvseA0hSLVOlw6SR9Aa8AKDbUJCu2PKmXv36DH6kmJ4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Currently, the PTEs and rmap of a large folio are removed one at a time. This is not only slow but also causes the large folio to be unnecessarily added to deferred_split, which can lead to races between the deferred_split shrinker callback and memory reclamation. This patch releases all PTEs and rmap entries in a batch. Currently, it only handles lazyfree large folios. The below microbench tries to reclaim 128MB lazyfree large folios whose sizes are 64KiB: #include #include #include #include #define SIZE 128*1024*1024 // 128 MB unsigned long read_split_deferred() { FILE *file = fopen("/sys/kernel/mm/transparent_hugepage" "/hugepages-64kB/stats/split_deferred", "r"); if (!file) { perror("Error opening file"); return 0; } unsigned long value; if (fscanf(file, "%lu", &value) != 1) { perror("Error reading value"); fclose(file); return 0; } fclose(file); return value; } int main(int argc, char *argv[]) { while(1) { volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); memset((void *)p, 1, SIZE); madvise((void *)p, SIZE, MADV_FREE); clock_t start_time = clock(); unsigned long start_split = read_split_deferred(); madvise((void *)p, SIZE, MADV_PAGEOUT); clock_t end_time = clock(); unsigned long end_split = read_split_deferred(); double elapsed_time = (double)(end_time - start_time) / CLOCKS_PER_SEC; printf("Time taken by reclamation: %f seconds, split_deferred: %ld\n", elapsed_time, end_split - start_split); munmap((void *)p, SIZE); } return 0; } w/o patch: ~ # ./a.out Time taken by reclamation: 0.177418 seconds, split_deferred: 2048 Time taken by reclamation: 0.178348 seconds, split_deferred: 2048 Time taken by reclamation: 0.174525 seconds, split_deferred: 2048 Time taken by reclamation: 0.171620 seconds, split_deferred: 2048 Time taken by reclamation: 0.172241 seconds, split_deferred: 2048 Time taken by reclamation: 0.174003 seconds, split_deferred: 2048 Time taken by reclamation: 0.171058 seconds, split_deferred: 2048 Time taken by reclamation: 0.171993 seconds, split_deferred: 2048 Time taken by reclamation: 0.169829 seconds, split_deferred: 2048 Time taken by reclamation: 0.172895 seconds, split_deferred: 2048 Time taken by reclamation: 0.176063 seconds, split_deferred: 2048 Time taken by reclamation: 0.172568 seconds, split_deferred: 2048 Time taken by reclamation: 0.171185 seconds, split_deferred: 2048 Time taken by reclamation: 0.170632 seconds, split_deferred: 2048 Time taken by reclamation: 0.170208 seconds, split_deferred: 2048 Time taken by reclamation: 0.174192 seconds, split_deferred: 2048 ... w/ patch: ~ # ./a.out Time taken by reclamation: 0.074231 seconds, split_deferred: 0 Time taken by reclamation: 0.071026 seconds, split_deferred: 0 Time taken by reclamation: 0.072029 seconds, split_deferred: 0 Time taken by reclamation: 0.071873 seconds, split_deferred: 0 Time taken by reclamation: 0.073573 seconds, split_deferred: 0 Time taken by reclamation: 0.071906 seconds, split_deferred: 0 Time taken by reclamation: 0.073604 seconds, split_deferred: 0 Time taken by reclamation: 0.075903 seconds, split_deferred: 0 Time taken by reclamation: 0.073191 seconds, split_deferred: 0 Time taken by reclamation: 0.071228 seconds, split_deferred: 0 Time taken by reclamation: 0.071391 seconds, split_deferred: 0 Time taken by reclamation: 0.071468 seconds, split_deferred: 0 Time taken by reclamation: 0.071896 seconds, split_deferred: 0 Time taken by reclamation: 0.072508 seconds, split_deferred: 0 Time taken by reclamation: 0.071884 seconds, split_deferred: 0 Time taken by reclamation: 0.072433 seconds, split_deferred: 0 Time taken by reclamation: 0.071939 seconds, split_deferred: 0 ... Signed-off-by: Barry Song --- mm/rmap.c | 46 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 365112af5291..3ef659310797 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1642,6 +1642,25 @@ void folio_remove_rmap_pmd(struct folio *folio, struct page *page, #endif } +/* We support batch unmapping of PTEs for lazyfree large folios */ +static inline bool can_batch_unmap_folio_ptes(unsigned long addr, + struct folio *folio, pte_t *ptep) +{ + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; + int max_nr = folio_nr_pages(folio); + pte_t pte = ptep_get(ptep); + + if (!folio_test_anon(folio) || folio_test_swapbacked(folio)) + return false; + if (pte_none(pte) || pte_unused(pte) || !pte_present(pte)) + return false; + if (pte_pfn(pte) != folio_pfn(folio)) + return false; + + return folio_pte_batch(folio, addr, ptep, pte, max_nr, fpb_flags, NULL, + NULL, NULL) == max_nr; +} + /* * @arg: enum ttu_flags will be passed to this argument */ @@ -1655,6 +1674,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, bool anon_exclusive, ret = true; struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; + int nr_pages = 1; unsigned long pfn; unsigned long hsz = 0; @@ -1780,6 +1800,15 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, hugetlb_vma_unlock_write(vma); } pteval = huge_ptep_clear_flush(vma, address, pvmw.pte); + } else if (folio_test_large(folio) && !(flags & TTU_HWPOISON) && + can_batch_unmap_folio_ptes(address, folio, pvmw.pte)) { + nr_pages = folio_nr_pages(folio); + flush_cache_range(vma, range.start, range.end); + pteval = get_and_clear_full_ptes(mm, address, pvmw.pte, nr_pages, 0); + if (should_defer_flush(mm, flags)) + set_tlb_ubc_flush_pending(mm, pteval, address, folio_size(folio)); + else + flush_tlb_range(vma, range.start, range.end); } else { flush_cache_page(vma, address, pfn); /* Nuke the page table entry. */ @@ -1875,7 +1904,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, * redirtied either using the page table or a previously * obtained GUP reference. */ - set_pte_at(mm, address, pvmw.pte, pteval); + set_ptes(mm, address, pvmw.pte, pteval, nr_pages); folio_set_swapbacked(folio); goto walk_abort; } else if (ref_count != 1 + map_count) { @@ -1888,10 +1917,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, * We'll come back here later and detect if the folio was * dirtied when the additional reference is gone. */ - set_pte_at(mm, address, pvmw.pte, pteval); + set_ptes(mm, address, pvmw.pte, pteval, nr_pages); goto walk_abort; } - dec_mm_counter(mm, MM_ANONPAGES); + add_mm_counter(mm, MM_ANONPAGES, -nr_pages); goto discard; } @@ -1943,13 +1972,18 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter_file(folio)); } discard: - if (unlikely(folio_test_hugetlb(folio))) + if (unlikely(folio_test_hugetlb(folio))) { hugetlb_remove_rmap(folio); - else - folio_remove_rmap_pte(folio, subpage, vma); + } else { + folio_remove_rmap_ptes(folio, subpage, nr_pages, vma); + folio_ref_sub(folio, nr_pages - 1); + } if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); + /* We have already batched the entire folio */ + if (nr_pages > 1) + goto walk_done; continue; walk_abort: ret = false; From patchwork Mon Jan 13 03:39:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13936681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EDEFE77188 for ; Mon, 13 Jan 2025 03:39:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AB556B0093; Sun, 12 Jan 2025 22:39:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 334336B0095; Sun, 12 Jan 2025 22:39:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D5096B0096; Sun, 12 Jan 2025 22:39:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EA8436B0093 for ; Sun, 12 Jan 2025 22:39:55 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7D6631C9343 for ; Mon, 13 Jan 2025 03:39:55 +0000 (UTC) X-FDA: 83001024750.19.E9A9E0A Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf12.hostedemail.com (Postfix) with ESMTP id 7B53640003 for ; Mon, 13 Jan 2025 03:39:53 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=My2FmB7A; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736739593; a=rsa-sha256; cv=none; b=I4gO/zDA6QRcRuI43XuLTA8EYgzhvqdXH4oqm2gCEg6d2wDQfhep70djfbw2jq1vc6Qdtb xSSAj4ifOkBuiIVAICw//TeqB4u96sgdaZ9GiVqDf7IgawP8VF3U/AK9mIPaQ9kT0aQjGc tFfjVQ9hs72Co+1+yM1nP/HKyfhJZIs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=My2FmB7A; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736739593; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t1XMRg54nQSp8/EH5otPlOs5Y//8NM069hYIc60Rxo8=; b=ttlcxW9jAgsaiaIN2PxuA0roZRKX8U07Deea5xkqCw1tDmh8qIwmID4L+whKWd3MgaEnBb u67vnC1A4OVUvHZOx+8fz4S6ClHJpZZ8tNmB0syGwQN4s6IghusXAsaSwhcmj2fK0SEGIu YgIa9edwgNvCcOdVaTCdPniHtvvblrY= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21669fd5c7cso66199655ad.3 for ; Sun, 12 Jan 2025 19:39:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736739592; x=1737344392; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t1XMRg54nQSp8/EH5otPlOs5Y//8NM069hYIc60Rxo8=; b=My2FmB7AKo/mQwEJKMxlMhcGx43lUAaA8eN+8TV5HTMPxDpNgAyaI6mI+Ad/TMXmfD TXQZPhAZHbOhpzoeuLvcanaBaTPgkV02R3Lhu5TP0MXvdONrHpDdacoIM+Wf3LSvFsyC Nxyf6T/8kFYFGbehyw1rB7UMlJedpTCSYpIBKSYPftzi+vcY2Ri5bWUGLwMNEsDKs3pb K3UCtpE5+vdeHgMwWf60VYE/XT3y2Roy2IhVAPcLFH3K9s2coK75s8MOp3z4rlDJXwqy 7uWunJjJXE+MOK8fpOdURbxDcEInttUfCGfssPj8+tNdTDXmgpzn43BOw2T6gUN2ubh9 BR9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736739592; x=1737344392; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t1XMRg54nQSp8/EH5otPlOs5Y//8NM069hYIc60Rxo8=; b=RFQ0sjWSnEDUUDMwuNk4dxZb42fGz2ufSwVhQ9cz6gs/7U5BcO9IPnV6XN2b64SEzr 5aUGQCO5Tvc/SpcJo0+FvRhPK93FHkb/01wcpL72n1Ah8org2ov4ATOzR3za5wSpTIeq bmsdzNiRq9yjiv5z1H0sq/REnye2MQLdTb9JQyYCA1nrppvNQRw6G+ynFwUPlJPuWuCh nrKF1gJ4fsWFWXt8BpT5dfsUtmmEv3dFde0qsoZ92gkkzaOjSszHeEqi0y/pAIaDeQyY aUmVSNMI7VuIPYnqXFAKZ98My2/P/C5IhNl0dyloE3UQvpztndinXyuWHDdsYZd4ZVC2 duUw== X-Forwarded-Encrypted: i=1; AJvYcCWpaaVhe2RAHNzBxgdY2XK7baQBCCLrOKyF88Abs7uNPd+9r0nxWc77TiTBgUib6yLmDNkIkzYktQ==@kvack.org X-Gm-Message-State: AOJu0YxSsdzFFGCkANY2yT2MXoLO6+O577bDu26E/Ta8lM8u1m6trQY6 ThSWugKzW8Cy6cfoBXmdJ9otg6IE8lxPUdo1Niv7sPNXy9KWNkhF X-Gm-Gg: ASbGncse8QXnSjd75WRumoGqns2lCEqulIz8UXmFN1332onQ/Bf4lhbUuOMQML6PqSG ycxpvJUziQnobhObjD4Wt7wTf0l3SsYxdr0hIeM+r8BKXbTv775FOQfa82Av0DhPuxgnArMWtA/ xY38SwjdBDR7hhQW0ts/asi25lcdtRZPqywkp6SmX0I0nmRhXTiUQrBLwx/r5PXxj5pPSAsUUpi p2HPlRZ7o7IGqdT6ngMxJUF0W6q0zgHT7kzEu8lChIW2DftahXJ7FFzhKM+V8TI783Wjqn9jBbV J2NfoOOt X-Google-Smtp-Source: AGHT+IFvTUatRCa49/9Qoa3okmsFGYsqNW658p8oE5OURLMqB1h00MlvT5OAmq5hWQRPMmGvVdLulw== X-Received: by 2002:a17:902:cec3:b0:216:6fb5:fd83 with SMTP id d9443c01a7336-21a83f69d84mr299035325ad.29.1736739592302; Sun, 12 Jan 2025 19:39:52 -0800 (PST) Received: from Barrys-MBP.hub ([2407:7000:af65:8200:39b5:3f0b:acf3:9158]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f25aabfsm44368405ad.246.2025.01.12.19.39.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Sun, 12 Jan 2025 19:39:51 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, ioworker0@gmail.com, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, v-songbaohua@oppo.com, x86@kernel.org, linux-riscv@lists.infradead.org, ying.huang@intel.com, zhengtangquan@oppo.com, lorenzo.stoakes@oracle.com Subject: [PATCH v2 4/4] mm: Avoid splitting pmd for lazyfree pmd-mapped THP in try_to_unmap Date: Mon, 13 Jan 2025 16:39:01 +1300 Message-Id: <20250113033901.68951-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250113033901.68951-1-21cnbao@gmail.com> References: <20250113033901.68951-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7B53640003 X-Stat-Signature: 6o776d841n8gdrtb7f6ixmdomi4syan7 X-Rspam-User: X-HE-Tag: 1736739593-659929 X-HE-Meta: U2FsdGVkX1+f2z9urunt53LzrxERudo7MNL1fz/k83LPyOC3fvEC4BktNw1yYh73mt+cj5NdZi0DnQ2VPsdpHLZe9yYQTYJVClNwNELXDlsTfh2AoqLncfRt3Gi5TnPnLJgfTbD+JsAb7Hnj+s3W7WKwoM9JCWt75DNoO9fNnkDix2lw8Fr8k52+iSmD4GV9IseDWkByc8P9CYpL4BhjHuo1MtFjqGl+jjRuxYNuX68Cb5KpeNvvpB0LxLa8Wj3/6oUhM7Q1jUPzmvqim/90Ag0gL4zsu8Eb8H12MX4fc1xxHwQ+6SwLUXHK2Gvh4yAaJoS2LwTFlJY69NWQ67qPsgndzDfh7VpCgQL0CWnkTme7dnPsfIS1Iynlf7cVt8m2xobsp4bzpNHHenyN0kD+aOgplWSxr67b/Ipde2k/ocjBKs/sofVeu/mLNO4e233HYDa8nH82cDnDrQ7J19iSz+tBdA4GFwyUPT3BqDtHFP/EBCxiHtwADohEGM6uMN2poVvu3si+qlGBY3DfwnFF9szkrjgXvkRCaN7GvEVpNDlYEzZJo1Xn/gm8rOSTUR0u4eXXmIrPiacN55GwUYg4/qtYoQSwXK0xySu5VmT/wLtTew10mo3ImSOCLEnpyLv9I3tPLJBY/gX1Gn3FpaDVl+NWwPsOEb1f3WfmoXUpKrOpq9uzIYlvXNgEYqVligeYrIsp56wZ8MeMi7e59OoHHE+iKJ5FLqLUA5SGb3WmAyv8FfSEi8uFDjEgEiirtagb3B1W3SvzUoTMBgMuMr3nyDr4qoNF0xTFRgwSUOehUKoSAoR74TQQ1VAA/9TCyegZImzbxckdnwdQSjutHQOHTPGtZV6Ud7JIU/+C4oT5YDeJGCcTSn7oOKL36DPRW2eyrtsOwSC0RKpiKmSB7qCjYACNjrq4nGBuphrgw91kc5U+OSwFojn4uHlpkP3sMv/BC4pgpanLVNKtXF4CXQC CHsTtLhG XI9f8HvglmVpMq1MlVBjGT20ZjxJHU1dk2GhuYbuki2hQEPl9bVzO5qdPJXGdlJP2MHZh/FaxCP/TrGOgPa1KfOsbRzs5eDv+DzZJE24b40gYY2I6xfVAU/T3tlbFnXFvwZlyCqDtK5tZTdNyRhfYcIZS2jaqppo0V9LymTQXEkf5T8pG5WNvDZ7PIQY6tREqPYz6XQLcsO1Jl24LujiORP7kWA4NNzwiLKB9Y0lrIb4QUZRQsDgowW1eijxw657+D9sDfeVlpXwsdDk/u4P6zZZJRjioiznIEElKYOHYgjAVMkgTQM9R1X1igsxsX0sxLYJzlsfd47VZxjca6ZGjdTPlUk9Y1z4htk278v2pRB6Pm6g9GsAjOeASFPcoOx23g1esV55VRV6TjHZjtpHXNhP5oI3SkS6NB9K8OjFBCKtwB05ls0vcEN6W4MLPVUgQ+GqlEpZ5HCeub82JrXP1rAImPrfsoJrwkVSacssUWOUwgZxfF9m4Kklycg/t6mMEkS7EdjLmJPSz7k4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song The try_to_unmap_one() function currently handles PMD-mapped THPs inefficiently. It first splits the PMD into PTEs, copies the dirty state from the PMD to the PTEs, iterates over the PTEs to locate the dirty state, and then marks the THP as swap-backed. This process involves unnecessary PMD splitting and redundant iteration. Instead, this functionality can be efficiently managed in __discard_anon_folio_pmd_locked(), avoiding the extra steps and improving performance. The following microbenchmark redirties folios after invoking MADV_FREE, then measures the time taken to perform memory reclamation (actually set those folios swapbacked again) on the redirtied folios. #include #include #include #include #define SIZE 128*1024*1024 // 128 MB int main(int argc, char *argv[]) { while(1) { volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); memset((void *)p, 1, SIZE); madvise((void *)p, SIZE, MADV_FREE); /* redirty after MADV_FREE */ memset((void *)p, 1, SIZE); clock_t start_time = clock(); madvise((void *)p, SIZE, MADV_PAGEOUT); clock_t end_time = clock(); double elapsed_time = (double)(end_time - start_time) / CLOCKS_PER_SEC; printf("Time taken by reclamation: %f seconds\n", elapsed_time); munmap((void *)p, SIZE); } return 0; } Testing results are as below, w/o patch: ~ # ./a.out Time taken by reclamation: 0.007300 seconds Time taken by reclamation: 0.007226 seconds Time taken by reclamation: 0.007295 seconds Time taken by reclamation: 0.007731 seconds Time taken by reclamation: 0.007134 seconds Time taken by reclamation: 0.007285 seconds Time taken by reclamation: 0.007720 seconds Time taken by reclamation: 0.007128 seconds Time taken by reclamation: 0.007710 seconds Time taken by reclamation: 0.007712 seconds Time taken by reclamation: 0.007236 seconds Time taken by reclamation: 0.007690 seconds Time taken by reclamation: 0.007174 seconds Time taken by reclamation: 0.007670 seconds Time taken by reclamation: 0.007169 seconds Time taken by reclamation: 0.007305 seconds Time taken by reclamation: 0.007432 seconds Time taken by reclamation: 0.007158 seconds Time taken by reclamation: 0.007133 seconds … w/ patch ~ # ./a.out Time taken by reclamation: 0.002124 seconds Time taken by reclamation: 0.002116 seconds Time taken by reclamation: 0.002150 seconds Time taken by reclamation: 0.002261 seconds Time taken by reclamation: 0.002137 seconds Time taken by reclamation: 0.002173 seconds Time taken by reclamation: 0.002063 seconds Time taken by reclamation: 0.002088 seconds Time taken by reclamation: 0.002169 seconds Time taken by reclamation: 0.002124 seconds Time taken by reclamation: 0.002111 seconds Time taken by reclamation: 0.002224 seconds Time taken by reclamation: 0.002297 seconds Time taken by reclamation: 0.002260 seconds Time taken by reclamation: 0.002246 seconds Time taken by reclamation: 0.002272 seconds Time taken by reclamation: 0.002277 seconds Time taken by reclamation: 0.002462 seconds … This patch significantly speeds up try_to_unmap_one() by allowing it to skip redirtied THPs without splitting the PMD. Suggested-by: Baolin Wang Suggested-by: Lance Yang Signed-off-by: Barry Song --- mm/huge_memory.c | 17 ++++++++++++++--- mm/rmap.c | 11 ++++++++++- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3d3ebdc002d5..aea49f7125f1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3070,8 +3070,12 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma, int ref_count, map_count; pmd_t orig_pmd = *pmdp; - if (folio_test_dirty(folio) || pmd_dirty(orig_pmd)) + if (pmd_dirty(orig_pmd)) + folio_set_dirty(folio); + if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) { + folio_set_swapbacked(folio); return false; + } orig_pmd = pmdp_huge_clear_flush(vma, addr, pmdp); @@ -3098,8 +3102,15 @@ static bool __discard_anon_folio_pmd_locked(struct vm_area_struct *vma, * * The only folio refs must be one from isolation plus the rmap(s). */ - if (folio_test_dirty(folio) || pmd_dirty(orig_pmd) || - ref_count != map_count + 1) { + if (pmd_dirty(orig_pmd)) + folio_set_dirty(folio); + if (folio_test_dirty(folio) && !(vma->vm_flags & VM_DROPPABLE)) { + folio_set_swapbacked(folio); + set_pmd_at(mm, addr, pmdp, orig_pmd); + return false; + } + + if (ref_count != map_count + 1) { set_pmd_at(mm, addr, pmdp, orig_pmd); return false; } diff --git a/mm/rmap.c b/mm/rmap.c index 3ef659310797..02c4e4b2cd7b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1671,7 +1671,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); pte_t pteval; struct page *subpage; - bool anon_exclusive, ret = true; + bool anon_exclusive, lazyfree, ret = true; struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; int nr_pages = 1; @@ -1724,9 +1724,18 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } if (!pvmw.pte) { + lazyfree = folio_test_anon(folio) && !folio_test_swapbacked(folio); + if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio)) goto walk_done; + /* + * unmap_huge_pmd_locked has either already marked + * the folio as swap-backed or decided to retain it + * due to GUP or speculative references. + */ + if (lazyfree) + goto walk_abort; if (flags & TTU_SPLIT_HUGE_PMD) { /*