From patchwork Mon Feb 6 15:30:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13130097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6985C636D3 for ; Mon, 6 Feb 2023 15:29:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A13746B0073; Mon, 6 Feb 2023 10:29:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C35F6B0078; Mon, 6 Feb 2023 10:29:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88B4C6B007B; Mon, 6 Feb 2023 10:29:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7BC506B0073 for ; Mon, 6 Feb 2023 10:29:00 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 50DE01A07B5 for ; Mon, 6 Feb 2023 15:29:00 +0000 (UTC) X-FDA: 80437250040.02.0B404D0 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf06.hostedemail.com (Postfix) with ESMTP id 4F0E6180025 for ; Mon, 6 Feb 2023 15:28:58 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=msHRiZid; spf=pass (imf06.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675697338; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RvSDe78RU2nTL9I6q9br/wM5C6UDPPKvFCHYVrqRQSA=; b=mQ57qyQPFbnawb2e9+Br2zWsKp1bTREvWmNPWPv5dx+T14S+zsQTKF6WinH4aXXgbQvz2v 4d+srTVqX+9xsKBkH8YhfEOL3BwGnIdpVBUiOQddQ4go0CeKltDsTteW6C9WboCUJC360X OWy2NCuuSRcgkQBDwg8JT+femN1akP4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=msHRiZid; spf=pass (imf06.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675697338; a=rsa-sha256; cv=none; b=pM3SbVeBG9PouTh9RPiB5txu8e5kUUxkvXnZwq5LO65QWRHkuLXgOXXxzgnXG/Qdj5UwkF 5QXdCukKeX1uDw+5P6GpUrEabPKuLsPSEozJX5iAQn5mYejaD5JmAgdW9gvWm3DngF5gpX sFd/bS84UwVK9xlP76tD1+U73ePafYY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675697338; x=1707233338; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TKcs1pJnuf9fzvs4mZBzCmsjSee88jH1Cx1fsFE5aTs=; b=msHRiZidZABgGnpr4aywG9uh1zkFpTEsN/4X0KHEmw31RILTw8zK/43B E6CxlrKLJ+wA80h4t7B7l06Q4J0PA2NqVyJkQ2oKviFlY9XcjYqrWnA8u RMz+G2J80/LaKZ19lrNPMOpgQeRn2BMFrq4l5F3OnqSFnL82x3Zg1qHZs 1gHo/paGRX4poeU5fwyG//bIUMGTJpWeGy0KxKCrOc7ibFxZXq6kroI0n 558twJN4LdxTpcjFEftkOTP2dyxBiuN9fXtKvgl8mp8iezo2nZ0nmBlaH rg8H3KDwTNa5bDILtoGfyAsNJKQxUc6vUXoQKKOJTWA4O63KRRgsEaq94 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10613"; a="312884032" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="312884032" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2023 07:28:56 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10613"; a="840381225" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="840381225" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga005.jf.intel.com with ESMTP; 06 Feb 2023 07:28:55 -0800 From: Yin Fengwei To: willy@infradead.org, linux-mm@kvack.org Cc: dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, fengwei.yin@intel.com Subject: [RFC PATCH 1/2] rmap: Add function to handle entire folio rmap removing Date: Mon, 6 Feb 2023 23:30:48 +0800 Message-Id: <20230206153049.770556-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206153049.770556-1-fengwei.yin@intel.com> References: <20230206153049.770556-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4F0E6180025 X-Stat-Signature: mepxenxbh3jn71ejb8k9ee11no1fouia X-HE-Tag: 1675697338-917759 X-HE-Meta: U2FsdGVkX18sXVKy2Tid8UTko6vcUHICyDwUP7J7r3f3dlDLiFJycqVBK3kYFZqkup+LQS4XvaFDylygYwV/83AYaT01cdHWfSLW+3dNwsWPpDMHUUQX4G34clQDf0WLY8GO6zPF72rVTqtxvAKNK3MeRDxy2qhEP4fWxGDv1hQnx4TNieCDGkdMEflL/GWxJNsf6R1gOQzHc/vAI6usii2c+AzK9BTuImmO7tVz2ZIo9U6QA0ShXy97f3KgbV0ogyeVOi1v1uFZW2mxRvViXQaBlfXNJsrgZ/TWgLDmq0AMDr8J0ziwzruP+rQfceu7cEIdJubruBWW+PPXLcAuB7bl3iNRheywHM17mvWnQzxPhsTjfP8oXoyL0qANnPHJ5YRADZdv7WoyRkt3gUkpGgOlWrhD+lJiQ9h950y41ZNedyIKND4y0z5M/vEhcU75Z/n+bu9JBioFQrKxLu9FTbtjqk694tJNVAo1P2tzzFaxkeivfzjitvhtwEmERJ9GphXjwJNbiyr8Ho/PQ25QKjaOzS4XQZDkXRFrJZmuF6x6neK8fpOzylS5TQBAg0MhjTJp/iLL8qgMGmxBdt7fOCClE+FLqgJmph03esVfNpUIZMjWku+a9PGurk8WokbgnoWXpmkjPfQIuCa3a+gpgO6quzj6nLp7j8w5vx3Lj5HKVjPhzIqUo9WWiv2HZW3RVb+5WRJdeztvfVGrZ6i6RGnBheZnsAHZHtup+r08riEkKMY3neHa/QMoUKtUsO6XPOP75dxOpiJr6U4BLU80rttPUQbuUosYoCIuwvoQDCCDgRq6aUi0HPrYnMv7CtH2XyPOKxO+C3cI+5kHilmSFLy4LtheVTa/GRZDTkNhZo+MlQsYF8ixdn7eLI5PWaRa0KQz8BJLdtcXpgIBcp+KkhbWDEtBoc10RzU2/5DtWLZ3dyYf7QVUw4zy3FuGwNdbe5Jtq2C1TwPR99s7niL wwLq9ulK bbhK0+D3gEB6Sy+aJxswKsdGy3uNn15bn2Gygmo0nNrcWTd1VG9bxWOQKwbW89vNzoOQN52fQ/DGVS6kgHu8Jw4qHIdh3qFqTmfG9A/X9wBdaztHN9q/CiqYbct08vfAlofnkBuIz1WcdlV5cKOgUPak+RwrB+obdRkG4eeNUkLkD2SM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add folio_remove_entire_rmap(). It will handle the entire folio rmap removing. Signed-off-by: Yin Fengwei --- mm/rmap.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index c07c4eef3df2..3ab67b33094b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1394,6 +1394,32 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, nr_pages, vma, compound); } +static void folio_remove_entire_rmap(struct folio *folio, + int *nr, int *nr_pmdmapped) +{ + bool last; + atomic_t *mapped = &folio->_nr_pages_mapped; + + last = atomic_add_negative(-1, &folio->_entire_mapcount); + if (last) { + *nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); + if (likely(*nr < COMPOUND_MAPPED)) { + *nr_pmdmapped = folio_nr_pages(folio); + *nr = *nr_pmdmapped - (*nr & FOLIO_PAGES_MAPPED); + + /* Raced ahead of another remove and an add? */ + if (unlikely(*nr < 0)) + *nr = 0; + } else { + /* An add of COMPOUND_MAPPED raced ahead */ + *nr = 0; + } + } + + if (!folio_test_pmd_mappable(folio)) + *nr_pmdmapped = 0; +} + /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from @@ -1431,20 +1457,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ - last = atomic_add_negative(-1, &folio->_entire_mapcount); - if (last) { - nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of another remove and an add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* An add of COMPOUND_MAPPED raced ahead */ - nr = 0; - } - } + folio_remove_entire_rmap(folio, &nr, &nr_pmdmapped); } if (nr_pmdmapped) { From patchwork Mon Feb 6 15:30:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13130098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A681DC6379F for ; Mon, 6 Feb 2023 15:29:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09FBA6B007B; Mon, 6 Feb 2023 10:29:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F42376B007D; Mon, 6 Feb 2023 10:29:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D95D56B007E; Mon, 6 Feb 2023 10:29:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C82896B007B for ; Mon, 6 Feb 2023 10:29:02 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8A029C07F3 for ; Mon, 6 Feb 2023 15:29:02 +0000 (UTC) X-FDA: 80437250124.10.28F58EF Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf10.hostedemail.com (Postfix) with ESMTP id 26191C001F for ; Mon, 6 Feb 2023 15:28:59 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=fDKWJQ5l; spf=pass (imf10.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675697340; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Qz55W5JPr1QUpwegUcb3lPxwpto9i0pJ+nW/ry2RwmQ=; b=q7kGiGZMsCoMPX2jyuqRqVwxw14YADVPErP8IIxl1eburvtPheiQrCKs7+C9q3MNrwYKGd ZJnh4+9yoduM/QdBKKlEMbzvdyf2pWWT+FNlkFcRHiMrjyGyFgpIkGkKAKgW9xAtQsiGpg yEmSCcI+BHRFoAD3Ah3irOShZKQB9BQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=fDKWJQ5l; spf=pass (imf10.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675697340; a=rsa-sha256; cv=none; b=RQi1yCbkUJ3qGjdE4uwznV+OcVWTKxrurr7IWV+r3/ovjEPVF0Cza81NEnpuq8kvkjiAtX 3+UTXbiB36+d2LvNAuE1L/xeymVl8ZO1rWgmiCimmj/i5AFI1oH+Gxf0mcveleWpfQ3h1s Sq8FhCbSZunpuENkaa8CNWp6xdtclUQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675697340; x=1707233340; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pW7kT/swC94zEOLUfbF+NK9jUZvajWU7xfcaVj3GkRw=; b=fDKWJQ5l3mKGg+IsDCjbzblwAM8BQ1BxLoKbWuuJKrVPP/czBmnWltLa EAnBHZKGKm9ze/mqGgdd18XshzlV/CMLRdqaq3lj1h3DENRf3+wxQfa+2 VjP0lxMQk2Jm07AAn4KLL7Sa8asfLc+SlJOeHlk9QV0LKeWDs+Rhu8hCx OndRxJl7D7DkMAh+1xXRWvIOdTEMJ0EcggGFFHyiiOZHHoGpjkVCaKcby PuTnakEYWj5cuH1hTRQ7eNQIYgSIMLBRqFWw9I7V8wyBIj7VDsOWqdRrK Sf7Rw87nH8GmCDn3w2nrTN7o0i2ZDgfGMm2MhoVpKoWfhCmwhxTMyZvTd w==; X-IronPort-AV: E=McAfee;i="6500,9779,10613"; a="312884040" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="312884040" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2023 07:28:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10613"; a="840381240" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="840381240" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga005.jf.intel.com with ESMTP; 06 Feb 2023 07:28:56 -0800 From: Yin Fengwei To: willy@infradead.org, linux-mm@kvack.org Cc: dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, fengwei.yin@intel.com Subject: [RFC PATCH 2/2] rmap: remove parameter 'compound' from foeio_add_file_rmap_range() Date: Mon, 6 Feb 2023 23:30:49 +0800 Message-Id: <20230206153049.770556-3-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206153049.770556-1-fengwei.yin@intel.com> References: <20230206153049.770556-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: wwkhebjwaahd7bz9df8fra9h93wbdh47 X-Rspamd-Queue-Id: 26191C001F X-HE-Tag: 1675697339-719456 X-HE-Meta: U2FsdGVkX19HgcbQP2T8bIRHGg3huGq0XQEVuJFFY2vt7N6vrfUH4wuVJH5UCyxe1e84KwxYQhfX8J5k0lis9MyEgR6w4JZEdKtqsgC6IKSHoY+8wzTFEe4KDA9g8HsFoOrqZECxlvD6sE+6EaMZ9xSjRAk/4JfIMwd8ghggJ84QgH32rRFdqLN7/pwHBvXKiXaW13Zr6mm4D9Mwy6GqnQgdhkpKjS3rC/TmdQR5rYOLAwXM9hzW5i73sau/uVU+pN1vkBqgooMfYExBZp2fVkMrVGf3nmEFGujRU2WoKCut6olCpjgZlVBCrDw4rbJ3EAhDjBJ0RrPrDF0AoLSVMzCzPk+n085U6aecanPTkcCOswCPI3yIe/QSN5BliA7I7drvCtWLiTG+gw7rHtWqCaM1BiHL9dIpJu/7M4JeCyxV266fzOH0AwABSdGG9mSb0xS9I6TFGYoGTDpDfr0dbgxkLXGnEkzRX9HhNheHjMZb6+1pWFnxXcjqfkmbMbHYcNY5XLfpuWh9RkSNglC3oTzdeVG9MViw5zL5/XSSsnb2AhkFHCBw5ezj5bpE6L72G7uxJ6CMPHp6UNaICr//KY987cuG4Q0iASxqaIlRT9o/nQTz51CY9+5xR9ham6hx3cq9YKqYnCt7IYrzAozwfPfqVdiSGa7JB7w1P1dCSBYi8X+Y/R0SENDno+VNMc+TYHVOjM/VEoIjfJir6G94mdgGF2p/hRcvkuBojgFfDC8yol/9MTSWP6wa33HBP8Ot3AXARL/CJoFthaJ/1lGnbzwXLtuAobX5s7JqRX3FC/HDZrQb6KqmbqZbXDlAPyyE8rdF59VOxp9hto8uuYPK6uuLxqf8MFoQwmNdcifk2dFdWn3opqNnaHblQCqB1P8b7PLXzu8R1hUK5Js4qqmJVYFzNpTeX7csaKYqx2KMoxv8lkNeP83ibggJYshdwYlR5GyPcshsOcBPr2CGUnM u8WAKh+r lVFDbH1GQjg7FBffTX9H8UUchsIDiY0+lxkhv9qMIm3QN0GS48yslK+buZAuYslATCeO+FzWrVyEwCYmZkR2phvRZxnvlvLtgIXA01RlFvrjFbSMe9x4Itml4lv2W9H5TNd4QXygFw7mq5YpS2kko4fpgvZryaW45igAmjjb/mkDBJYH0PXd/tV94BM+pFSDZmgcKYZg5GJ5KYgTLyH1njBUBDNi3pGxjt3nK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove parameter 'compound' from folio_add_file_rmap_range(). The parameter nr_pages is checked whether same as folio page numbers to know whether it's entire folio operation. Convert the folio _entire_mapcount to page mapcount to handle the case the folio is added as entire folio but removed by removing each page. Signed-off-by: Yin Fengwei --- include/linux/rmap.h | 2 +- mm/memory.c | 2 +- mm/rmap.c | 51 ++++++++++++++++++++++++++++---------------- 3 files changed, 35 insertions(+), 20 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 974124b41fee..00e2a229bb24 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -199,7 +199,7 @@ void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void folio_add_file_rmap_range(struct folio *, unsigned long start, - unsigned int nr_pages, struct vm_area_struct *, bool compound); + unsigned int nr_pages, struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); diff --git a/mm/memory.c b/mm/memory.c index 51f8bd91d9f0..fff1de9ac233 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4270,7 +4270,7 @@ void do_set_pte_range(struct vm_fault *vmf, struct folio *folio, pte_t entry; if (!cow) { - folio_add_file_rmap_range(folio, start, nr, vma, false); + folio_add_file_rmap_range(folio, start, nr, vma); add_mm_counter(vma->vm_mm, mm_counter_file(page), nr); } else { /* diff --git a/mm/rmap.c b/mm/rmap.c index 3ab67b33094b..23cb6983bfe7 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1308,24 +1308,23 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, * @start: The first page number in folio * @nr_pages: The number of pages which will be mapped * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page * * The page range of folio is defined by [first_page, first_page + nr_pages) * * The caller needs to hold the pte lock. */ void folio_add_file_rmap_range(struct folio *folio, unsigned long start, - unsigned int nr_pages, struct vm_area_struct *vma, - bool compound) + unsigned int nr_pages, struct vm_area_struct *vma) { atomic_t *mapped = &folio->_nr_pages_mapped; unsigned int nr_pmdmapped = 0, first; int nr = 0; + bool entire_map = folio_test_large(folio) && + (nr_pages == folio_nr_pages(folio)); - VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { + if (likely(!entire_map)) { struct page *page = folio_page(folio, start); nr_pages = min_t(unsigned int, nr_pages, @@ -1341,15 +1340,13 @@ void folio_add_file_rmap_range(struct folio *folio, unsigned long start, if (first) nr++; } while (page++, --nr_pages > 0); - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + } else { first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); + nr = nr_pages - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; @@ -1358,6 +1355,9 @@ void folio_add_file_rmap_range(struct folio *folio, unsigned long start, nr = 0; } } + + if (folio_test_pmd_mappable(folio)) + nr_pmdmapped = nr_pages; } if (nr_pmdmapped) @@ -1366,7 +1366,7 @@ void folio_add_file_rmap_range(struct folio *folio, unsigned long start, if (nr) __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr); - mlock_vma_folio(folio, vma, compound); + mlock_vma_folio(folio, vma, entire_map); } /** @@ -1390,8 +1390,8 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, else nr_pages = folio_nr_pages(folio); - folio_add_file_rmap_range(folio, folio_page_idx(folio, page), - nr_pages, vma, compound); + folio_add_file_rmap_range(folio, + folio_page_idx(folio, page), nr_pages, vma); } static void folio_remove_entire_rmap(struct folio *folio, @@ -1448,15 +1448,30 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { - last = atomic_add_negative(-1, &page->_mapcount); - nr = last; - if (last && folio_test_large(folio)) { - nr = atomic_dec_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); + if ((atomic_read(&page->_mapcount) == -1) && + folio_test_large(folio)) { + int i, nr_pages = folio_nr_pages(folio); + + for (i = 0; i < nr_pages; i++) { + struct page *pg = folio_page(folio, i); + + if (pg != page) + atomic_inc(&pg->_mapcount); + } + folio_remove_entire_rmap(folio, &nr, + &nr_pmdmapped); + + nr++; + } else { + last = atomic_add_negative(-1, &page->_mapcount); + nr = last; + if (last && folio_test_large(folio)) { + nr = atomic_dec_return_relaxed(mapped); + nr = (nr < COMPOUND_MAPPED); + } } } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ - folio_remove_entire_rmap(folio, &nr, &nr_pmdmapped); }