From patchwork Mon Dec 18 07:06:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Yu X-Patchwork-Id: 13496284 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF51FC35274 for ; Mon, 18 Dec 2023 07:06:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C2F76B0082; Mon, 18 Dec 2023 02:06:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 571E26B008A; Mon, 18 Dec 2023 02:06:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 460D46B008C; Mon, 18 Dec 2023 02:06:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 338496B0082 for ; Mon, 18 Dec 2023 02:06:44 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 04AB3140939 for ; Mon, 18 Dec 2023 07:06:43 +0000 (UTC) X-FDA: 81579056328.05.20DCA1E Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) by imf11.hostedemail.com (Postfix) with ESMTP id 881A540005 for ; Mon, 18 Dec 2023 07:06:40 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf11.hostedemail.com: domain of xuyu@linux.alibaba.com designates 115.124.30.132 as permitted sender) smtp.mailfrom=xuyu@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702883202; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4q7rDA3+HDyO0BnBZwU46CdStWZm7aDSlwsQuGWu+lE=; b=da/Hg4QxVo+WwuJ3mDk/FqNS2N6DO1f3CLKRrc3ZqGcBzmieoY0N6lwg4qEceacUJDEVZs vcg5pd1LLfQa81DGD5CzNowXHi9LY5r9rXHlfd5xiCBJJQ8NdO35mnINh0drAZGfciKPOW 8GazrDz0rRPHEDOjxZk2umlWSxEevpc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf11.hostedemail.com: domain of xuyu@linux.alibaba.com designates 115.124.30.132 as permitted sender) smtp.mailfrom=xuyu@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702883202; a=rsa-sha256; cv=none; b=0iIFtszcykRTZslVuqLZ5wjJCpVTDSilUXU8aFAXzXdhIjmjw9HgJFo2mEIGua/cBzwPHy nR+Ip0dlDf8Iptd04XTmdfE03Sv5PFW1R1y5nKUVCKcNSnRQLJz6y7Epu08UJbNCNlQlyP rktUIyxfZkMx30JKQXQ0OLro8CK/3qI= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=xuyu@linux.alibaba.com;NM=1;PH=DS;RN=2;SR=0;TI=SMTPD_---0VygwMCR_1702883196; Received: from localhost(mailfrom:xuyu@linux.alibaba.com fp:SMTPD_---0VygwMCR_1702883196) by smtp.aliyun-inc.com; Mon, 18 Dec 2023 15:06:37 +0800 From: Xu Yu To: linux-mm@kvack.org Cc: david@redhat.com Subject: [PATCH v3 2/2] mm/khugepaged: map exclusive anonymous pte-mapped THPs by pmds Date: Mon, 18 Dec 2023 15:06:33 +0800 Message-Id: X-Mailer: git-send-email 2.37.1 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 881A540005 X-Stat-Signature: 1priss6bhspmf479ueh3b8qaex7fyxwa X-Rspam-User: X-HE-Tag: 1702883200-995095 X-HE-Meta: U2FsdGVkX18/ncwz/VIH0P5Gd+2J6PxkTgqhgSI8cMSvLrCY1+LVeFMfEoM2A900UXMzO45dltn58sVg5d6pHx9JmHExUTu2aSleIfGWiovTKYGrmB/MgzLQrLcqrcF6Qf9TCnafwtU3uuz2HjIkA6EOYhXTJLstcW8kCQoekKhKKyN3/dLbyMJ8rIDUfUC3t9nO5QuEpraCwcHkinDHTXE3adzdIyNQ7Es0uX1avfgc6d0m4Z413grw0vt8wB8JNrOEIulskFrnYb4x+GHLjlBIS21Q1jBm/VKU588xAMpYSKVHtHT40slrkNQhVO3tBM2x7qzgWEd+DP+pRAuHcYggx4vvpZkeAUvRebAx2VwN4AJbKLK8dHUMH0BtyBRn7d2gx5qNcO5tTAIQYZvUiq/wzUhTUEwfHr8V9nXabGFuNnanel9tBAjBAWyHQuH4LZNZ84FxVbghJrN+OIbCWhg5UuAN/qaha/d9RvzSU6SV0rLoQRPGRXD5fM9TQVuE6ERFKY/W6uFwCS3iQPAI8IqOIKfAZUR6QRUAgspa3hJeGV3M+bgnwMX6E8DvFYwcNMcKBfKZAn+BHAlN7ECCvYiknS8JDcUjpUhANyeyZmVenZfU/MBjuwOykxTLJOPpCtlD97RcvuqN68g83pE48jSKKnS4ujVPu3CQALQLuXSdbCyXTCBJilfSwb7MgyBo6vi769niceVs0K1d28w3w8qvIsH7UJd7tJkg+JpI+mLxA/0fcunIucvJUDJDqgFuQq78qRuJmyWMz11ZMeul4mBwKmwOCYjeXFOJ+93pzmGR/cyBL6zwuXXrfevChx2Md1JclHjodkK6t9vvObEedBmV/IC9LV0WiQWdyKEQ0X377RfxGiBjQPiTNJPUMaQIQJmYXzlbKBaUwuHvrgL5t/3vocIDqvAmm2BoMDsZ6i25+AOXToy9Bg4x0XqWqiYLIvknvzFMwT0x1mHJ6W6 MNwsIYyh ioRViT+JUO7dgNluA73toAvOQ8CPZnqXuVP0o8ZgHi8z3dWv5sE6ir3cOd/7lzaPGrUWjnNUSqdZDfhzvGlbYpOFu92wB9iQCh2Hs2JlTyR0x/RI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This adds another case, as David points out, which is suitable for mapping anonymous pte-mapped THPs by pmds. When all subpages are PageAnonExclusive (PTEs may be either R/O or R/W), we can clear PageAnonExclusive on all tail pages but the first (head) page and collapse to a R/W PMD with VM_WRITE or a R/O PMD without VM_WRITE. Signed-off-by: Xu Yu --- mm/khugepaged.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 57e261387124..d843c1e3ec39 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1285,6 +1285,7 @@ static int collapse_pte_mapped_anon_thp(struct mm_struct *mm, unsigned long addr; int exclusive = 0; bool writable = false; + rmap_t rmap_flags = RMAP_COMPOUND; int result, i; /* Fast check before locking folio if already PMD-mapped */ @@ -1366,8 +1367,14 @@ static int collapse_pte_mapped_anon_thp(struct mm_struct *mm, * Case 1: * No subpages are PageAnonExclusive (PTEs must be R/O), we can * collapse into a R/O PMD without further action. + * + * Case 2: + * All subpages are PageAnonExclusive (PTEs may be either R/O or R/W), + * we clear PageAnonExclusive on all tail pages but the head page and + * collapse to a R/W PMD with VM_WRITE or a R/O PMD without VM_WRITE. */ - if (!(exclusive == 0 && !writable)) + if (!((exclusive == 0 && !writable) || + (exclusive == HPAGE_PMD_NR))) goto up_write; /* Collapse pmd entry */ @@ -1390,7 +1397,9 @@ static int collapse_pte_mapped_anon_thp(struct mm_struct *mm, * false-negative page_mapped(). */ folio_get(folio); - page_add_anon_rmap(&folio->page, vma, haddr, RMAP_COMPOUND); + if (exclusive == HPAGE_PMD_NR) + rmap_flags |= RMAP_EXCLUSIVE; + page_add_anon_rmap(&folio->page, vma, haddr, rmap_flags); start_pte = pte_offset_map_lock(mm, &pmdval, haddr, &ptl); for (i = 0, addr = haddr, pte = start_pte; @@ -1401,6 +1410,10 @@ static int collapse_pte_mapped_anon_thp(struct mm_struct *mm, ptep_clear(mm, addr, pte); subpage = vm_normal_page(vma, addr, pteval); page_remove_rmap(subpage, vma, false); + + /* Clear PageAnonExclusive on tail pages */ + if (exclusive == HPAGE_PMD_NR && i != 0) + ClearPageAnonExclusive(subpage); } pte_unmap_unlock(start_pte, ptl); folio_ref_sub(folio, HPAGE_PMD_NR); @@ -1408,6 +1421,8 @@ static int collapse_pte_mapped_anon_thp(struct mm_struct *mm, /* Install pmd entry */ pgtable = pmd_pgtable(pmdval); pmdval = mk_huge_pmd(&folio->page, vma->vm_page_prot); + if (exclusive == HPAGE_PMD_NR) + pmdval = maybe_pmd_mkwrite(pmd_mkdirty(pmdval), vma); spin_lock(pml); pgtable_trans_huge_deposit(mm, pmd, pgtable); set_pmd_at(mm, haddr, pmd, pmdval); @@ -1595,7 +1610,9 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, out_unmap: pte_unmap_unlock(pte, ptl); - if (is_hpage && (exclusive == 0 && !writable)) { + if (is_hpage && + ((exclusive == 0 && !writable) || + (exclusive == HPAGE_PMD_NR))) { int res; res = collapse_pte_mapped_anon_thp(mm, vma, address,