From patchwork Tue Sep 27 16:29:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chih-En Lin X-Patchwork-Id: 12990926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60BF9C54EE9 for ; Tue, 27 Sep 2022 16:28:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ECC318E00E0; Tue, 27 Sep 2022 12:28:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E7BC68E00C1; Tue, 27 Sep 2022 12:28:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF5CF8E00E0; Tue, 27 Sep 2022 12:28:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BDC0F8E00C1 for ; Tue, 27 Sep 2022 12:28:10 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9A9FB140EBF for ; Tue, 27 Sep 2022 16:28:10 +0000 (UTC) X-FDA: 79958397540.11.C0C76F3 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf29.hostedemail.com (Postfix) with ESMTP id 4AE4212000C for ; Tue, 27 Sep 2022 16:28:10 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id f23so9551552plr.6 for ; Tue, 27 Sep 2022 09:28:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=4E5FbfG/IuR+E8W1emv8V3grpVK/EnZv9IRpD1bffsk=; b=R6qS6gl1BdIlAy4Pc+DF6Tnk7r09A1Hzh3RhF+jg8a3vWDHO/Z8Z0F/8KLFmNT4UZz p/zFN3RfTqWi2DRtOVtFciqpCdIdmtpT6DJVVtxBQq6UgU6IFoSyjC2EQZwD12QjI2rP N6sKt/2wIihFM0xukfEBIwASir6D+SsWWfgEpZfNYQlep+bU6+y1VL+nU1RUxcpcZbzj HDJDak4eIuiXg+OZKVbWp7xuXqSnIxgyQjH4X/XP702GEzT5h+djQNgxeLZCe8y8DeX2 1QEo27PMnsy6vuvoS48eRrJKETldKeSTA/zrFTpSxL5TIlzzEkP0pCRtT4W1+NWE4eY6 RLZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=4E5FbfG/IuR+E8W1emv8V3grpVK/EnZv9IRpD1bffsk=; b=JG4ZUBfPPFomSubBqT7Xg5XfjVlzmDM60Rfy5vRF8x2ycgeok3irIdTYQZMmNv4MAt ZBqFAd7wM3y515dE0QZlbk93fYW51eU6rlnkpx44LBsoigUeYdzfVaFI2bG/GHrIUjys V5q/Ifmbhg86mXYE2Jg2gYeVQqlqr8MRcXLw4jgbD6cpjO8A49L3jlo4zONeqkaBYb/M SkxC3qC2rAVWMi/goYbhkxuWXGECCXyEfGj5XJ5/+TixGUNxQ7QOQlInDRsJJiVxJpxG 03H0IA8d4XDpyoBCD+V179cQ15oO//FbJl6YdXjyKGzK7xDliEZNs2Pg/VyvxU5RlO5U sfXw== X-Gm-Message-State: ACrzQf0aJEZZ2y5IJ/xIqDG9SRUZcVUSbr1tbF/RIQQy1LjDWmxfKZtl vbXfsflPrKS9fXGqljUyJCI= X-Google-Smtp-Source: AMsMyM4f2hvZ08Sspjy753JXqV501MbKNQc+IDnf+X9RFXObXCyHmtTBbnBpVaxK4l6Y+BwjCYmecg== X-Received: by 2002:a17:902:ce81:b0:179:f3fe:7fef with SMTP id f1-20020a170902ce8100b00179f3fe7fefmr2502835plg.119.1664296089240; Tue, 27 Sep 2022 09:28:09 -0700 (PDT) Received: from archlinux.localdomain ([140.121.198.213]) by smtp.googlemail.com with ESMTPSA id 9-20020a17090a0f0900b001f333fab3d6sm8602360pjy.18.2022.09.27.09.28.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Sep 2022 09:28:08 -0700 (PDT) From: Chih-En Lin To: Andrew Morton , Qi Zheng , David Hildenbrand , Matthew Wilcox , Christophe Leroy Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Luis Chamberlain , Kees Cook , Iurii Zaikin , Vlastimil Babka , William Kucharski , "Kirill A . Shutemov" , Peter Xu , Suren Baghdasaryan , Arnd Bergmann , Tong Tiangen , Pasha Tatashin , Li kunyu , Nadav Amit , Anshuman Khandual , Minchan Kim , Yang Shi , Song Liu , Miaohe Lin , Thomas Gleixner , Sebastian Andrzej Siewior , Andy Lutomirski , Fenghua Yu , Dinglan Peng , Pedro Fonseca , Jim Huang , Huichun Feng , Chih-En Lin Subject: [RFC PATCH v2 6/9] mm, pgtable: Add COW_PTE_OWNER_EXCLUSIVE flag Date: Wed, 28 Sep 2022 00:29:54 +0800 Message-Id: <20220927162957.270460-7-shiyn.lin@gmail.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220927162957.270460-1-shiyn.lin@gmail.com> References: <20220927162957.270460-1-shiyn.lin@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664296090; a=rsa-sha256; cv=none; b=LJoF/4+uDGm1FJ/oVNVle0fytqPz3ZgCUZYnklEpE9hz5DvgeEJTC+y2Peda4LzwmZORBz ND6k+bS9Blo/Mw02Hi32YydUy72tls4nbLZz3M0YoDQJurpTnRlVCwVmrZJZyQgWGwj+M7 f1/kMc/PoLXpvIe48swpU0Y/ki0OBWw= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=R6qS6gl1; spf=pass (imf29.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664296090; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4E5FbfG/IuR+E8W1emv8V3grpVK/EnZv9IRpD1bffsk=; b=hUj/MdS4gvP/7RqlerEfUREKU05RzLUhtJgE/gDdmuUhYtZUzxY9gfHIJAsdVePFyK6XN8 vpBQ3zA+mvIOciVPe3wP9OBDAuP8HcM4V6xxAUV3j5jQXNu8IzVCfZiYKwbP42V7sc/1MX uIro5KEBkPW5ueJhn9Xr8TVRYUh8WMo= X-Stat-Signature: oq8xp7uw464f93cm9ecdbxtbu38bd4h8 X-Rspamd-Queue-Id: 4AE4212000C X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=R6qS6gl1; spf=pass (imf29.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-HE-Tag: 1664296090-520199 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For present COW logic (physical page), in some situations (e.g., pinned page), we cannot share those pages. To make the COW PTE consistent with current logic, introduce the COW_PTE_OWNER_EXCLUSIVE flag to avoid doing COW to the PTE table during fork(). The following is a list of the exclusive flag used. - GUP pinnig with COW physical page will get in trouble. Currently, it will not do COW when GUP works. Follow the rule here. Signed-off-by: Chih-En Lin --- include/linux/pgtable.h | 18 ++++++++++++++++++ mm/gup.c | 13 +++++++++++-- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 8b497d7d800ed..9b08a3361d490 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -656,6 +656,24 @@ static inline int cow_pte_count(pmd_t *pmd) return page_count(pmd_page(*pmd)); } +/* Keep the first bit clear. See more detail in the comments of struct page. */ +#define COW_PTE_OWNER_EXCLUSIVE ((pmd_t *) 0x02UL) + +static inline void pmd_cow_pte_mkexclusive(pmd_t *pmd) +{ + set_cow_pte_owner(pmd, COW_PTE_OWNER_EXCLUSIVE); +} + +static inline bool pmd_cow_pte_exclusive(pmd_t *pmd) +{ + return cow_pte_owner_is_same(pmd, COW_PTE_OWNER_EXCLUSIVE); +} + +static inline void pmd_cow_pte_clear_mkexclusive(pmd_t *pmd) +{ + set_cow_pte_owner(pmd, NULL); +} + #ifndef pte_access_permitted #define pte_access_permitted(pte, write) \ (pte_present(pte) && (!(write) || pte_write(pte))) diff --git a/mm/gup.c b/mm/gup.c index 5abdaf4874605..4949c8d42a400 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -634,6 +634,11 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, mark_page_accessed(page); } out: + /* + * We don't share the PTE when any other pinned page exists. And + * let the exclusive flag stick around until the table is freed. + */ + pmd_cow_pte_mkexclusive(pmd); pte_unmap_unlock(ptep, ptl); return page; no_page: @@ -932,6 +937,7 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address, pte = pte_offset_map(pmd, address); if (pte_none(*pte)) goto unmap; + pmd_cow_pte_clear_mkexclusive(pmd); *vma = get_gate_vma(mm); if (!page) goto out; @@ -2764,8 +2770,11 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!gup_huge_pd(__hugepd(pmd_val(pmd)), addr, PMD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) - return 0; + } else { + if (!gup_pte_range(pmd, addr, next, flags, pages, nr)) + return 0; + pmd_cow_pte_mkexclusive(&pmd); + } } while (pmdp++, addr = next, addr != end); return 1;