From patchwork Fri Jan 28 04:54:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12727933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF101C433EF for ; Fri, 28 Jan 2022 04:54:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 908FA6B008C; Thu, 27 Jan 2022 23:54:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 891466B0092; Thu, 27 Jan 2022 23:54:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E4106B0093; Thu, 27 Jan 2022 23:54:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0007.hostedemail.com [216.40.44.7]) by kanga.kvack.org (Postfix) with ESMTP id 5B1A06B008C for ; Thu, 27 Jan 2022 23:54:33 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 144B1181059D9 for ; Fri, 28 Jan 2022 04:54:33 +0000 (UTC) X-FDA: 79078480026.07.D4D775A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 9AB5F180014 for ; Fri, 28 Jan 2022 04:54:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643345672; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r0OLEdIrUqYcaY6DYkFVTZPLD4+1hLYjm8QOXt2Ctgs=; b=KS0rCxbUCeGWpqdh3MnrO9A1mZgVFH+ymefxG+6bCb6t7XB9L4SS6XfjU8H5EkdD0VfLef krK/FHiO5R+zsL0DnOB+DLjdR/ZdgngzSO4Pg+/Znb8aFef7UssGBxcnj8S9/29zftmqpH sLEXpgSdTzHNStNBJZsvVq2cmWUMeJw= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-639-GO06Lr_ZMqqWuIREHUmbRw-1; Thu, 27 Jan 2022 23:54:31 -0500 X-MC-Unique: GO06Lr_ZMqqWuIREHUmbRw-1 Received: by mail-wm1-f70.google.com with SMTP id n7-20020a1c7207000000b0034ec3d8ce0aso2392468wmc.8 for ; Thu, 27 Jan 2022 20:54:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r0OLEdIrUqYcaY6DYkFVTZPLD4+1hLYjm8QOXt2Ctgs=; b=fjti5C7Xo59SI4/RPLooTy0MS7yj6rhJYt2WAT7+uoQtCp5vq0r1bC0vBv17A94GyK 2233E8OmOChpYIKkHqpv67iO9LabsKPpqqV9D3ZsU0FUUq5Tpz0QMgwfXWMx773Pzg9t eOPG3ZnGea3JajjvJTu8Qq9EtL6lmZC0junnZXGH9pSNd/MhOo2QUlYdhN0EiyN/mXzP BZcdtq6nS2qRC9zoNwtI8t/8VLtjMHveGgfoDDklaXcV9J0kO/j04Bv+5dD0igwoIEzU ZGgbJvwmASyaf10EKc8G2EHvZV3XKwVXoxydyUcBhDQ6hj7MTs4uyii9UV53k88NPyat 56SA== X-Gm-Message-State: AOAM532/sGdYyEdwm6Po2una2ft879W+pvFNxjLwiGLc7TQbO5+FuOO8 jj+y5Yk+SMKpy66rs3OeiQlC87xWmznr2yjMBf8zB87znSjv/AsRx5lo5IRi2HIlhqopueVTmwT 4NcNMdfp8GZET4l/hUYhyVcs9AeitqKRYiLEmQl+Zv/5w/YumCDuEjxy6SUhe X-Received: by 2002:a5d:6da4:: with SMTP id u4mr5486429wrs.611.1643345669693; Thu, 27 Jan 2022 20:54:29 -0800 (PST) X-Google-Smtp-Source: ABdhPJzNjlbcr557L0JzcCx8cJn1ypoGGBxeXy3L3cPs1G3/t3aoCJ8N911EUy9hAc8Wvze/o8vnfQ== X-Received: by 2002:a5d:6da4:: with SMTP id u4mr5486406wrs.611.1643345669412; Thu, 27 Jan 2022 20:54:29 -0800 (PST) Received: from localhost.localdomain ([64.64.123.9]) by smtp.gmail.com with ESMTPSA id i13sm814014wrf.3.2022.01.27.20.54.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Jan 2022 20:54:29 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Alistair Popple , Andrew Morton , Andrea Arcangeli , David Hildenbrand , Matthew Wilcox , John Hubbard , Hugh Dickins , Vlastimil Babka , Yang Shi , "Kirill A . Shutemov" Subject: [PATCH v3 1/4] mm: Don't skip swap entry even if zap_details specified Date: Fri, 28 Jan 2022 12:54:09 +0800 Message-Id: <20220128045412.18695-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220128045412.18695-1-peterx@redhat.com> References: <20220128045412.18695-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 9AB5F180014 X-Stat-Signature: 6ohckoek5ugmq9mxghm5i5dsuiwr3wnb Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KS0rCxbU; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf06.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=peterx@redhat.com X-Rspam-User: nil X-HE-Tag: 1643345672-941044 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The "details" pointer shouldn't be the token to decide whether we should skip swap entries. For example, when the user specified details->zap_mapping==NULL, it means the user wants to zap all the pages (including COWed pages), then we need to look into swap entries because there can be private COWed pages that was swapped out. Skipping some swap entries when details is non-NULL may lead to wrongly leaving some of the swap entries while we should have zapped them. A reproducer of the problem: ===8<=== #define _GNU_SOURCE /* See feature_test_macros(7) */ #include #include #include #include #include int page_size; int shmem_fd; char *buffer; void main(void) { int ret; char val; page_size = getpagesize(); shmem_fd = memfd_create("test", 0); assert(shmem_fd >= 0); ret = ftruncate(shmem_fd, page_size * 2); assert(ret == 0); buffer = mmap(NULL, page_size * 2, PROT_READ | PROT_WRITE, MAP_PRIVATE, shmem_fd, 0); assert(buffer != MAP_FAILED); /* Write private page, swap it out */ buffer[page_size] = 1; madvise(buffer, page_size * 2, MADV_PAGEOUT); /* This should drop private buffer[page_size] already */ ret = ftruncate(shmem_fd, page_size); assert(ret == 0); /* Recover the size */ ret = ftruncate(shmem_fd, page_size * 2); assert(ret == 0); /* Re-read the data, it should be all zero */ val = buffer[page_size]; if (val == 0) printf("Good\n"); else printf("BUG\n"); } ===8<=== We don't need to touch up the pmd path, because pmd never had a issue with swap entries. For example, shmem pmd migration will always be split into pte level, and same to swapping on anonymous. Add another helper should_zap_cows() so that we can also check whether we should zap private mappings when there's no page pointer specified. This patch drops that trick, so we handle swap ptes coherently. Meanwhile we should do the same check upon migration entry, hwpoison entry and genuine swap entries too. To be explicit, we should still remember to keep the private entries if even_cows==false, and always zap them when even_cows==true. The issue seems to exist starting from the initial commit of git. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Signed-off-by: Peter Xu --- mm/memory.c | 45 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c125c4969913..4bfeaca7cbc7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1313,6 +1313,17 @@ struct zap_details { struct folio *single_folio; /* Locked folio to be unmapped */ }; +/* Whether we should zap all COWed (private) pages too */ +static inline bool should_zap_cows(struct zap_details *details) +{ + /* By default, zap all pages */ + if (!details) + return true; + + /* Or, we zap COWed pages only if the caller wants to */ + return !details->zap_mapping; +} + /* * We set details->zap_mapping when we want to unmap shared but keep private * pages. Return true if skip zapping this page, false otherwise. @@ -1320,11 +1331,15 @@ struct zap_details { static inline bool zap_skip_check_mapping(struct zap_details *details, struct page *page) { - if (!details || !page) + /* If we can make a decision without *page.. */ + if (should_zap_cows(details)) return false; - return details->zap_mapping && - (details->zap_mapping != page_rmapping(page)); + /* E.g. zero page */ + if (!page) + return false; + + return details->zap_mapping != page_rmapping(page); } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1405,17 +1420,29 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } - /* If details->check_mapping, we leave swap entries. */ - if (unlikely(details)) - continue; - - if (!non_swap_entry(entry)) + if (!non_swap_entry(entry)) { + /* + * If this is a genuine swap entry, then it must be an + * private anon page. If the caller wants to skip + * COWed pages, ignore it. + */ + if (!should_zap_cows(details)) + continue; rss[MM_SWAPENTS]--; - else if (is_migration_entry(entry)) { + } else if (is_migration_entry(entry)) { struct page *page; page = pfn_swap_entry_to_page(entry); + if (zap_skip_check_mapping(details, page)) + continue; rss[mm_counter(page)]--; + } else if (is_hwpoison_entry(entry)) { + /* If the caller wants to skip COWed pages, ignore it */ + if (!should_zap_cows(details)) + continue; + } else { + /* We should have covered all the swap entry types */ + WARN_ON_ONCE(1); } if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); From patchwork Fri Jan 28 04:54:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12727934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60B6AC433F5 for ; Fri, 28 Jan 2022 04:54:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C14DA6B0092; Thu, 27 Jan 2022 23:54:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BCF2F6B0093; Thu, 27 Jan 2022 23:54:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3FB46B0095; Thu, 27 Jan 2022 23:54:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0052.hostedemail.com [216.40.44.52]) by kanga.kvack.org (Postfix) with ESMTP id 92E036B0092 for ; Thu, 27 Jan 2022 23:54:38 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4F44695260 for ; Fri, 28 Jan 2022 04:54:38 +0000 (UTC) X-FDA: 79078480236.04.206B14F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id CE92440014 for ; Fri, 28 Jan 2022 04:54:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643345677; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X4WXHh7pV56dMHSpF/O5K7LVTClsvKIQ79sBisfSRRA=; b=NwLxpM3xsdpQRDDPRuXsKZsvIwme0k4TWM+FtVyDrXervUyNetKC5XrrKb3DP0MP11WjOc 5CjN0BKwYpEQ2zUMblNGedazIW8hhBl0JL0ddU30EkGy3Bh77qYhdWy60IdolzfUanV2v7 nWW8/qLgNScGCEe+kaeucWEd/SrN/sk= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-498-arKQiqOjONy13JKnoPSofA-1; Thu, 27 Jan 2022 23:54:36 -0500 X-MC-Unique: arKQiqOjONy13JKnoPSofA-1 Received: by mail-wr1-f70.google.com with SMTP id k7-20020adfb347000000b001dd761d46c7so92131wrd.4 for ; Thu, 27 Jan 2022 20:54:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X4WXHh7pV56dMHSpF/O5K7LVTClsvKIQ79sBisfSRRA=; b=dJIbfc8LK7dpOI8pGov5SekNHG3/Z9GEQvWub+w1setarzFYJ/tlACd5NV7d9GMRH5 DhszqHJIop0mQUVYrs4cEU/x80Bheiur/XJXKoccvAkT7eibDKr7yA1h4YeLIjJoJYN7 x8iuQoGo7n1x9hkBLLA8rfqfU8EhPt2exmU3yhR7HCHeRTa8tGamnUcPqagWa87TDzQH M1nx4mj86EPaQ4WqjKZgODUWL1LVE9j85oK5PxECpejLSP0BEO7GCdxQq8rLmFlnGCST /QC2uvtNdvz82ytUiFd7lLnVcHDqlz5rlM8F2Msaj4LRUmVwBiTleYyoqwKLIU7COH6f tlNQ== X-Gm-Message-State: AOAM530B6YCVJzuy/TvOBEszvJT+LeABZAMzllshTO7Xn8KwJ2muTFj+ +e8R1IjAqH+oqAfxqMuewwE6oUum0/u0frr57I5leFFUc8dJZkzv+w9saWHFeRlSZ2kUnaAKSP1 +E04D6yT7n2IZo33C8XLWNAhYuPmwPi7o81Fd0TjkqmDEqHIDTrh5lhqmRxsk X-Received: by 2002:a1c:a90d:: with SMTP id s13mr5787863wme.32.1643345674676; Thu, 27 Jan 2022 20:54:34 -0800 (PST) X-Google-Smtp-Source: ABdhPJzUZOPum7MMLETcw25d/iBx+n7SScJ596aZWjbvcjraz7IAo8lenVlYxR/jtIOkJT4M5mmwpg== X-Received: by 2002:a1c:a90d:: with SMTP id s13mr5787840wme.32.1643345674342; Thu, 27 Jan 2022 20:54:34 -0800 (PST) Received: from localhost.localdomain ([64.64.123.9]) by smtp.gmail.com with ESMTPSA id i13sm814014wrf.3.2022.01.27.20.54.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Jan 2022 20:54:34 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Alistair Popple , Andrew Morton , Andrea Arcangeli , David Hildenbrand , Matthew Wilcox , John Hubbard , Hugh Dickins , Vlastimil Babka , Yang Shi , "Kirill A . Shutemov" Subject: [PATCH v3 2/4] mm: Rename zap_skip_check_mapping() to should_zap_page() Date: Fri, 28 Jan 2022 12:54:10 +0800 Message-Id: <20220128045412.18695-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220128045412.18695-1-peterx@redhat.com> References: <20220128045412.18695-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: nil X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: CE92440014 X-Stat-Signature: qdxkg3eda366wgmbcc1xz4o5u7acn8a8 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NwLxpM3x; spf=none (imf04.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1643345677-657828 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The previous name is against the natural way people think. Invert the meaning and also the return value. No functional change intended. Suggested-by: Hugh Dickins Signed-off-by: Peter Xu Reviewed-by: David Hildenbrand --- mm/memory.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 4bfeaca7cbc7..14d8428ff4db 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1326,20 +1326,19 @@ static inline bool should_zap_cows(struct zap_details *details) /* * We set details->zap_mapping when we want to unmap shared but keep private - * pages. Return true if skip zapping this page, false otherwise. + * pages. Return true if we should zap this page, false otherwise. */ -static inline bool -zap_skip_check_mapping(struct zap_details *details, struct page *page) +static inline bool should_zap_page(struct zap_details *details, struct page *page) { /* If we can make a decision without *page.. */ if (should_zap_cows(details)) - return false; + return true; /* E.g. zero page */ if (!page) - return false; + return true; - return details->zap_mapping != page_rmapping(page); + return details->zap_mapping == page_rmapping(page); } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1374,7 +1373,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, struct page *page; page = vm_normal_page(vma, addr, ptent); - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); @@ -1408,7 +1407,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, is_device_exclusive_entry(entry)) { struct page *page = pfn_swap_entry_to_page(entry); - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; @@ -1433,7 +1432,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, struct page *page; page = pfn_swap_entry_to_page(entry); - if (zap_skip_check_mapping(details, page)) + if (!should_zap_page(details, page)) continue; rss[mm_counter(page)]--; } else if (is_hwpoison_entry(entry)) { From patchwork Fri Jan 28 04:54:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12727935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 214C8C433EF for ; Fri, 28 Jan 2022 04:54:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADC866B0093; Thu, 27 Jan 2022 23:54:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A64F26B0095; Thu, 27 Jan 2022 23:54:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B7A86B0096; Thu, 27 Jan 2022 23:54:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id 7A99B6B0093 for ; Thu, 27 Jan 2022 23:54:44 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 2E5DF8249980 for ; Fri, 28 Jan 2022 04:54:44 +0000 (UTC) X-FDA: 79078480488.22.340333F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id B47F2120012 for ; Fri, 28 Jan 2022 04:54:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643345683; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H1DVV/S8YSQjtZLKWoD0HN3kRcNDUylmDF/rAE9teCo=; b=fbtW0ZdNo98b36St0hQUTiMiA+4nGu0r4EgdzUkbg/Au67uaNOG5hjnjjkdu3pNys3OxZ0 K+FmjRdg4bBEcrSM8UMsn1nnRQoxxcTftcfFqf1l7HQjNNixCcHdGOrU2SB7Kk2hmtXLJ3 R/dkOccZb5Z5/GsR/sDwV6pEeMKgusk= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-106-Wquw1MOWMuKEFsJ9IbIHJA-1; Thu, 27 Jan 2022 23:54:42 -0500 X-MC-Unique: Wquw1MOWMuKEFsJ9IbIHJA-1 Received: by mail-wm1-f69.google.com with SMTP id s1-20020a1ca901000000b0034ece94dd8cso5308317wme.5 for ; Thu, 27 Jan 2022 20:54:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H1DVV/S8YSQjtZLKWoD0HN3kRcNDUylmDF/rAE9teCo=; b=C3CBKGwVN5/aF9LJvGEe0OxTz0fVhuOVWwGReY9VzpYS6SFC7EhckhTV7w65lXM0HE XobPzIk18u6K7zGgDBhxDXRxLYbId3o06XgIe5DDeRey7Tvc2SDpldx8DGRE05od2PfQ xS2iNSaaSuSqBERLEDIQLAhXDCwV2CKBjD82gVVV3R01WE355eksVpyv1xSEK2V5yNpY jTgHTQhM/OlMbAoAOCOhJxQlNNdDGH/lU1dJ2uErHtmOMe4bT2oLvBR5XYpJB9J39fgV Im9pbCPvC/SztgY1m5JB4kIudYtFuPyhlmzyQy0D9+OFnMmyQNvPxMhvJIcIkZ+V9qnB DpwQ== X-Gm-Message-State: AOAM531hh+81h0wEdvc+IYcwDHs6oEcEMzUEpCThgUPF9xPIrNXkCv6n cdAdRvgfRrBLp2Lksf9msmLHr86Vp45Sd+V021zbDnO/qHMM2+QX5gi2831KRBoNYq1zAr3jhEk aAJ88ELrIOP8ZuCP+1GuF/wxjuoSX+PwAsA6DVOqgIVYGZQFg7CjynJChaLh5 X-Received: by 2002:a1c:2b06:: with SMTP id r6mr14345447wmr.4.1643345680401; Thu, 27 Jan 2022 20:54:40 -0800 (PST) X-Google-Smtp-Source: ABdhPJxSmguUCLAJc3uPMchWo2O4qWDyEBL0Tp/mOd4YhtXSNiPDsTt43pf61km0GV/X6qLuqocw1g== X-Received: by 2002:a1c:2b06:: with SMTP id r6mr14345423wmr.4.1643345680044; Thu, 27 Jan 2022 20:54:40 -0800 (PST) Received: from localhost.localdomain ([64.64.123.9]) by smtp.gmail.com with ESMTPSA id i13sm814014wrf.3.2022.01.27.20.54.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Jan 2022 20:54:39 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Alistair Popple , Andrew Morton , Andrea Arcangeli , David Hildenbrand , Matthew Wilcox , John Hubbard , Hugh Dickins , Vlastimil Babka , Yang Shi , "Kirill A . Shutemov" Subject: [PATCH v3 3/4] mm: Change zap_details.zap_mapping into even_cows Date: Fri, 28 Jan 2022 12:54:11 +0800 Message-Id: <20220128045412.18695-4-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220128045412.18695-1-peterx@redhat.com> References: <20220128045412.18695-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B47F2120012 X-Stat-Signature: q9ztnypktp4xjij4gupcj8kgpxpxb9ti Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fbtW0ZdN; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf29.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-Rspam-User: nil X-HE-Tag: 1643345683-63483 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently we have a zap_mapping pointer maintained in zap_details, when it is specified we only want to zap the pages that has the same mapping with what the caller has specified. But what we want to do is actually simpler: we want to skip zapping private (COW-ed) pages in some cases. We can refer to unmap_mapping_pages() callers where we could have passed in different even_cows values. The other user is unmap_mapping_folio() where we always want to skip private pages. According to Hugh, we used a mapping pointer for historical reason, as explained here: https://lore.kernel.org/lkml/391aa58d-ce84-9d4-d68d-d98a9c533255@google.com/ Quotting partly from Hugh: Which raises the question again of why I did not just use a boolean flag there originally: aah, I think I've found why. In those days there was a horrible "optimization", for better performance on some benchmark I guess, which when you read from /dev/zero into a private mapping, would map the zero page there (look up read_zero_pagealigned() and zeromap_page_range() if you dare). So there was another category of page to be skipped along with the anon COWs, and I didn't want multiple tests in the zap loop, so checking check_mapping against page->mapping did both. I think nowadays you could do it by checking for PageAnon page (or genuine swap entry) instead. This patch replaced the zap_details.zap_mapping pointer into the even_cows boolean, then we check it against PageAnon. Suggested-by: Hugh Dickins Signed-off-by: Peter Xu --- mm/memory.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 14d8428ff4db..ffa8c7dfe9ad 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1309,8 +1309,8 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) * Parameter block passed down to zap_pte_range in exceptional cases. */ struct zap_details { - struct address_space *zap_mapping; /* Check page->mapping if set */ struct folio *single_folio; /* Locked folio to be unmapped */ + bool even_cows; /* Zap COWed private pages too? */ }; /* Whether we should zap all COWed (private) pages too */ @@ -1321,13 +1321,10 @@ static inline bool should_zap_cows(struct zap_details *details) return true; /* Or, we zap COWed pages only if the caller wants to */ - return !details->zap_mapping; + return details->even_cows; } -/* - * We set details->zap_mapping when we want to unmap shared but keep private - * pages. Return true if we should zap this page, false otherwise. - */ +/* Decides whether we should zap this page with the page pointer specified */ static inline bool should_zap_page(struct zap_details *details, struct page *page) { /* If we can make a decision without *page.. */ @@ -1338,7 +1335,8 @@ static inline bool should_zap_page(struct zap_details *details, struct page *pag if (!page) return true; - return details->zap_mapping == page_rmapping(page); + /* Otherwise we should only zap non-anon pages */ + return !PageAnon(page); } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -3403,7 +3401,7 @@ void unmap_mapping_folio(struct folio *folio) first_index = folio->index; last_index = folio->index + folio_nr_pages(folio) - 1; - details.zap_mapping = mapping; + details.even_cows = false; details.single_folio = folio; i_mmap_lock_write(mapping); @@ -3432,7 +3430,7 @@ void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t first_index = start; pgoff_t last_index = start + nr - 1; - details.zap_mapping = even_cows ? NULL : mapping; + details.even_cows = even_cows; if (last_index < first_index) last_index = ULONG_MAX; From patchwork Fri Jan 28 04:54:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12727936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CED54C433FE for ; Fri, 28 Jan 2022 04:54:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 620F66B0095; Thu, 27 Jan 2022 23:54:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A9F66B0096; Thu, 27 Jan 2022 23:54:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44B966B0098; Thu, 27 Jan 2022 23:54:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id 354AE6B0095 for ; Thu, 27 Jan 2022 23:54:50 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id EEAB08909D for ; Fri, 28 Jan 2022 04:54:49 +0000 (UTC) X-FDA: 79078480698.05.2A93720 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 6876418000E for ; Fri, 28 Jan 2022 04:54:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1643345688; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rxjE8nQUBN11kwzcRplXdSY7IFspDWL4Rd9u3ETZvZU=; b=DpPVS8bRsgEYbjmaC38U/rf8j4I/NsAvg1E7UMlIYqKiegCGXC+aOf15udJmkwuSt7uotI wor6Hsp1DdwcUQmRTVT7fk+0lBzDqZ0OwRWPExQjI2WJXGDUFmNLsHrZwdvID36/dL0Wkh ssVIGnbxEngY/O8qg7SQ0MW2pwhSq1I= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-358-LhNsHIXBP4qDOW5i57j4jg-1; Thu, 27 Jan 2022 23:54:47 -0500 X-MC-Unique: LhNsHIXBP4qDOW5i57j4jg-1 Received: by mail-wm1-f72.google.com with SMTP id j6-20020a05600c1c0600b0034c02775da7so1140485wms.3 for ; Thu, 27 Jan 2022 20:54:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rxjE8nQUBN11kwzcRplXdSY7IFspDWL4Rd9u3ETZvZU=; b=vTcj9ea83uxY/+iXszA8n4VKxTNVe1yhI+/j/H1h8q+mT9sBsAqY5kl8JcPwawdZRS +PI2Bn8fphjjGhOhBqfB7RZPU3XGLUhSM50N2WTi3P8RYVoU+Fz/q0+xVw/MUhTW6q+y UXQogRbDqngd1CR0xjQJx/d0UPnp3/g6gsZmTDPjFoZwsxaniUxubi3oFigmE9i1j9YW CyxYZm8p0mDUNSV+MROjO4kf/yoS2Uvway5RUWGUdMBINdMsbhr70MyZSHpMbjLYBPg0 Obj0ycZWowfQvNTXSn1orGiPPlXRqGVDAM+Sit7FE0uvqNo5F8bBbMRcJy7JAiqSDrJQ ValA== X-Gm-Message-State: AOAM531aK+GAG1atg0gmNlVNcdXC2X7q7TgWVxRTgHPhftyLMuRmtF/r SHIh+SzJ/VOm5qje16yRXcfaC2+m2SpkYJ5bOS/doEYkfB3I1GY7StAogc6K8Mrb64enzeOBAX4 yLMnbXrhREfvrlo18JVqt4JRy+/ZFvus1wx5X70dVYHhjWgGI0YGTJ2R0EPtU X-Received: by 2002:a05:6000:1a89:: with SMTP id f9mr5884829wry.251.1643345685554; Thu, 27 Jan 2022 20:54:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJzvpMqKTZBmuLjr5YO2EFIJM9G9DWCeuPiKqQMCAYXt0G0OqDw+Xws+L0tBDAR/T/8LmGetKw== X-Received: by 2002:a05:6000:1a89:: with SMTP id f9mr5884805wry.251.1643345685283; Thu, 27 Jan 2022 20:54:45 -0800 (PST) Received: from localhost.localdomain ([64.64.123.9]) by smtp.gmail.com with ESMTPSA id i13sm814014wrf.3.2022.01.27.20.54.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 27 Jan 2022 20:54:45 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Alistair Popple , Andrew Morton , Andrea Arcangeli , David Hildenbrand , Matthew Wilcox , John Hubbard , Hugh Dickins , Vlastimil Babka , Yang Shi , "Kirill A . Shutemov" Subject: [PATCH v3 4/4] mm: Rework swap handling of zap_pte_range Date: Fri, 28 Jan 2022 12:54:12 +0800 Message-Id: <20220128045412.18695-5-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220128045412.18695-1-peterx@redhat.com> References: <20220128045412.18695-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: nil X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6876418000E X-Stat-Signature: sw5e8e85cb7hmemqgu35tw71cy6nrgba Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DpPVS8bR; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf06.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-HE-Tag: 1643345689-914726 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Clean the code up by merging the device private/exclusive swap entry handling with the rest, then we merge the pte clear operation too. struct* page is defined in multiple places in the function, move it upward. free_swap_and_cache() is only useful for !non_swap_entry() case, put it into the condition. No functional change intended. Signed-off-by: Peter Xu --- mm/memory.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ffa8c7dfe9ad..cade96024349 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1361,6 +1361,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = *pte; + struct page *page; + if (pte_none(ptent)) continue; @@ -1368,8 +1370,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, break; if (pte_present(ptent)) { - struct page *page; - page = vm_normal_page(vma, addr, ptent); if (unlikely(!should_zap_page(details, page))) continue; @@ -1403,21 +1403,14 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, entry = pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { - struct page *page = pfn_swap_entry_to_page(entry); - + page = pfn_swap_entry_to_page(entry); if (unlikely(!should_zap_page(details, page))) continue; - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; - if (is_device_private_entry(entry)) page_remove_rmap(page, false); - put_page(page); - continue; - } - - if (!non_swap_entry(entry)) { + } else if (!non_swap_entry(entry)) { /* * If this is a genuine swap entry, then it must be an * private anon page. If the caller wants to skip @@ -1426,9 +1419,9 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (!should_zap_cows(details)) continue; rss[MM_SWAPENTS]--; + if (unlikely(!free_swap_and_cache(entry))) + print_bad_pte(vma, addr, ptent, NULL); } else if (is_migration_entry(entry)) { - struct page *page; - page = pfn_swap_entry_to_page(entry); if (!should_zap_page(details, page)) continue; @@ -1441,8 +1434,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, /* We should have covered all the swap entry types */ WARN_ON_ONCE(1); } - if (unlikely(!free_swap_and_cache(entry))) - print_bad_pte(vma, addr, ptent, NULL); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); } while (pte++, addr += PAGE_SIZE, addr != end);