From patchwork Wed Feb 16 09:48:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12748344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CA4AC433F5 for ; Wed, 16 Feb 2022 09:48:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D57F6B007B; Wed, 16 Feb 2022 04:48:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8853C6B007D; Wed, 16 Feb 2022 04:48:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7861F6B007E; Wed, 16 Feb 2022 04:48:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0161.hostedemail.com [216.40.44.161]) by kanga.kvack.org (Postfix) with ESMTP id 6A0E96B007B for ; Wed, 16 Feb 2022 04:48:53 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 237CC894CE for ; Wed, 16 Feb 2022 09:48:53 +0000 (UTC) X-FDA: 79148168946.15.B5DC920 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 7E2C91A0006 for ; Wed, 16 Feb 2022 09:48:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645004932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r0OLEdIrUqYcaY6DYkFVTZPLD4+1hLYjm8QOXt2Ctgs=; b=bnWmuNK7ZH3+P2nYPLbVQMDnt3FgmCA8z4gA6xmEfVLqTRYip1lwzjAUc6gSLH3c30C+qj EA83DxdqWoohe0SaCfz9g3sAFsLzCzMl0WKMVq0QotNAJMVtgWsgnRMxhN+VKm8fw16vog em+BPCW4bp7tHW0fNEdFfFtXe+mp+NY= Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-155-5ZgI5VFnNu2HSUFcvJV9yQ-1; Wed, 16 Feb 2022 04:48:50 -0500 X-MC-Unique: 5ZgI5VFnNu2HSUFcvJV9yQ-1 Received: by mail-pj1-f71.google.com with SMTP id f6-20020a17090a654600b001b9e4758439so1104573pjs.1 for ; Wed, 16 Feb 2022 01:48:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r0OLEdIrUqYcaY6DYkFVTZPLD4+1hLYjm8QOXt2Ctgs=; b=W9Ic4mN3MexBUj4D/H41nUL+2kyRFcIoL7unDUzfYDfZ2Qidm7YrmbDH9bBfTiWsYv iOPNrK7l0RpH/+iseB0xryC1IgUY0AeY2SiXGoxoD/TVEuoyE5eNZuxTivepVhXNz8ed QO0dFyWYq8SreNNz6PQ+XDmwUpmUYYPK6+NrAc0kXdqDbjF1MQ29yjbpG5yOODKaEOfm 66XwG6PYQycYe68+FN/0+N2IPrKa2i2+vdzDSO0d1tKl3Tq+ueSl4/VCIDSu/WiaCXok vydR+gxE8s5uzB317KAZNEaV9C4UfPlaCUZEgLGfmGpwjtYpIQ08r7AIGnfDYU9IrWG6 FbAA== X-Gm-Message-State: AOAM531OJw43SqtJGNXgofHKx7J5O433MVgcwtfol8IOMZgaC1oNnyXy pxeKvI9LTJ+6Xva7sLE3U2iLzx7+4/gvBb0HjdWP2lRrTyax4v5f8EOVpE3Uc6oSaaKZwHG8wod VaRlK41g7D4c= X-Received: by 2002:a17:902:6acc:b0:149:8f60:a526 with SMTP id i12-20020a1709026acc00b001498f60a526mr2088573plt.25.1645004929853; Wed, 16 Feb 2022 01:48:49 -0800 (PST) X-Google-Smtp-Source: ABdhPJy8ro/PQOMeJ2c0+5tdk79xjZJksrx44v8baDKpAy/uaeFgFPs2vRvtMpMFMhDbUf7ALQrB3g== X-Received: by 2002:a17:902:6acc:b0:149:8f60:a526 with SMTP id i12-20020a1709026acc00b001498f60a526mr2088543plt.25.1645004929405; Wed, 16 Feb 2022 01:48:49 -0800 (PST) Received: from localhost.localdomain ([64.64.123.81]) by smtp.gmail.com with ESMTPSA id qe7sm11567835pjb.25.2022.02.16.01.48.36 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 01:48:49 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , "Kirill A . Shutemov" , Matthew Wilcox , Yang Shi , Andrea Arcangeli , peterx@redhat.com, John Hubbard , Alistair Popple , David Hildenbrand , Vlastimil Babka , Hugh Dickins Subject: [PATCH v4 1/4] mm: Don't skip swap entry even if zap_details specified Date: Wed, 16 Feb 2022 17:48:07 +0800 Message-Id: <20220216094810.60572-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216094810.60572-1-peterx@redhat.com> References: <20220216094810.60572-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bnWmuNK7; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf19.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=peterx@redhat.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7E2C91A0006 X-Stat-Signature: 4u6m6bkuz7z8o878inu7pr3npizcf6q8 X-HE-Tag: 1645004932-247898 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The "details" pointer shouldn't be the token to decide whether we should skip swap entries. For example, when the user specified details->zap_mapping==NULL, it means the user wants to zap all the pages (including COWed pages), then we need to look into swap entries because there can be private COWed pages that was swapped out. Skipping some swap entries when details is non-NULL may lead to wrongly leaving some of the swap entries while we should have zapped them. A reproducer of the problem: ===8<=== #define _GNU_SOURCE /* See feature_test_macros(7) */ #include #include #include #include #include int page_size; int shmem_fd; char *buffer; void main(void) { int ret; char val; page_size = getpagesize(); shmem_fd = memfd_create("test", 0); assert(shmem_fd >= 0); ret = ftruncate(shmem_fd, page_size * 2); assert(ret == 0); buffer = mmap(NULL, page_size * 2, PROT_READ | PROT_WRITE, MAP_PRIVATE, shmem_fd, 0); assert(buffer != MAP_FAILED); /* Write private page, swap it out */ buffer[page_size] = 1; madvise(buffer, page_size * 2, MADV_PAGEOUT); /* This should drop private buffer[page_size] already */ ret = ftruncate(shmem_fd, page_size); assert(ret == 0); /* Recover the size */ ret = ftruncate(shmem_fd, page_size * 2); assert(ret == 0); /* Re-read the data, it should be all zero */ val = buffer[page_size]; if (val == 0) printf("Good\n"); else printf("BUG\n"); } ===8<=== We don't need to touch up the pmd path, because pmd never had a issue with swap entries. For example, shmem pmd migration will always be split into pte level, and same to swapping on anonymous. Add another helper should_zap_cows() so that we can also check whether we should zap private mappings when there's no page pointer specified. This patch drops that trick, so we handle swap ptes coherently. Meanwhile we should do the same check upon migration entry, hwpoison entry and genuine swap entries too. To be explicit, we should still remember to keep the private entries if even_cows==false, and always zap them when even_cows==true. The issue seems to exist starting from the initial commit of git. Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/memory.c | 45 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c125c4969913..4bfeaca7cbc7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1313,6 +1313,17 @@ struct zap_details { struct folio *single_folio; /* Locked folio to be unmapped */ }; +/* Whether we should zap all COWed (private) pages too */ +static inline bool should_zap_cows(struct zap_details *details) +{ + /* By default, zap all pages */ + if (!details) + return true; + + /* Or, we zap COWed pages only if the caller wants to */ + return !details->zap_mapping; +} + /* * We set details->zap_mapping when we want to unmap shared but keep private * pages. Return true if skip zapping this page, false otherwise. @@ -1320,11 +1331,15 @@ struct zap_details { static inline bool zap_skip_check_mapping(struct zap_details *details, struct page *page) { - if (!details || !page) + /* If we can make a decision without *page.. */ + if (should_zap_cows(details)) return false; - return details->zap_mapping && - (details->zap_mapping != page_rmapping(page)); + /* E.g. zero page */ + if (!page) + return false; + + return details->zap_mapping != page_rmapping(page); } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1405,17 +1420,29 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } - /* If details->check_mapping, we leave swap entries. */ - if (unlikely(details)) - continue; - - if (!non_swap_entry(entry)) + if (!non_swap_entry(entry)) { + /* + * If this is a genuine swap entry, then it must be an + * private anon page. If the caller wants to skip + * COWed pages, ignore it. + */ + if (!should_zap_cows(details)) + continue; rss[MM_SWAPENTS]--; - else if (is_migration_entry(entry)) { + } else if (is_migration_entry(entry)) { struct page *page; page = pfn_swap_entry_to_page(entry); + if (zap_skip_check_mapping(details, page)) + continue; rss[mm_counter(page)]--; + } else if (is_hwpoison_entry(entry)) { + /* If the caller wants to skip COWed pages, ignore it */ + if (!should_zap_cows(details)) + continue; + } else { + /* We should have covered all the swap entry types */ + WARN_ON_ONCE(1); } if (unlikely(!free_swap_and_cache(entry))) print_bad_pte(vma, addr, ptent, NULL); From patchwork Wed Feb 16 09:48:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12748345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C237C433F5 for ; Wed, 16 Feb 2022 09:49:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D1596B007D; Wed, 16 Feb 2022 04:49:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1800F6B007E; Wed, 16 Feb 2022 04:49:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0490E6B0080; Wed, 16 Feb 2022 04:49:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0162.hostedemail.com [216.40.44.162]) by kanga.kvack.org (Postfix) with ESMTP id E9B096B007D for ; Wed, 16 Feb 2022 04:49:06 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A9B389276E for ; Wed, 16 Feb 2022 09:49:06 +0000 (UTC) X-FDA: 79148169492.18.43E81F6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 26F5F1C0002 for ; Wed, 16 Feb 2022 09:49:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645004945; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=trYMhQgsw7O8qP2aA5tHFBOoRzKbhngS39UuP85axlI=; b=Uv6trUpjNpmXncbfkASgd01XuxLuEvrZ8X/Klngc0fesEmnkEbB24+3NjGYSkc8ULuVxqn +YmNHZ8pEfgIj221l83hO56Ysnf7Q01HLhq+bYSqbHTJWHzguxHTN2wI/orsazHSNrYhOX oLQQ/QRqxrvdizxPjWezjgP4Qckf9kw= Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-241-WCcDg_R2Oy2CqrQFJaghlQ-1; Wed, 16 Feb 2022 04:49:04 -0500 X-MC-Unique: WCcDg_R2Oy2CqrQFJaghlQ-1 Received: by mail-pj1-f72.google.com with SMTP id b9-20020a17090aa58900b001b8b14b4aabso1183704pjq.9 for ; Wed, 16 Feb 2022 01:49:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=trYMhQgsw7O8qP2aA5tHFBOoRzKbhngS39UuP85axlI=; b=WVKMO1l8b/+sy8/+IxLcWaD3tnwBeX+P91KMHMqoIhFE8QnuMp5Yjq73I4UMUbwAw2 Il0yqyeY2R8KAS6H2ttV2T/Jp4Vetv0p0b4hj2H+HED42cjdqIX4PPCyBifeEMP7E3yK IOib/uipxlnzk1CL8VdiRqiTghNXW/3oByQJyYhqKqOBUXi64S1qU+K7c3mgidPyrzwW sLAqFOSbxSLmouvfRLM6rmUIV1NKvmI9FDRYK43hFeNYfFgf01mEXnqs6DFmQyaD0Kgd 06IS6vnwKotZmuEWLporeIAh9tNodAPsPeS702UFJfXAVug2y7W2wZ6EXUVX+DgNQuDv acXw== X-Gm-Message-State: AOAM530w8HTMaQg9H12oLhkGrENa3riVaIed48zYXJWrikFr5KbY4fEm My9hqPRNNVOnLaWLcxyGU7bIlciVEGWeawtM57pflFc2LHbJskHCFbhqQKaIZbsRFn/8asYHZMI Wt4NYraRyfBA= X-Received: by 2002:a17:903:228a:b0:14d:aa04:2278 with SMTP id b10-20020a170903228a00b0014daa042278mr1850392plh.58.1645004943456; Wed, 16 Feb 2022 01:49:03 -0800 (PST) X-Google-Smtp-Source: ABdhPJwp9Rn/eyC2VMT306FaqLkb/8rUoQRaxAT2awvvDCzl/7i40J6BsqMQKml7qFHRkaY20DhzAA== X-Received: by 2002:a17:903:228a:b0:14d:aa04:2278 with SMTP id b10-20020a170903228a00b0014daa042278mr1850363plh.58.1645004943140; Wed, 16 Feb 2022 01:49:03 -0800 (PST) Received: from localhost.localdomain ([64.64.123.81]) by smtp.gmail.com with ESMTPSA id qe7sm11567835pjb.25.2022.02.16.01.48.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 01:49:02 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , "Kirill A . Shutemov" , Matthew Wilcox , Yang Shi , Andrea Arcangeli , peterx@redhat.com, John Hubbard , Alistair Popple , David Hildenbrand , Vlastimil Babka , Hugh Dickins Subject: [PATCH v4 2/4] mm: Rename zap_skip_check_mapping() to should_zap_page() Date: Wed, 16 Feb 2022 17:48:08 +0800 Message-Id: <20220216094810.60572-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216094810.60572-1-peterx@redhat.com> References: <20220216094810.60572-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: i7m46tq7fy4bk4r43f4637qmq4qdpoys X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 26F5F1C0002 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Uv6trUpj; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf20.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=peterx@redhat.com X-Rspam-User: X-HE-Tag: 1645004945-330094 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The previous name is against the natural way people think. Invert the meaning and also the return value. No functional change intended. Suggested-by: David Hildenbrand Suggested-by: Hugh Dickins Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/memory.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 4bfeaca7cbc7..14d8428ff4db 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1326,20 +1326,19 @@ static inline bool should_zap_cows(struct zap_details *details) /* * We set details->zap_mapping when we want to unmap shared but keep private - * pages. Return true if skip zapping this page, false otherwise. + * pages. Return true if we should zap this page, false otherwise. */ -static inline bool -zap_skip_check_mapping(struct zap_details *details, struct page *page) +static inline bool should_zap_page(struct zap_details *details, struct page *page) { /* If we can make a decision without *page.. */ if (should_zap_cows(details)) - return false; + return true; /* E.g. zero page */ if (!page) - return false; + return true; - return details->zap_mapping != page_rmapping(page); + return details->zap_mapping == page_rmapping(page); } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1374,7 +1373,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, struct page *page; page = vm_normal_page(vma, addr, ptent); - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); @@ -1408,7 +1407,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, is_device_exclusive_entry(entry)) { struct page *page = pfn_swap_entry_to_page(entry); - if (unlikely(zap_skip_check_mapping(details, page))) + if (unlikely(!should_zap_page(details, page))) continue; pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; @@ -1433,7 +1432,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, struct page *page; page = pfn_swap_entry_to_page(entry); - if (zap_skip_check_mapping(details, page)) + if (!should_zap_page(details, page)) continue; rss[mm_counter(page)]--; } else if (is_hwpoison_entry(entry)) { From patchwork Wed Feb 16 09:48:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12748346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CD77C433F5 for ; Wed, 16 Feb 2022 09:49:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D04D6B007E; Wed, 16 Feb 2022 04:49:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1803D6B0080; Wed, 16 Feb 2022 04:49:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 048526B0081; Wed, 16 Feb 2022 04:49:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0233.hostedemail.com [216.40.44.233]) by kanga.kvack.org (Postfix) with ESMTP id EBCC66B007E for ; Wed, 16 Feb 2022 04:49:21 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A218690F7C for ; Wed, 16 Feb 2022 09:49:21 +0000 (UTC) X-FDA: 79148170122.10.AD3CF5A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 1BC481C0006 for ; Wed, 16 Feb 2022 09:49:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645004960; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oZfOtFEoi6jPUWDy50tF9fEsGXQRQqyIANlBUu6Ts5c=; b=P3W6Mhi9W7rK8w9pWo4uY4HKBi58T2YRncPcGL0fVTutLiUySYnKjtPH0HBRohIzaQ5wa5 I88G5cNWxSZDzBq/KNsx94G8abHVhL/LVUQ8gfr/Jex4awi8ihd5FwDhKWlY0RMJFQR+Vy hdcbC28JHqQvwwjuaa1/YTyZamaUseU= Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-531-OLUQymyhNX2sAVfHEKw2GQ-1; Wed, 16 Feb 2022 04:49:19 -0500 X-MC-Unique: OLUQymyhNX2sAVfHEKw2GQ-1 Received: by mail-pf1-f200.google.com with SMTP id f128-20020a623886000000b004e152a2c149so1206455pfa.5 for ; Wed, 16 Feb 2022 01:49:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oZfOtFEoi6jPUWDy50tF9fEsGXQRQqyIANlBUu6Ts5c=; b=GU+WWKZ8CZnRQrgWT6JpDrFF/oQbzwukNv/cUE1gIUCLNBgciShRi8iacZ6QMdzYat FIA9KcRqKDT33f5PMLn0LLcVnfqoMbK+cHbzYIqAbysudkGTQocOWMfkr6NNWEHmbJ++ m2QhmPYzmeFakiSIg9Hb6uP/v/i/DqWf0WqMjA9IF3d1uYw1Vo5/h+DdsqHPD1ycVZWQ 2/XgBIuTFAadh4Q3R57uqgraCikeRtbxV1+77wsl42oLt1KRNOllSn1Jw/vRf+JrpqXe rWkA65wqXFbIKlGHbGprtL34o+MaY2FiZmuo+MKoH9Kc7LaH9+5fP0pE7C2f8YHlByps byxg== X-Gm-Message-State: AOAM533xJ/n66S7gY3mdTIMoaAZgldKfHh342FRBX8hRyPWCTGezYojs m8kjDeENKmjWYr5VQCKrZojUkweTMy0yPhuulL+PBAb24M8LeLC27M02XXdZUZRiPcqNPtcNPKd QGmrzfN+16i8= X-Received: by 2002:a17:90a:e7ca:b0:1b9:66c8:982c with SMTP id kb10-20020a17090ae7ca00b001b966c8982cmr784796pjb.30.1645004958303; Wed, 16 Feb 2022 01:49:18 -0800 (PST) X-Google-Smtp-Source: ABdhPJxr7nMoK1mV1Zu2nttOBw8/qblPs/Y5s3WL1TIEPfmM1m5wytDOSBV+uQUMnFliIrLh3OSfJg== X-Received: by 2002:a17:90a:e7ca:b0:1b9:66c8:982c with SMTP id kb10-20020a17090ae7ca00b001b966c8982cmr784784pjb.30.1645004958039; Wed, 16 Feb 2022 01:49:18 -0800 (PST) Received: from localhost.localdomain ([64.64.123.81]) by smtp.gmail.com with ESMTPSA id qe7sm11567835pjb.25.2022.02.16.01.49.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 01:49:17 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , "Kirill A . Shutemov" , Matthew Wilcox , Yang Shi , Andrea Arcangeli , peterx@redhat.com, John Hubbard , Alistair Popple , David Hildenbrand , Vlastimil Babka , Hugh Dickins Subject: [PATCH v4 3/4] mm: Change zap_details.zap_mapping into even_cows Date: Wed, 16 Feb 2022 17:48:09 +0800 Message-Id: <20220216094810.60572-4-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216094810.60572-1-peterx@redhat.com> References: <20220216094810.60572-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 1BC481C0006 X-Rspam-User: Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P3W6Mhi9; spf=none (imf20.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: garq7qbwfypndn7okz16nnxrnykrt3si X-Rspamd-Server: rspam03 X-HE-Tag: 1645004960-200997 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently we have a zap_mapping pointer maintained in zap_details, when it is specified we only want to zap the pages that has the same mapping with what the caller has specified. But what we want to do is actually simpler: we want to skip zapping private (COW-ed) pages in some cases. We can refer to unmap_mapping_pages() callers where we could have passed in different even_cows values. The other user is unmap_mapping_folio() where we always want to skip private pages. According to Hugh, we used a mapping pointer for historical reason, as explained here: https://lore.kernel.org/lkml/391aa58d-ce84-9d4-d68d-d98a9c533255@google.com/ Quoting partly from Hugh: Which raises the question again of why I did not just use a boolean flag there originally: aah, I think I've found why. In those days there was a horrible "optimization", for better performance on some benchmark I guess, which when you read from /dev/zero into a private mapping, would map the zero page there (look up read_zero_pagealigned() and zeromap_page_range() if you dare). So there was another category of page to be skipped along with the anon COWs, and I didn't want multiple tests in the zap loop, so checking check_mapping against page->mapping did both. I think nowadays you could do it by checking for PageAnon page (or genuine swap entry) instead. This patch replaced the zap_details.zap_mapping pointer into the even_cows boolean, then we check it against PageAnon. Suggested-by: Hugh Dickins Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/memory.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 14d8428ff4db..ffa8c7dfe9ad 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1309,8 +1309,8 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) * Parameter block passed down to zap_pte_range in exceptional cases. */ struct zap_details { - struct address_space *zap_mapping; /* Check page->mapping if set */ struct folio *single_folio; /* Locked folio to be unmapped */ + bool even_cows; /* Zap COWed private pages too? */ }; /* Whether we should zap all COWed (private) pages too */ @@ -1321,13 +1321,10 @@ static inline bool should_zap_cows(struct zap_details *details) return true; /* Or, we zap COWed pages only if the caller wants to */ - return !details->zap_mapping; + return details->even_cows; } -/* - * We set details->zap_mapping when we want to unmap shared but keep private - * pages. Return true if we should zap this page, false otherwise. - */ +/* Decides whether we should zap this page with the page pointer specified */ static inline bool should_zap_page(struct zap_details *details, struct page *page) { /* If we can make a decision without *page.. */ @@ -1338,7 +1335,8 @@ static inline bool should_zap_page(struct zap_details *details, struct page *pag if (!page) return true; - return details->zap_mapping == page_rmapping(page); + /* Otherwise we should only zap non-anon pages */ + return !PageAnon(page); } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -3403,7 +3401,7 @@ void unmap_mapping_folio(struct folio *folio) first_index = folio->index; last_index = folio->index + folio_nr_pages(folio) - 1; - details.zap_mapping = mapping; + details.even_cows = false; details.single_folio = folio; i_mmap_lock_write(mapping); @@ -3432,7 +3430,7 @@ void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t first_index = start; pgoff_t last_index = start + nr - 1; - details.zap_mapping = even_cows ? NULL : mapping; + details.even_cows = even_cows; if (last_index < first_index) last_index = ULONG_MAX; From patchwork Wed Feb 16 09:48:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12748347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D174C433EF for ; Wed, 16 Feb 2022 09:49:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EAE36B0080; Wed, 16 Feb 2022 04:49:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 79A8A6B0081; Wed, 16 Feb 2022 04:49:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6633B6B0082; Wed, 16 Feb 2022 04:49:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0181.hostedemail.com [216.40.44.181]) by kanga.kvack.org (Postfix) with ESMTP id 571DF6B0080 for ; Wed, 16 Feb 2022 04:49:39 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 12FDA181AC9C6 for ; Wed, 16 Feb 2022 09:49:39 +0000 (UTC) X-FDA: 79148170878.25.9B6BB81 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 793B340009 for ; Wed, 16 Feb 2022 09:49:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645004978; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rxjE8nQUBN11kwzcRplXdSY7IFspDWL4Rd9u3ETZvZU=; b=F6fSmJyVuJcJcp/FrmasSucZk+m7W0TL3zK9P5lklWOES1l9f2uiCsyq9ZbkMQmRq9NOyk Y5IzoahAG2DgBteR+V5Xx0CqDwqQWv0I3X9e9m+dO9aLGRDaaUq8+r0dZXS6iWtdjrsJdO rVHR3dt+P0FwWiKLg0AYzyWgwQqUAqs= Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-669-XSpMsdOROJOMKpXMYDTEbA-1; Wed, 16 Feb 2022 04:49:34 -0500 X-MC-Unique: XSpMsdOROJOMKpXMYDTEbA-1 Received: by mail-pj1-f69.google.com with SMTP id w3-20020a17090ac98300b001b8b914e91aso1230151pjt.0 for ; Wed, 16 Feb 2022 01:49:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rxjE8nQUBN11kwzcRplXdSY7IFspDWL4Rd9u3ETZvZU=; b=hVRP3wPbXfUge4qtPYECeMhLaWywG5gYKeh9E+2nznpT42URtJBYyhXOsIOeFkBbqW 3jNYiFLC7L0J/RL18UWIE6DYC7iEONSp9ER9Poj7qcSBxzvnqfaz+QJ1HtjS+GpVy2zM iOUFaKQYVAI1FZemT4RGVY3WB9/fgIaBRH4lWpUyYLTVa6x6kZm2bfT8VrB9xTknMwRK E6wtZzEonM0tYO4GftX8Ny+C9os7/L02pmDZZMyDWnqftXuVmGSomI0Ek2iwuFiDIMii Xj/tPQKP/K8/b09waWObZldqTMkzTZT0kvQLAJFH+Goq1tSrv3q05KOTCKfckKvbqlbL z9JA== X-Gm-Message-State: AOAM530YO1m6f6JATKBOCaD0DD9zi/J77EJLnnoNrLtEfTnIjUMjzgSb FMAv/GHBkqSvGeJ1gzRuJLqiMT7vCOUoSpBpXO/Gcs88dh0ODjEbBmukPtGjf8P6KcZrH3TllN3 0IthO0ZptfMI= X-Received: by 2002:a17:902:7892:b0:14e:c520:e47d with SMTP id q18-20020a170902789200b0014ec520e47dmr1620153pll.105.1645004973346; Wed, 16 Feb 2022 01:49:33 -0800 (PST) X-Google-Smtp-Source: ABdhPJyU/EfLN18Cu4rotd2bXfEqIGUN8J9OwxkyYql/PYAkRf8rWS5DdGmn2URaeB50VX2oPQZFOw== X-Received: by 2002:a17:902:7892:b0:14e:c520:e47d with SMTP id q18-20020a170902789200b0014ec520e47dmr1620133pll.105.1645004973050; Wed, 16 Feb 2022 01:49:33 -0800 (PST) Received: from localhost.localdomain ([64.64.123.81]) by smtp.gmail.com with ESMTPSA id qe7sm11567835pjb.25.2022.02.16.01.49.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 16 Feb 2022 01:49:32 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , "Kirill A . Shutemov" , Matthew Wilcox , Yang Shi , Andrea Arcangeli , peterx@redhat.com, John Hubbard , Alistair Popple , David Hildenbrand , Vlastimil Babka , Hugh Dickins Subject: [PATCH v4 4/4] mm: Rework swap handling of zap_pte_range Date: Wed, 16 Feb 2022 17:48:10 +0800 Message-Id: <20220216094810.60572-5-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220216094810.60572-1-peterx@redhat.com> References: <20220216094810.60572-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 793B340009 X-Rspam-User: Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=F6fSmJyV; spf=none (imf07.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: ik9bgzd75qo8imqd31jo3a4fcqnkr5g6 X-Rspamd-Server: rspam11 X-HE-Tag: 1645004978-791331 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Clean the code up by merging the device private/exclusive swap entry handling with the rest, then we merge the pte clear operation too. struct* page is defined in multiple places in the function, move it upward. free_swap_and_cache() is only useful for !non_swap_entry() case, put it into the condition. No functional change intended. Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/memory.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ffa8c7dfe9ad..cade96024349 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1361,6 +1361,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = *pte; + struct page *page; + if (pte_none(ptent)) continue; @@ -1368,8 +1370,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, break; if (pte_present(ptent)) { - struct page *page; - page = vm_normal_page(vma, addr, ptent); if (unlikely(!should_zap_page(details, page))) continue; @@ -1403,21 +1403,14 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, entry = pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { - struct page *page = pfn_swap_entry_to_page(entry); - + page = pfn_swap_entry_to_page(entry); if (unlikely(!should_zap_page(details, page))) continue; - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; - if (is_device_private_entry(entry)) page_remove_rmap(page, false); - put_page(page); - continue; - } - - if (!non_swap_entry(entry)) { + } else if (!non_swap_entry(entry)) { /* * If this is a genuine swap entry, then it must be an * private anon page. If the caller wants to skip @@ -1426,9 +1419,9 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (!should_zap_cows(details)) continue; rss[MM_SWAPENTS]--; + if (unlikely(!free_swap_and_cache(entry))) + print_bad_pte(vma, addr, ptent, NULL); } else if (is_migration_entry(entry)) { - struct page *page; - page = pfn_swap_entry_to_page(entry); if (!should_zap_page(details, page)) continue; @@ -1441,8 +1434,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, /* We should have covered all the swap entry types */ WARN_ON_ONCE(1); } - if (unlikely(!free_swap_and_cache(entry))) - print_bad_pte(vma, addr, ptent, NULL); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); } while (pte++, addr += PAGE_SIZE, addr != end);