From patchwork Wed Nov 10 08:29:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12611563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8930C433EF for ; Wed, 10 Nov 2021 08:30:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8CC7D610A3 for ; Wed, 10 Nov 2021 08:30:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8CC7D610A3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 346A56B0071; Wed, 10 Nov 2021 03:30:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2CDBC6B0072; Wed, 10 Nov 2021 03:30:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16E176B0073; Wed, 10 Nov 2021 03:30:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0021.hostedemail.com [216.40.44.21]) by kanga.kvack.org (Postfix) with ESMTP id 0584C6B0071 for ; Wed, 10 Nov 2021 03:30:11 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id F233C824C439 for ; Wed, 10 Nov 2021 08:30:09 +0000 (UTC) X-FDA: 78792348180.17.C51408C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id C030D509555E for ; Wed, 10 Nov 2021 08:29:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1636533009; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zc9KdahAguXkghVeJqdY3150Du1QUqNhK9U4RYp1RCU=; b=HUp5dSG7adgN3Af7MLPs7T8lCAA+mDmCj3lFX3IjjIqyUSYj2Kq9P0OD0r7ydjmOZdzeR2 UFcAFvKQghc1g8xBNX1CDNLCrckYWauTfUW9Z4k92aTxm2ZBHuegoQI8TUYwiXfPH4bT+E JnKXgCV8YDMjVEGz9TTfETqLgCg4lCk= Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-329-kaG4y54gMD2M45zGs7Qhxg-1; Wed, 10 Nov 2021 03:30:07 -0500 X-MC-Unique: kaG4y54gMD2M45zGs7Qhxg-1 Received: by mail-pg1-f199.google.com with SMTP id f7-20020a63f107000000b002db96febb74so1157698pgi.7 for ; Wed, 10 Nov 2021 00:30:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zc9KdahAguXkghVeJqdY3150Du1QUqNhK9U4RYp1RCU=; b=ObhpRCgaI+dUVGBjMw3unyczAx1ogS4yfiQLD7okYZEtPDOZHSWpn79HuDfOjzBDkm +wE5qylFiH++drbbm3LfZN+qwCjNe71B2x6Ou2oUMazGLR+syFMFVCrh1Va2ODtN6Bhf lmLQzmvo5balzQjkk3pLTMtcSyeK15vnZ8HE5CXXFZuin2Nzeq3A0fXal5bWWuRdDn44 qxVJq8r7pqBC9PgS3N5rEi5yA9Fs5iXvMBTTLrEkn8A6Hxk+sq1wvKl4JtJEkFQBma3f FHrbN/UzOz0yx7gtmn6uUoWleVcaepx7meYhlys368mxYXTteD2QQD/1IXsROwrC+fd7 pGTQ== X-Gm-Message-State: AOAM531AM7p0sHYqLAW8IEdm0jY+xBshWPaDlzyWzZYKVgMhCTylp9Nc MKFqqrttuNwJL8FJWcKxZ43JP/LwOXtc7QslbjnoAxIsn8cWgvgFDavwb35neaK/SsL1wY6wFy4 10h3kCGw7VP0= X-Received: by 2002:a05:6a00:234f:b0:3eb:3ffd:6da2 with SMTP id j15-20020a056a00234f00b003eb3ffd6da2mr14429772pfj.15.1636533006778; Wed, 10 Nov 2021 00:30:06 -0800 (PST) X-Google-Smtp-Source: ABdhPJz5G/zio2lUCBLScLXJPHr7YuBo+SEEpRo7ak25/Vhyewmy2QoxXzy6zMkmUgeA4ABnG5roEw== X-Received: by 2002:a05:6a00:234f:b0:3eb:3ffd:6da2 with SMTP id j15-20020a056a00234f00b003eb3ffd6da2mr14429751pfj.15.1636533006522; Wed, 10 Nov 2021 00:30:06 -0800 (PST) Received: from localhost.localdomain ([94.177.118.35]) by smtp.gmail.com with ESMTPSA id s7sm23709986pfu.139.2021.11.10.00.30.01 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Nov 2021 00:30:06 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: peterx@redhat.com, Andrew Morton , Yang Shi , Hugh Dickins , David Hildenbrand , Alistair Popple , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" Subject: [PATCH RFC 1/2] mm: Don't skip swap entry even if zap_details specified Date: Wed, 10 Nov 2021 16:29:51 +0800 Message-Id: <20211110082952.19266-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211110082952.19266-1-peterx@redhat.com> References: <20211110082952.19266-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C030D509555E X-Stat-Signature: acssyfypq45wzgzqsnr7tc3su4ob4x1a Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HUp5dSG7; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf01.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=peterx@redhat.com X-HE-Tag: 1636532995-896535 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This check existed since the 1st git commit of Linux repository, but at that time there's no page migration yet so I think it's okay. With page migration enabled, it should logically be possible that we zap some shmem pages during migration. When that happens, IIUC the old code could have the RSS counter accounted wrong on MM_SHMEMPAGES because we will zap the ptes without decreasing the counters for the migrating entries. I have no unit test to prove it as I don't know an easy way to trigger this condition, though. Besides, the optimization itself is already confusing IMHO to me in a few points: - The wording "skip swap entries" is confusing, because we're not skipping all swap entries - we handle device private/exclusive pages before that. - The skip behavior is enabled as long as zap_details pointer passed over. It's very hard to figure that out for a new zap caller because it's unclear why we should skip swap entries when we have zap_details specified. - With modern systems, especially performance critical use cases, swap entries should be rare, so I doubt the usefulness of this optimization since it should be on a slow path anyway. - It is not aligned with what we do with huge pmd swap entries, where in zap_huge_pmd() we'll do the accounting unconditionally. This patch drops that trick, so we handle swap ptes coherently. Meanwhile we should do the same mapping check upon migration entries too. Signed-off-by: Peter Xu --- mm/memory.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 8f1de811a1dc..e454f3c6aeb9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1382,16 +1382,14 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } - /* If details->check_mapping, we leave swap entries. */ - if (unlikely(details)) - continue; - if (!non_swap_entry(entry)) rss[MM_SWAPENTS]--; else if (is_migration_entry(entry)) { struct page *page; page = pfn_swap_entry_to_page(entry); + if (unlikely(zap_skip_check_mapping(details, page))) + continue; rss[mm_counter(page)]--; } if (unlikely(!free_swap_and_cache(entry))) From patchwork Wed Nov 10 08:29:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12611565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32A47C433F5 for ; Wed, 10 Nov 2021 08:30:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D585761076 for ; Wed, 10 Nov 2021 08:30:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D585761076 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 7FF046B0072; Wed, 10 Nov 2021 03:30:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78A046B0073; Wed, 10 Nov 2021 03:30:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DB766B0074; Wed, 10 Nov 2021 03:30:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id 4C6046B0072 for ; Wed, 10 Nov 2021 03:30:15 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 107177FADD for ; Wed, 10 Nov 2021 08:30:15 +0000 (UTC) X-FDA: 78792348390.07.2F874E4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 71A4130000B0 for ; Wed, 10 Nov 2021 08:30:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1636533014; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/Mopq8eqVCQt3k2LgKCs45drQOHTZZHv/QrpqMo5cqQ=; b=SJ/COEOC4XqhEWxuW1OC128Ydbws+6pZXt+ZMTpH84X+pALwEB7Fl8gOqzLhRSKsRregNt KoNQmJMNzPjXzV58no6tCLvwK9Iyb+DcDhR+7wDWzdiBb44qj2O4iDh3yZ7ZM15kuvMm1Q u8el5HGaX0hgXyRZcUCmtMoji9ikBKE= Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-484-xG1sNY2uN3ChXTt64vvJNw-1; Wed, 10 Nov 2021 03:30:13 -0500 X-MC-Unique: xG1sNY2uN3ChXTt64vvJNw-1 Received: by mail-pj1-f72.google.com with SMTP id iq9-20020a17090afb4900b001a54412feb0so662055pjb.1 for ; Wed, 10 Nov 2021 00:30:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/Mopq8eqVCQt3k2LgKCs45drQOHTZZHv/QrpqMo5cqQ=; b=hhMKk4qqcrB6DGWkdDk4ip9IQjxiIurd49QXQObZMp1iSB+ww4FhdQ20KHtjNNDTyx IwoS1D03rbJ28lzROLTVvo3RF0Fu1vEjcOhr4Uxs9WtGq9mA/m99bQxqeHGxkYa1agSB RJxJZV60pxhMhKVKtLj1Q+Po3gyAt8taUCVK9MvzFIEzPTn95qHa5RtA7mRS+kIem2af f4OG86dD9MjNfZ103gGFgBiJnXidoNII/YSmovTNYPhiYoZb0DCDUw0NgUDcI0xNOubr jr0JVakLyvrH4pNWSMlCOGYxxndmXPTBDzJy2+g8TGHD/LDJ3yFcxY9Lu/IQ6nssOns2 2epA== X-Gm-Message-State: AOAM532hyROy4UHiF/80CaiH0yEgz+jvowfjjoykOoe6+ZrO33EaZfmV AUAn2Exc0LDS6t/HPBkXU+6k/Hg9u7l6UsqclnYqYbyRqXgoJCj+NC4FEyen4fM25O98z6KggDc rRC2hp66kmwI= X-Received: by 2002:a17:902:744b:b0:142:46fe:7fbf with SMTP id e11-20020a170902744b00b0014246fe7fbfmr13548609plt.83.1636533011790; Wed, 10 Nov 2021 00:30:11 -0800 (PST) X-Google-Smtp-Source: ABdhPJzDXOtXqxrc8KYS1LcooPeq0n9NmJXaoIsx8EsNg1RzMgg48pAAzDRcIkBGlG9WlDWha2HVhg== X-Received: by 2002:a17:902:744b:b0:142:46fe:7fbf with SMTP id e11-20020a170902744b00b0014246fe7fbfmr13548581plt.83.1636533011533; Wed, 10 Nov 2021 00:30:11 -0800 (PST) Received: from localhost.localdomain ([94.177.118.35]) by smtp.gmail.com with ESMTPSA id s7sm23709986pfu.139.2021.11.10.00.30.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Nov 2021 00:30:11 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: peterx@redhat.com, Andrew Morton , Yang Shi , Hugh Dickins , David Hildenbrand , Alistair Popple , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" Subject: [PATCH RFC 2/2] mm: Rework swap handling of zap_pte_range Date: Wed, 10 Nov 2021 16:29:52 +0800 Message-Id: <20211110082952.19266-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211110082952.19266-1-peterx@redhat.com> References: <20211110082952.19266-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 71A4130000B0 X-Stat-Signature: k8hgdubwdrgmcpi5jkz1xwwhtnynqnwk Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="SJ/COEOC"; spf=none (imf03.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1636533006-244979 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Clean the code up by merging the device private/exclusive swap entry handling with the rest, then we merge the pte clear operation too. struct* page is defined in multiple places in the function, move it upward. free_swap_and_cache() is only useful for !non_swap_entry() case, put it into the condition. No functional change intended. Signed-off-by: Peter Xu --- mm/memory.c | 25 ++++++++----------------- 1 file changed, 8 insertions(+), 17 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index e454f3c6aeb9..e5d59a6b6479 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1326,6 +1326,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = *pte; + struct page *page; + if (pte_none(ptent)) continue; @@ -1333,8 +1335,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, break; if (pte_present(ptent)) { - struct page *page; - page = vm_normal_page(vma, addr, ptent); if (unlikely(zap_skip_check_mapping(details, page))) continue; @@ -1368,32 +1368,23 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, entry = pte_to_swp_entry(ptent); if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { - struct page *page = pfn_swap_entry_to_page(entry); - + page = pfn_swap_entry_to_page(entry); if (unlikely(zap_skip_check_mapping(details, page))) continue; - pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); rss[mm_counter(page)]--; - if (is_device_private_entry(entry)) page_remove_rmap(page, false); - put_page(page); - continue; - } - - if (!non_swap_entry(entry)) - rss[MM_SWAPENTS]--; - else if (is_migration_entry(entry)) { - struct page *page; - + } else if (is_migration_entry(entry)) { page = pfn_swap_entry_to_page(entry); if (unlikely(zap_skip_check_mapping(details, page))) continue; rss[mm_counter(page)]--; + } else if (!non_swap_entry(entry)) { + rss[MM_SWAPENTS]--; + if (unlikely(!free_swap_and_cache(entry))) + print_bad_pte(vma, addr, ptent, NULL); } - if (unlikely(!free_swap_and_cache(entry))) - print_bad_pte(vma, addr, ptent, NULL); pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); } while (pte++, addr += PAGE_SIZE, addr != end);