From patchwork Tue Aug 9 22:00:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12939964 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 193E6C25B08 for ; Tue, 9 Aug 2022 22:01:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37A748E0002; Tue, 9 Aug 2022 18:01:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3033D8E0001; Tue, 9 Aug 2022 18:01:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17D3D8E0002; Tue, 9 Aug 2022 18:01:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 014F58E0001 for ; Tue, 9 Aug 2022 18:01:06 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C31E01C6A5D for ; Tue, 9 Aug 2022 22:01:06 +0000 (UTC) X-FDA: 79781425332.01.F2D8945 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 05B4A100177 for ; Tue, 9 Aug 2022 22:01:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1660082465; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=2Y05UawN7N/AGIQ2Hdz9OrZ5faEMa1jRZA1Sh+qd1pw=; b=YFkpljzD161MoKaW0HSQhe8Mh4Wmm4n24dNx5U2sddYrH+FREOwU/zruyC+rs7f/n+au/P sRp1HjC7exDnFIDaEpgybIrPuB4+WRTzy8pwmoCu3dmts0nhMtX9Ovkoq92A9+kID1Rlha oMqWNA4gPlEHZTQ8lACx2mHLtizhz8Y= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-183-II3YbT_3MASk29MTVdizgA-1; Tue, 09 Aug 2022 18:01:04 -0400 X-MC-Unique: II3YbT_3MASk29MTVdizgA-1 Received: by mail-qk1-f198.google.com with SMTP id br44-20020a05620a462c00b006b92ca79985so8137217qkb.1 for ; Tue, 09 Aug 2022 15:01:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc; bh=2Y05UawN7N/AGIQ2Hdz9OrZ5faEMa1jRZA1Sh+qd1pw=; b=23/42t7Qj2+a8tLCP5ieCDh37HbarvpmYF29xXJ69/SntNbfuG+ltca+G69hiudOCb l8RLNEQn/V2zwl/UZvreoIMlNk7tXzEIWPgIl94CL/VO/5KRrQ8wQr0NkFkoTlDQ5Kxi 2K51tE7XnIpV85ykqjURNo8+lZOhtzENziC5cHEHXWNrfLjPLhZ9WvecY0y27Py2GRDE fnth1LxLq3xEeeW4CBlunj7syb13pw+CS3RJt32MmGMzJ8rOrAMUaFStJJ6uGrFc6QaP IFabK6RsmkHkB670wVsGzlM/1jZdm8IEMy7+qvSk7CFnLoQdQC/UeAvfRqvwg5OiKPCJ AmWw== X-Gm-Message-State: ACgBeo1mZxSIjoAuBa5+zpQoVSn6ypG60ay1qBZ0GCS0iJqTodYFCCdx s1AjDAs14IFjmkXy+nfpilwFJbNTKoyB+JNTrnhh60LnwI6r6v8qvK9VhG8h4xI/C3/LsGYqaSY dTIGSa/qX4Qsoxsv+13SihsFJzo8NhyUDcLc28KRQVuEke8lHw2IKE+oWqcm8 X-Received: by 2002:a05:620a:2b8b:b0:6b9:43ca:4a6 with SMTP id dz11-20020a05620a2b8b00b006b943ca04a6mr9354376qkb.346.1660082463507; Tue, 09 Aug 2022 15:01:03 -0700 (PDT) X-Google-Smtp-Source: AA6agR7sp5xe6cpJkByfw2vmK3F81PdNa3RHkuG6pyvhf2+9jfQmNT/KF3UoFptTnCY9sWk3mgI8Yw== X-Received: by 2002:a05:620a:2b8b:b0:6b9:43ca:4a6 with SMTP id dz11-20020a05620a2b8b00b006b943ca04a6mr9354332qkb.346.1660082463063; Tue, 09 Aug 2022 15:01:03 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id b16-20020a05622a021000b0031eddc83560sm10563383qtx.90.2022.08.09.15.01.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 09 Aug 2022 15:01:02 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: peterx@redhat.com, Minchan Kim , David Hildenbrand , Nadav Amit , Andrew Morton , Hugh Dickins , Vlastimil Babka , Andrea Arcangeli , Andi Kleen , Huang Ying , "Kirill A . Shutemov" Subject: [PATCH v3 0/7] mm: Remember a/d bits for migration entries Date: Tue, 9 Aug 2022 18:00:53 -0400 Message-Id: <20220809220100.20033-1-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660082466; a=rsa-sha256; cv=none; b=QGnoConsrqqk9p3bmCjgPqTTvXl6c+LypBVNOP06tUzHmT9Cvsc+O3new0ELhZV3t/C04N z9dwM8eNxnPBwVyo2tYFbNf1Fs0nlYxqdhClPISZIHg7AGhnmrmFp2UeLNVTYoBj86CeU/ hy58qHiSh8uduy8f3KoznoIe/5HTOUE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660082466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=2Y05UawN7N/AGIQ2Hdz9OrZ5faEMa1jRZA1Sh+qd1pw=; b=ECNvZ3U18GJyOTrfPvbANaaCpvV8iTxM5UYNRCkCmDDN+Bzq/OxzpYH02noi2qXfiD8Jmr cyQymSB++7JHVHr83P4OWC47DFAgMOsgcIO7XETtPbWSkEXT4HO5I0ybI6I++THJfLUM4j e64HXfXPJfWt/t4VDuGVTa6SBYCWQCc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YFkpljzD; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: beacnrwpfj5acmmcw8wgts1wqdk53shs X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 05B4A100177 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YFkpljzD; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1660082465-822701 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: v3: - Rebased to akpm/mm-unstable - Use BIT() [Nadav] - One more patch to add comment for "ifdef"s [Nadav] - Added one ascii table for migration swp offset layout [David] - Move comment to be above "if"s [Ying] - Separate the dirty bit carry-over of pmd split to separate patch [Ying] - Added two patches to cache both max_swapfile_size and migration_entry_supports_ad() at the end rfc: https://lore.kernel.org/all/20220729014041.21292-1-peterx@redhat.com v1: https://lore.kernel.org/all/20220803012159.36551-1-peterx@redhat.com v2: https://lore.kernel.org/all/20220804203952.53665-1-peterx@redhat.com Problem ======= When migrate a page, right now we always mark the migrated page as old & clean. However that could lead to at least two problems: (1) We lost the real hot/cold information while we could have persisted. That information shouldn't change even if the backing page is changed after the migration, (2) There can be always extra overhead on the immediate next access to any migrated page, because hardware MMU needs cycles to set the young bit again for reads, and dirty bits for write, as long as the hardware MMU supports these bits. Many of the recent upstream works showed that (2) is not something trivial and actually very measurable. In my test case, reading 1G chunk of memory - jumping in page size intervals - could take 99ms just because of the extra setting on the young bit on a generic x86_64 system, comparing to 4ms if young set. This issue is originally reported by Andrea Arcangeli. Solution ======== To solve this problem, this patchset tries to remember the young/dirty bits in the migration entries and carry them over when recovering the ptes. We have the chance to do so because in many systems the swap offset is not really fully used. Migration entries use swp offset to store PFN only, while the PFN is normally not as large as swp offset and normally smaller. It means we do have some free bits in swp offset that we can use to store things like A/D bits, and that's how this series tried to approach this problem. max_swapfile_size() is used here to detect per-arch offset length in swp entries. We'll automatically remember the A/D bits when we find that we have enough swp offset field to keep both the PFN and the extra bits. Since max_swapfile_size() can be slow, the last two patches cache the results for it and also swap_migration_ad_supported as a whole. Tests ===== After the patchset applied, the immediate read access test [1] of above 1G chunk after migration can shrink from 99ms to 4ms. The test is done by moving 1G pages from node 0->1->0 then read it in page size jumps. The test is with Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz. Similar effect can also be measured when writting the memory the 1st time after migration. After applying the patchset, both initial immediate read/write after page migrated will perform similarly like before migration happened. Patch Layout ============ Patch 1-2: Cleanups from either previous versions or on swapops.h macros. Patch 3-4: Prepare for the introduction of migration A/D bits Patch 5: The core patch to remember young/dirty bit in swap offsets. Patch 6-7: Cache relevant fields to make migration_entry_supports_ad() fast. Please review, thanks. [1] https://github.com/xzpeter/clibs/blob/master/misc/swap-young.c Peter Xu (7): mm/x86: Use SWP_TYPE_BITS in 3-level swap macros mm/swap: Comment all the ifdef in swapops.h mm/swap: Add swp_offset_pfn() to fetch PFN from swap entry mm/thp: Carry over dirty bit when thp splits on pmd mm: Remember young/dirty bit for page migrations mm/swap: Cache maximum swapfile size when init swap mm/swap: Cache swap migration A/D bits support arch/arm64/mm/hugetlbpage.c | 2 +- arch/x86/include/asm/pgtable-3level.h | 8 +- arch/x86/mm/init.c | 2 +- include/linux/swapfile.h | 1 + include/linux/swapops.h | 145 +++++++++++++++++++++++--- mm/hmm.c | 2 +- mm/huge_memory.c | 24 ++++- mm/memory-failure.c | 2 +- mm/migrate.c | 6 +- mm/migrate_device.c | 4 + mm/page_vma_mapped.c | 6 +- mm/rmap.c | 5 +- mm/swapfile.c | 18 +++- 13 files changed, 194 insertions(+), 31 deletions(-)