From patchwork Wed Aug 3 01:21:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12935111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70001C00140 for ; Wed, 3 Aug 2022 01:22:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0C968E0002; Tue, 2 Aug 2022 21:22:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BBB588E0001; Tue, 2 Aug 2022 21:22:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A82E58E0002; Tue, 2 Aug 2022 21:22:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 95AF08E0001 for ; Tue, 2 Aug 2022 21:22:04 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6BC0E41399 for ; Wed, 3 Aug 2022 01:22:04 +0000 (UTC) X-FDA: 79756530168.25.F42D235 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id E226240101 for ; Wed, 3 Aug 2022 01:22:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1659489723; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=eRfVsx2J8bSWaoY/W9Xo3vXxtbJlknv2N+vt75Xmtjo=; b=dcrLmpp33nbSGuxzztzVNm+UvO5WBsoNMPmRHeoRiHQ1DtBEiCb1oIifuNcCUW0gmaHG/z NOfE8/lsDgxUkNNHPZawuI7EOR7FUDnHQedCqO6kJC7TOaS2lKR85V62GuxXc33iUrLaY7 xY1SciJJzZ7AvO4MgOW+GXfXBLNRjXA= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-387-nWswPoNGNCqqB-JdjKKrMg-1; Tue, 02 Aug 2022 21:22:02 -0400 X-MC-Unique: nWswPoNGNCqqB-JdjKKrMg-1 Received: by mail-qk1-f199.google.com with SMTP id u15-20020a05620a0c4f00b006b8b3f41303so4774287qki.8 for ; Tue, 02 Aug 2022 18:22:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=eRfVsx2J8bSWaoY/W9Xo3vXxtbJlknv2N+vt75Xmtjo=; b=t0N/P6X6WcejoZMvid5HkqVGyAjKOc3GlpPABOHPh7DHFCxddOLLgdgZgFfsatKc/K pqM7FzkhCpClPIFyupaXf/NrbkDoWh37pcPFRQDZZpDinrXflL75YgBbBem60IFsY4v8 aUWHN4dEU1n05ARVC/FZX5wKd4TEDUGTmLBbiMgqIsoRxSqmWDhkvp4mCw3z9ZhRzLIP O7YkOROUkHL07wFLKkPw4YkqR0uYPmclEsGzISAvGCwnVcrWA9n97bUe0PXJdr+nVkKg gy15HlXWjW5ubr6b82uChrhbJvJK6H/HEOyyDNneCdHGSBVQC/af9DT/Zd72N5niQfo9 VQMA== X-Gm-Message-State: ACgBeo32Nu7FWhyAgXRyM9U+Wd9wtse4VJJza1cUJuEXfzhQFvLA7gmz FRjUFzfgWCTfpNwQZG5Tz/EN803tfLPkX3fHjoU035b8/pKKfOU4HjyZlO9Njc2Osquehmd0TSt XDzslEWUQi6c= X-Received: by 2002:ad4:4b30:0:b0:476:f037:8a2 with SMTP id s16-20020ad44b30000000b00476f03708a2mr3602048qvw.2.1659489722085; Tue, 02 Aug 2022 18:22:02 -0700 (PDT) X-Google-Smtp-Source: AA6agR5+ooUzYb3QOMUHiraBGk10183ECWADOXu+fEo4pH9ZjvPmFfzGi/YyA273l8lJmBRTRc+4KQ== X-Received: by 2002:ad4:4b30:0:b0:476:f037:8a2 with SMTP id s16-20020ad44b30000000b00476f03708a2mr3602031qvw.2.1659489721757; Tue, 02 Aug 2022 18:22:01 -0700 (PDT) Received: from localhost.localdomain (bras-base-aurron9127w-grc-35-70-27-3-10.dsl.bell.ca. [70.27.3.10]) by smtp.gmail.com with ESMTPSA id s16-20020a05622a1a9000b0031ecce4077fsm10188675qtc.31.2022.08.02.18.22.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 02 Aug 2022 18:22:01 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrea Arcangeli , Andi Kleen , Andrew Morton , David Hildenbrand , Hugh Dickins , Huang Ying , peterx@redhat.com, Nadav Amit , "Kirill A . Shutemov" , Vlastimil Babka Subject: [PATCH 0/2] mm: Remember young bit for migration entries Date: Tue, 2 Aug 2022 21:21:57 -0400 Message-Id: <20220803012159.36551-1-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659489724; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=eRfVsx2J8bSWaoY/W9Xo3vXxtbJlknv2N+vt75Xmtjo=; b=BmUh/nkf/iCFx4+TzoA1o4ahhuTkvuBlWjoYgns2UYdZPTYnmhZt/CUOZsDpOIm2qfi+xX b73TStOmuUH46RxaG30TDXkOeDrU2XLo90utXHWjrNVxWb/oNdhMrNonWFOPBsVVfySWA7 GykcsEOQmLQ3a4BogwyEclHAOi2wrxA= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dcrLmpp3; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659489724; a=rsa-sha256; cv=none; b=gPhVurC0XBqBsg7hWZgyB3GP3yt5QY1N97jQQgF7mCLXXANylPTIbuuOcJeUraQLdtF9Gg P1poDorEyjT5IItqsdaz1tdQVz5EFIbrc5I4cy046Suwbs72J6OWGlyXJbvT1e9fuyKG2s WgfccjTYSst69tyPfhORwl2wlKDYi/M= Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dcrLmpp3; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E226240101 X-Stat-Signature: 7q79cbndqmxxwt3puco6xuxx1knhaq3z X-HE-Tag: 1659489723-673015 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: rfc->v1: - Fix build for arch/um where MAX_PHYSMEM_BITS not defined [syzbot] - Add VM_BUG_ON() in swp_offset_pfn() to check swap entry type [Ying] - Use max_swapfile_size() to detect swp offset size [Ying] - Posted patch 3 separately, dropped patch 4 rfc: https://lore.kernel.org/all/20220729014041.21292-1-peterx@redhat.com/ Problem ======= When migrate a page, right now we always mark the migrated page as old. The reason could be that we don't really know whether the page is hot or cold, so we could have taken it a default negative assuming that's safer. However that could lead to at least two problems: (1) We lost the real hot/cold information while we could have persisted. That information shouldn't change even if the backing page is changed after the migration, (2) There can be always extra overhead on the immediate next access to any migrated page, because hardware MMU needs cycles to set the young bit again (as long as the MMU supports). Many of the recent upstream works showed that (2) is not something trivial and actually very measurable. In my test case, reading 1G chunk of memory - jumping in page size intervals - could take 99ms just because of the extra setting on the young bit on a generic x86_64 system, comparing to 4ms if young set. This issue is originally reported by Andrea Arcangeli. Solution ======== To solve this problem, this patchset tries to remember the young bit in the migration entries and carry it over when recovering the ptes. We have the chance to do so because in many systems the swap offset is not really fully used. Migration entries use swp offset to store PFN only, while the PFN is normally not as large as swp offset and normally smaller. It means we do have some free bits in swp offset that we can use to store things like young, and that's how this series tried to approach this problem. max_swapfile_size() is used here to detect per-arch offset length in swp entries. We'll automatically remember the young bit when we find that we have enough swp offset field to keep both the PFN and the young bit for a migration entry. Tests ===== After the patchset applied, the immediate read access test [1] of above 1G chunk after migration can shrink from 99ms to 4ms. The test is done by moving 1G pages from node 0->1->0 then read it in page size jumps. The test is with Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz. Patch Layout ============ Patch 1: Add swp_offset_pfn() and apply to all pfn swap entries, we should also stop treating swp_offset() as PFN anymore because it can contain more information starting from next patch. Patch 2: The core patch to remember young bit in swap offsets. Please review, thanks. [1] https://github.com/xzpeter/clibs/blob/master/misc/swap-young.c Peter Xu (2): mm/swap: Add swp_offset_pfn() to fetch PFN from swap entry mm: Remember young bit for page migrations arch/arm64/mm/hugetlbpage.c | 2 +- include/linux/swapops.h | 84 ++++++++++++++++++++++++++++++++++--- mm/hmm.c | 2 +- mm/huge_memory.c | 10 ++++- mm/memory-failure.c | 2 +- mm/migrate.c | 4 +- mm/migrate_device.c | 2 + mm/page_vma_mapped.c | 6 +-- mm/rmap.c | 3 +- 9 files changed, 99 insertions(+), 16 deletions(-)