From patchwork Thu Apr 18 13:44:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13634764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2A0FC4345F for ; Thu, 18 Apr 2024 13:45:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 591746B009A; Thu, 18 Apr 2024 09:45:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53E946B009B; Thu, 18 Apr 2024 09:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B9FE6B009C; Thu, 18 Apr 2024 09:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1A0576B009A for ; Thu, 18 Apr 2024 09:45:10 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0C6E0A0EDC for ; Thu, 18 Apr 2024 13:45:03 +0000 (UTC) X-FDA: 82022773686.28.345A1DC Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf13.hostedemail.com (Postfix) with ESMTP id 3460120021 for ; Thu, 18 Apr 2024 13:45:00 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bBqdZNyB; spf=pass (imf13.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713447900; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jgQ7Fv4qoOj50C93E985pB5vnNpjpXSugVdGk65lNOA=; b=UK+3kOsrhFFBwG5N5MQZrEInhglwp/08joYpaFB3o5twMZktFF1cVINKTIbtdnT3Yy2yQw sNCX7dQdc/RNBHuGUSkg4i2OjmrHGdtbqHdt9hHSAQz2cqLZhjBPVxHnmn81P5RjTqMDCf fM46jeYZqqf2eP/+j++IohqfEj2Z8jw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713447900; a=rsa-sha256; cv=none; b=Vvo/2fgWgIyu7scIDIcfBFLC4QWEEPhrMbB+w/BV0YODgN++Ul19XBKPg8gQ3KeOlAc1Xh T7MeLMlAHuMqD+bqJkNxzJGn1HZOzFEz05VrDZDBDFGOP0HEKyLNFDYi2nOtguRtkhXj7K pd0se+LqXL/lEorWeRbC5GvzTh2wiGI= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bBqdZNyB; spf=pass (imf13.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1e4266673bbso8193165ad.2 for ; Thu, 18 Apr 2024 06:44:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713447899; x=1714052699; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jgQ7Fv4qoOj50C93E985pB5vnNpjpXSugVdGk65lNOA=; b=bBqdZNyB3jeTs/bBNPcTiSEQSnXj2JgSNfgPlWyAIMZksPMmByM6ZwocPZmOd7Vsw7 k+Bt1hHQHsd8YzqT0Gp8oRRfx7pOqtvD29VlUM0RXUnY64uchlyLe6F1sUR8mBLHYOou HRBDPLLJiMp0dNHstb04QrfkaDQHUmd8DG/VJbRRsMKjuvmWVHt0+GJ1CmixEPKqf9aK hir8gNBFw1x9pM25ySPXHxDTUdwSj9waT/AsHyqJR4kni/iF1EZkW47uEigYImHqLMCQ mBhB+JPK0aWFedi4ZOEEEViVzipm7LKHDIUkyNVQK2RUPR74aCLdh/8D53qhwb3+T+A+ L+WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713447899; x=1714052699; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jgQ7Fv4qoOj50C93E985pB5vnNpjpXSugVdGk65lNOA=; b=Wd0TSsIxJkBvHXgeEwGkwRnBXaHHokgaf7Z0hFnbS31V2u7Jwk54mDQTV05++9II4j aShNP3AMswyjJOK31qJlqrtSLWfOakY/9UxhVdTCgWO2ZU3EaKGOictsopPM5tFPZw9a jzm8CYsgE+XwQk6/nIlITJbnYa8cCS/Y0vW5odp7XHTgzlTuhR95CES5MFH2M3Ia/n5/ iukIY/C3ZeGmPXRaO8LMFyK2AC6XOHYObKzg8UVcpQ2lHzsXmJ6RSvRmQnJZvEGNUtjk IYLF1v3aDwvJ7HWxE/857O3APvpkc4wSHfUc44VsLmduB3gNVBtSXwyCjnY6ju3yZi9M t6Ow== X-Forwarded-Encrypted: i=1; AJvYcCVClG+jqlRPo0dwrjClIPubXvCzrW2JnI6kPd20h1HQ7mCgbieGsCDqHX6Ezg5dB0aFDGA518qW5uHg164T71cIu7g= X-Gm-Message-State: AOJu0YxssrOUIasTITUt9mh2gwsksJkKqGH9YyNwgQ/oAyBzE0/GQX2j gWWlrF60TnKtV/p8fpt8fe4P8Kj4gaMv72y89L42eslSIMFDWFXs X-Google-Smtp-Source: AGHT+IGGqaeZJIZi8LM0BZvZXjbrMIOv72J2g3c12uT7zMGxJh9Xc/5vCUs+3eTCkX6ArwbYFFIuZA== X-Received: by 2002:a17:902:c20d:b0:1de:f91:3cf3 with SMTP id 13-20020a170902c20d00b001de0f913cf3mr2714045pll.55.1713447899052; Thu, 18 Apr 2024 06:44:59 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id d8-20020a170902b70800b001e4fdcf67desm1504837pls.299.2024.04.18.06.44.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 06:44:58 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v10 4/4] mm/madvise: optimize lazyfreeing with mTHP in madvise_free Date: Thu, 18 Apr 2024 21:44:35 +0800 Message-Id: <20240418134435.6092-5-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240418134435.6092-1-ioworker0@gmail.com> References: <20240418134435.6092-1-ioworker0@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 3460120021 X-Rspam-User: X-Stat-Signature: cs4fumpu358jbmdps875nxhk9hb5yt96 X-HE-Tag: 1713447900-120198 X-HE-Meta: U2FsdGVkX18zGzWpF6dlW193gRP4OuTiaPrOAcl3dU1dwVV/PRQBeMYVtd8qSSouKIsHRtVqJEXzb3fFeeoQGYvFjXAxz3YIw8u3jZqjlR6IV9ApHORls2gXVbPlvObfOBAb17a9m8tjqA/z85jD83eQjLJyCRbxIBWILNd0FDS2KZ3bw5yYJvJflXyOx6riX4wCgXBpDj1NQR582j7cHPJioNq922uzY26K46odmSC25DttBXyENBTrxVOFDJdd+bbCvMSlgLx2yJnDpB/Ui2sQz2qQ+M+acXMD/2UU2Fqlzy1IkRFzktknzK7K+cR7NSkFfTWWyVyTwxRCwt6W/cYSuSYTcuDesshjsXxnarKtaZiYOvWJX0MIweVnBEHwmzuXYiD5biN3WiqtaI9IEtQAsYSD22GyCw/uJbve88R779/0t5E9OIP7fTJ87S8XnL68964Vg1MqHkB/0PqgqERj2TaEptpaW7Xr8NcbePRyWFOZ6NvmrrMRz5oJW4GoaEgJnBYwlxT+vnGqt8bLKwhZSuRheXcw859yWw3hMTa4I8DCMr9aA3nFKl7nHGeL9WXemwUsJY1sLsBTxg8PqRYc1X2VS0lQjw3kHtZlhknlfpq7CpDqZ1HoZa5AT39msQh0hL9vXXfP2wQCqZKcP46NYkqS35Rr1dn/KpeoywI5fwouvXBnq5Nb7TerKhDCmXZXbi/I+aM/EsEB8/O+onGAgxHQhtZqRDOOsJbhRtDYdcmGpsMNH2iHbf4nauZs4M//0qF1NJjQGlELzvLG5z/WbyjPWiv7kymp5auqbYygp3A66xMj0Eata1Yw+MAUHJnaE0+ONNK7WFNdcVFmFRPRsropETiEsfia/LxLFHfc2d2vO5djxZ88odrxZv1zBntHzRpMJg4Xc+y1UjbzcGE+ekecWbVmIVhFTNn2X0qKJG/BXx5M5oKoFPZOiQWpF6ZNP+j2Y3I3w8Q/jNK NzAKcVeh ugw0P8qYixJhs1Q1sEIhTGaddjUWdCYc7DgeSHlqIzLFeRTA6SmV3215HJx+EZVYp18PHV4xH4J0do3l3fbv1tQrW8bAyIowDF0VBYycQ0ULvQ19U0M5UJLsg0T6nc+xkOWkXmFRXSQfdn9zn3WmKn/nnz5wLnbl364qzDbp9qnTnmZi0gg3MNSMOLbLI3u43VO4DNCfFeDGRS49SYr4BzAZL1cOHr7LEDLPcQ9sPm60kmms8BeGfR7r6rPyco89x8sL/R5DL995/nP5z+yZEksoDh48TXyooIad0jrNYT1r0+76ClV8Wohi5Y9X2gecWAJBSpzQ1FsyoGf7rqWs2zDZS1o7SWUbRkj7gtE4cZgH5lQ0Ha83KeD85SBqy1XOnnzUjLNv2jOqqlMZFQCtEbGYrYH1fP46u8BFKIA2JB73brpgsLsi2A3pVrjsGCE4J02MEEOpBWnqzwpS347NMXpjnJK2gJJ/9+qywbxYhee0cWC6UEEKoaTYbQJQNsnL4BgfER77jVlVrHCuK3PqR5Y612acjENbY230tTQq7a3VvSgQHTbXsUpKoRpU0EC6KnYOVE2FTUDq1vq8O+16w8ELLwsjSVKbkeVAbxCx0KwaaYDo/ILzdvbQuZUe563sb0/mx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch optimizes lazyfreeing with PTE-mapped mTHP[1] (Inspired by David Hildenbrand[2]). We aim to avoid unnecessary folio splitting if the large folio is fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% [1] https://lkml.kernel.org/r/20231207161211.2374093-5-ryan.roberts@arm.com [2] https://lore.kernel.org/linux-mm/20240214204435.167852-1-david@redhat.com Reviewed-by: Ryan Roberts Acked-by: David Hildenbrand Signed-off-by: Lance Yang --- mm/madvise.c | 85 +++++++++++++++++++++++++++------------------------- 1 file changed, 44 insertions(+), 41 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 4597a3568e7e..ed125ad8a21e 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -643,6 +643,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) { + const cydp_t cydp_flags = CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY; struct mmu_gather *tlb = walk->private; struct mm_struct *mm = tlb->mm; struct vm_area_struct *vma = walk->vma; @@ -697,44 +698,57 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, continue; /* - * If pmd isn't transhuge but the folio is large and - * is owned by only this process, split it and - * deactivate all pages. + * If we encounter a large folio, only split it if it is not + * fully mapped within the range we are operating on. Otherwise + * leave it as is so that it can be marked as lazyfree. If we + * fail to split a folio, leave it in place and advance to the + * next pte in the range. */ if (folio_test_large(folio)) { - int err; + bool any_young, any_dirty; - if (folio_likely_mapped_shared(folio)) - break; - if (!folio_trylock(folio)) - break; - folio_get(folio); - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(start_pte, ptl); - start_pte = NULL; - err = split_folio(folio); - folio_unlock(folio); - folio_put(folio); - if (err) - break; - start_pte = pte = - pte_offset_map_lock(mm, pmd, addr, &ptl); - if (!start_pte) - break; - arch_enter_lazy_mmu_mode(); - pte--; - addr -= PAGE_SIZE; - continue; + nr = madvise_folio_pte_batch(addr, end, folio, pte, + ptent, &any_young, &any_dirty); + + if (nr < folio_nr_pages(folio)) { + int err; + + if (folio_likely_mapped_shared(folio)) + continue; + if (!folio_trylock(folio)) + continue; + folio_get(folio); + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + start_pte = NULL; + err = split_folio(folio); + folio_unlock(folio); + folio_put(folio); + pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + start_pte = pte; + if (!start_pte) + break; + arch_enter_lazy_mmu_mode(); + if (!err) + nr = 0; + continue; + } + + if (any_young) + ptent = pte_mkyoung(ptent); + if (any_dirty) + ptent = pte_mkdirty(ptent); } if (folio_test_swapcache(folio) || folio_test_dirty(folio)) { if (!folio_trylock(folio)) continue; /* - * If folio is shared with others, we mustn't clear - * the folio's dirty flag. + * If we have a large folio at this point, we know it is + * fully mapped so if its mapcount is the same as its + * number of pages, it must be exclusive. */ - if (folio_mapcount(folio) != 1) { + if (folio_mapcount(folio) != folio_nr_pages(folio)) { folio_unlock(folio); continue; } @@ -750,19 +764,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, } if (pte_young(ptent) || pte_dirty(ptent)) { - /* - * Some of architecture(ex, PPC) don't update TLB - * with set_pte_at and tlb_remove_tlb_entry so for - * the portability, remap the pte with old|clean - * after pte clearing. - */ - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - - ptent = pte_mkold(ptent); - ptent = pte_mkclean(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + clear_young_dirty_ptes(vma, addr, pte, nr, cydp_flags); + tlb_remove_tlb_entries(tlb, pte, nr, addr); } folio_mark_lazyfree(folio); }