From patchwork Thu Nov 7 20:20:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13867062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEB7DD5D688 for ; Thu, 7 Nov 2024 20:20:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4131A6B00A7; Thu, 7 Nov 2024 15:20:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C37A6B00A8; Thu, 7 Nov 2024 15:20:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28A976B00A9; Thu, 7 Nov 2024 15:20:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0CD526B00A7 for ; Thu, 7 Nov 2024 15:20:43 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B6021AB44B for ; Thu, 7 Nov 2024 20:20:42 +0000 (UTC) X-FDA: 82760415528.08.1FB8DDB Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 64A2212000E for ; Thu, 7 Nov 2024 20:19:52 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BQwL7bgL; spf=pass (imf29.hostedemail.com: domain of 3FSEtZwYKCEwC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3FSEtZwYKCEwC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731010755; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=l7+Mjo89P4AdQIIV//Uu9js0oVBe923zMULMp7p6G5Y=; b=R9utwY4MqrTiuskeAMVseJiEJjUH18UDlp5MxX/0fUYEYe60StiXH6RLNyS3PHVp3OBz5L FW1AtEHhhowj5FVpgpUCVLiatnJym6pml6R/Kw07BQgk6gaa7rWBWcLLp7Ql4LVzca09Oo x5yA2djbK9tIyMOeL9EBLFEi3b/hjYs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731010755; a=rsa-sha256; cv=none; b=nwMSMphQr654vHwipCLzbCkmqr7GwFeQIeThBYAyAqttOCNgSToKkf4jfTT8aTGsrWWW11 pIS74UbLpIUAkZOjKQwIf+atdPLRUpbBL3NcqicUGkPJF9T5rGlmzQTmHr1NeEA7KiVXUM aetG1cc7uMOOExjSzK7QtZChDrLVW9k= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BQwL7bgL; spf=pass (imf29.hostedemail.com: domain of 3FSEtZwYKCEwC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3FSEtZwYKCEwC8Dvo2u22uzs.q20zw18B-00y9oqy.25u@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e33152c8225so2940645276.0 for ; Thu, 07 Nov 2024 12:20:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731010838; x=1731615638; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=l7+Mjo89P4AdQIIV//Uu9js0oVBe923zMULMp7p6G5Y=; b=BQwL7bgL/WTb25TCbrltp0xKS7qWNsUVCZEqDgnQmC/au25V66a9J/IiiTG+FrSCWz hZkXJ5BhoHgdMmyr4n57PFdzGsmtj91xjB9rrVWlLEDu//s248Vrzf82GPg5G4f0z7hq ossnz6Z+PaURLsiUHdFKeVJNmlEPXa24FUh8ZOiNTlA0JFZHO+Y7FH5H5NF3M+yFbFJ6 UKA+Mw0xBZvUU7A/u2sdKKexMH+psdRDihCd9raRBXcpBensUJjzKogd1j4/bJZtvw6n X1f7V3bVABLHqh+VftZhP1lpPpI65Nw/zUdliFkh3kdF7v/BxcNuBEr5KMU8E8YrHt8U 0Ctw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731010838; x=1731615638; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=l7+Mjo89P4AdQIIV//Uu9js0oVBe923zMULMp7p6G5Y=; b=aIaQYqQg+YFl4gIhUX5GJiCQ1hgpPhiU5+YBefNXR54/qvCk/XfIJFhKetKHb5rijE /gLBbconQtQ8ZwOE1u7mkrvZFye+mKCOYTPOp9FQsjd6ir0DcuJyCFCCO19PHiLnWBT3 jzRsm9lxhnAyyZ548bXF/nIgKzQvWIR727kmg/2j33bWIAKFzTTAetQg6xoT1bwBTOjC TQF5CGwFPSj0aaOXLcURxy6m3jCu5qOjFcOCgnbtiDpOCKUlB5A9Rcm766jtPnfbhdTF +5LFMzIgyCY81h36WWqiBREpvzWgReKzjPShq/z9jBaUWy5Sd4K2sUjIMR16Pb/BUnfn GiWQ== X-Forwarded-Encrypted: i=1; AJvYcCVXO70xJxAcAZnWkez6saXL+apvuGSFISIlzNDNKjpfXbXu0uX15N23YNsT3p1aRcqBWdvp+raBQA==@kvack.org X-Gm-Message-State: AOJu0YzjCWuP5FGSNfXGM0djwZzKDnTLeQVSavcTYQuifsM9J0ermDp8 RV4pLaA9vvmbHy6f9/B00v4iTSDYXpvgmmdiM9ZcqSCiAyVa1bwloMgEA+CSOo6TXvy+vDY9Zzf KmA== X-Google-Smtp-Source: AGHT+IGMlauM2Ccm7lpcq+NmFLTze2u+1oA+ThJ4YgG3NG0DPe4hsPEtWI2lkBjtgT/O9I9Gq0F9hjCGH4Y= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:a4c0:c64f:6cdd:91f8]) (user=yuzhao job=sendgmr) by 2002:a25:dc4a:0:b0:e25:5cb1:77d8 with SMTP id 3f1490d57ef6-e337f8ed8bbmr193276.6.1731010837952; Thu, 07 Nov 2024 12:20:37 -0800 (PST) Date: Thu, 7 Nov 2024 13:20:27 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.277.g8800431eea-goog Message-ID: <20241107202033.2721681-1-yuzhao@google.com> Subject: [PATCH v2 0/6] mm/arm64: re-enable HVO From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Rspamd-Server: rspam10 X-Stat-Signature: p8sswhdy98brjbyreqzk87ipwgn8qssn X-Rspamd-Queue-Id: 64A2212000E X-Rspam-User: X-HE-Tag: 1731010792-93139 X-HE-Meta: U2FsdGVkX18aS8YeMZ8B3E3CoYYTh/5RHWCVREqQAwNPnvrEq+ZtNfFDCKJoPhEOgMVKwa8TSb+b3gcwYy3ykp+h/YRCHGIwlFktW+hgtNb3rqL9V7HYIb4tUnu8aTsciB+jX4pltMv5MtxIu60JlTlJWL0dD4r//YNtrdV67DDZhlpb5+kUtkVZmdlRukZ2cu2lN4PqsK33UfqfVh+1SfY/EaP77GQKpB/kh0wcv+MX85wn9xfzx5sqgmro5CcC6PULmg17jl0vrympA7t1ecWDa5O47127mWvx1U5846oFEJjzcMoW2nYHV6UYj33z3i7ECqaBiI0nznmxYULc66P/PBnHE8A9jIDXyeX73JgLPXWFnTMxa7rBTkuiBw6wBnnh8S6uI8KPG0HqTxbmrNw4mKURbnBF9mEHhplOqyNFLu9LhmRqGyIJ+7a/ea2Dulm8jwBKNMxN6MBoWi1Z0a5ryDRylUpJ9Epxi3yZwqTjMulH/LiamjXRTPxA6dVNnB6HGVdysz3nHIcngj8HPIOOJWN/XnsSw59vnN+zonsIFaiytse8AvC6R0L2Uav+h39omLOxsU/b41HDE48MfMswKZY/lvdbpaQzvkYNLTjxExGh8ni+D4eloK26PtQUSBJU1x/66YfEsjBCuosg1k8NsfnIM5jU0n/F1f5ikM7WOGYwqnbKTrPTMCnkF427UAOwylO+kGv1uz6Vc8MF8E3iY5r12YdfB8KIS8i/Kv5MVbF6PvnCvqgbt1zmT4GEaqYjchOYTR6uifVJzCXzysedogyQBSe2jBrN/2LlpRcOVvDMtbF0zosss+xwaO982YgZUh89DVP1RgCCIXdoG+ByBL4oQivPWJX03m6DHsb9kxfBP8soNNFX9OWU51MT2gMjL6fgEBfZT1Y/LH3cSRXcAnEgeKqT06Fk9pE1CXUTFHKTGwekw6m+zcrVVZKZHSVYU4xldol+xrEUUb0 Q3QtkAlx cnjUMekC+CE75hGUTgH2LuXF/rmoqDnUEsqTzKXLv/Gxz41DQUqsWq2CrqNKTrydeBsBFvPiTrilDpjnMBCeymzSpnmWDidX2lochEg3Ghu21vP7f2UzSTLtDRJ/XI/7X0SQjXk92DehKTU//RxJkDUH5Bev4rp2+dFsIVME6LcqIc1Y77PTzf2w8yY77NDWTQMHyXLWJgXqIQkOGz9y8tFNncCb9Hz4f1KDIigCNFeT9FjJbJjJdI5qPM0zWSR1HUV1U9Lx9eSnGVrRZWlD0KIup6RqsHuHkoN3ZtFHKTViGjAnbjGoZ/1At1ifJq/Itg/kH1365RD5Y7tcjKNZpOF0fTq2UHG11yUYYSPkfcYO1WDPTREEgbVlst134B8uG3URpGIpqyLBP1khv1igNPPfzbF0+7ZzcKAbHpoHfOSin6p3H5ZuZcccLeDo9Ekhgry3YRhgjG+GBlWK6WRRzMdvpNNbFLT+RL8lgIJW1pk9W//QqbCQYMgfWCNnUV6bO8tZAtS+nK/5amRN1kcthD/1TlpUuvL6EybviT5ruWHhUno8eyWJ3b3YQeYcgUXv1mvts8X5uQyl/rZbirO6dyyhoK03g5L+sS25Hd8O3r3WH4aH1yyXFEfGqFHgRUC0ZOq8yz/Bo2RvunfgVK+AIlSbk/pBEII4xWjGq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: HVO was disabled by commit 060a2c92d1b6 ("arm64: mm: hugetlb: Disable HUGETLB_PAGE_OPTIMIZE_VMEMMAP") due to the following reason: This is deemed UNPREDICTABLE by the Arm architecture without a break-before-make sequence (make the PTE invalid, TLBI, write the new valid PTE). However, such sequence is not possible since the vmemmap may be concurrently accessed by the kernel. This series presents one of the previously discussed approaches to re-enable HugeTLB Vmemmap Optimization (HVO) on arm64. Other approaches that have been discussed include: A. Handle kernel PF while doing BBM [1], B. Use stop_machine() while doing BBM [2], and, C. Enable FEAT_BBM level 2 and keep the memory contents at the old and new output addresses unchanged to avoid BBM (D8.16.1-2) [3]. A quick comparison between this approach (D) and the above approaches: --+------------------------------+-----------------------------+ | Pros | Cons | --+------------------------------+-----------------------------+ A | Low latency, h/w independent | Predictability concerns [4] | B | Predictable, h/w independent | High latency | C | Predictable, low latency | H/w dependent, complex | D | Predictable, h/w independent | Medium latency | --+------------------------------+-----------------------------+ This approach is being tested for Google's production systems, which generally find the "cons" above acceptable, making it the preferred trade-off for our use cases: +------------------------------+------------+----------+--------+ | HugeTLB operations | Before [0] + After | Change | +------------------------------+------------+----------+--------+ | Alloc 600 1GB | 0m3.526s | 0m3.649s | +4% | | Free 600 1GB | 0m0.880s | 0m0.917s | +4% | | Demote 600 1GB to 307200 2MB | 0m1.575s | 0m3.640s | +231% | | Free 307200 2MB | 0m0.946s | 0m2.921s | +309% | +------------------------------+------------+----------+--------+ [0] For comparison purposes, this only includes the last patch in the series, i.e., CONFIG_ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP=y. [1] https://lore.kernel.org/20240113094436.2506396-1-sunnanyong@huawei.com/ [2] https://lore.kernel.org/ZbKjHHeEdFYY1xR5@arm.com/ [3] https://lore.kernel.org/Zo68DP6siXfb6ZBR@arm.com/ [4] https://lore.kernel.org/20240326125409.GA9552@willie-the-truck/ Major changes from v1, based on Marc Zyngier's help: 1. Switched from CPU masks to a counter when pausing remote CPUs. 2. Removed unnecessary memory barriers. Yu Zhao (6): mm/hugetlb_vmemmap: batch-update PTEs mm/hugetlb_vmemmap: add arch-independent helpers irqchip/gic-v3: support SGI broadcast arm64: broadcast IPIs to pause remote CPUs arm64: pause remote CPUs to update vmemmap arm64: select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgalloc.h | 69 ++++++++ arch/arm64/include/asm/smp.h | 3 + arch/arm64/kernel/smp.c | 85 +++++++++- drivers/irqchip/irq-gic-v3.c | 31 +++- include/linux/mm_types.h | 7 + mm/hugetlb_vmemmap.c | 262 +++++++++++++++++++++---------- 7 files changed, 362 insertions(+), 96 deletions(-) base-commit: 80fb25341631b75f57b84f99cc35b95ca2aad329