From patchwork Mon Oct 21 04:22:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843546 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DABBED3C93E for ; Mon, 21 Oct 2024 04:24:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:Mime-Version:Date:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=djmWeapaSQGC8nHmrg76kCEnRDM6khIeNx/vxZy7BEg=; b=WfQ/tgv+UV9HwTomxH3WEl+FQ2 IL1SnW1WqnodJT6qwPYl8EQDCndXCgbTWg8HXhPCaRhnI/pD913ungu2yDTU7nXdtL98BYKUOrEoT DOb/Lq9281hM5teVURLTFUxTuGmLuGDO9Fm3At+liRvT8CXFJhM9/z4blCS4bQmK6xYIQc+PKPA8X d1f3MbKW0slWvWTR/vAQlRlLWIkvvtNqzdkhQXvoMMBfv3EBz61rQjZ1gnjsGvwEk7X3NjQ3yLDhz Io4ks0GI5xexolgDagE6Hev/7cam77TB+ZmZFblc9Z13V+XQ1YVlXb8ihCfmm4FcYOKvBg+ICQ+9J +5Leyr1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t2jxY-000000061C3-06Kz; Mon, 21 Oct 2024 04:24:00 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t2jw0-000000060qO-1ky4 for linux-arm-kernel@lists.infradead.org; Mon, 21 Oct 2024 04:22:26 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-e2946a143efso5294069276.1 for ; Sun, 20 Oct 2024 21:22:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484542; x=1730089342; darn=lists.infradead.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=djmWeapaSQGC8nHmrg76kCEnRDM6khIeNx/vxZy7BEg=; b=RWHXCAkzsZVMjHTpDsN9w5vi80QvoNsdoKw9jvQoVM2zQYOSDqs1wbEZ76YgL5Xnt1 AwN0A0yI/f88J/80RvMOTGJIFhwi9xTLe78swSwpolfz2i+neVKPvx3NQhtrrjkLTcNT 5Epax9bB1kXnxp3rMYmUU4SkKO59LQ/YPK3ugSjeXkZKk1gYW8tZWmHW4AoWSAzfnr28 AXWTrJMfGwSHXcu5ZW/9f+LTFC5EKblyxAZx5IOvVYeANrN8CxYBGA9bBtPKH6RGV5Xo cXedxBI9BeuBkuW8c1XV0EaH+mSum4S9XHNn507E+wFIO0NUwfuAj41stQUSUFcZ4bDk sLIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484542; x=1730089342; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=djmWeapaSQGC8nHmrg76kCEnRDM6khIeNx/vxZy7BEg=; b=mElzaSKKPhIXLDkbLT30CC2i5pW3fbch34IPnwwMz63RI6RuyshDJWcMvWoz6zHDKx wddFwN0i7LehnDsBGQ1+rjY2XWWwi/rHOO6WSXYawuV1XDGEVy9e2iYV7o6D1fSX+eNx 9aqZ0Q61p6JUruH6i/SRJtjAzREu2n4qPn0kA1jD69QTLd8bcHcZVhdlzL1Dstx/wIGb hyj43b3XhRGPGTPJ/xwVsL5NjXvIrzdKBobnzxFo12WSQifQnKs24Iin1FEXvgHTIQRI sb895rFsyJk6c5tBVgb1svRfcT0fQLd6C1zrWOa0ToBJuf6cotBn75JfWlCmcDhI408s G9kA== X-Forwarded-Encrypted: i=1; AJvYcCWdZHBhidq8U13qUhDCm0j7q+adp8ZhRWY8jrHwOkB19hovHAhrTkVbOly7aTWFqXwdFrLtiGdj53auyZ/eXkZ0@lists.infradead.org X-Gm-Message-State: AOJu0YwU5+La0hN7DNo5samZBJHIYff80gOIU69YxqEnWxt5wUEZfwc4 29nurE8H9RxKXiBqbZ+6XmPh9teeQqq0aKqsUFdqSDJc8FkRgv9pc4YVBmaDcr7UdRh/Rt3vosp hnA== X-Google-Smtp-Source: AGHT+IGG1IXCwOdLAyki3IzhZW4dyFxYFnYAhLqXXbmqpD2Dv/xXTcGUjx179gaMsBuxYaYLTRxOt7OLXTk= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a25:72c3:0:b0:e29:ad0:a326 with SMTP id 3f1490d57ef6-e2b9ccc8449mr48383276.0.1729484542310; Sun, 20 Oct 2024 21:22:22 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:12 -0600 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-1-yuzhao@google.com> Subject: [PATCH v1 0/6] mm/arm64: re-enable HVO From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241020_212224_510497_C33C9D56 X-CRM114-Status: GOOD ( 11.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This series presents one of the previously discussed approaches to re-enable HugeTLB Vmemmap Optimization (HVO) on arm64. HVO was disabled by commit 060a2c92d1b6 ("arm64: mm: hugetlb: Disable HUGETLB_PAGE_OPTIMIZE_VMEMMAP") due to the following reason: This is deemed UNPREDICTABLE by the Arm architecture without a break-before-make sequence (make the PTE invalid, TLBI, write the new valid PTE). However, such sequence is not possible since the vmemmap may be concurrently accessed by the kernel. Other approaches that have been discussed include: A. Handle kernel PF while doing BBM [1], B. Use stop_machine() while doing BBM [2], and, C. Enable FEAT_BBM level 2 and keep the memory contents at the old and new output addresses unchanged to avoid BBM (D8.16.1-2) [3]. A quick comparison between this approach (D) and the above approaches: --+------------------------------+-----------------------------+ | Pro | Con | --+------------------------------+-----------------------------+ A | Low latency, h/w independent | Predictability concerns [4] | B | Predictable, h/w independent | High latency | C | Predictable, low latency | H/w dependent, complex | D | Predictable, h/w independent | Medium latency | --+------------------------------+-----------------------------+ This approach is being tested for Google's production systems, which generally find the "con" above acceptable, making it the preferred tradeoff for our use cases: +------------------------------+------------+----------+--------+ | HugeTLB operations | Before [0] + After | Change | +------------------------------+------------+----------+--------+ | Alloc 600 1GB | 0m3.526s | 0m3.779s | +7% | | Free 600 1GB | 0m0.880s | 0m0.940s | +7% | | Demote 600 1GB to 307200 2MB | 0m1.575s | 0m5.132s | +326% | | Free 307200 2MB | 0m0.946s | 0m4.456s | +471% | +------------------------------+------------+----------+--------+ [0] For comparison purposes, this only includes the last patch in the series, i.e., CONFIG_ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP=y. [1] https://lore.kernel.org/20240113094436.2506396-1-sunnanyong@huawei.com/ [2] https://lore.kernel.org/ZbKjHHeEdFYY1xR5@arm.com/ [3] https://lore.kernel.org/Zo68DP6siXfb6ZBR@arm.com/ [4] https://lore.kernel.org/20240326125409.GA9552@willie-the-truck/ Yu Zhao (6): mm/hugetlb_vmemmap: batch update PTEs mm/hugetlb_vmemmap: add arch-independent helpers irqchip/gic-v3: support SGI broadcast arm64: broadcast IPIs to pause remote CPUs arm64: pause remote CPUs to update vmemmap arm64: select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgalloc.h | 69 ++++++++ arch/arm64/include/asm/smp.h | 3 + arch/arm64/kernel/smp.c | 92 ++++++++++- drivers/irqchip/irq-gic-v3.c | 20 ++- include/linux/mm_types.h | 7 + mm/hugetlb_vmemmap.c | 262 +++++++++++++++++++++---------- 7 files changed, 360 insertions(+), 94 deletions(-) base-commit: 42f7652d3eb527d03665b09edac47f85fb600924