From patchwork Mon Jul 11 03:46:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 12912781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79A30C433EF for ; Mon, 11 Jul 2022 03:46:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8E7848E0002; Sun, 10 Jul 2022 23:46:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8973B8E0001; Sun, 10 Jul 2022 23:46:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 739628E0002; Sun, 10 Jul 2022 23:46:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 610B68E0001 for ; Sun, 10 Jul 2022 23:46:40 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2BB04336FD for ; Mon, 11 Jul 2022 03:46:40 +0000 (UTC) X-FDA: 79673432160.26.57C0DA9 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf04.hostedemail.com (Postfix) with ESMTP id C53794001E for ; Mon, 11 Jul 2022 03:46:39 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id b2so3398728plx.7 for ; Sun, 10 Jul 2022 20:46:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qmBhjPhdel7Ory0h+LtHmaoT8CpQHwHtjRSzAczsGS8=; b=UY5oZcsZHFdGH9KcetDN8dqr2Mf0qiwqMqlLdkQ+0PNZPPWK8nytWfBrVg68lCyEAN jrVXhCD+p/9vhueaNcEswnstaaqhbi96KvKB1URC0IEAGTbb7bQ3L2sJKv8fP4/q3fdz qnhDuiRe5eJunmVyfmUC7y6G6huMy6rKKY2vpo7MUN+GN+8Yg69bDGSpiExtmJG6nmwc HGrj6W5Zy4EILKOUdTTBtFEUayRpo9P6vRZIShjByCsnQmdWy7cHknQIBwTn6y+6JaHt 7Jslc7TiuDWWVL3D1WVwoeXWF4xp0s+zIBfTFhbW89tgNx9jEYuB6VyOj7QkOyyjH+tF MEOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qmBhjPhdel7Ory0h+LtHmaoT8CpQHwHtjRSzAczsGS8=; b=XTmiXDadGbMyalq49psxaQLWNx/zmRNyierbzdr8brRJOVLEXbNX/IkYWd11VMF70u h66Las2GgjBZYeta64KfCFhiIm4nUHISgq18N4npzuiYcmpoCMZsMzOJSQqwoHBgvIIv f43SETJ067tN0Rws/52buvOHqJp3aCpJuTHsS4oEsOJFLYcTxjLyXorLg0clPe2WZRhO r0TNk+BsrNK9r32wRTL/KIi6HaoBOuDeyFfi7tDDI799nOdMmrGvBmKSK+BWT+dxyP75 MT+5EUkGuIfo8eC86rJDvSrXZ2gBDdXH7j+9gQDMTD+mcf4mURcfF7E2RutY3CiLOjmT AmyA== X-Gm-Message-State: AJIora9GehNSKiHSQXXJEAeu8Lmcidia+3kraecIV5SCqSgw2GEeAWE1 gMfNrgZto4WUQeEbGV9bLEU= X-Google-Smtp-Source: AGRyM1uXqBxzEnWgTK+S6bqpqoqltGNYamS7CTaFawotYp6lLuQkplQPx4DboSyk03U1rFf9/7Vqqw== X-Received: by 2002:a17:90b:1b48:b0:1ef:a90d:eae1 with SMTP id nv8-20020a17090b1b4800b001efa90deae1mr15050227pjb.68.1657511198328; Sun, 10 Jul 2022 20:46:38 -0700 (PDT) Received: from localhost.localdomain (47-72-206-164.dsl.dyn.ihug.co.nz. [47.72.206.164]) by smtp.gmail.com with ESMTPSA id a13-20020a170902eccd00b001664d88aab3sm3447949plh.240.2022.07.10.20.46.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:46:37 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org Cc: corbet@lwn.net, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, yangyicong@hisilicon.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Barry Song <21cnbao@gmail.com> Subject: [PATCH v2 0/4] mm: arm64: bring up BATCHED_UNMAP_TLB_FLUSH Date: Mon, 11 Jul 2022 15:46:11 +1200 Message-Id: <20220711034615.482895-1-21cnbao@gmail.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657511199; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=qmBhjPhdel7Ory0h+LtHmaoT8CpQHwHtjRSzAczsGS8=; b=NSkdl//vuDtXlNGIueNfpruHId+hOPNTvgDLP984bKc03NccFYLr5CZkfcVDnI14i+NDJF xdEIVvETpMFFuiUFmZHnbsTKt3BLS/L4Nwy3kevhGALOzBg6JsV7sB5bMWgceqfscr/l31 C6DM4TS18vZiQPNWG1Rk71cUxaRVdag= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=UY5oZcsZ; spf=pass (imf04.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657511199; a=rsa-sha256; cv=none; b=emYXF3MIA5+2/ZzeFliGKh3BO5VIqv0nB1VbVo/FhR/kY+A81i4MY4+c6BdPU627aOyzU6 XXmujwFFjmwztz5vsfdkrvAIGHvkfGvdXe9JjP0Ukk5JGo9m2r+h2HHEcaTjmS694ApnyA KJxh5XQlosdF2jx5qj5Lfpw+16hDvpk= X-Rspamd-Server: rspam02 X-Stat-Signature: yd9qbkm6heohcrigxd6rr87o53h9eugh X-Rspamd-Queue-Id: C53794001E X-Rspam-User: Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=UY5oZcsZ; spf=pass (imf04.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1657511199-666631 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Though ARM64 has the hardware to do tlb shootdown, the hardware broadcasting is not free. A simplest micro benchmark shows even on snapdragon 888 with only 8 cores, the overhead for ptep_clear_flush is huge even for paging out one page mapped by only one process: 5.36% a.out [kernel.kallsyms] [k] ptep_clear_flush While pages are mapped by multiple processes or HW has more CPUs, the cost should become even higher due to the bad scalability of tlb shootdown. The same benchmark can result in 16.99% CPU consumption on ARM64 server with around 100 cores according to Yicong's test on patch 4/4. This patchset leverages the existing BATCHED_UNMAP_TLB_FLUSH by 1. only send tlbi instructions in the first stage - arch_tlbbatch_add_mm() 2. wait for the completion of tlbi by dsb while doing tlbbatch sync in arch_tlbbatch_flush() My testing on snapdragon shows the overhead of ptep_clear_flush is removed by the patchset. The micro benchmark becomes 5% faster even for one page mapped by single process on snapdragon 888. -v2: 1. Collected Yicong's test result on kunpeng920 ARM64 server; 2. Removed the redundant vma parameter in arch_tlbbatch_add_mm() according to the comments of Peter Zijlstra and Dave Hansen 3. Added ARCH_HAS_MM_CPUMASK rather than checking if mm_cpumask is empty according to the comments of Nadav Amit Thanks, Yicong, Peter, Dave and Nadav for your testing or reviewing , and comments. -v1: https://lore.kernel.org/lkml/20220707125242.425242-1-21cnbao@gmail.com/ Barry Song (4): Revert "Documentation/features: mark BATCHED_UNMAP_TLB_FLUSH doesn't apply to ARM64" mm: rmap: Allow platforms without mm_cpumask to defer TLB flush mm: rmap: Extend tlbbatch APIs to fit new platforms arm64: support batched/deferred tlb shootdown during page reclamation Documentation/features/arch-support.txt | 1 - .../features/vm/TLB/arch-support.txt | 2 +- arch/arm/Kconfig | 1 + arch/arm64/Kconfig | 1 + arch/arm64/include/asm/tlbbatch.h | 12 ++++++++++ arch/arm64/include/asm/tlbflush.h | 23 +++++++++++++++++-- arch/loongarch/Kconfig | 1 + arch/mips/Kconfig | 1 + arch/openrisc/Kconfig | 1 + arch/powerpc/Kconfig | 1 + arch/riscv/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/um/Kconfig | 1 + arch/x86/Kconfig | 1 + arch/x86/include/asm/tlbflush.h | 3 ++- mm/Kconfig | 3 +++ mm/rmap.c | 14 +++++++---- 17 files changed, 59 insertions(+), 9 deletions(-) create mode 100644 arch/arm64/include/asm/tlbbatch.h Tested-by: Xin Hao