From patchwork Thu Jul 27 18:55:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13330588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95B87C0015E for ; Thu, 27 Jul 2023 18:59:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 309FE6B0072; Thu, 27 Jul 2023 14:59:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 293FD6B0074; Thu, 27 Jul 2023 14:59:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 135D96B0075; Thu, 27 Jul 2023 14:59:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 056996B0072 for ; Thu, 27 Jul 2023 14:59:08 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id AECB54052A for ; Thu, 27 Jul 2023 18:59:07 +0000 (UTC) X-FDA: 81058304334.12.A0A182F Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by imf10.hostedemail.com (Postfix) with ESMTP id C4003C001A for ; Thu, 27 Jul 2023 18:59:05 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=IdGa66Ut; dmarc=none; spf=pass (imf10.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690484345; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EDamHVk4ujUG+CHbAQWQotzld5TgznJ3XAxjQYSqNSU=; b=69zkiBIoCj7GijGmcMk0C01cR125C0C+bbr+6b2R8F3XSwDd1kzAA7q/hh0Q+uM5QTVnhg QpaUNLj4CRN9POUb54jeDL1wjoSujJiduMuJU+S6Djwzd3eiVdGE7NhEUIbkapSig8h+vA 2vVh4ZPgjW6GUqZ87pr1XesCrsTFXZk= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=IdGa66Ut; dmarc=none; spf=pass (imf10.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690484345; a=rsa-sha256; cv=none; b=v8dqcrOq1jcILaK7tl4dBfzxdQgxbMQXWtwjTXNf8OqGa0m/5AHUlyRW91IA49+uD53okS LISQjk3pvBtVw+iE4Xr5PKtcJL1pNhy6MrurDj5cDHpQWtBoIKzuHU5a7URwN6pIoeXJNa +R+MXXepjOHUSjKau0JNgfOUolNO3QA= Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-3fbc244d307so14246615e9.1 for ; Thu, 27 Jul 2023 11:59:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1690484344; x=1691089144; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EDamHVk4ujUG+CHbAQWQotzld5TgznJ3XAxjQYSqNSU=; b=IdGa66UtIIFbYYRT34X7nXZCEdzcXTUXP42pHU+fRDakhGzxioo4R0X4pMQHNU3UvT cb6zYYiY1GSCrHL23POq5GqGsmx16xdh+ORbhy0jbLlh3508NDq2BU8MNyVptXqxHU3m z6Huh+dgLHgTmMh6/GFFtZgMKDfLTF/lLUhuu6lFFzZ5FfNz2M+UalL7Nxy2wcTt4Urm vHdUKMcDMqL0Rjs+nJEbZgdzu00ikS2T1OAVYH9IyrUU52fcpQTId9PDN+EBqrinVVD3 YmbFuFktcTPCtmoQ7c19/s3iT4UVf0BCy+pzRa9E9o/mm8ujQqggxhdzJL4AmSxTKYKG +0qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690484344; x=1691089144; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EDamHVk4ujUG+CHbAQWQotzld5TgznJ3XAxjQYSqNSU=; b=K9bHZ6adL266Z1T3iZJZB6UTZK966D+D/Mp43vpN8plK8s4CnSRhdWnR65spJyR9pQ 30j3MviWcgnc4WzlcMdkM1NzisN0BRjgAP4mTYwKLl2tGZ3tCzFdGdnvRi4tb1ySXHY4 la7nPxH+dT4gHPMEla+jY2RMUn6QhGqCu+alyQU0k4lX2oUr0Onwx+Wgk4tAv8ej9IDa ejINN1TrjFTGuv+Lkg+9sXVDxR5vzgYvRmAgR2GTOzuJOcVRS+FayVwRXJNCowY8CATK C9tdePtSbeALk9IReykrkvPo3iCBU9AWYsFSzi5mzB8OYknQGe/glh7psPJWhYYkrAVp L4sQ== X-Gm-Message-State: ABy/qLahVK4Zr27k/CvIocRBT6gHbmJys5uoXK8Fc8P+OvPWjIxe8SZn zO08n8TZMSqoMmeV8tOcNdEwjQ== X-Google-Smtp-Source: APBJJlHhcGHjdWFgZg+N+yKc2Y2lhGlFYBwO6VQdAW4Af4g86WHeGB5c7zLJbHluWyguraWCOhnK6A== X-Received: by 2002:a05:600c:aca:b0:3fb:a937:6024 with SMTP id c10-20020a05600c0aca00b003fba9376024mr2398881wmr.29.1690484344310; Thu, 27 Jul 2023 11:59:04 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 2-20020a05600c22c200b003fa96620b23sm5376517wmg.12.2023.07.27.11.59.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 27 Jul 2023 11:59:04 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v2 3/4] riscv: Make __flush_tlb_range() loop over pte instead of flushing the whole tlb Date: Thu, 27 Jul 2023 20:55:52 +0200 Message-Id: <20230727185553.980262-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230727185553.980262-1-alexghiti@rivosinc.com> References: <20230727185553.980262-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: omaaeqiaqta6opdfknab4kmi1pjm4brb X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C4003C001A X-HE-Tag: 1690484345-694663 X-HE-Meta: U2FsdGVkX19789QCtrwLn36cYJIFAFu/23YFE1Hvkv1izOtXdltir/U6K70G2dbFTMnyEwTinCRzZAgHCA7sW29ymtjUWZd6ElQ8cSAOsN7nSKDHLNTcVefVnTF8STD4QCDHu4MIkYSnF4JTpklhe3507jD7XBhBV1sotb3PLVMwnWE/8k5LxVGzAyBBo7joItodJwuFpwRlM0oGQal1r/eAMMz7Aug++e9Go73l2kWXJiybo9aoHGJWXUb/cW/HOi6oUVHShtUqVSvLNwWYBBEmhO35K+176FRtdYu14ZNUL0jcbs3s7Di9D6F9e1+DWTY/vM+toqJ3gz0pkZykfJez9zBFUFet5qF7JzyTvktPzW3G6wQUI5Yh5lRztgoLHkjFlvYD7cdmZmovJYkBsT1HKQzXTXwUM+91NTAAfe1uZqBp49EIPB3dxwudDN564yWPua+X+/uVnqM99tIeDHCB/BSeR0cUXj5AgTIyMLPVvamgSqd4L9EB5WICXjBX64LIDsifIjc0VsABkkMF7AUbgAihFdyxGdb4eIsw8xxVDZyqKfTm5gKDrh5EAt3Kl8IRCWQjlQwOyTOZQGzWM9ugkoLpnP9JBPLsqoWPtC/Q9WCRdmF3+ryB4uBaTh/bz5uQ8ewctWLcYu65q6i/rS98wrZGLIvKtXrMH7ym0z2PKfw6nxCSTG6pbLL8exH9JQTf63h+4wpvqnA1Kz6Bu8IoaFa/j0WQ0wEE65Iv34UBj/Spz7IPxFrvfMBuFWSEu7qkTY1u7FiWmVzU/TEVcBHCezwq4PufuG+e5mWfEeLrb6JLTeNtrcLSm5Zcc5PEjCdJPvTlE8jd3mJJLRHYvzRfBMVvqSQjlRyV3N0smkP50TQGfQxQzHKkvtD8dEvqdmzQ9mFiYQJSN54/RjrxEWhHXMcxjD/hpKfbDctLTls93YktMsu+HP6w0kFdwMpmk/Oco/nvmQd80ZH9xWe h5lplMik LHFaKK6x+O5AOHfO/iKj+Zvc2VtZB/u+SJbfthyQcVNZHvLgdeM/OX2V1KoktMLUpagHpTyAxpgGrS9t31++aoc7Cqn4mM0hGCFI04a7sLPxlsHYyAK5vFL98Q4Tz5iraMw1bUq4U6KC0f1gQLUt4kru8MeLga8n9jihqh2NdJcOj9uyObbfJDPuKKzKdPiNzA+a18eOKcCa4kQL4IxiamQCIfwEdRsEAUDPM0Qj4MN4M4ev8kGi3WkjeQDcLruNrQIev7YXX5JVJM4e70A4+bqxSjkYhO9bOjegqlb0agc3erdxEtsW7IlDOMLiz780NS6Gl7ilb6133clNmkelVYhk8S5X7JXLrkpBExw9RAGG4IqygYkv6pvDWeDpICv0WxhdG3G7A8lM/qTf79FfyeTHAcGp8pz6mD4Ep/ANYrpOiCAZD4Qaay8VJIDQdZU/6nt8kFOLkUJkfvPRwLnctSDFSL5FPADDMq+y5QaR5zpNABROmTcpJ4OI/L873nWe0RHOOeawuCBnTGoSgwfHRRCAvnTvyL8iQtR+sJcANTr9/8uc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, when the range to flush covers more than one page (a 4K page or a hugepage), __flush_tlb_range() flushes the whole tlb. Flushing the whole tlb comes with a greater cost than flushing a single entry so we should flush single entries up to a certain threshold so that: threshold * cost of flushing a single entry < cost of flushing the whole tlb. This threshold is microarchitecture dependent and can/should be overwritten by vendors. Co-developed-by: Mayuresh Chitale Signed-off-by: Mayuresh Chitale Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/mm/tlbflush.c | 41 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 3e4acef1f6bc..8017d2130e27 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -24,13 +24,48 @@ static inline void local_flush_tlb_page_asid(unsigned long addr, : "memory"); } +/* + * Flush entire TLB if number of entries to be flushed is greater + * than the threshold below. Platforms may override the threshold + * value based on marchid, mvendorid, and mimpid. + */ +static unsigned long tlb_flush_all_threshold __read_mostly = 64; + +static void local_flush_tlb_range_threshold_asid(unsigned long start, + unsigned long size, + unsigned long stride, + unsigned long asid) +{ + u16 nr_ptes_in_range = DIV_ROUND_UP(size, stride); + int i; + + if (nr_ptes_in_range > tlb_flush_all_threshold) { + if (asid != -1) + local_flush_tlb_all_asid(asid); + else + local_flush_tlb_all(); + return; + } + + for (i = 0; i < nr_ptes_in_range; ++i) { + if (asid != -1) + local_flush_tlb_page_asid(start, asid); + else + local_flush_tlb_page(start); + start += stride; + } +} + static inline void local_flush_tlb_range(unsigned long start, unsigned long size, unsigned long stride) { if (size <= stride) local_flush_tlb_page(start); - else + else if (size == (unsigned long)-1) local_flush_tlb_all(); + else + local_flush_tlb_range_threshold_asid(start, size, stride, -1); + } static inline void local_flush_tlb_range_asid(unsigned long start, @@ -38,8 +73,10 @@ static inline void local_flush_tlb_range_asid(unsigned long start, { if (size <= stride) local_flush_tlb_page_asid(start, asid); - else + else if (size == (unsigned long)-1) local_flush_tlb_all_asid(asid); + else + local_flush_tlb_range_threshold_asid(start, size, stride, asid); } static void __ipi_flush_tlb_all(void *info)