From patchwork Tue Jul 11 07:54:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13308218 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 738F7EB64DC for ; Tue, 11 Jul 2023 07:55:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 065C08E0002; Tue, 11 Jul 2023 03:55:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 015868D0001; Tue, 11 Jul 2023 03:55:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFBEA8E0002; Tue, 11 Jul 2023 03:55:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D0DC98D0001 for ; Tue, 11 Jul 2023 03:55:41 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9E6B41A014C for ; Tue, 11 Jul 2023 07:55:41 +0000 (UTC) X-FDA: 80998571682.02.FBA1BED Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by imf06.hostedemail.com (Postfix) with ESMTP id BF151180017 for ; Tue, 11 Jul 2023 07:55:39 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=NCoKYaei; dmarc=none; spf=pass (imf06.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689062139; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/EU/iMG6yg29uBo51A2iBmY9OZkda2o1/b9d0LOiq1U=; b=8TLCN6CR9aiQML3ZwCuZuF59/iFf5BwHcSYsFaXPEN4gKpEkUhrUygCFY4wlKI8bK5OAky NqF9BD9lIuQcEntiJoIuWEsC2s0YUVQYEuSGcLApG4mJTCKlBREYzZgj5V6fqivD6LLMLN OceAsFblPIWLm2RYAuFrLqmjKnrKCWQ= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=NCoKYaei; dmarc=none; spf=pass (imf06.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689062139; a=rsa-sha256; cv=none; b=d1XhNBgl5/zhOi8MzXkspD2lSsB1beZhYcz+6mKfaVcH0UEzh2it4MlL8JZ9uE2tsWYqOx Nx/gl0O/kpBn9GLEwv9+JE0EXdqMuxKa4ez4cP2UbfWleOrTuoPcGl1af1aQm+gULqGVZT Ix1kKFPFhdoBs1AO62uRZJCcdUjHTjM= Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-3fbdfda88f4so55147255e9.1 for ; Tue, 11 Jul 2023 00:55:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689062138; x=1691654138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/EU/iMG6yg29uBo51A2iBmY9OZkda2o1/b9d0LOiq1U=; b=NCoKYaeiv1nFsZu1kkN9IKONZ0ywf8MrhYyLoYAG7Q6i/1MQkq4oT+ftSJXO49RKyX ACt2ggo37pYi+8YgPL8PJ6dxHo9tA4/bLupo3TXiy9URZ4wDKkU28wB9BygUOld8z2e/ 3zc8DUiMdWOgQtJ0Ndn4UV5Li6hfcg8Kga3ZZhjNakNirjxP7y+vfmb9LrM9PCGhIPob Hdzs0ngha48Jpks5P0qe+0HhpMg4tmNZJB4wqnH4M1bv2Zwy3CCAuPi3okrxJTIKbojW InsOcPcSYd20w6iszei4mg1pDK4LCzlrLwfCY1fjznaJUncNDxte3J8oxZxhilbHxfFV 6g/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689062138; x=1691654138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/EU/iMG6yg29uBo51A2iBmY9OZkda2o1/b9d0LOiq1U=; b=gmxVFyWOLDliOSF/06d6RmXH3AyDmrCenwqQlMWuQyrT5HG2e4qZfjO6r3UW/DkUhD 4lgGa5ubCVkBIfdo4SdN250/bVjvOJvH28qPd2avyph86ev8WKTnw2bCtku/wevgmxYv 03czzKpbOEroZnYKq2UPbvtSAbjxRRgsiZBXbmzA7d8qxfAWPtSLImHtKxwQMMp030Be mKkjlBNyi9hms3M+ZTRIo/axnLkVSwLkRuSU8NwKiwiGFxkfEu2laeuF41I8qYmCOc0R TTQZfG9C1JW/TE/SSzYnGSoEotCX/a69UjJAZB6qO+CeP/xVohxNvuaDCHMR1luY6JSy 1S5A== X-Gm-Message-State: ABy/qLYBDXFC2jiGWxkNGAgr97t8yq+2pD5ez6Y4J/uU7oM178CkOC0N dsnOhYn04hRFyD6Xmc6fagWfCQ== X-Google-Smtp-Source: APBJJlFW7UA1mVn1ZcXadTPXanJoND7bXUx+Kn681JHSgt5iPRc/xDSnmY7M3nrypuHmGJz4jg4M6g== X-Received: by 2002:a05:600c:b5a:b0:3fc:60:7dbf with SMTP id k26-20020a05600c0b5a00b003fc00607dbfmr9944380wmr.41.1689062138206; Tue, 11 Jul 2023 00:55:38 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id f19-20020a7bcc13000000b003fa973e6612sm12317248wmh.44.2023.07.11.00.55.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 00:55:38 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 1/4] riscv: Improve flush_tlb() Date: Tue, 11 Jul 2023 09:54:31 +0200 Message-Id: <20230711075434.10936-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711075434.10936-1-alexghiti@rivosinc.com> References: <20230711075434.10936-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: BF151180017 X-Stat-Signature: krnjktrzg8mxfo9g9jnsbwjnpq9tibgc X-HE-Tag: 1689062139-43372 X-HE-Meta: U2FsdGVkX18djfXgbiS9CLTtIK3pQviyTx27jD8ZwSIkhdQqE20awggfX7AIGRa/zS0PSmXlO1vtRZsOhthrFqrcPcYq6lz6HJ9BDPk5wgA53/e+TMFDC/Xe7mq61wpzmuBvLkRMvkC6D43zVDAB3u3+dS8y2UrHxbgQ/qpw6AGK5bMILcECa3DFDGWCXFt93by3nx26l2S4YEsPaGuZ1wxI/DV3iYlFh4swQbJb5xwOEyMptD+yc8sppmtnxlUcOqiVVUe0JzsnyV9ER1sm7iPEtBnl+Gx1lQH+dQbG0qYtRxRRFGPhmzrelN/Qw/Cj/xXD8l6I7IAuKmLYPf3Dqw9qoFfBILnOxY6/kvuqjc5hgmi4PXgqBkReKHGQJh7vRd3TU5poJrHBvoAhIRybFEf79BFGUreEip43vweNe2JDHa/Q+FASiUoGZgp87z+x2L0mesXQ1GbxaKdFELh/TZ7ikdNdoZGjytd3d0r0kiRFZDzz7jlSghJ4yXvJpLeuaBPggiAMc0gqzzJb1i5eULwBm0JVac7rKaTpwnzrTY7B2HpVeW2Vg5K6cJXkbeUHp/HoK4qz8E6UvIWOZthRRuqQs92bPagKEy9vyTZiIh2HqokjdhL3Y3NdsJKDUm1g2abKq4KW/nLE6ozxfeHo8V0UGMhLanboHeAeYeGRntOhlw/574GSrWEv47qba8VoIZ7lQ7BWDlCICxYm1pAqMlaSW1tZEA8Pe41w8Gs8XDPPrbKlcX8r2pPx2hL/BOyxxnie/V4EggyCflTK+3k6KFAHJm1SIOaigJ/RGBodBblPE/Zx15rXaSsJoJr1QBdsZh2IBkQ1BumzGjOOkEXiVo7uqnrKi3eEHaaAejer9+Xrg5eAhp6/f/XvQMaBMKnzrXqb0sisIdX8qDRHJPm9p5UPsd2QAd0/e7rBjNL5b2fr7kbLQZXbdbAI74iN08FOnK7sn1DeDX8LDYlV3hb foH0Z0Ld u88ej3XiZyhU7NbJS49nHDxt1LfBpuZJWJTIbeeNAXfaz3TpPKY8ZZPVx21Vm51qLOd2wjqD5NGDnmbHfJWp3CQgeQHFB4PgVzGJvsYxaGrSZ/kyKM9BeuvWDtkTr+H/xPhNVVG9hk0NlzVt+wZEodInZDOxe7ySEgkqc/IRI0Vsn9d5HS7qmX5zOip4AqCfRdjSGJ139LhG1ScyrEclmTpJ8ADXteaRpfv6EBb66kf+nZk1NegKBfqAWaiYYQM5BjXJ1hzj3C+TSPc12bj6nHZIpXdGaATwiNG1baKeg/CoNlvP63lfy0J7t6dvzTq+rXuCN62XFYr3rTUyaf5Fiq3xUoptXwLuyHGohwFUAuaTaOKt7BZo4k1n6l+xSz0EmcUI/FEdq/zIup6u9/YckkASajOwdmTv0v7krdZZBH024jxtDIvPUxodBoauIy9DauuL0GbBzWoCKApsN/LadYQkPfcsQgJbgdV6s/gZmAQYUGpvR1YkYWgV+Mg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For now, flush_tlb() simply calls flush_tlb_mm() which results in a flush of the whole TLB. So let's use mmu_gather fields to provide a more fine-grained flush of the TLB. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/tlb.h | 6 +++++- arch/riscv/include/asm/tlbflush.h | 3 +++ arch/riscv/mm/tlbflush.c | 7 +++++++ 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 120bcf2ed8a8..3373dcf0f413 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -15,7 +15,11 @@ static void tlb_flush(struct mmu_gather *tlb); static inline void tlb_flush(struct mmu_gather *tlb) { - flush_tlb_mm(tlb->mm); + if (tlb->fullmm || tlb->need_flush_all) + flush_tlb_mm(tlb->mm); + else + flush_tlb_mm_range(tlb->mm, tlb->start, tlb->end, + tlb_get_unmap_size(tlb)); } #endif /* _ASM_RISCV_TLB_H */ diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index a09196f8de68..f5c4fb0ae642 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -32,6 +32,8 @@ static inline void local_flush_tlb_page(unsigned long addr) #if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); +void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, + unsigned long end, unsigned int page_size); void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); @@ -52,6 +54,7 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, } #define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() #endif /* !CONFIG_SMP || !CONFIG_MMU */ /* Flush a range of kernel pages */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 77be59aadc73..fa03289853d8 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -132,6 +132,13 @@ void flush_tlb_mm(struct mm_struct *mm) __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } +void flush_tlb_mm_range(struct mm_struct *mm, + unsigned long start, unsigned long end, + unsigned int page_size) +{ + __flush_tlb_range(mm, start, end - start, page_size); +} + void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); From patchwork Tue Jul 11 07:54:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13308221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 399E5EB64DC for ; Tue, 11 Jul 2023 07:56:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A29D48E0001; Tue, 11 Jul 2023 03:56:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D9798D0001; Tue, 11 Jul 2023 03:56:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A1A58E0001; Tue, 11 Jul 2023 03:56:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7D1BB8D0001 for ; Tue, 11 Jul 2023 03:56:42 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4939A120149 for ; Tue, 11 Jul 2023 07:56:42 +0000 (UTC) X-FDA: 80998574244.12.6DEE092 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by imf26.hostedemail.com (Postfix) with ESMTP id 6FDE1140005 for ; Tue, 11 Jul 2023 07:56:40 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=Qf1MDi4I; spf=pass (imf26.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.48 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689062200; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FofimiyN+7gyzGEdPyrBP2skR/4a1H3RZbKcHwnJwCU=; b=0S4PjUgNX6xKN4n5OEsQNcksm+oSQ4RefzK6BX3mzrxpHV5hcl+rxDSzOVT5Oc+4Urljpm 0+th5zuQYRU2b2DG0X3A1g//XVf8gGemBDJqipcAlJa4ohLoXLyj249X9D7qjQgKM8z+Kg XVOEDU1renDFVKKFDANkHM2+JEle4kM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=Qf1MDi4I; spf=pass (imf26.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.48 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689062200; a=rsa-sha256; cv=none; b=fsyk0pc5Im5GrIY7lyInrN/uKBOMR/DTkVOffBi4O5UloYWlOqmjIvALDzTh9X8/oXo9R3 pO62yYBVZKnOE7H+p+ezpqg6gsc+10w5RHelCCSSRzzS1PdYK45wI/owOZb0Mq6InGBYyS SWpoNLBaV9yUbWVuNQr7BQap11L7Rns= Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-31438512cafso5831003f8f.2 for ; Tue, 11 Jul 2023 00:56:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689062199; x=1691654199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FofimiyN+7gyzGEdPyrBP2skR/4a1H3RZbKcHwnJwCU=; b=Qf1MDi4IQVSgi+mVmNSDLBvvWhfd8ba6Ayo48Gj0+fuhQZ4V9G61dBcMO//cFS1/fC eMTYYy06UvyotO+oVGCBq6zI9sY1LeYU01nJRoZJ+JLckfsVKJMM97nPklKyDZRE3HOu TVkWiZ1s6BeKrYLS2wuPUnzE9wyBrK5LdHaTtNHXVs67A6vQfZbIP4oSE43hkg/e5u/I R82FjYAOaShi8TnwHOG++346KF4aekyWodyFCqKGj8HPeSDtbX8x88W80pqGP3MBU7wD 2AUgLCJEZ6IWj8TTYNHFpXz8+pCuX+wIU15tPfBynjH9x67d68/tJq305rfx6T581tDJ NCkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689062199; x=1691654199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FofimiyN+7gyzGEdPyrBP2skR/4a1H3RZbKcHwnJwCU=; b=jLeN6jkG3BvvoI0HbONesv5g2mwRNOpEFT/ZwbbzPwiZFSzMcJ10fRCq808ttPfzT1 VtgC1UkVgC3OL38HGRoVteT/UUcm/Lm112DorKMFEj8ww3h6bcg1eAL+RPKnea92m7re Ufo4RPyUQSlbva0ga9o3pA8IXwnue0WsLpt/dmAcajsR0BE9AvCMiy++KWntfsOJPWNF guxQwxalTvFTHjg53wUhjdmJZCvspYFtXV5vA4Xdd0P63NS0Q2fbctB0/nplc6CPlPPg NomtKEt5Yxbz2SJUK2iPCN7K1kc+dM+RBCZxqSS/ULWFwVl2TGq7uCtbhcgwchwsVPQF N9gw== X-Gm-Message-State: ABy/qLZIK/Pvz0y+LsB0j/V+Ww3kmpPNLHJRgY2tHOMK93v48gZSZ9a0 nQDTxK/97NVrc/L0hf4cayHBhA== X-Google-Smtp-Source: APBJJlFgOrHK/zRfOp0MLvbYM0U1zfW7SDksh40rSfKhwgzyv7cSwo1z6lrEN1sxG+rtuvh+ttt+lA== X-Received: by 2002:adf:f052:0:b0:30a:a15d:eb2f with SMTP id t18-20020adff052000000b0030aa15deb2fmr16274589wro.3.1689062199178; Tue, 11 Jul 2023 00:56:39 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id p8-20020adfe608000000b003141f3843e6sm1489099wrm.90.2023.07.11.00.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 00:56:39 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 2/4] riscv: Improve flush_tlb_range() for hugetlb pages Date: Tue, 11 Jul 2023 09:54:32 +0200 Message-Id: <20230711075434.10936-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711075434.10936-1-alexghiti@rivosinc.com> References: <20230711075434.10936-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6FDE1140005 X-Rspam-User: X-Stat-Signature: t3z87xnro94kby4g4r8g79eeir9gtiw6 X-Rspamd-Server: rspam01 X-HE-Tag: 1689062200-66949 X-HE-Meta: U2FsdGVkX1+s+bgmGi43ogOyl8TDKuVgKpNdV4nF2gal1m9L/yqqPhSjqUnZbUhYegB1amRFhVTW/nD/FdJK+/JrwuTkHXAgdn0LrW1EdzF62pZK1H2GH7FPFqFOqaTjL5TKHVd3NBEDx3C6EL8qRuHQsz9NXW3YVPvR92cqq3+wqfHmxfLO90iMtbQWdEDEtDo09wkWsJHTfyUCJc2m4KV1V5l98G3KC4R403OVsmas7iTwpsz/kzWSbk2CA8M+m7w3vbLi/7NojJ5h3OnS5YmMXwENUeiJqVAfFsdMbjhYN7t0UHYUjxL0xhPLE7CjDxSPiLBaXSSQac3AIbahL6mdic3Ub30ZrqMvebTzHevSTUCWZxz4YcH0Yhn+BiMB7AZdrE/n6D5YFBzneTTjV6nc9Ll5LvZXrV/hJiNg4pNK34i4qRYOIkT5Nn46dfzrI3a2lWVRWXQHfcjPSoWnA/x5LrNHwgMb+7z3TX5i6OWwoZ0ccVACIH5PlB5URId1BRle88nECdBiVFKgsFYYSJ8ltEKaiqI18r5A5uRD+7CyszSI0OZXZHpQbbHq9itt8qJn5zOAyYq1KRNY5HYpjLlqt7TbAMa2vxTQ8p2LKivRUy+w2yAkqQuVHYlgaIBKW0315QsLHVfWPNCrfL0+lu8w9F5Cz0FHdpm5B9khY0crK0z5n/OL46/dHiWJd0kPm+bH2NI1A5IuNw/26CFHTEE0w0KjeQVKTkjLcsPdbuwks/cHVnbzst9DNMzMD2UJ8MW9tj1MXrAETI8IZfHzmvf7fROE3iRTMO2zeWGBxmvYYH1g7FgQsTUr6b81D1Oq+7oHBe8/PCQxEt1ZU3DNHqghSpkyU8ssoZKjH76N6ox1Ou0y+clCJ+jfwN7zO94caszprKKm5B53+HP4jaFjzfO2uJmFDT1jpUee6lVjBm22RR3x2vAT00xK+AdKX42L0CvLJw46xUj7kgI+84j fwIZ5LP9 E9eu6gv+jfEp24awNE3ui2Ryo1mercn+cV8PU3HXrMC0UElW0svSkx34+HyjtdHGoJFv5wuhnSWbsTrez8eu8F+EO597ozTZ3t52/BCA/AdTZK72Fy7VcjfxJQsVdpfSAxn7dKego4ko0gotzQDfEE//QbeIq2W0HQbKNkGc6Dt/4f03ofw1SvmB7Zco/ZB3YNvNXh11VA8KLn5hE1/eD8ktRdk5oQdikOR6hHSuYZNiOXmIg771sBl1XppO2u8IRixHfdMLMvWeReDRA8njg4dhcLGzvcVx85Lpo+QG9d3Ewf7nUyE3CM3wVQtsHcRuwM5pB2ekDlRefopHPxkW/kB8/gZYOJFDEYpxjS7hb5Ge5suI83bSSpBsV2sGHnMHSuFUgRe3uI4u4t3a74z6abW7uQbEamdBWWFdH7MTx4/N7uBdr68BgWnU3YqW6EwhqKdi6DNXpHRTlXgKz4JvzbUbAi+FePZergBOEPYca2RcioOXLGZ5LdKZR5w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: flush_tlb_range() uses a fixed stride of PAGE_SIZE and in its current form, when a hugetlb mapping needs to be flushed, flush_tlb_range() flushes the whole tlb: so set a stride of the size of the hugetlb mapping in order to only flush the hugetlb mapping. Note that THPs are directly handled by flush_pmd_tlb_range(). Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/tlbflush.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index fa03289853d8..3e4acef1f6bc 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -147,7 +148,14 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); + unsigned long stride_shift; + + stride_shift = is_vm_hugetlb_page(vma) ? + huge_page_shift(hstate_vma(vma)) : + PAGE_SHIFT; + + __flush_tlb_range(vma->vm_mm, + start, end - start, 1 << stride_shift); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, From patchwork Tue Jul 11 07:54:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13308222 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30276EB64DD for ; Tue, 11 Jul 2023 07:57:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B62638E0001; Tue, 11 Jul 2023 03:57:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B12518D0001; Tue, 11 Jul 2023 03:57:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DBD98E0001; Tue, 11 Jul 2023 03:57:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8EDC08D0001 for ; Tue, 11 Jul 2023 03:57:43 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 60C8480148 for ; Tue, 11 Jul 2023 07:57:43 +0000 (UTC) X-FDA: 80998576806.11.FDBDF8A Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by imf01.hostedemail.com (Postfix) with ESMTP id 823B840010 for ; Tue, 11 Jul 2023 07:57:41 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=jpjsGIvH; dmarc=none; spf=pass (imf01.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.44 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689062261; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+GXjLhe5TXUta+EXxk7AgfyYmiNiXB9g5OSSIlBPGEA=; b=32NuS16B+E6pVE+tdkbex7PIcWMas5rpmKx8A8AHqO9uvD5Ya70gN4wLiupuf8wPds2RB/ NFNHMZO5YhpTSdQEM9sxOLMPEWZ+neHGzdGmR00lfZXDQWmamsl/MxSocsQBZzKVluqpgV bn4slVLKn2LL8BfvwQDYszph1AA6cj0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=jpjsGIvH; dmarc=none; spf=pass (imf01.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.44 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689062261; a=rsa-sha256; cv=none; b=0iGZjVu3na/DbJhCaJmv/NMu8DGtLf9xFcQiAo49asX3lJrgD4rls7mr6CIL7GXeScNtyb 0mCHPcK3Lb/0uDkR7S/WHZK6NzPI6vEXxbse+KWz7m4MFmkFUXFJEe29x4HZDF2ufIxzPS 2SWA4fhQmlXds0moX4qzzJRI3NUofXk= Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-3fbc656873eso67201535e9.1 for ; Tue, 11 Jul 2023 00:57:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689062260; x=1691654260; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+GXjLhe5TXUta+EXxk7AgfyYmiNiXB9g5OSSIlBPGEA=; b=jpjsGIvHWDXwtmenwjPw+yleh84wYF4Gde1Y36TPhdybnF0jpwSLd/aOo7CFMtSuDK ScyvlYS80l2vcQ9apidNHokPdNcKdGvUFOz16EWh6HyNXWzuR/QaMrKyDRVuWrs51yKx skZgZTUZjD6cFKSOYgb53UN64gLFvysjjqU8dLz9nE2+HJHAHGf48tinF7oqdJtBI4pc Et4wuuECz5m6cPYyXZRudVvVIj9AKfqmjpYZWVzHlfwOhzfYW2aebgP/Fu+HS+8C0bcV XDhilVqCMXb407/A4iXWplI1yIn5lQwUah0v2s5df1KU/cDtaR7kCadJjP9Y4QwOQKtB s4Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689062260; x=1691654260; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+GXjLhe5TXUta+EXxk7AgfyYmiNiXB9g5OSSIlBPGEA=; b=HlX5uP8CKOPUEj25Wi7LbtO76eQ9edD6Ok8MurlQCuuypgrHgsB3se1y/9aFVQbng0 6BUIMR5Aa+YUA39uJjYn5QBD44Nyvjrh5cnAKHDEC12oHPWEIIiid2vJiUksBYILyhjX DNIrc/KIwz1i8I8VifiPnGFN/DniIfoMXJ2bVXRPAkX5LfnbTa269Il0PpNh7oBe0BIr b9v6hm9TX5MnyrFPluLU6oodnjAxWF6cymbomKMZW2TjjucnTsDEf16gxTfhILMNZ/2x tV1OGIL7NSNIWaPJYCrStf+ENDKT4DqSLIEbIIE045Whfyh9fTs5pmV0BNDqndVosNLa J6Jg== X-Gm-Message-State: ABy/qLb0voSAYd401sux2XBLp6sg1tZ7BDJQ69npjSUs/bNXGrI9XfOp dQlp/+pu9cU1+XTyOv6s5aw3vw== X-Google-Smtp-Source: APBJJlFF+JXKqksjRHwPA1L44xQAHuT0wraSNDH0ZMzzf07N8OaPZzuThlMge3mXhbU6gGV1bjdvMg== X-Received: by 2002:a5d:4404:0:b0:314:46cb:880a with SMTP id z4-20020a5d4404000000b0031446cb880amr18276011wrq.28.1689062260118; Tue, 11 Jul 2023 00:57:40 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id t16-20020a5d49d0000000b003143bb5ecd5sm1499978wrs.69.2023.07.11.00.57.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 00:57:39 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 3/4] riscv: Make __flush_tlb_range() loop over pte instead of flushing the whole tlb Date: Tue, 11 Jul 2023 09:54:33 +0200 Message-Id: <20230711075434.10936-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711075434.10936-1-alexghiti@rivosinc.com> References: <20230711075434.10936-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 823B840010 X-Stat-Signature: a63izr1xwzxdipay8hesntpaau8p6kym X-Rspam-User: X-HE-Tag: 1689062261-870140 X-HE-Meta: U2FsdGVkX1+84SqmAThKD8DCJXVKRIRDnKhdpNbRWOVsYnveg3WXmh+TNBwx0tSNOpNJMZqlkRaMAA5vAjMFkiNms/ohiPBt/zKr93cSYP8E8UC8bcUO0qGmhxBiouvZTAaEC4YmFvYFJ2UloYX9++QS7cn73OyGabnh8fu3TaTBbmX5R6QVkahwK9WuYvsFhf3k68sQ4haNgHnOS+SPgwZN1OO56O1fJFM4iMFuEI4JCJ5+hSDE39aRMf19uDtzVoRwmSGvLJFKl1a/Y0POFDlTPUgoQCARpKqQiH3BS7z/cEpEMOWCxGT36sqkoyng1I7L1oPkBdSodPQlSJ/42wgp4NQVlh8eo8w59/ZK8NPpSHXOYZa4/dfzsZoPAuc84AicpNTKfEha5Xd42PyeU0hGP5plSj21uCsWcH4Dh2Bvo23lh4MhxhQWsNFFcLptCNpOTQ29+ZtYqXE1e0mA6W/SOkFzWr0qbk75yARIXQe0VZEG8rfzyGJyXiON3E0QqQA+B90FPXLLXAKT8UJzzaDD6a/9x5HHTLfgMQersHkQNtdTV5DMjLjrHRpnh9ciQ6gPIBEXopsDfpMYbWQU5Mga35ENvRgNAD0LX9K8K6RLmPE3zROxJAXt88iSj3z6bDGJMW5RfyGcdwfZBEqWYAz3SCqs0mtz2RBGFj0ktBpwxG5FH6xXCunqKgMjRlismAO3/eKIEVXaK+0FCq0GMs+MTcSl7oV1xgtF0fAhQBPNZ+lfnojtz5bIEARsld8UmAZHTYJhjMDQXpJXf1PvEDZ8azpZa037pEDZjKRJJnDjXCe49AIe3kv4Asjy1WMdgA+gVRRyRMALR3LfTl7FwAIPiIPTQPScgMLfiwRrdKRiBN4NGUSSrVL+1ftR63XZP/z+OLx7gV2NzwGoz00hImgahPzObWcECKpAgUL00oQKwKYtt4J4rNVo9HF/Muxexr0NFwlS7uAzUeO6WcF Oc3MpUAI ug1NPTCpyFfynpA5O6RbdeocFvPhiKvDk+Fe9aCmUxHbp8kwChmvsWlQUdCEfYuJi9o+WT/PvRbsMFcTPZaFigkx8RCrLxDA85Yg3Yf3pUt0BO9p3THvkRjFoWPk9atJEapzd9487oDt30pR8+b0HdaILKHrOzemXL8hy9RxIKuz4iR9uQRm/zzPfq7b5FGQkgGtDTXAlkMQsj8oSV7xzUj9M9f6uMUh/tppKI09wcEDpzEiP40lmb6DXmZ3XgpqtdlF+qtcFLJgx/TAhQJj2RqgRQaC89MEfy2JGlW8uBxqoX8xyOhyDj/B8Sv+bzBpP8ThOcKPvZfWcjDBFgVt2P93/rXFjPTm8iEuqIP0TbNysf0Yvz+e9q9M7Y/L8q3BiGq24+BcCiC2bEHrZUQ07fw3koeRZzYK++Prhsf2sKQtgjMJB8nx+TPN8W0oPNQQ/igkJgpPncGnZwLgTDUnewPNdpmNvQl9ISFX/jP2yx9bDs6HzdYzZiMWJdfH8cHioXjBCtLiBA29CgLXe92jlkfwyy7kgaNN4LYvPbg9hXbyUMoc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, when the range to flush covers more than one page (a 4K page or a hugepage), __flush_tlb_range() flushes the whole tlb. Flushing the whole tlb comes with a greater cost than flushing a single entry so we should flush single entries up to a certain threshold so that: threshold * cost of flushing a single entry < cost of flushing the whole tlb. This threshold is microarchitecture dependent and can/should be overwritten by vendors. Co-developed-by: Mayuresh Chitale Signed-off-by: Mayuresh Chitale Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/tlbflush.c | 41 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 3e4acef1f6bc..de61ecaa218a 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -24,13 +24,48 @@ static inline void local_flush_tlb_page_asid(unsigned long addr, : "memory"); } +/* + * Flush entire TLB if number of entries to be flushed is greater + * than the threshold below. Platforms may override the threshold + * value based on marchid, mvendorid, and mimpid. + */ +unsigned long tlb_flush_all_threshold __read_mostly = 64; + +static void local_flush_tlb_range_threshold_asid(unsigned long start, + unsigned long size, + unsigned long stride, + unsigned long asid) +{ + u16 nr_ptes_in_range = DIV_ROUND_UP(size, stride); + int i; + + if (nr_ptes_in_range > tlb_flush_all_threshold) { + if (asid != -1) + local_flush_tlb_all_asid(asid); + else + local_flush_tlb_all(); + return; + } + + for (i = 0; i < nr_ptes_in_range; ++i) { + if (asid != -1) + local_flush_tlb_page_asid(start, asid); + else + local_flush_tlb_page(start); + start += stride; + } +} + static inline void local_flush_tlb_range(unsigned long start, unsigned long size, unsigned long stride) { if (size <= stride) local_flush_tlb_page(start); - else + else if (size == (unsigned long)-1) local_flush_tlb_all(); + else + local_flush_tlb_range_threshold_asid(start, size, stride, -1); + } static inline void local_flush_tlb_range_asid(unsigned long start, @@ -38,8 +73,10 @@ static inline void local_flush_tlb_range_asid(unsigned long start, { if (size <= stride) local_flush_tlb_page_asid(start, asid); - else + else if (size == (unsigned long)-1) local_flush_tlb_all_asid(asid); + else + local_flush_tlb_range_threshold_asid(start, size, stride, asid); } static void __ipi_flush_tlb_all(void *info) From patchwork Tue Jul 11 07:54:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13308223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65017EB64DC for ; Tue, 11 Jul 2023 07:58:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC0E78E0001; Tue, 11 Jul 2023 03:58:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C71A08D0001; Tue, 11 Jul 2023 03:58:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B39428E0001; Tue, 11 Jul 2023 03:58:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A540E8D0001 for ; Tue, 11 Jul 2023 03:58:44 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 61ACDC0145 for ; Tue, 11 Jul 2023 07:58:44 +0000 (UTC) X-FDA: 80998579368.07.843E391 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by imf22.hostedemail.com (Postfix) with ESMTP id 5F64DC0013 for ; Tue, 11 Jul 2023 07:58:42 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=w92bWwP4; spf=pass (imf22.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689062322; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YDYIXf8sIKLPYejGQ8O6uVDYRyye9psJA+Q5PmU0G6c=; b=QLZ5FaOM455mnQvoPTgDVbb9wf6FtMQDTPt342qdl3niyUAA1MlXdOkQd2Cpv5VguvBZ7q 9+NqPUQw/DAF5iQ8/qjZ9O4yx/GsEf0/VPW50+zl7aJqjkxUPYByVdZpDDgNfS0e8nQd/1 WwSDIOG+c3o/cGIreK/8VMgOb6PL/5A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689062322; a=rsa-sha256; cv=none; b=cc6bpiZqQW3i7pXoxU5aQJPpTRoQRccgnXGNbpSpHyDlPZLyP3wwwIuPbALIwrW2aazWM1 rh5vGjCSSFH//axcupGttk1UfH0zon6+hRT1WR3zDjvmgUgQpMKjlWLXfw5rCQhT1iEWdA rDjKWcKGyOEGV4XsEV/XRt9l0cj0z2g= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=w92bWwP4; spf=pass (imf22.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-3fbc5d5742bso59639785e9.2 for ; Tue, 11 Jul 2023 00:58:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689062321; x=1691654321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YDYIXf8sIKLPYejGQ8O6uVDYRyye9psJA+Q5PmU0G6c=; b=w92bWwP4/B9Yg8nTudHNREZgu5pw/Y+pQ/5HeoGML42uuWQbv8zilzfmgXbkrCeUXJ BigOw37ZeE+XRBcUme8MnQpOGaTDa7wH4BCEBKoFhYCZ+hVEXq0Q3Mv/yS9fBPm22IAZ 5H2e5BzVRS345P78IGjkr7Sg6L606PPnBu/yLKT3L5E04/awzbxsIVShbm8PjQD8fNKG cq982sPP3LSK8cemVb1+wFEr0xgVkryUn7u3N0cBuFOQEhKyoy/uauDpQuDZhIntYloR KCSkuNsDO2zpLsRgyBdBNuZUfDREeDfyFDyDnHp6qvfQoWIgJOr3FntAg4EIsSB3wIGx QgAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689062321; x=1691654321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YDYIXf8sIKLPYejGQ8O6uVDYRyye9psJA+Q5PmU0G6c=; b=EQQF98Nq7QT/oV0A2eYOps1Cyq8hkefrWIDJIvgpSCHuQDTghIjDBSkRavmIYRMQ+N tkjH6pwDlHc0Fe6ABAPdvMu9AK3lA12dP4yYU3WS3w4xP6b5WSh5zOxkjFNjV2ez0y1K BKQgQupbUJ2phK+8zEkjZzdNf+RrXp7mStJw1lN4Stffete270sUuj8+OPGykOjtcHfd /qhymxOhawbTqa5WVnLKRCGapdaBjLvVE2+DFY3BXz1MCXwz82M0SthJUvukRxUVs1xN zmWSvFX6atoufOiaf5qbqaHDWB8Acym02Tzo7wFxhEjP0fZQh8c8G736gGlGs52mhm2z ILSg== X-Gm-Message-State: ABy/qLYfwg6A/4POnWK8oNvzly4xzn4RXCphvJLwlpeLJ957nic5xyPh CvmTMI3Lm7HqI2M3c1v8RMgePQ== X-Google-Smtp-Source: APBJJlFsBtStWYZgknujylpz0AhL0BNK0IKMFzV1Mxb+JDn6KB3EiMXLptQkKqpS3M+govHnZDRwFw== X-Received: by 2002:a7b:c38b:0:b0:3fc:524:e80a with SMTP id s11-20020a7bc38b000000b003fc0524e80amr10779950wmj.18.1689062321091; Tue, 11 Jul 2023 00:58:41 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 12-20020a05600c020c00b003fbfa6066acsm1750168wmi.40.2023.07.11.00.58.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 00:58:40 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 4/4] riscv: Improve flush_tlb_kernel_range() Date: Tue, 11 Jul 2023 09:54:34 +0200 Message-Id: <20230711075434.10936-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711075434.10936-1-alexghiti@rivosinc.com> References: <20230711075434.10936-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5F64DC0013 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: a93cwixtwwgbxhm6xkzwdnwgmc5r6yn6 X-HE-Tag: 1689062322-464103 X-HE-Meta: U2FsdGVkX19lb8Di9cmYDU4ILIYdh35eJU66+3f+6COLZPmQ07dpZOl5uEYDPUuyFTFjRkT2xtdh5RmhWpZLUTmwOoaW1hXw7n9xrZVhQ8NFFXUfT7Ir/zRf+cQdvubbhpwtGJ0ENKbfZAVGigQ3JVVt8zFfT8TlgEGYVF/J0qkBfEVwDa+AG8l5DEReT4rBVCHFCvTrc7PhwbQUwYwNcDOgHShhmdJvaRBOYUtjPKAWmYHCelppJY67bGCUSkX6BjSlaUAWSx0rpKJTusGGglBTj60cXHtOts+266hWJFsOtHhWDNDxB54ZnCHxiEckkwd9MFHYX7QkUlGcKFZ7q37CLDBfV/760gvAyGJmhrr7aKbKOguEcoA+O9O7CgNZJTt846YDzqwjcTMc4hl5Ir4vU9+L7wpCfPN2rrDTSUIXf4gmsktVX0NQUcTxkxn+WS0gnKcc5iOWpPesA3nAocjf1NVLnq01JQn9EKW1gUvYzp0EJbC+8L76Xk3uKDYPUjNbaKihmHcTuNMEjRB1Bg0ACgZkgYUJKfq6djXFxD2GyrtV0sQHNjWYCfVYuQS6Xx6P7a0rBVe9UD3uf0+khaAOGu7l/jrLvTV+R3C6MrcIk+D8gXngyxN5a8mzsjpORBNMjVo/SE7eeiRXKjMClg2cf1GUrr1vmqG5NK+CaVnSncxEugbNn06EDG8vHslvf5H+aNVhLNl4/iXtR/y2z5JdsmDnbWRwK1WhynzDn1639VuHVabAzYO/M77ErKg8m03OjyGthD3scN0iRAc+bBG5Ts/DHyJ+/68Hz1CQKbYeFieNogGRXaA7/XQIM3HOzpAv8LdrQMeeudqnrSjg6RCKrSpABcANXGiiiyfcTmUVsUSGvgHZXhzaY50y1Nv1YH4ARFhwelMBeTtUL3dYZ3Ip5XUfavbjM0RUQGDLmUB04TsFBZPS4zplXyw+SPWCC7NggsICygooOZSFQgh hUWBY5rx FsbRu0FQHHJpcwOrlLkEpyvFsyY2/1nQpu8guBDox/yUi+iSiJbfWzZPKRH/x7PznQCOAvc1EcZBaDm7xwC2mY0kyc2HnIl5RIpddsIxCx4/N0zsiuc1vq/y2TMw0jkVNqdF5oYuZZY1Rmn81tjXTsargf/P9emGizydwF392eNwRHGH9tvpgi0LKcq6PQtDkLF1QJ1ZQ7P4wci1rc6OKw8P1ukmGdwz/4bXsOjpeucIt5mlwRQyeZfZ/cxeY22asyUDBLitsw7N2ZrI3gBh+P6rykj7eRVqVWQpZbvDT4Y2hBXKPkTNXmrL4f5HqmmA65FK8BbaAQXZNZ6xt54+rCWQ2oLK2z3xspAbESi6k8gXf3LKJeLfuqU2FEtJdc7Edfo4eytoTp9aAWtPLhpUB2cYezwd9Z9pzXVVtwWz5ZtjKnRSVnf/5B67i2fYWIh7LdU9rb+2xAlin5/aDzf5A0mGBQ7mabNmgq/nPBtzOrJtGhvPYPsSryrrEXA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This function used to simply flush the whole tlb of all harts, be more subtile and try to only flush the range. The problem is that we can only use PAGE_SIZE as stride since we don't know the size of the underlying mapping and then this function will be improved only if the size of the region to flush is < threshold * PAGE_SIZE. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/tlbflush.h | 11 +++++----- arch/riscv/mm/tlbflush.c | 35 +++++++++++++++++++++++-------- 2 files changed, 32 insertions(+), 14 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index f5c4fb0ae642..7426fdcd8ec5 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -37,6 +37,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +void flush_tlb_kernel_range(unsigned long start, unsigned long end); #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -53,15 +54,15 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, local_flush_tlb_all(); } -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ - /* Flush a range of kernel pages */ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - flush_tlb_all(); + local_flush_tlb_all(); } +#define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() +#endif /* !CONFIG_SMP || !CONFIG_MMU */ + #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index de61ecaa218a..07cfed83bec8 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -117,18 +117,27 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { struct flush_tlb_range_data ftd; - struct cpumask *cmask = mm_cpumask(mm); - unsigned int cpuid; + struct cpumask *cmask, full_cmask; bool broadcast; - if (cpumask_empty(cmask)) - return; + if (mm) { + unsigned int cpuid; + + cmask = mm_cpumask(mm); + if (cpumask_empty(cmask)) + return; + + cpuid = get_cpu(); + /* check if the tlbflush needs to be sent to other CPUs */ + broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; + } else { + cpumask_setall(&full_cmask); + cmask = &full_cmask; + broadcast = true; + } - cpuid = get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; if (static_branch_unlikely(&use_asid_allocator)) { - unsigned long asid = atomic_long_read(&mm->context.id) & asid_mask; + unsigned long asid = mm ? atomic_long_read(&mm->context.id) & asid_mask : 0; if (broadcast) { if (riscv_use_ipi_for_rfence()) { @@ -162,7 +171,8 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, } } - put_cpu(); + if (mm) + put_cpu(); } void flush_tlb_mm(struct mm_struct *mm) @@ -194,6 +204,13 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, __flush_tlb_range(vma->vm_mm, start, end - start, 1 << stride_shift); } + +void flush_tlb_kernel_range(unsigned long start, + unsigned long end) +{ + __flush_tlb_range(NULL, start, end, PAGE_SIZE); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end)