From patchwork Sat Oct 28 23:11:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EE15C4167B for ; Sat, 28 Oct 2023 23:13:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 671A76B02C7; Sat, 28 Oct 2023 19:13:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FBA46B02C9; Sat, 28 Oct 2023 19:13:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2EE466B02CA; Sat, 28 Oct 2023 19:13:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1C79B6B02C7 for ; Sat, 28 Oct 2023 19:13:46 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0378FA049A for ; Sat, 28 Oct 2023 23:13:45 +0000 (UTC) X-FDA: 81396424452.12.EB16032 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf26.hostedemail.com (Postfix) with ESMTP id 38B4314000E for ; Sat, 28 Oct 2023 23:13:43 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=h9hANiwC; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf26.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534824; a=rsa-sha256; cv=none; b=V22/eqzDy1JaSqU/rrimtx8/5Jm+FB6qBfVvBkjNoxZQymHWUE3Pwn9G454h7PxmBfsh8y ZnW1kQO8aqOmtjhHGEFDxyB3Z0EwVH2TtIMqZtL6hPeCJYyyH7S/UdTjb1u/kKID3iNJXr SYTbdn+kFKE6aWDzI3tyuZan160nS18= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=h9hANiwC; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf26.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2EsAJ3DvynnbB6jUfsinRvKaNQwCusxxXujfrC8RtXI=; b=4wWDz98Uq8Srv5pMg5b/FuuLt+B0RHpX+z1S5wmLakIl26v+cVaQbr2+nNFW9BCLdzyPUh wwB+RwyzDrtDvHjVjkDlzh4bpQ2rz17aPGj4aUTFAx5EQ+ETOAxkrfg4H/ouhL7fyuag5p TxWB0jBgijU4AgtiwKngsh2OHAavdq8= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1c9b95943beso29439925ad.1 for ; Sat, 28 Oct 2023 16:13:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534823; x=1699139623; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2EsAJ3DvynnbB6jUfsinRvKaNQwCusxxXujfrC8RtXI=; b=h9hANiwCCf1WxgF7bTFKjdDfaqNbHzWooHLmrt6tIs1KS8NWQ7BcLboU7u0PiVKpQ1 mrcxNcqXbAl9FDQRYKD994aTKsdCEh8FhcBSGB4yvNCnZJETYoLcinxoUiX+prZjpVOq EMnoZ7BLb69rP6Sk/9cKgRIcXwGFI5NiU22sz4j4b6d0N7xvmdxgW4EGszVdU3goQnNz zvzayGUBUz2M9Vz5HZtd0ty7XFRhWho6iKxIfwjlSNUmEQ/alniJdMxG9rwTQ/zmX3kg mVQfqhLKwuBRCNPmgrqCdsqvGTCH30pmrykrlBQBHG3KJkKfBjdTfkJ2FliT3m/feyt6 qt4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534823; x=1699139623; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2EsAJ3DvynnbB6jUfsinRvKaNQwCusxxXujfrC8RtXI=; b=r8XcSb/Q0AEDBRkrvgPfjUaoJSegaxGppKGjetX60wfQA+8oRPMEBvJqBLYOn1WPKr dvXCbBMThk3XiNHxa/Va3fccS3LxqrD9/MJZuMRP2fL5WBffBWeA+36gzkSDPKJvcrvs Foq8AkBew2ioHu89npKZUwwHQAKDrT1IbMP3hai9DK6sohQZ3/6WWssKdlD2AmAFCYzb mITkaPE07oS+oLXrXtqoyUcUSyYLosGu1rQHSKq83G+5CT1uxaqX3/tLffo6gL3j1rCg lt2hJ/Aq9gd28UV68Mhax79r1GByN7d1TCXU0b5l4kewVXP6ywxEzJGc8lFsP7/T8vtT I/5A== X-Gm-Message-State: AOJu0YzIDR5fnb8abfXblTRkCCvFWAqTeETF03IWpSA+UFOG2Easl/wK SgShG6rXduWRVgfT28PcJlWINg== X-Google-Smtp-Source: AGHT+IFw/0gO2jty39BfEl53e0TAm8jThBxaHB3Pi6HPPc0H5aL70GOaasoPy4Ex/mKBNftI1RCeRA== X-Received: by 2002:a17:902:cf41:b0:1ca:8252:a91 with SMTP id e1-20020a170902cf4100b001ca82520a91mr7454451plg.41.1698534823149; Sat, 28 Oct 2023 16:13:43 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:42 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Jones , Lad Prabhakar , Samuel Holland Subject: [PATCH v2 01/11] riscv: Improve tlb_flush() Date: Sat, 28 Oct 2023 16:11:59 -0700 Message-ID: <20231028231339.3116618-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 38B4314000E X-Stat-Signature: zaznggxujxps1hz7iu1ysiishhfop9gf X-HE-Tag: 1698534823-980467 X-HE-Meta: U2FsdGVkX187A/kIfnGpuacns6ZjN5jBw3vg2NiU4H0/FX4FF2+bbg1iiECSEuQI93TBp7D8CEvnW4jIyvWzbXlqcGpHDtE6qBziTIJXBv75YgJtRt/kzCecQnQHt34powtmAWdd2tO0LJ92lWQuBHfvZExCYSo1jKT5C/tMdU6QN8Hp48mhP1XU6avPSFXWPbdzaQ1ZHdPIQC5QnzesxDrBp2LMO60a8CwZbg1TwNGdO275rdhLUWX94FieaOar94xUg4IrArXJCBtfRLsn0nJnftN3xrPwtLs03HKysUkjNodDnsbRYht6+AbIrOQOkA50HHchPP1pnMblU7+PTtYc6QuHCno7ndpbn2kqVxwYyyKKsXKMxye24jtG4lPm8ggKdD76WU5n5uPtqpNYJIdYtVqhBfxtZJVxsazLBf3+bRXOYVlL77E+OWFLDni0ads2Pu1IE3a0a5264bVtR9RbMGjkPDlkmOzbXz/Y9Ht4qyotlSPpm9uFnzJ6V8pie8Ni7q7SQAdZPC4+IBLBHxGRYYBGHGXg0N8xUE5dZJklwErxux8HoCBiNXWbVeCUFhsPHeFKf2Ye9YC1+/C/5qQaRBkOctsc+rzO2XkYAfPSIDTOdP7U2RaRKxnqVWPhwTkjbBQ19c/BuwVToB7I3mROyVTCS9ERCql04EMZXR5j2HlJQXmA2XeyAUCro9Vv6EFcQ4g0AnCujPlNUCXRt1oWHcG3sGuXBs5wFowXF6VEXu8TYUS6+t6VsghqdG1bKT2rukpPxwpj0tf7JR+nrAtOQi4nrtrNCZjmLvksMHkinXjT4bRZmRtGNzmV5IAVnYxAtURyVShorotUL98lhXVr+lwdcZvQIj+CW4WXsge64ACrP4GIHHWSLiYUkggLENLPTC8rFL2L/zeqsHDzXmoSzYeYmb6Mi0/W9evVJRQaXwjcQ6HMZNuGvyWr6jhqX+gBBmJ8Nvn/W0RAQop v9Kliwhq 35ZaCLRhoPIGatoO0ZiiOc65/5/8zVzfnV2eKyRHFU1hW3zZivJZLtzt308erokf3LlMIZHbv8VRn8K9BEEcu6rVwJQMbgccD80d0V/S5mvEd2NdxvCU56Z7wgrVQhXzr1nLxt8S2hCrNeuDJDnz/9OpvRkpuo348K3bF6wavRNvQ6GwSei+EpfHjHq1sJ52MDmSOFNhYZ35IKjW4b+idn6Ip+Jeymzj+c7PMsxq5NKvfM5gS1347Lf56SZYSLOLvrS+aqxcikEEmdwSdXkNFF7CcVWYyUjqkCmwtbozdDQq4nWWiZB7OEkNK6E/0n73tkHsZE5LS6iX5mnlSxY/pSKv8XN6GmdQb9MGv0qTggO9qYIUQt7atgHUCTsx9XtdRpyGsKCNkdbz4Rt2OO4TRTwucM+dSDnh5w9IRWM07O9r9snZTUq2eDddM0AqyLvSe5pp/J5yWWKrCfqhqt5qY3zQ/1i3RG82rDO6zlQq9WerxpXqtxlUpub4ZO6xnUVLZLCr37Ypo2y23XL8kG+kJydwv+PUMRht8lXOVTQdLgBoeBht3zJAnWHtIx8YogsCZlb0tG0xgBnOZeIU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000088, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexandre Ghiti For now, tlb_flush() simply calls flush_tlb_mm() which results in a flush of the whole TLB. So let's use mmu_gather fields to provide a more fine-grained flush of the TLB. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Tested-by: Lad Prabhakar # On RZ/Five SMARC Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/tlb.h | 8 +++++++- arch/riscv/include/asm/tlbflush.h | 3 +++ arch/riscv/mm/tlbflush.c | 7 +++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 120bcf2ed8a8..1eb5682b2af6 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -15,7 +15,13 @@ static void tlb_flush(struct mmu_gather *tlb); static inline void tlb_flush(struct mmu_gather *tlb) { - flush_tlb_mm(tlb->mm); +#ifdef CONFIG_MMU + if (tlb->fullmm || tlb->need_flush_all) + flush_tlb_mm(tlb->mm); + else + flush_tlb_mm_range(tlb->mm, tlb->start, tlb->end, + tlb_get_unmap_size(tlb)); +#endif } #endif /* _ASM_RISCV_TLB_H */ diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index a09196f8de68..f5c4fb0ae642 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -32,6 +32,8 @@ static inline void local_flush_tlb_page(unsigned long addr) #if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); +void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, + unsigned long end, unsigned int page_size); void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); @@ -52,6 +54,7 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, } #define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() #endif /* !CONFIG_SMP || !CONFIG_MMU */ /* Flush a range of kernel pages */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 77be59aadc73..fa03289853d8 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -132,6 +132,13 @@ void flush_tlb_mm(struct mm_struct *mm) __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } +void flush_tlb_mm_range(struct mm_struct *mm, + unsigned long start, unsigned long end, + unsigned int page_size) +{ + __flush_tlb_range(mm, start, end - start, page_size); +} + void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); From patchwork Sat Oct 28 23:12:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439610 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A61D6C4167D for ; Sat, 28 Oct 2023 23:13:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F25CF6B02CA; Sat, 28 Oct 2023 19:13:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAD616B02CD; Sat, 28 Oct 2023 19:13:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D29E96B02CF; Sat, 28 Oct 2023 19:13:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B38DC6B02CA for ; Sat, 28 Oct 2023 19:13:47 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 941811CB445 for ; Sat, 28 Oct 2023 23:13:47 +0000 (UTC) X-FDA: 81396424494.03.F679421 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf17.hostedemail.com (Postfix) with ESMTP id C73CB40004 for ; Sat, 28 Oct 2023 23:13:45 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=jmZ0tQWK; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf17.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534825; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=kguaAV6mm40UKTtYsL9Xv6tvyel4SsmmI5uYo5XPeQ++reaOYBXU6Gf8ADuww5gqeUMQeo RHfpLwsiqGZQn5SKSCcakA1uVXW160sxc2C6IUGlWwNSM2Tsfi5+s8kpwJ3GDclX+brn17 0+fgmrW5xNeYii7omf/FgOHI6nk1QQk= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=jmZ0tQWK; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf17.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534825; a=rsa-sha256; cv=none; b=HTpKqRUwRmh1WZ4m/465ETms21OKRTKkEjh7h9hGMOuWasuL6F4r9meZa1elRd72Z4SYDj weY0/12vxo6J/KWFMzOsyJok9CTT2fTs0+lMoIC+H6K1rdEASPN0ZG+u91oIT1LXEtRvs5 RRVG4/celI6i2KMt97fsvDcoOhl2XKk= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-692c02adeefso2868166b3a.3 for ; Sat, 28 Oct 2023 16:13:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534824; x=1699139624; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=jmZ0tQWKIdIkV6pbj6H+mmbvBqe+7No+YEdehSaS2aCUV1aHU9mN+B2lMgyBnDchz2 l5Sw8GveNB4PPUecHVAjo/7AO+qtsOrG5gaP9j+itcgBm0glpSSvkX+LtnJkiqQMdlQ4 BcvMKTYZxjDRQNHQ8Jx6mSfkEd9Czmik8A/QZ4CKBfytH2S1bLl+EN82BcEuuE108kb2 wGtndh3cSSANbnsyRSSK46kkI1qquFxiXSHaT0rkrpeaCbqPLrjDjYw2o6MF72Uxw7V5 jOjaX4HGIMU/8DM1AO3fcdRbQUUFEWesEUvta//K2qAA2XsJHRk7/sc32+PtqYzf1y2y hLYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534824; x=1699139624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=E+s4QGdkqbRcsWFH0dLPuKslJxgidJcrOwOMwrYOj8CRRPz9+3yPRX9sGEg9McV0fl dev73iZuVTUArzcV8gfEq/BAxuCUxin/G+mw1Mm+GSG0NX9e1mOmNQoC69V3xPlGFR0n QsEnZFa/wmitBy/bjHRf4aFhluznEKryZna8DpEftYnrxBpU9JLDnjWnwWsb3xA7UdQr nmAIQIeQ5vzJKmWDamk5jNgFpKqaqr1p8NvUm+Hi6joLop6AjdzQTpjhjvWu638kOJIN E1hg1kGZ+FUDY0xsd6h5vTQkFBraKMF3U0AjXcop3zhP0b9srZl/h+gkNaQ/LLuhATuZ u1tg== X-Gm-Message-State: AOJu0Yxp1i11T0NJqynWX4gNfhrC/U+WwWYPicrLPSPZbP2mHogkyxid TcXqyW+FlMJZOjpvDgfhdPLPxw== X-Google-Smtp-Source: AGHT+IHFtwqkvMLL8vTXw7lsMTaEWiNW/nYYuBPFSSXqkCgtp1/aoWCoS2LcxRJlFYSU2eWn0N2URQ== X-Received: by 2002:a05:6a21:3b45:b0:16b:a5fb:eee5 with SMTP id zy5-20020a056a213b4500b0016ba5fbeee5mr5984191pzb.28.1698534824609; Sat, 28 Oct 2023 16:13:44 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:43 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 02/11] riscv: Improve flush_tlb_range() for hugetlb pages Date: Sat, 28 Oct 2023 16:12:00 -0700 Message-ID: <20231028231339.3116618-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: bpp59cqwnjcr69ji94bokyyydgcs9use X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C73CB40004 X-HE-Tag: 1698534825-842275 X-HE-Meta: U2FsdGVkX18+QtnlX2o8oa6OT844G0Clulk0rYzMy4rwrqgEjlJ7SYXLsQwukrNoyT/aefA+S4xwXT0kJRJjODWs+DaMKgrZL8M/jgf5edAg8CKVVDk9FL83BcLQr1VypLMsnQk7wFeNLvGveWLEVT+ZB2bqea4TrGgSH97RamTHCCmmmj7fUuPHiYGfrsD+iEdgyLcYbBrVktcvkDgCvv9ysgNHwoXhPT728Nk/Cd29vtovPPtaF7rdHTVKSDnHeQlClGWQxeRXllWpPd8P5bA9y96NNVWAx6z3feK2Qdb/WtxYwH33BdawlquHiUWZUWR6jOtY+XXdEkzSw2gtb+SmUqCkHZrhzJSh8MUj9E28omHsWAibuva6gx0B2SFG7ngC2s3JIQQ57o+U+yJOM3KgPNWPoSCOCr6prj/YlvH+G7NPbxOJhRv6nRwIgOSiGjdHVfnDdyZRTlO6v2QyIyua8Bz+Rth2MSRy+tKhXE3zXzpEulxamKdqzJV6TtZXAZ4e+B/RNBggWM7AR5adzJCpDJKC3XKmslviPSFvAv2Qo8Z8knYYAJgs7QWJaI4V/2c4IcYI3ESYs+s8sgd3EMl3Q0t2/6UX7H62osDFttQRJey1FY3lAexrQa9DbaKCTSqQ8XxTnVG5SxLZq+ke+Gito1tSeFzyv+jSEJnwVEL7XMKFjNm4rrWN0EV2TahJGJo7hgBfjEgO+2jE7r5qEA5RROkQTkhCAZtumHvp6cGNGfiOtY1b0Vpp7f0cir3jsYOaVWLsgLN09yz8nDXNzfk23a0eXysYAkSLAwH1M04BXFVHKEYTS7kaNAKNuZrcgXzQ3db3fbrIaAVOYXNOsmPk6Qjy7vPDbB1nWaia+ey1G0IuJ/WhGBxaAKuYP9l5ONEOHTbbYRxtBfWPp2dlishl80OfmjoPVM0Mvrrmu6G2nI6jv832F4SczUDSYrMwSGTS7xhxp9SQLP6jbgw Lgpi4jL5 UKKlvioGdOdHjHYhhSvLgCXPbWb/WRYez0utlQnaoK9vswkKBEmoV+rABZSlCmNss8fijOdxERbYptkIjgZS1q9KuJ1g9NdYMYNAxrwYR/NvUvFJigJO5EoyewbT5mTFFOoPMbU4zbr5YXvhZV0vSKYWoAs15SsGZuqmM0iXi66j6yvik8TPL8ao7JEBuKkf6EcHY0P+sW2JK2aUz9/RBCUrpcif+aJy9b4XQPm092l75bwVA0LsFAo/7vBO67Gb67Mx7K7yiuIilWcGjRk1hkSBstWgIF57qhuDps8sQNJnXaVixQ4slhQpnGWOYakpsR38lsHPP5YQSrBI2c/5fEKLj+ojeuHQKE39/6zsnXXj8JI3KvC5xT6i9DyhUyfCj6T7pY4PB+Zzsj36y3OkYEigwFVo2sEYa+dYDyB66hH2elpRi9OvBVg2jc+7UJzYFIkIPJ2UWSGYmt4QzDB9tfb9yKpqCRhKJQS9NVCIfMAK6f1VD17Prn9dPdg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexandre Ghiti flush_tlb_range() uses a fixed stride of PAGE_SIZE and in its current form, when a hugetlb mapping needs to be flushed, flush_tlb_range() flushes the whole tlb: so set a stride of the size of the hugetlb mapping in order to only flush the hugetlb mapping. However, if the hugepage is a NAPOT region, all PTEs that constitute this mapping must be invalidated, so the stride size must actually be the size of the PTE. Note that THPs are directly handled by flush_pmd_tlb_range(). Signed-off-by: Alexandre Ghiti [Samuel: Removed CONFIG_RISCV_ISA_SVNAPOT check] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/mm/tlbflush.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index fa03289853d8..b6d712a82306 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -147,7 +148,33 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); + unsigned long stride_size; + + if (!is_vm_hugetlb_page(vma)) { + stride_size = PAGE_SIZE; + } else { + stride_size = huge_page_size(hstate_vma(vma)); + + /* + * As stated in the privileged specification, every PTE in a + * NAPOT region must be invalidated, so reset the stride in that + * case. + */ + if (has_svnapot()) { + if (stride_size >= PGDIR_SIZE) + stride_size = PGDIR_SIZE; + else if (stride_size >= P4D_SIZE) + stride_size = P4D_SIZE; + else if (stride_size >= PUD_SIZE) + stride_size = PUD_SIZE; + else if (stride_size >= PMD_SIZE) + stride_size = PMD_SIZE; + else + stride_size = PAGE_SIZE; + } + } + + __flush_tlb_range(vma->vm_mm, start, end - start, stride_size); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, From patchwork Sat Oct 28 23:12:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DE31C4332F for ; Sat, 28 Oct 2023 23:13:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8E5A6B02CD; Sat, 28 Oct 2023 19:13:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BEE236B02CF; Sat, 28 Oct 2023 19:13:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A19EE6B02D1; Sat, 28 Oct 2023 19:13:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 881406B02CD for ; Sat, 28 Oct 2023 19:13:49 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CCB2F120903 for ; Sat, 28 Oct 2023 23:13:48 +0000 (UTC) X-FDA: 81396424536.28.BAF48D1 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf11.hostedemail.com (Postfix) with ESMTP id F157D4001F for ; Sat, 28 Oct 2023 23:13:46 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=Wc+XkRjC; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf11.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KwX4pskItPHUM4maFDdS5O3Er1CtrhdQQ55njchpgeM=; b=n/J5qoK3IcB/VMWiRL8ImPgVfcRCr5scAA3CkLFLNiQjZCYjzVY2Il28yBDKgvPQi3TXtw +C/gnaHwloLvreZrxdMuKnrzgasqe9UHaXQPQLm062/0Ow2t8IA5zhkgD0aCc6DuC4zcXi 2rIlI2Q5+DRKxVIvSPoqFfGtnD1BDHk= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=Wc+XkRjC; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf11.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534827; a=rsa-sha256; cv=none; b=bSwTaooFkvezbChc/nB6aujy1dPt7+ao9SWWzZJ1C4qgy+qLRcXRtEWHU7yG/5/DDeUrGf 88xcKd8S2Eo/Q/4B+iEITm3Rc1i1+OATwhw/4ARKxgmsM/xA1Rczi298pr3U/dhNwyYZnj 9zErvXPsk5xtksg8B7aVjZpF5ZItpUw= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1c77449a6daso28325555ad.0 for ; Sat, 28 Oct 2023 16:13:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534826; x=1699139626; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KwX4pskItPHUM4maFDdS5O3Er1CtrhdQQ55njchpgeM=; b=Wc+XkRjC9QV2IyGwaKXob1evf4Yqn1gxDq8lFrIyYhbpcdqaOJbhaab6egsZgEqhxs ZUaX8LG4Z4ME8YLDJxtp7vC69na2WKlisAeUCvyGb18/C4tPOjV+TTP5UDxx/9xf47RY vb8ZOXyTXYSm4BSxuqob8+EB0ywxdTGvEpDLeb6yrl+SnPicjTT3v9y5yK1E9v7wmIwj 4xYOH0bBqiw+iclW5XbPvpChMSzRMUSQAlfk2jZspOOG1jWS9f6ikgitiRgR6oVtyPIF SXCG1fidzHTXCDisGsV4andCtEc2prV+wtWvmClZInzYh8jQ+dlZbotuv2IBkBwn7oDr lZSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534826; x=1699139626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KwX4pskItPHUM4maFDdS5O3Er1CtrhdQQ55njchpgeM=; b=bqr2NxsZhfa6HdPv7TtfaGVa4NTEB6upLuLHVUwxOtSaDeeDTGEFu00T9u5ilba5E/ asBDIaDbJ8Bxw7gPCwK5xPerrn0e0a/wo03DByerf9jhUirHOtMYUVIQCCPETvGg5yCG Hsox6hW/cTMqRhYc/aj/D/XKL0d6em8iu5hgoROvbz00MT7vIV4oGSFlmOfxgd2a0Hr0 5rkvZ0uzKC7OasJ6+a/wF+6sLdVila5+b/QCzuMSDPTe3Ec52KKmLkttje5Vfanlvtn/ RKCthGuB17JPt1OZhLt52GvLYB+k5PNgGnN6ac7xOc5+B1rQPpr8z825gVa/HuTCFegc +n/Q== X-Gm-Message-State: AOJu0YziEZXjsVdfoOPhCwx090ASXVd8Ddva/FTNfCGOd+fC9vfuV+wT qRjkRyc5ybudIS46lznO14AuIQ== X-Google-Smtp-Source: AGHT+IEP1RLLbzkV6SDyhDMWypu6mUWTgp+BfL6QnGjrDAvgGu3DpZZT/BgKJhnh9HCZ+gZpZ5q05A== X-Received: by 2002:a17:902:8c85:b0:1c9:c0fa:dfb7 with SMTP id t5-20020a1709028c8500b001c9c0fadfb7mr6367498plo.57.1698534825930; Sat, 28 Oct 2023 16:13:45 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:45 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mayuresh Chitale , Andrew Jones , Lad Prabhakar , Samuel Holland Subject: [PATCH v2 03/11] riscv: Make __flush_tlb_range() loop over pte instead of flushing the whole tlb Date: Sat, 28 Oct 2023 16:12:01 -0700 Message-ID: <20231028231339.3116618-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: F157D4001F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 4wkz98h1r4btn577quyyjq7j38ojdz3q X-HE-Tag: 1698534826-949034 X-HE-Meta: U2FsdGVkX1/Q0vCTNEcaddIgq4u3s2dOyBpMf2fh+BXZLuBS7+ZZmCxW8I4YYh7wsNXSTN0De+5JI9VaRiMu3PLUzHRu/NH/v6UPDdZ66/s8dFeOpx0YS+ifrzzaBjL/aDYDPlaI4/PmrHd3bQyo9GFUwQLurn6HQIPvb6BVBBsNTXFWtgQ2JUVe0T4xvf+nqyOfSAmxDd1DgS7Bl5fljLTpsNVBG6jFxJD5gACPKB7ohSzIE0pFQkBMey56JWWuou9jVKLhqgD3Mdo/eUYXiggkF3JrsaF35sY19RNO+OjtkHaw24thC8xQ0RNznRfxhb2dD/b9Hxzodg4KsIeSsw1qUI27KhjYTnCxJ+6qnzMoJttIBXhnuWKBKqnlpOmR8YHJ2bAhRc8z/lXT7U9SaZui491zJCK+8hjhYgdfiA5+Ly3MZGhViFMLbh7v1/OfHUr1T7z1yLfa87wypbNWYbvumq/Hnwmgmwd7U+TNm7iO7fm8zbjGK8nKo7KjB1+tqu9ttQ4jVMyCGivx0Ar8c/MA+1eHzpiBLHf7DBSkoCVTse2cLwL5yzsJD4HbjZGwdRlc9Ksevyb5hLoS9mmL4sFiiHu05/XblEU1SieithXbWUffQYs0/i2GF64Tv5nEfwTdXmYo8/PmNqXpcxNpIggsCmcCnDnfhiG7z7ShS/Z6wckku/paCSRBk2oPLbCnuMWcJd36aZvUCgS4wj4G4SgHTBuMJ6psWa1b20vFQOvVyfEjoheIeYNcY9l9xHlxwECUggOMtNEHbz03jUuEpI5tE0v+FZ/jR/ZGEYR2YvXVHhBniZAGkk72qY0hPWKWp++UxLvpstK5zJClJ1LaW+2aY7wRoS7pN2pw0LEDWsUXRUeDaA6HGCQMKOE2hwk2YLrk9cI4iYEnOamrIxOXkr5RzPuVHWkSyF93H749jbQ87Ikn7FCbv8zndS6aC852ntHAEugY+k3hLzl+nj1 pOg6rf0q Qcgahf1LWBT80lahG7FOF8yDMLTEpFn39VsJpm9vz2bjwYq9s/ru0FA061nWmKWw1SzuSE6YLMOlyMLDdqqEEs6hbh5wOHazAEPrGv7j0DfSkw6VdH9URaSdkmrN+5DZbCDNlOWGYoxUwplNiIHfwYmF7xeg9c2/c5328kRtoj+HU5/GVNgrrvLeTUGAXplAcQLyh2hWx7I/EaXg/WuHCK8smoRjp3tLJ6xa7wj2DpntCM0d7RGzlmLOysFyx0PRN4XPYXcYt/70ZLVk0YQMy1ram0dmXPdb2Ehacl30x6Ja97BHenJ4ItyDsLtueRaYIORI4NHIBr9p50CY6GPV6sYf4fCMZDkRZf9LJib5FtuXn+yQTjCS7IpsZ+R7sa3astK8VVoTL1GULG93nERzSyZkJMCE+5n8EvewHvircTk8tApHcd9ozdksUCr/KoztzzHysSYNWJ3VHmBtSgN1uW69TPdRdvkbilqkVh6v/HexGq75scWbDzMma1f60mp2D87CzBa4A9XkVv/1kXe3YCXZ6cMJjVNkNG1Kn/232bzaHAJ8ZVsG1l8ckpWKQhqWknxLAhTd0JoTfjTY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexandre Ghiti Currently, when the range to flush covers more than one page (a 4K page or a hugepage), __flush_tlb_range() flushes the whole tlb. Flushing the whole tlb comes with a greater cost than flushing a single entry so we should flush single entries up to a certain threshold so that: threshold * cost of flushing a single entry < cost of flushing the whole tlb. Co-developed-by: Mayuresh Chitale Signed-off-by: Mayuresh Chitale Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Tested-by: Lad Prabhakar # On RZ/Five SMARC [Samuel: Fixed type of nr_ptes_in_range] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/sbi.h | 3 - arch/riscv/include/asm/tlbflush.h | 3 + arch/riscv/kernel/sbi.c | 32 +++------ arch/riscv/mm/tlbflush.c | 115 +++++++++++++++--------------- 4 files changed, 72 insertions(+), 81 deletions(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 5b4a1bf5f439..b79d0228144f 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -273,9 +273,6 @@ void sbi_set_timer(uint64_t stime_value); void sbi_shutdown(void); void sbi_send_ipi(unsigned int cpu); int sbi_remote_fence_i(const struct cpumask *cpu_mask); -int sbi_remote_sfence_vma(const struct cpumask *cpu_mask, - unsigned long start, - unsigned long size); int sbi_remote_sfence_vma_asid(const struct cpumask *cpu_mask, unsigned long start, diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index f5c4fb0ae642..170a49c531c6 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -11,6 +11,9 @@ #include #include +#define FLUSH_TLB_MAX_SIZE ((unsigned long)-1) +#define FLUSH_TLB_NO_ASID ((unsigned long)-1) + #ifdef CONFIG_MMU extern unsigned long asid_mask; diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c index c672c8ba9a2a..5a62ed1da453 100644 --- a/arch/riscv/kernel/sbi.c +++ b/arch/riscv/kernel/sbi.c @@ -11,6 +11,7 @@ #include #include #include +#include /* default SBI version is 0.1 */ unsigned long sbi_spec_version __ro_after_init = SBI_SPEC_VERSION_DEFAULT; @@ -376,32 +377,15 @@ int sbi_remote_fence_i(const struct cpumask *cpu_mask) } EXPORT_SYMBOL(sbi_remote_fence_i); -/** - * sbi_remote_sfence_vma() - Execute SFENCE.VMA instructions on given remote - * harts for the specified virtual address range. - * @cpu_mask: A cpu mask containing all the target harts. - * @start: Start of the virtual address - * @size: Total size of the virtual address range. - * - * Return: 0 on success, appropriate linux error code otherwise. - */ -int sbi_remote_sfence_vma(const struct cpumask *cpu_mask, - unsigned long start, - unsigned long size) -{ - return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA, - cpu_mask, start, size, 0, 0); -} -EXPORT_SYMBOL(sbi_remote_sfence_vma); - /** * sbi_remote_sfence_vma_asid() - Execute SFENCE.VMA instructions on given - * remote harts for a virtual address range belonging to a specific ASID. + * remote harts for a virtual address range belonging to a specific ASID or not. * * @cpu_mask: A cpu mask containing all the target harts. * @start: Start of the virtual address * @size: Total size of the virtual address range. - * @asid: The value of address space identifier (ASID). + * @asid: The value of address space identifier (ASID), or FLUSH_TLB_NO_ASID + * for flushing all address spaces. * * Return: 0 on success, appropriate linux error code otherwise. */ @@ -410,8 +394,12 @@ int sbi_remote_sfence_vma_asid(const struct cpumask *cpu_mask, unsigned long size, unsigned long asid) { - return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID, - cpu_mask, start, size, asid, 0); + if (asid == FLUSH_TLB_NO_ASID) + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA, + cpu_mask, start, size, 0, 0); + else + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID, + cpu_mask, start, size, asid, 0); } EXPORT_SYMBOL(sbi_remote_sfence_vma_asid); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index b6d712a82306..e46fefc70927 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -9,28 +9,50 @@ static inline void local_flush_tlb_all_asid(unsigned long asid) { - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); + if (asid != FLUSH_TLB_NO_ASID) + __asm__ __volatile__ ("sfence.vma x0, %0" + : + : "r" (asid) + : "memory"); + else + local_flush_tlb_all(); } static inline void local_flush_tlb_page_asid(unsigned long addr, unsigned long asid) { - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); + if (asid != FLUSH_TLB_NO_ASID) + __asm__ __volatile__ ("sfence.vma %0, %1" + : + : "r" (addr), "r" (asid) + : "memory"); + else + local_flush_tlb_page(addr); } -static inline void local_flush_tlb_range(unsigned long start, - unsigned long size, unsigned long stride) +/* + * Flush entire TLB if number of entries to be flushed is greater + * than the threshold below. + */ +static unsigned long tlb_flush_all_threshold __read_mostly = 64; + +static void local_flush_tlb_range_threshold_asid(unsigned long start, + unsigned long size, + unsigned long stride, + unsigned long asid) { - if (size <= stride) - local_flush_tlb_page(start); - else - local_flush_tlb_all(); + unsigned long nr_ptes_in_range = DIV_ROUND_UP(size, stride); + int i; + + if (nr_ptes_in_range > tlb_flush_all_threshold) { + local_flush_tlb_all_asid(asid); + return; + } + + for (i = 0; i < nr_ptes_in_range; ++i) { + local_flush_tlb_page_asid(start, asid); + start += stride; + } } static inline void local_flush_tlb_range_asid(unsigned long start, @@ -38,8 +60,10 @@ static inline void local_flush_tlb_range_asid(unsigned long start, { if (size <= stride) local_flush_tlb_page_asid(start, asid); - else + else if (size == FLUSH_TLB_MAX_SIZE) local_flush_tlb_all_asid(asid); + else + local_flush_tlb_range_threshold_asid(start, size, stride, asid); } static void __ipi_flush_tlb_all(void *info) @@ -52,7 +76,7 @@ void flush_tlb_all(void) if (riscv_use_ipi_for_rfence()) on_each_cpu(__ipi_flush_tlb_all, NULL, 1); else - sbi_remote_sfence_vma(NULL, 0, -1); + sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); } struct flush_tlb_range_data { @@ -69,18 +93,12 @@ static void __ipi_flush_tlb_range_asid(void *info) local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); } -static void __ipi_flush_tlb_range(void *info) -{ - struct flush_tlb_range_data *d = info; - - local_flush_tlb_range(d->start, d->size, d->stride); -} - static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { struct flush_tlb_range_data ftd; struct cpumask *cmask = mm_cpumask(mm); + unsigned long asid = FLUSH_TLB_NO_ASID; unsigned int cpuid; bool broadcast; @@ -90,39 +108,24 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, cpuid = get_cpu(); /* check if the tlbflush needs to be sent to other CPUs */ broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) { - unsigned long asid = atomic_long_read(&mm->context.id) & asid_mask; - - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = asid; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { - local_flush_tlb_range_asid(start, size, stride, asid); - } + + if (static_branch_unlikely(&use_asid_allocator)) + asid = atomic_long_read(&mm->context.id) & asid_mask; + + if (broadcast) { + if (riscv_use_ipi_for_rfence()) { + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); } else { - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = 0; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range, - &ftd, 1); - } else - sbi_remote_sfence_vma(cmask, start, size); - } else { - local_flush_tlb_range(start, size, stride); - } + local_flush_tlb_range_asid(start, size, stride, asid); } put_cpu(); @@ -130,7 +133,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, void flush_tlb_mm(struct mm_struct *mm) { - __flush_tlb_range(mm, 0, -1, PAGE_SIZE); + __flush_tlb_range(mm, 0, FLUSH_TLB_MAX_SIZE, PAGE_SIZE); } void flush_tlb_mm_range(struct mm_struct *mm, From patchwork Sat Oct 28 23:12:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439612 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6971AC4167B for ; Sat, 28 Oct 2023 23:13:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11B286B02CF; Sat, 28 Oct 2023 19:13:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 055FB6B02D3; Sat, 28 Oct 2023 19:13:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D76D56B02D7; Sat, 28 Oct 2023 19:13:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C23C36B02CF for ; Sat, 28 Oct 2023 19:13:50 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 75F9A1A0463 for ; Sat, 28 Oct 2023 23:13:50 +0000 (UTC) X-FDA: 81396424620.18.4D7B078 Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) by imf21.hostedemail.com (Postfix) with ESMTP id 8608F1C000D for ; Sat, 28 Oct 2023 23:13:48 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=D0sCXu6U; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf21.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.160.45 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=56YUmquglDtdI2AgsFp8Dfqen0eRNcOUaE781b7rud8=; b=HbUF9GZ+CrdEieOMHrVgtdyXbI1B6X2Q/WYc0nYVUgZ280VfdAEKzM6wqphuLoKFK3xKtT 8Q1RsrIr1HlRM9RHggX6L8ETCw0hKfY3CavcPUx5/GRmpNjCyn6TK1WnBWWXziW339LpHS Q76ad1A1PRv70IuExOvaClA7vZW+RBU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=D0sCXu6U; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf21.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.160.45 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534828; a=rsa-sha256; cv=none; b=Aq+XNvpEIFIAgqqbrZFSAqsc5kxYBetUgooB61kwj08CHvps0dvyVjYZCF40UGRC6GhKh2 sPBrA9VSAzHnnHJXRGIpEBNMOMAmWsN/gHWvHkqv2Nn8A3BYcCMKluXOmu4818RWhobFK2 R1OxFdxWvlqjJ91m/aBZj0g6pUCh14Q= Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-1ea4a3d0a44so2219252fac.1 for ; Sat, 28 Oct 2023 16:13:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534827; x=1699139627; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=56YUmquglDtdI2AgsFp8Dfqen0eRNcOUaE781b7rud8=; b=D0sCXu6UF0A7uqDY7XpqxCd1QBmW75goOAVDMnvOe5uGZcsZqD1fsm2X523YhVz3D3 4UD1tQGiwlvQ3R6dsnILwviCghwbHu53VMYQmi+eGxK3tBiCetQcjr/g75IKz+Zp6D+R 92//36ja4xc5qyT+tcfB4k4YIJqG02bNHGDkaocy4/NvGT4+056T5lQSopN8DacIU46e iSrdP4v6+cHxBieO4nPnPZVLEyaDQNC6bz6hqK9dq0RM7eT4va49Lr7O0Ma5cM/3sG2l kh1v8ye54eN0YQ4CIFx/VX5Up3KTnBogSgPCN0pYOfze9X61NwsFK8B5ZkxHFVjW2Qm6 R9lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534827; x=1699139627; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=56YUmquglDtdI2AgsFp8Dfqen0eRNcOUaE781b7rud8=; b=kfzauEjDBKFei0cEERQoYjHciEvUQCmgZJEfqcHXZPnMmAdZ94cUCA2X2c+eX0mKPH TjyGNpVqPZI/5hjynBTFkny3QlqtEBEy8OAOu2Wsas3S7KtiWCi5huK1z5A2yt6DIUc7 nGOBIyv9ZV1aaR/JSlIc1n0QGEesz581kpRNsIkp7TYCaFm6WFZ9BzNgZXFetwH0+tbz coXllNg6VSNYRHqyaxsroFkbYEyAFhoeJvPAvKoaayLUIOYhaIMzewmNKbCadH6WE/cX NvmFXQFb/+t/U54ysHU89aHsWEQcxeN5DuiaWMU9oAw9XEUdeJvnPqDfy/6/cLBcPojo eAow== X-Gm-Message-State: AOJu0YyTBpN9VDlCl+O/4CaI8V5AsxvMmFg0M4O0DBZTrNmLVE3nOemO mjzKW38eZ+OsJnezDs00dHzbMw== X-Google-Smtp-Source: AGHT+IGq5Mcxzf3tN07mJTIgv1Wxotaz2Uaine2ZZiRfV+MdLhpoDUKwMV8rRW6x4wnAP4zHnD4SGA== X-Received: by 2002:a05:6358:903:b0:168:dc03:5b90 with SMTP id r3-20020a056358090300b00168dc035b90mr9086199rwi.23.1698534827455; Sat, 28 Oct 2023 16:13:47 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:46 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Jones , Lad Prabhakar , Samuel Holland Subject: [PATCH v2 04/11] riscv: Improve flush_tlb_kernel_range() Date: Sat, 28 Oct 2023 16:12:02 -0700 Message-ID: <20231028231339.3116618-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: pxz3173yp5xbfrgyecihf8ary6x7dfzs X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8608F1C000D X-HE-Tag: 1698534828-22245 X-HE-Meta: U2FsdGVkX1+SB/IG6IWcbeYzS90ajEr30rgyCj3/8yL9DISI2tZxS0zOsQ8bkyj1SA62NwH67SlhDp/0y9sAqzj/YGAEPbM8ER0YcI6NgKdokqWcP5N14+U8iFuLP4tg3G6KnR6jLhGdTvKGu24zkBICEVczUC057rVmOktDnPazhnWmcbSyGFU5KuilWJwsJX85ERrAupsmikGNMZUEL7a3sbTRUi+JDwdJYDFcuwL2xkkGAPX+MmpjhXs2bGtAw5JUE/G2HkibDzplMzwU2NcGLvrvJkkKyjaD6meAZuH12PpYjoA3yIEfpQ4MA0t+8PkzHNb2Tp4bTdNkpXu6LHngmVkBjKfcAyVG5Ryb4tYufJAFoTWA/XJSgCzUz+s9hj9KbHN52KcRiUsJ53D9upISA8y2yxH7ndCVJ3Sq5VHl5k58yf7ejavnCARHDKx+FyeBntk2BhrvRytssk+chqwJ38UWWRSXnSMJ5NvhDPkOFakqv4pVfiNRurQeYAc2FrrLzzyI1yrsWvPORbEl69UoJfSL21RzOf5zhRp0DVhJL/oR4vffiiSwaf4WCkZ0PB9VaNjU9UTgM7ZyYYEdlcYVRRVL+85q+zyjOq1b6tPAegPCsgb7X9VUQfeq/n/j8Yjx+KeU+uUnBH+cZOTUlJbkfQ1UkAV54cMRZzSHjVQ//4KmZvbyOIrZjPHfaYvD/7y+mIbLr1B5gdnFY0i58VHPVU2TxMvGRGU+mKtAr0qfcoxJlxprVcgH6DC56rCj4U/MoUVzql1rw8bDUc0w9KakqMNtm7ed9RMY0bL7tyYOi4QakOcd5PE6SjnLFm/I5W3icmhR2DethSvgLjwmsJFFWoCuU1Yvf35pzco/a7bvHtJwZfR/HP1a8HO30PbR72czZb0bDXyI//LvQy/yV6qkUEpvSrW+atq85Z7yn8ekzeXBgjEWstorD4qARsGLBlbb60LMPBDUvPhM2uw M6LcYuHw 8hS0Zadbuue4QNCgzSzFYD57uM17Apjn0TD7s5RphhD3yLJO0QNpt0W5Kv1O0DNuLgFf4qzKiqWpELw2IL7yVMokfs5JzMvkpFhayWsDLStVwHEWbJ+v035f/zZeoYG39U/ust+7XhBbNbqPixZ7Y6+xHXCegR8P9nXN6dvYvRUwi3NMEF1lj6JFX6GLvtuSzPLqVkeuJIYSd53RmnK8/mMEBiEWJNLbuibbG3fC2tsS6ou4MtEBcEmt3czwhQzuoFlDbIdTDGDvX1PvJOIuDpE7beKcF8mx+rWr1zVejo87xlicTDL/rE7yrT6u1yJmGjzHcN3wwer5RAikCtmasICpOofhMpHimvyCyCGmW0gnJ8sKwRtGfIQYdrUUWXU44Kf+4S7mzS/sSKy9kGDrp/6kM8acgETl+yMi19QB8+/J1T7uuBVGilv0KZukpFnGbv4UPxwh4Az4voeuqeAlBV2rN91RuY2mnQmcL9EJIevyAYRJ8dHIiXnasrMotzrRmqpXypC13txVVDKP6/9ywoskh0slDeU3NbSCw0tcImExGi9uMGQaXfk/EDNMmqL8egG2PWC/8sgjk/uM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexandre Ghiti This function used to simply flush the whole tlb of all harts, be more subtile and try to only flush the range. The problem is that we can only use PAGE_SIZE as stride since we don't know the size of the underlying mapping and then this function will be improved only if the size of the region to flush is < threshold * PAGE_SIZE. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Tested-by: Lad Prabhakar # On RZ/Five SMARC [Samuel: Use cpu_online_mask and merge if statements] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/tlbflush.h | 11 +++++----- arch/riscv/mm/tlbflush.c | 34 ++++++++++++++++++++++--------- 2 files changed, 30 insertions(+), 15 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 170a49c531c6..8f3418c5f172 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -40,6 +40,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +void flush_tlb_kernel_range(unsigned long start, unsigned long end); #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -56,15 +57,15 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, local_flush_tlb_all(); } -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ - /* Flush a range of kernel pages */ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - flush_tlb_all(); + local_flush_tlb_all(); } +#define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() +#endif /* !CONFIG_SMP || !CONFIG_MMU */ + #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e46fefc70927..e6659d7368b3 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -97,20 +97,27 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { struct flush_tlb_range_data ftd; - struct cpumask *cmask = mm_cpumask(mm); + const struct cpumask *cmask; unsigned long asid = FLUSH_TLB_NO_ASID; - unsigned int cpuid; bool broadcast; - if (cpumask_empty(cmask)) - return; + if (mm) { + unsigned int cpuid; + + cmask = mm_cpumask(mm); + if (cpumask_empty(cmask)) + return; - cpuid = get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; + cpuid = get_cpu(); + /* check if the tlbflush needs to be sent to other CPUs */ + broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + if (static_branch_unlikely(&use_asid_allocator)) + asid = atomic_long_read(&mm->context.id) & asid_mask; + } else { + cmask = cpu_online_mask; + broadcast = true; + } if (broadcast) { if (riscv_use_ipi_for_rfence()) { @@ -128,7 +135,8 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, local_flush_tlb_range_asid(start, size, stride, asid); } - put_cpu(); + if (mm) + put_cpu(); } void flush_tlb_mm(struct mm_struct *mm) @@ -179,6 +187,12 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, __flush_tlb_range(vma->vm_mm, start, end - start, stride_size); } + +void flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + __flush_tlb_range(NULL, start, end - start, PAGE_SIZE); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) From patchwork Sat Oct 28 23:12:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC773C4167D for ; Sat, 28 Oct 2023 23:13:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3406E6B02D3; Sat, 28 Oct 2023 19:13:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C5886B02D7; Sat, 28 Oct 2023 19:13:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A48D6B02D9; Sat, 28 Oct 2023 19:13:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E266D6B02D3 for ; Sat, 28 Oct 2023 19:13:51 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B1E03807EE for ; Sat, 28 Oct 2023 23:13:51 +0000 (UTC) X-FDA: 81396424662.06.320044D Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf01.hostedemail.com (Postfix) with ESMTP id CE78940007 for ; Sat, 28 Oct 2023 23:13:49 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=lHjyCIKO; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf01.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7iD5kVeuyzksz9qFDj4nk3gJEMMhm0SiL+jEgfo9gAo=; b=1JD7hTVnRgAhHqMoT/+GA+tWgJIfkYzDZo2v5Bio4qv3nxKxm5I4cBWZBosZbRoNZ4/xtm PYUNzDiQEpzpGBre/3iGHYXtgy4iMpB5/qa69gF3o5HIOo8Qg5nq/x78fCF/VbXyUvjEVj bww9TUNsKg8b+Aw3AY6RO+7oG1WYQMM= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=lHjyCIKO; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf01.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534829; a=rsa-sha256; cv=none; b=gfF5iAgm03fed6FPcErRufdAEFBA1iN70zn69evwwRCzb0rjyYwV/T+oEXTcKP9PxaMA/p VTNFAmRMc+/AiiC81E/e4c2A7BHtw4LG4WOS+9VO+v3yiuLuBi5+HI1mkwPeEzhm+9/ZzB V10SUXuKuxihUdLYTi/77w57vFrgPh8= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1cc30bf9e22so6051065ad.1 for ; Sat, 28 Oct 2023 16:13:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534829; x=1699139629; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7iD5kVeuyzksz9qFDj4nk3gJEMMhm0SiL+jEgfo9gAo=; b=lHjyCIKOt9C6qR0SC2oC7Frie1OQ3GZoXy/OZELdawDgcdbKecl9CoUhDfjaQhJYVv 5mcdXaiH5GP3L7UOIiv0lYf+KQQS9LWMciJHyAcZxc4xKSlUmsStOm8Qgn1QphwNDcPl V9YBXeC+J86xHel3WolHGIJh2RU91k0t/49Y9sUgkY5opON2g57f1JxKgjTYKitfA0YR Ea1TFxmd5NyeyQ3PHwq/drSpsPn3phLVUg8v7gKbHVVDMxGDKUT/FScwB4N6juHrItp0 k4Jh9SPDEwRnIDJIsJRJym+MZLVwV5jzYBhAyYYT6IIvTFgcPrF0M3y+oByYqgqo7y3+ RQow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534829; x=1699139629; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7iD5kVeuyzksz9qFDj4nk3gJEMMhm0SiL+jEgfo9gAo=; b=U2wOvpz4/j2S3GZFfcV+MGAZwxnhtqqB3A/j77h50qcJkSFWguHuV/U4MhA3OuFInC qTWiB5GV+aYa0WXxdwNgWePOiICvTXc8bSQ1YcMvfMbysf5lAwlL8K9ZXe8bXOWFTVnU 9UC2IQ0ZpduwbU4qDjrheAKVAFbFY4GILx5ggc9EV8TLdNOztRxEk2I2wK8N4gRLZhGP rs8xiH8seNePtq5Y/OD9H5gA6VRueJNuJDR2GvkW4YLPjvf/yeKShwU/1PkH+RsVJMO2 Y1bupeDl6JaYQnOgsz35TNJmOyz+FPNqpwnY9f76ACKBpw5IRtvbhIA8FzcYk2qhga7z BXYA== X-Gm-Message-State: AOJu0YzWH3xvwZP3GFMIaMGB5aXNmWbLndZaDMCjy+HFgbnFGhrb4fi7 9WGR6EPypnGv5JynLSGk49fMzg== X-Google-Smtp-Source: AGHT+IGZfjM6oWIducpDDQ7hM02X3ekD2Mrbs3+cVpCJdtVQrXLcSNbo2i119+LD0HVTSm0uB+XkgQ== X-Received: by 2002:a17:903:11d1:b0:1cc:f41:8f82 with SMTP id q17-20020a17090311d100b001cc0f418f82mr8777913plh.16.1698534828754; Sat, 28 Oct 2023 16:13:48 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:48 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 05/11] riscv: mm: Combine the SMP and UP TLB flush code Date: Sat, 28 Oct 2023 16:12:03 -0700 Message-ID: <20231028231339.3116618-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CE78940007 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: rujej8fpp4os97f386ec8z4b83jp35gm X-HE-Tag: 1698534829-568356 X-HE-Meta: U2FsdGVkX1/q7R7Svie/VHsWv5PEV8Kwix+e7+24RY4CojoFZ3PfdAl9mfbHAejKOL6jUdVTi+jIRppHvrsCWhsNqvTiJrWsaVbOYHoFh8syrPenQWLBfOe7IcQBAbCNm14tJPkrRrXd+xIBnOHISW6XtnBk2RAFbTvALW4e1jHRf6xdttUsvqjZiKAx9IqE/1n28rM466RWYRV7h1fLUulVm5yLlOubnHxj5cMPmBmzaiWroEfvI5lVeOvBmOFx9fm/m1MomGHrUkRm1ZbbMt1oaUepytSHqhYjX4FuCt30W5GWbf9O67CLm8k75JyYxgpmsFhrKr6HP5mkDSCwiUYUvFmxLJCxahg9OgDtDL+Mbr/sHcqfSmsObBj+q8jt55jFRhQzAfBf6ekTIQ4hzErXQF5arZTQCjwokOOAX4MENZ/7Qlxr7t3Uxr4iaRx7xL4tHCyNQySQA41nLSLAukwCpkrRyJ7WQqlTh1wWtkgXNnqcolGiX68BfTnvcrQq0JrZp9jIeZPYJmf20mqsHywL2SiVG0H4T5O7HqfUJSmwfwGLFTAEKL4NPVLaZyOUjxneelzPNfRK+N0Q2OWXvelpvkbUCtU3UZc1XVVsINeH5PwTZ2hovwAAkHJ5Qc6ntURdO8vbN1z3O+92E5fYvjqTLvoOHAaC4FbnMkoAIxPZz/hhJFpdfo2KJEH96yb9SpKG54k6duZD9en9sQhjqcsO+pF94aZdk/ETmCBXargKEPYRIBZkpz+ruwK4IQsm7zu8zsEB/lY4JFfECK6MMLgIwQ4XspJYIy9dc1XAhqFtrjt0foDq2XS0CBHJAVq3lGriwzghQaUG90B0QN8eyAbEp7Yk5zP+CBm0rgxLL6T8nh9WMvLbfMVFp1WYb0q0enNMcJUIwEYvOO6pp8v8jWY+a93YKMO6FYEplduydJwh76LzNEUOjOW8UtVdCQc5g2qkeHJ8iFk2xzF36JS TwrF9p/N pdJO7iuRykg70sUipA2i31poTcYwv+9IBE26cVgT0+lG7ounGvPAustEmeui6z/enaxTvGT2Wa6rGgyHSo8FH7EqrbnS7Ub7JxHHcFL1avbFsVxF3aBopVvwq3L3/SMMu8TRtxiQcCp8l3Ml3T+FNmFqSE8XQZlfBnA7sP8eJlTZPRHN4a3dQte3o8Hdetc/JPhK8ZX57wIycr4TsV1Gfui8NpNQzLZFiw1YB+pPHYkZlly0Zyb6siVbvs9mQxRJqtWLOlo5NuDo8UsWs1sLFXEdA0A2fThUG8QUh2vfR8syVApnEtmiRtaFWQohUBqf5RVc5behAth8rTwyK44jcxEM/CLW1GiLVgB9hB5tPcGVjl4fKK5k7IzfJtxnwSozcCEkWMPy+vaTwqbA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Signed-off-by: Samuel Holland --- Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/include/asm/tlbflush.h | 33 +++++++------------------------ arch/riscv/mm/Makefile | 5 +---- arch/riscv/mm/tlbflush.c | 13 ++++++++++++ 3 files changed, 21 insertions(+), 30 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 8f3418c5f172..317a1811aa51 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,13 +27,12 @@ static inline void local_flush_tlb_page(unsigned long addr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) +#ifdef CONFIG_SMP void flush_tlb_all(void); +#else +#define flush_tlb_all() local_flush_tlb_all() +#endif void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int page_size); @@ -46,26 +45,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 9c454f90fd3d..64f901674e35 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,15 +13,12 @@ endif KCOV_INSTRUMENT_init.o := n obj-y += init.o -obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o +obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o tlbflush.o obj-y += cacheflush.o obj-y += context.o obj-y += pgtable.o obj-y += pmem.o -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) += tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..22d7ed5abf8e 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -66,6 +66,7 @@ static inline void local_flush_tlb_range_asid(unsigned long start, local_flush_tlb_range_threshold_asid(start, size, stride, asid); } +#ifdef CONFIG_SMP static void __ipi_flush_tlb_all(void *info) { local_flush_tlb_all(); @@ -138,6 +139,18 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, if (mm) put_cpu(); } +#else +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) +{ + unsigned long asid = FLUSH_TLB_NO_ASID; + + if (mm && static_branch_unlikely(&use_asid_allocator)) + asid = atomic_long_read(&mm->context.id) & asid_mask; + + local_flush_tlb_range_asid(start, size, stride, asid); +} +#endif void flush_tlb_mm(struct mm_struct *mm) { From patchwork Sat Oct 28 23:12:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32C6EC0018A for ; Sat, 28 Oct 2023 23:14:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5895A6B02D7; Sat, 28 Oct 2023 19:13:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 50FA06B02D9; Sat, 28 Oct 2023 19:13:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33CD66B02DA; Sat, 28 Oct 2023 19:13:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1E0586B02D7 for ; Sat, 28 Oct 2023 19:13:53 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id EABF34014F for ; Sat, 28 Oct 2023 23:13:52 +0000 (UTC) X-FDA: 81396424704.01.E6E4D23 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf30.hostedemail.com (Postfix) with ESMTP id 23AF780002 for ; Sat, 28 Oct 2023 23:13:50 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=QvmlC+7Z; spf=pass (imf30.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534831; a=rsa-sha256; cv=none; b=MGiFfOGPUVujhZzRjiFov6SjFvzJVZ64voC6Zc4a0+NkxL7GfU49mbuy1ZmVvSGREyfITM ZS9ViBzv3dVk1cKYIj/FzxtdChvgxXNyusoIFgCUN/NRzI/Ba9LGjCqZtJmM/xFoGtyMRx S29Mwk9CLHfwykDZuTxA61rTje6xKk0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=QvmlC+7Z; spf=pass (imf30.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534831; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UP8DHxv66mJzZKknfJMFXXbwrunx+Hl9Ii6vLteV5ng=; b=yTpWAfPtatPhNDxxCIzoyFq3qZI9FwBH8MvQR7p9LLSv7O+B8Zmu5P8mSJAFYqzA4cPPz3 orXw65Z9kacF47fFIdF6mzGHZyBHitauGHpzOGkx9i3yxJ8NJDQr+yRzsAWWFH9LUpbf3k 29Ne4ZWzcCSjMuMbIc13HtISshqklbA= Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-5b93ddb10b8so1388368a12.0 for ; Sat, 28 Oct 2023 16:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534830; x=1699139630; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UP8DHxv66mJzZKknfJMFXXbwrunx+Hl9Ii6vLteV5ng=; b=QvmlC+7Z0MNKqGfzzc49vfhTdGnvigC+l2UyHfq0S5PY+pZQURm2604EHlV6Jz2W5t 5T7zt8hB9BWpEfQALxCgHnAnzE8/ZGdgQCybHXCWS3g53GrqD0OtvWbIIOo2TnPJ4B1i oh4fmojjUBow0v/KjyWMoU4UQkuzviapRaz3gLXUsIIpz0Myd8NyNvsBLml4xL7gpr/k 8KZkwQoHs2p8205S+6MMwGynNqT2VXN09dMif7v62JQ1mMX+uJyAv4VyfipCfQAzjL7Y GHEy6avTiJqV/Ksn8o2IU6xylz2Pq9Oessg/sOSQNxo6pUteXs2vsauH5zXmdn3IqfXy Bf6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534830; x=1699139630; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UP8DHxv66mJzZKknfJMFXXbwrunx+Hl9Ii6vLteV5ng=; b=bYPlbPbXCuAdzv16SDfPn4cbmHz0ZUaKbc4Hv4QXzArjMQP0MX5Js2t3zrxZfm6pI/ SQvmIwEmxLxn7Na+hWWR1iYzNna2B8wHrxNuVy+nfDo2q+X+dUdAgmJf0g3GCxTtnunU CCBQKxa956VNr/qcQBC/RGZ8GAhD3+4+O1XTIwTLX7H8b+cqBnNGrqp57hGQbcq8Y0ll 0i7EpGAt1aZZ4eVuJUkmn0TXqdM/JIlugXUa6RvjkYrFDnoAq2iqFHteRQf9u9UF5c9r Sum9PTiUb1ydL/sk2vvbxornOJ/QwIT/IMyVCwKefKG8ccGj3SsVlsKil5IIo5CU+AJW Df+g== X-Gm-Message-State: AOJu0YwcrjEoygRgbl42/567tyKvqo3SY05S6mVmEmrf3jyZWOuSBpcP qrEq5by89ewrGw8LGHfKmoAmiA== X-Google-Smtp-Source: AGHT+IGnkbRXw2gehdqQUYovjjCjlvZO0f8kuIzIEbkVVgufqh5Fzfqz26Y3uN2f14MpzFpb+9CFvQ== X-Received: by 2002:a17:90a:1506:b0:280:2a3f:9938 with SMTP id l6-20020a17090a150600b002802a3f9938mr2800689pja.12.1698534830039; Sat, 28 Oct 2023 16:13:50 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:49 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 06/11] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Sat, 28 Oct 2023 16:12:04 -0700 Message-ID: <20231028231339.3116618-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 23AF780002 X-Stat-Signature: emtise6m83bjihmtmj31zxmeod1oz39k X-Rspam-User: X-HE-Tag: 1698534830-86322 X-HE-Meta: U2FsdGVkX1/sozyrG29uBygEsgGn+GVlPHewQ46ySZaCz61MHakW+uPLDgzddOFVW38lXBoGqf7FgaoTvukERkZ/pztfTuHjJ+Qo0pphq0Ygp4l2ztt65P3m+rIAquJ4qTuIxUx6ZDsO5DxNMd80pWOkDU+tR+2gBhl1fCrjwELzER2rfbsr2NHpyfMyEA0JBb4SZJHEviVCEP0OwXvQwM2tMiF1R3V12MthL24KZJbyIogu0YLUiq7TtqrrT+es1DTrF3XhFYU+127pWQskjrapkkg5WZovqTaQGUI863Zq8GBFQu3rCUavf/o1N4z4jbM6BQwMcN25E0P9sHKcyR23jxCGg9p0JV3HDdtW8wlaqcj1P1Ctt2C0VupMI6Y5/5Yl09qdhmNuvA5Es0aic3GJrYCl8y36IQ1Gh8IDKAryN5r0bXrCIJX/zc/fCWBcHlvwgHV49bqwy5MPwsgZS+yvyqpX6J2bXpOIkSzeJA//X1NvY0nOjsDnV8w2wufV3vkX9fyVYbaZavDxcMdET8JW1h+h7aLZQz3/5YWJr91iKk6++K0bY14nFt4a1a7Mz29iKAHiwf7Cu/tTFakjHCKgPqmrALkCTzfwRK6Y/7Ca3mQQ331ix5EfzGETBnIEpawKP+ZvjndcU71xe8DJ2sGYsa6sP1RyI5jHY0Pu5LP4JcbqWQ/qa5/pNhBNKx1e78umhmF6StxsPjEvdnTbWznybHtDzy5l3n1ta53k+v32TGaFuU2RZifr47/dPkAMIF4/NWtXOMUm2cTuthtWEfzjoRjrRdo58Engz+T/2J4N10+8XjaJ0ojqNIQZeMFOgxIC9jbgNOjQU6S2Bvbwp4/XiokOumsqnTqPCxdu2Ibq9x2FmGRac6IFU3VsAYeyKH45I54zuqpu6kGOXrsUH/bRRzO2nRsgc1SIuDa38Wb7gu/FC1lD3FbGbWB+rlm7OUdna4PqdJoyckA+xCa UeqZGhih tw92VG69gvoyCjUqarUQBSGWboLwanD8eYanC0Q+eHLIblJrd1c6aSM741V7c/MhnKXI1EKBtW0614nRm4WL4fyukXBH8+gKjQipSL6Q2cqHtwlCQhxXsD8gNorr/qJnlB58eKTBYv0Hs/cU/+3Pf8rTzitbm3A06rlClfWydHdSh344mbwxleJDe3WQp2FJTQ9Sf3DbzjX6uLowKWAoV2elPk1KcLuZI8bOxqArD+IwgVTTfLPM3xMh79gjqqTioMjhj2l8/TMZUGuWzn6J7Z2HwNnenzQdVn94dUYukkKwmn7QF9TUvYhmu2mHVkTOMS5yltZoHR93bXEGJzu0toqik0Rd0NwYGiXC4MiPz985hcfLsaTtSJaz/Q7uUMr1kqhBBcyH/84SoFgQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 != x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index b55b434f0059..d3f3c237adad 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -44,11 +44,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 317a1811aa51..e529a643be17 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } #ifdef CONFIG_SMP diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 22d7ed5abf8e..0feccb8932d2 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. From patchwork Sat Oct 28 23:12:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9CB4C4332F for ; Sat, 28 Oct 2023 23:14:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86E876B02D9; Sat, 28 Oct 2023 19:13:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7ABFF6B02DB; Sat, 28 Oct 2023 19:13:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D5C16B02DD; Sat, 28 Oct 2023 19:13:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 466656B02D9 for ; Sat, 28 Oct 2023 19:13:54 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1A3CE4091B for ; Sat, 28 Oct 2023 23:13:54 +0000 (UTC) X-FDA: 81396424788.22.45C49BA Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf08.hostedemail.com (Postfix) with ESMTP id 4E34116000F for ; Sat, 28 Oct 2023 23:13:52 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=H695ZLci; spf=pass (imf08.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534832; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uq0QdLYJN1f+TxJhAAAn9k5Zt34COvQfV/ogec/O0wM=; b=2Gfl3MjaWE4sd2uuuNaQgFl4H4YcLnlQbDLQP74dwDUyYIDApQm50DMZxCeivj4fooOONW Pzhu5o+N5TueX3iQdhCqG8WQDc5eCDJOF+8ATNtVb+b2j6ZQqIIlnMXZPJ/SZPrrmy0Ca2 gcDPdZ9qYoK6mFpDxNwBzUCTeHMkHoI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534832; a=rsa-sha256; cv=none; b=qrUSGOauTEBdU18VJD2Up/2YGAe+d6tv8DzITG15qCHkZ28GJiDdkRJIRIjEUev2QFXho5 A6HMPFeXfblPYbBCiGckzQZjD1Vef2GhaLBYZj90vbJEZ2JJyJxWCk3YoVaKPA599VxuOO S7xbUje+u6qBcbnRTa7iGpXh04k/QMo= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=H695ZLci; spf=pass (imf08.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1c9b95943beso29440405ad.1 for ; Sat, 28 Oct 2023 16:13:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534831; x=1699139631; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uq0QdLYJN1f+TxJhAAAn9k5Zt34COvQfV/ogec/O0wM=; b=H695ZLci8pEFW9TbYE1xQSzFlOv1cZRna33ed8wvZz0C2yyLENjaBzCGlNKd+2+dAE 1OVyBKBY53XAOvrGi9nOH+GPN6p07m7doFOwKIAys7JTDHabTbYp+wMet0aECHBgMKUe 1ZeoScqXeAEGHCZmQ9JFPeDaBXE5fTABaV15wocHQp2ClopPtLRbnbnc/8py7Ocdhitx 2+8/o1lb6u/exsfK8kcxnayiqTMY8Rae1kToNe2WYW9ERUZXJpgC/ycAQHTN5ehkrklI xFbV1K9gssLaO5+kenBktcjVjjY2FzTZcimLYv69QIW6Aby1O8UoHWhZqX54OKSWzmXC GO3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534831; x=1699139631; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uq0QdLYJN1f+TxJhAAAn9k5Zt34COvQfV/ogec/O0wM=; b=ZQFjXQDqaUUogwjkyVjTW7ANh98HCEbVu3OL5DQxsJU0uscyKT2qMYjuxLc3r9+QhJ 275XYNsscaV5/ndcffL2SFBzIJTC2YlgfRoDGpSSUnnN3UC/cUbCH3L+1jDXiSn2q2Nc IX/JLogQqSMDYM45eMW1FE2dkvAE8ltnLkc6GBTlpuQVTzWeEdOzuNgB78Z/p+iY8k+/ xu10Fb8jprjhKnXruFNuRCx+RfDGCPUNCBQ6cs0UH9TtzevosidDl3uhHyiVfxVRJsKO 7GGxG09Je5AGxsQzen6dJPTqwBFmAvx+BOw2W2Dlnnwk3wwFy+nKt0ASusU3uFcQETwy dNSA== X-Gm-Message-State: AOJu0YzL6XJ/2q3OqTycS6pzAAv1I5s8AIr+XNjie+hpEMVOKujdcqEl lCAQYFWq2UKDy+Y+QoBJba1MyQ== X-Google-Smtp-Source: AGHT+IGqJfNl+X6UXX6MsfY+UkH1b50Ad4rlkkQrptVRUTaYY/ZhDSmvjBpzXH88ANRSqVScovaqNA== X-Received: by 2002:a17:902:6b8c:b0:1c9:bf14:4f94 with SMTP id p12-20020a1709026b8c00b001c9bf144f94mr6530418plk.44.1698534831309; Sat, 28 Oct 2023 16:13:51 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:50 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 07/11] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Sat, 28 Oct 2023 16:12:05 -0700 Message-ID: <20231028231339.3116618-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4E34116000F X-Rspam-User: X-Stat-Signature: 3rtc67xjywtipumfc71y1ayy33wdi8ei X-Rspamd-Server: rspam03 X-HE-Tag: 1698534832-539079 X-HE-Meta: U2FsdGVkX18/GZrntRFsCZEche/R1tVU1IFU+e8KS904iCqpEI9hMxi5D+o7bfcNywcU7pVXrBMY+cZD+HjKJJO8rVwuo5ykp2TqrMz3QJeN+hGPLbUg9x+8M5mHGCTElvKC6BCZfoKVHdyfFj7TPwE9NRK0UG/Zv6xJKY4p/5dZOz/n1SWJG2zv6aPHMyi7CCC7p2jLNBnhhK/jSwmVan9IZ7XRNQ1tUhMdv7Kw7KyPpBHHh+66Htw5LzEXWzZns//yy9GJaAcyig6/nqsNHcjC9VJW/RFuL4VKMilvQgXfUzFdwa3AgIcillhkV99qQvAvsVGK28jWS34csAgEYll31adPBIpmeX0mupV2pzcn20iHMS9+SarGdv6FrMMypCyTO0y+Owk8c3i4oO2U/0WENfi2PbuDvl5+SyWYxVGAJS+iqtLlXgQK6F83PJLuHZLCvZZ5uB/l49VCiYEJhaBN17zgkwqIYGrBsx7+gBQeb+JlZNJBrfQcG30nGy3Hwxjx8Xl7/lEYpcuH6shEMP4dGqMPQMTEuEluREXh7yvp0AfdDGXbnWxQgdH0UhAr0uJi3t0ftvptng8/3wMgJjEWvwWxJNCLwhXHgZORFnRX36H0vzWHRzRj5inQVIiwqWD6QSA0YI4Ke8fHoYFdlioTk3psXcBkFaU9v3FHfoP+XzZ9AxXY5TlC8xU/KvfKX1aKCbfwFxjJ5j1+wEFPCpEkr2Q49SxclfTsWkN00OQ16DbSQAquXeKtE/1Emgj+3GjyBDWFFHunCswgy3l5UHZjlpkgSJ/t9IXBFPNgDwLNbiAzg/KstdDi/h/rR5zLcJFUGkxfsZfGaVggY3o8KF2mk0ouRPpLvA0JEJBDpgRSAWTAuctkbwvC5CzlzNBtKm/dhRELbI5cfCRlTvzSx2G3BGWs/C9NIv4GwzLPT1Eq3jWiyuV7LhhGKkF/yV2ZIvYEoAKfvU7SVexp29x /MiZC+R4 2UN6cw3bk/rNsZm7ENjfwcuuk82ceOX8G7UTICusUXvMw0e8yE9wEWe1WZJzsYpW7hlp7jFy93OF76JE3Z7qBeurXgXmsXCtPhwdZhvK/UxamdqTFrbwm8YoDEG2uUViCC61+sO+aVlDgMpb0sMr8wQu/HrLbDZCcuOT76ciyYEi52U+jmIY2dvL7qkCcSZS21rr0z6FVAqPSX3irzDleFuGgBfwMWiBxs/Ie957J9HxAKLlSBTStTQVn8ihM+Vgua4DPua+3UJn3AGJIXxPQiD4KT+IPUvufGVxSlGIEGcG6E0KG+ixqB3GpFHIfpkEOxe0GK/IcFhrNFeEazN4DnzIKi9374LI8XFkts91A0zdqCv0T8FaRJpcpEQRSD8Z7TuMZaxVn2CBGo/Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 4 ++-- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 217fd4de6134..43d005f63253 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx == 0) cntx = per_cpu(reserved_context, i); - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) = cntx; } @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); if (cntx != 0) { - unsigned long newcntx = ver | (cntx & asid_mask); + unsigned long newcntx = ver | cntx2asid(cntx); /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) */ old_active_cntx = atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) == atomic_long_read(¤t_version)) && + (cntx2version(cntx) == atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) /* Check that our ASID belongs to the current_version. */ cntx = atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) != atomic_long_read(¤t_version)) { + if (cntx2version(cntx) != atomic_long_read(¤t_version)) { cntx = __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0feccb8932d2..1cfac683bda4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -91,7 +91,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; if (static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; broadcast = true; @@ -123,7 +123,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long asid = FLUSH_TLB_NO_ASID; if (mm && static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + asid = cntx2asid(atomic_long_read(&mm->context.id)); local_flush_tlb_range_asid(start, size, stride, asid); } From patchwork Sat Oct 28 23:12:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81BF1C001DD for ; Sat, 28 Oct 2023 23:14:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B56466B02DB; Sat, 28 Oct 2023 19:13:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEAD96B02DD; Sat, 28 Oct 2023 19:13:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E3FE6B02DF; Sat, 28 Oct 2023 19:13:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 74F1A6B02DB for ; Sat, 28 Oct 2023 19:13:55 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4541D14087C for ; Sat, 28 Oct 2023 23:13:55 +0000 (UTC) X-FDA: 81396424830.19.5607E4B Received: from mail-ot1-f47.google.com (mail-ot1-f47.google.com [209.85.210.47]) by imf17.hostedemail.com (Postfix) with ESMTP id 8067440006 for ; Sat, 28 Oct 2023 23:13:53 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=Okj1T4kA; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf17.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.210.47 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534833; a=rsa-sha256; cv=none; b=e4eU6ubtYY3EzUb2aMOl5BiDpPN0LUA5iyL7HdRqI5IADixOzXtZ7+VVZhjPk3yG6zMmOQ 6XCw1MUxXM8GXJbIhsufaX83vzqhbHy+soy3sCwOVvCqdYf3renCKSaNEXt3kPwbdlJT+K ViEUa+eeZyy0X6RxVE2cPmPhMMi+SXA= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=Okj1T4kA; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf17.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.210.47 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534833; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nfPkHia9ku1mIe8MIuwK1ts/4i7vxeqzbyNHZF4YvJg=; b=XjCXSDDppPKFe7NgEQV+yy/2GKyP3el1Aq6J+zfLp1QE5MkwlwSJ3Z8Gk4CFqz7avAPJuY FlS1ezihJ7jC/ePLYL6Qeb7S2ADlP2h4EWHYG7sz+fo3+L4T42ddpYTpTJtx3tTfAnC1OH zIUnqbYH5vReHM0IfBEf/ZO9l4sCRvU= Received: by mail-ot1-f47.google.com with SMTP id 46e09a7af769-6ce2b6b3cb6so2234147a34.3 for ; Sat, 28 Oct 2023 16:13:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534832; x=1699139632; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nfPkHia9ku1mIe8MIuwK1ts/4i7vxeqzbyNHZF4YvJg=; b=Okj1T4kA1CGFUTzclvAQ7z/hWpYx29wqpK4FaWWbT+lpSmmfQ3+uSwFI9jTiTYqplU DPqGXTSCJfImCsE+mJVwpC1DMuSujYK/GLD5RNRnclBkQhbHYnjfmgwfGjTORV+khDff azsavRAdHFDzofrkl91T5kzlgZF/1k3UPyPb/Lcsnh16claLSRFMVx6UhmznkTScxW2W fsQvYvvIiq3AJKcvyiUN+I6VFhXtoe682AQmC28V8fx1xbxqcBwpt2w9G7jfJlQmuK5O PYqDPfu8DoBYfGM0ZtofHm7rTkHiONilSNq4S4FDBAY7aK6zR8GzgxDeTo41A7X4xnyj 7tMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534832; x=1699139632; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nfPkHia9ku1mIe8MIuwK1ts/4i7vxeqzbyNHZF4YvJg=; b=LSn08tU3rzyHw4jnLauYjLL34gJs+XcpE0qOwd4VkQzsEiKQIMe0vqjQARhVUmrvsE WvYAMJty/4HAOxaWNOqYYBrUY2cHbGiSxNpAy97f4Fhr2WBUznhvauGOO5DatQAhml97 gW16o/cuIvFy/URFr5T3ySdZBsPiqI0EvwmXEQ9D/DkZDMJ3hHlprR4B9jTU2hw28Vrd ftQvclH6WcCwolZK79bQn1vOj2PJYgJAPb5ff/AdEeWebKQpMN6/BJqVHQTPanoc1hpp 3PMvINHemKrjC/s5up8I/xHqMIgoG1R5FUEGsx9cUzRY0+K8ayTIUr1p8bdtmqKtSTUt OusA== X-Gm-Message-State: AOJu0YyxhgvYVNAyuX44TAwbb8alceRXYp7pUz6722cEGKPmhC5KuczF wiAyOhf4FJoIut+tKyGkelsm4Q== X-Google-Smtp-Source: AGHT+IERITWcnpZawWSuuFpc/cjj/Gs86LMrNqv2CdnNpOWafKnnt4wsBVVnxZL60abLMgaX5a0ZCA== X-Received: by 2002:a05:6830:1e30:b0:6bd:c7c3:aac2 with SMTP id t16-20020a0568301e3000b006bdc7c3aac2mr6567117otr.18.1698534832734; Sat, 28 Oct 2023 16:13:52 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:52 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 08/11] riscv: mm: Use a fixed layout for the MM context ID Date: Sat, 28 Oct 2023 16:12:06 -0700 Message-ID: <20231028231339.3116618-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 8067440006 X-Stat-Signature: brtjq4qwc4n85azahhnpeogxqxketpsk X-HE-Tag: 1698534833-597490 X-HE-Meta: U2FsdGVkX18FzFstjFiWm13uFK8UIUjQfkuW9a9C9m6rzjWlmLyO5TsqjjWJrpLj6NmhaoRVrtTBMyo0g/v1rYzBUiPUBJ+y6J7xUtC9LpBjJCSZozwuC+7UvfE6fvW60JkSKKKxwxqynSetU/Af33k3cRHwAEnm+KLmrxEZTp2S4tCaWO5SJ6P8JZxYLEUMpEmxj88d4mX2bZjiGP3L7nR8xmrDqQKOeroxu4DZvRNFpTF7leLN+DRCEb1URxep5hH3EoJg+bWu7WmYgozwFrgrP4+2CIxNqLKr8ZGQpfFoh5R693tiatRTLpwdDTpsQPaTl1GWBq/ETiG0FNlCaij5DLpNsPQ5gF+qGUGRCxaygcEFU87IE2DgVNZdZzqC7rt+M8ZNTpe6lmwM8m9DLS9h7BTgZbUjITxk9ovj9i72xWnQF2Wg09ityhOiatZYC9ThDsZ7AVwNjLaBTsf+qyWHV9U4kZR+QY5IJsCtRBd0jsLWeTmtGFVP8iJmAAK6K4rJagLlHBsRZInlrARPvDYJFKnKcZnmbaYC7vgTKkW1x1OV7l6CyKIANvzYyBS+cWK8sKUwjsh4FuCFhZdTSCySTPQvN3yPB95jxUDFoSOINECvtAeGDSuQkFjcQcZILsA48wzp1HrBsmIwsrVphvUuHT1jpWAlvD7Xroi7HBHwP3mi4kG5vfjNdo8HHxQd4UosjWL8nHxY2eJZ2jHu5P5VPLo08e+hn9K/BO13MzL2obaO5JJtW4Dmwe/BWbBWl18B+see7eyediWItz22bM1ASE+HaEvGSIEM4g+3DiFnzA38jBGX4zXWOWpDzOSSvM7HIA9mDm/EopzpOFBRMBV6loiWkL31zm35PVlbVV7k+Gm+BT1G9aYoBZjEC/acrwojqz49J1kURcA+Ivyn4RGIKUfqngwI60vgiXs3gR2h8e79QmTdTJcra6dodXqAO/s8oWaJkjiGcYhSs0j lJpEo5VP x9Ze5ETpO+Tyx+NrG+toJBbFZv4Iw0uPwYdsCZQ1rd5GnWx2ReGFnV7iqaQ1BEj7DKXgAMy5HDjrk5sWcxPS5S5uPyRK2oNYOIqv+JCKSWYzT1vlYlOBKZ8df0pQHkiU23F42217naJZnkDqHB0Vp9TDGRAASQsYx5c56TZ+uDQ5w4HzU7CgNJRQxMxXOkFzccuSacybY6SKuUGz9kRqwGMTZtZbl5yYErsZt7Ejb2h7LrC7rDEWUoKydrN+5171hwioZ32gveqBAkHROPnFvT3UFxLPJbpcs0UirWlNaqaGsLtwZ6Req7VHAsdJ/uWq3bN4bZd8o5JzyA1I5YhQ7I8nbbIf9pfTIkO8fGU2hhdq0RmrpM9FXnwbuKw+diS87TzpQk9inxzTmmbQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000150, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index e529a643be17..62d780037169 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43d005f63253..b5170ac1b742 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; static atomic_long_t current_version; @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; /* We're out of ASIDs, so increment current_version */ - ver = atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver = atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_version); /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids = 1 << asid_bits; - asid_mask = num_asids - 1; } /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); context_asid_map = bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) From patchwork Sat Oct 28 23:12:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06B2CC001B0 for ; Sat, 28 Oct 2023 23:14:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F24A6B02DF; Sat, 28 Oct 2023 19:13:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 27E896B02E2; Sat, 28 Oct 2023 19:13:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03DE36B02E3; Sat, 28 Oct 2023 19:13:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DE9786B02DF for ; Sat, 28 Oct 2023 19:13:56 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BE6B31605CA for ; Sat, 28 Oct 2023 23:13:56 +0000 (UTC) X-FDA: 81396424872.08.931C92E Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf16.hostedemail.com (Postfix) with ESMTP id 08B43180006 for ; Sat, 28 Oct 2023 23:13:54 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=Obw8PH1F; spf=pass (imf16.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534835; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=7Me4z8gPQOOpyYHwgfTNMOATdjnxynjEEyHm1yT6sQO2gePhZnGFw7tmPjKLsg9C9bWZ64 kLZnAeu9sFv/JlZoLA7yBM7OXrfdbmT3Z5VRPFtcWzDh2uVubCGg9J0mY6QE+VKlSoQicU uxoXWkzkWUjKi5eVnTEA1v/xbU37tDo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534835; a=rsa-sha256; cv=none; b=beXh14ak5g4jOsiGSwy12WZKQusC/EfkqV+jDegVPF3F/9xGxJIEDzG7bmVHnkQhqPQvGF OTqgfH7G15HiaS25UnPL/bGKC6d6Juc0fmYMmZG2BuD2TaQNlvvcQDOgaqXyIfhsZz69RE vjGL0SB8KTARIm570zFZrKWv3d4ZEuo= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=Obw8PH1F; spf=pass (imf16.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1c5cd27b1acso29402315ad.2 for ; Sat, 28 Oct 2023 16:13:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534834; x=1699139634; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=Obw8PH1FPrb745h1whuW05m2BPrRuyEq14zfE9nqvOpcTspXX98QUQBkcF6jn8RiWx URjLzpxOzOrZ8im4ct8csYz0mHFSW6WOBgNkEZnloEyzvKIbnY8rqQqmPvHIA//g1z6p rT8lzZWRi1Rsl3AXGK69IP/B4VnU+IGpw5hT+XhRfQUXnyxqeqvIcomx60AsSbpJXOhD IkND5HCK5Kn5GAFaDYXJvmkYmhsY38dMj1tNTGNnTCMfF9u5NaGzz2Khy7myPTPc98IZ hcomvQKyjhOUKNgfepRxVMQTq4wCKyZ2z54b0gcHY8YYPVlOe8UUK5fruJtB1PBlwkxE TtJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534834; x=1699139634; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=vYizqxbt1n4gUSdseR7Ty5Ghnx1jNAfnyUTT77QEF7Gpu9LK/+GIeHfGxCApXg6+l6 iXuUoXavGgoRY3TAuCv29/SjM/4PjncxKVoOGOJuBvkrqCPrj/ieYYLU/jY7h01HT9f/ vF6iyAGiCEGcxqlhGpxvtfvSmgGKDl0FgiidvfZK8FOaLhrneBhVaaqMJjr5M76Mtgx9 +ThoEGlmOnTH1aEdvUWx5vY8RbwqcJkAhyq07VFbVpWL2gDgqh8T1ppxt4zloOrePlJN ZBq2fLES7/oIAjKQtdLu5RTbZLrU+/P3rHlotD2pMRGm+7I3Yrbksav8QDxdVKJv8jfb QErg== X-Gm-Message-State: AOJu0Yz27L4KmsbjakJaV0R6OFAgnAKUneMsIqr+nVhi8rzOW78izAsO EcgIcBYuFn8MY8nlZQ+wpdWvsg== X-Google-Smtp-Source: AGHT+IEXxDKHTizsqyBXjr5oqt/mWPSp5qFgTjueov73uPEcJh4MjB79gaQaqVj5Le1ZHH1qO9h3pA== X-Received: by 2002:a17:902:c751:b0:1cc:29ef:df7d with SMTP id q17-20020a170902c75100b001cc29efdf7dmr3951181plq.65.1698534834011; Sat, 28 Oct 2023 16:13:54 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:53 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 09/11] riscv: mm: Make asid_bits a local variable Date: Sat, 28 Oct 2023 16:12:07 -0700 Message-ID: <20231028231339.3116618-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 08B43180006 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 764q87ngc5z15ujbzkjbpb7ooehryqar X-HE-Tag: 1698534834-284883 X-HE-Meta: U2FsdGVkX18dD/Ar6QSFXMUc0vVB8FEG43Ec83Tgum0QyeXlhtKp9uldp9+zqul7j52Ge8nyFvOx4mggt6s8Mxmbwpchen7bA3tHQdTjyI2WsPaqZdzET94awBQXJtDnHfdBsJE39FJEL8z0Ahd0atvWewdj1YWbyG4Hb5Jnb0f3agRChWEbnYGmslwoXKOjlrr28jyZ+L5Q9jlMftijlHBw2RF6WAngEd7sgrm81QVL2c2nuoFN4ogetcX2XHRQFi0r+oncaXOqCetYIv+ib2amvjOcCGh8W3V96pIxDRTkc79nHBe/6WloiSPFic3f0EaDMBN2k7yfXr6EcxfZRy+YAjDRxCQEgM1LtP84wDjpxjOZ0IZFIqyRO95pdoeqIHE+PF8jXpSp41BIAUU42jpBzZrON9fjQ3oDtbDaMlGIBAXFdXcPLEnSrr7yq4B+Tm9W4ryB1vlxbSaW2exSWGH8TfEJfwywMc9AlC5iUUoZSNWd3oNR/6yK43ZXfnthbgjr1JhLzOQXk73zmQNKJCdtjxgYztwV67UU7Kg9vep6S0qjtPh2WZJGPtlXcg3wsvYrlzjlsmcPuNzn5STSkV/fy58QlPHNKO+uVFKHGndp2NF/fKjTAoYC4Vxi91h0O7utVNWUJyota4AEf9M/GwXUpEpUuCY9g5UCwdw3FOZVuMakidKX5D2eln2bVJ7F0sOj51o60RRxdVSgkXl8hy1j8kzKlkjMTcwwaf/uAn+Pyl9tjCKxMuUZvTq10FCBn3OBlbX5uV53m3OET0QxUS8UneCMZLs26CpKgKu4DFRnJ7sgdFqhA0InTVHi8hO4giZDxm8n5Nb3MnxIcik3FKursQlNkpPtumWPKgC3JCaVBuTCt5xpZTygYNlrxEi9tmNMKa8Y6PcxybHsl/yrx+0Q+W/6AaHi8H62IvQCOmLC1qZzCYhxTyTYxbiS94wiEQ2eNLMWDnYGIYFrtos jDa7xiz2 8NgtxHcHuDfLbLy5yGRdlTVPgMdWMstu6sx3CwHcZUEZakN0qf78FQfsgWrjLKystQtlGCs3KXkxn/wrrWX404xDQsQk3NsMevo80K2kIgSX63RYa0brJqlh0qiu3RdGuHIjftGPNMsxjMhA8KNLfjss+QRWt6wOaHTyP9yYJgGQnjwZ8VKAv5eQ0lv0ZpcemLAE9WLz/6LB7G7IOU/Qp+H55i3vjYMpl3mmbwgP6W/sFll2CzIdOY7OGtCzJWqXDzmifoeQRN0WWRyZQ1Ly2gGK8dir/OYyN4LOA9HDUHhwzVoa1X+hEfhicMScEeDMZKCfUG18i81kTe0BiSuuMdX0sag3UCCBg5PveozemHR6Gi+xONV3KFSSlywapQnTR3XUx7veOWVgmV5rP1WCU1IMeForn58F4cCSqEtiGY+XANwRQxjVoRDNsHLAiuhfNF4vE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000037, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This variable is only used inside asids_init(). Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b5170ac1b742..43a8bc2d5af4 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); -static unsigned long asid_bits; static unsigned long num_asids; static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; /* Figure-out number of ASID bits in HW */ old = csr_read(CSR_SATP); From patchwork Sat Oct 28 23:12:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70C41C00A5A for ; Sat, 28 Oct 2023 23:14:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B7F536B02E2; Sat, 28 Oct 2023 19:13:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B07FC6B02E3; Sat, 28 Oct 2023 19:13:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 930C26B02E5; Sat, 28 Oct 2023 19:13:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7B4276B02E2 for ; Sat, 28 Oct 2023 19:13:58 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 57A0B1A0463 for ; Sat, 28 Oct 2023 23:13:58 +0000 (UTC) X-FDA: 81396424956.25.1CE7CE2 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) by imf15.hostedemail.com (Postfix) with ESMTP id 8566BA0004 for ; Sat, 28 Oct 2023 23:13:56 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=W8PfBv7m; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf15.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.161.43 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534836; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=fE3CAdf9uNR+8NXkOESdg+YzzJbVsHppNjOV2VSgArtgGdW5tji+NWGwB4gV80FptU/rey iFFMm/BXLfdosOnkPQ0VyC//3ByZhYsxZwvf/QPzwC4rpNcHeHR8uDoNzkTrvpk6FQiRYg adQeddKb4eEWi4NN0AoLQpARlADEQ6I= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=W8PfBv7m; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf15.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.161.43 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534836; a=rsa-sha256; cv=none; b=Bu2MAMBGYLhIgI6iW+WeiXyWDZKeoJUUj5gpTjbQ+2RwTqnK7tg+fziQD1Z1s05Adc/ZlA rkB74HGYn2fvMs/2687tcTN3pH44j+lqMAWhP5HdNUnbDwrAWWRRF8Mxy1vvt/mqLh1lJZ MAFyaD3T82PWg9uuvVUKvjpv/RTSSzs= Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-581d4f9a2c5so2072595eaf.0 for ; Sat, 28 Oct 2023 16:13:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534835; x=1699139635; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=W8PfBv7m3Pyj6UHezWakpWQckBv3Yy/2TvsgAt/rCurKF1uchRqsa6rxb0Nw7qCp2k /DwdC62VVnBEft7X+T6f1Fm3vCd4vSiMVZZj0MXYRSWbA/ETXht/WJI0reREqnS+mOut ridwv2k4UJ99RmbyeimquwXznuFZZu35VPSPQ/2BrWDSg4MX5kPLPt20xaaF1I8QM0WM toHWuQlCyP1G7kkVhj5N4lUPXN80DzyfQqvl1BoDbqmWSPL0/uXpWhjlDehEWxA1/1Ct 3AALTw7MfDIGrxfZIt1NLLrTyXqid4CNrJVkR6o9D7lHmntYlX/OTKiuCwAruCla6Rsw AO3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534835; x=1699139635; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=G9xjXIhgrjCCohtHr1dHQsMpQ4X9qmPVLmJmEQhMRFNf2XXMY/3sfCiqHt6tB7Ozb6 fDgKF8QG2Kv/oRA8r3o9IgIFK+cunjQ/JPm2/DYfBFSiVRVaS9qj/SlRNRFJyUKDty+q UacpqZWEZx1Oj/ecuGeyHJN2MPmHEIqCgszFhsBgfQyhZp4ANOq+K5UblS+UR0dahuSz oFmJoN3wARM1St8cPDZo4gl6hdI1MewYPG3/Xu079fOhfm+a+JplU+BJ3qnFptrU7HZA K6+zlxbtn+BbpAB7DADBDjUlYx/3RR6o2trsIHzyp+3DlMjKa8XuXVuzRZSp8LkIYwC/ hU8w== X-Gm-Message-State: AOJu0YzGTgkQZcwb5oHvX1K+qmLbzhmzkFsbHi+YOdpLGIlLays+claP qJgjRaimBrQKyQTOoMlE3F9lq/yuEqIcqFfWJn0= X-Google-Smtp-Source: AGHT+IEpjr2TCXm5am55hbUBmsnjcw94CHvXcnlavbn7wyDrZYS6wclRzO1RWsfOFMjAji60OjxQDg== X-Received: by 2002:a05:6358:5e08:b0:168:d6cd:7b2e with SMTP id q8-20020a0563585e0800b00168d6cd7b2emr6151237rwn.29.1698534835540; Sat, 28 Oct 2023 16:13:55 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:54 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 10/11] riscv: mm: Preserve global TLB entries when switching contexts Date: Sat, 28 Oct 2023 16:12:08 -0700 Message-ID: <20231028231339.3116618-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8566BA0004 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 3cbxa5q1akq7sibst41fdypsh7e5qoss X-HE-Tag: 1698534836-275808 X-HE-Meta: U2FsdGVkX1+tXPfsmJaiP+co1GAILUqQiwgRkVAQq/ICz9ClTvGnNVSLkltNezhj7J7+yaileav7q3ZbwRftC+2CF/lQNqY/bcHcEYEPhrPxLnpBbajcsKOhdCenolPyVkDlaDQpFTNEIv7P68gV08n29P+SUR4aPpeyfsRUulYAdW7El3sLLrbFVxTSd+KGuT9D4W1IXjO6mUFfwWTsYeY8MnzUyvUk+medEwy95X5xqAm37KH1HgxXqQfzKb6Pfy77nEWpHCLKr8ro0ymuGKXWeh4MjUuyZb/kyEz8KUB4wttZgyji7q0fxRHXgQkXJJs0WvgYEXaOAYJ1S+hsZGBBK+klzYZSONbc/9wDAEAqzB7uaWpqIGQWsYDK3CgvKiZ0GpfLJJ/mKyRqSjaB50J7m3COJ67A2KZT4kaksqQEJxACoBR7l0JrudsnNZhgcwXaMre51DU9IBsOvrxwgPau0iVAHeJRssy/msVpdcwoQG8kLiUlP9VnNm32baBJdOhubimD+Tfmh4uV85dM+Va4HE8vKjiEhGrbK3FcEQFuEUfBv2ZioznbR1dYX4K5WNKwnM0WD7CMoPW7fwIleuF+8sLKZc+snEOr+qi0bzY+34QsHppG5SkYHKQC/YXdrl4g8PecVDS8fw+oNqW3dFlpWRAYRIeokHelecNJ5BTyECI06XwqOB+r0qCJTLIwlWS+r+QhjT9B6yO/ccEllG3x9/YDmHXKCpSgEBnwZCsrGOL5iPKB4C0hv2YTCk+wAiph0sFL1Y0N2qLnFSGLg8CtSqPKkDSk/T6uv/tuiGITk5iooIef7nuJpcfUfc52zLsxChtCxyDc3ifmpwfgzV5a8+/QNgc++ih5u5JjRghXya8mvGRFrL/gwxPnrBp/bmVHM1MoGn1PGVouChVYGlILlvul6Ded5enwLGxXemLZ0PM0B2BO1YxvqGYIi56f4R6KyvDF4sX6a4TB0lW cKDguPzB 7wMfS/IUuAoRGsO6moHfXqaAT8ZdvWQofeqNXa24WvLzUnNEkUFl7wKMesaRfxbtO3EgvFRW18IFy+OF1XUAvAex8wjCn3GGFOejgzmwcnv1bZJxCj/K2wigjzbWGKA2XXewwcEtaIe//UNCSqiLwUh4tSbawX21ukJ3Rc0dOOCXCp7sYFEYJQfogtt10WMnk/lusUeRjRH2EzTQ7fA2LqnDiLokT7WmF8b3xsVVzyNounvMnBXWVYxHTG39dWv+uqGcNPB8RbuJKdDKMNkM72r/3CB4kViroPqZTWubmhJIYP0/9wnTIguKYSE5G5+Eaey0cBjaK1w3sz655jjLrqKjVcmcEnO2akNLcNd4oPSRvRo3rRloj0kVHslPaK2Pjz/CJ8jnVmt00KSk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.229746, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=x0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43a8bc2d5af4..3ca9b653df7d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } static inline void set_mm(struct mm_struct *prev, From patchwork Sat Oct 28 23:12:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCC5CC4332F for ; Sat, 28 Oct 2023 23:14:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22D7E6B02E5; Sat, 28 Oct 2023 19:14:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BD076B02E7; Sat, 28 Oct 2023 19:14:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 030986B02EA; Sat, 28 Oct 2023 19:13:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DB2B26B02E5 for ; Sat, 28 Oct 2023 19:13:59 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id ADD2B1CA9DC for ; Sat, 28 Oct 2023 23:13:59 +0000 (UTC) X-FDA: 81396424998.18.2BA44E1 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf14.hostedemail.com (Postfix) with ESMTP id D9D44100002 for ; Sat, 28 Oct 2023 23:13:57 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=a6ypL30g; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf14.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698534838; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RYq2JBnvfDkZMtFq+gFMcak9lyRAM53Oqx2LdQZI3vI=; b=lqzwg6PQwxsVUrbmuuncIc4elU9zFrOIRVYmRlwncCkXJQbQB/YhygeRjjTxLSQ7FfNSPU BujZdw9UJyZPtWI3swL5YwwnjDf6NKGj3r0uuR+Dgdt0ITqhQCpSw1hTiHaauBcZcf4f6u CbIL4xUfEm6EcjVTel8UFBatS2/ZYvA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=a6ypL30g; dmarc=pass (policy=reject) header.from=sifive.com; spf=pass (imf14.hostedemail.com: domain of samuel.holland@sifive.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=samuel.holland@sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698534838; a=rsa-sha256; cv=none; b=BBmfqUrMQSOoOSMgfASaUBMwMRVYH1XzHDBo/Ftg7ALr3HA4sByeixsBE0vqPlLkdoqb7e aMqedgoaknbKFK6G8+G4jQV/tulsYdMsiAQjEKPcFa0QzRrC7d1mJZ/ZJO3ZGdsUc1rXzM Rt+N0D0ixlCz0g1v5zdn3JnDBS+r6l8= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1cc1ee2d8dfso17082315ad.3 for ; Sat, 28 Oct 2023 16:13:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534837; x=1699139637; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RYq2JBnvfDkZMtFq+gFMcak9lyRAM53Oqx2LdQZI3vI=; b=a6ypL30g4ioKgIUD3LoVMXoxgEa8xdSVLZcfPNyz3LsF+6Wvo2b3AtN9rR9oXcVrZ0 VvN0Iy0IVnjEqJA7CFc/JUTX0VdS16ICwVDbmbOG4m8AgXUs07zp/HROBKmPs1pepbWB 3QKxoCuSK6SsobAQDWoSjvHZu7D6CiNFf6x0R6dZIBHeVFOwHxNE4BlyI5q1jbOWLbN2 reJUXy0GJm/yd+S2m49RUgpuCbmR6knW1l8Zj/uStfuqbubFZkROlCO53tak5l3E5SwL GJ5RcaDaSbRHMVrkn+2PuzqzmI0pXQQ6sjpKvX1vqQ/amewnRayQAlr2xzVmWds+oYAx rHzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534837; x=1699139637; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RYq2JBnvfDkZMtFq+gFMcak9lyRAM53Oqx2LdQZI3vI=; b=NKfa3hKHEbY603/7dt75j6SiSDFt2WZxziSam5QZAiuYypO/ISZN1grPnUmmPqArfX /2kdztX2LmI05ZA8GWG2/FiStfLsWaF+Drqdlr1wvweW4YhqpjBZ8JNUeiLrKdJ6PKF8 t7nYiaJ8cbCFCOTRg2Kb+Ccd1O06lljQIsEmBmELIrlCyzng9SrKaL/pFsXYIerpXSrO D20czza/AWLRfkqp7UcbikAzLinSKZ2BvQ6kLWBkIP4nHAhfAyrVLo75Zx9fnu8dmFb6 ywvgngkwQZRkZs60pWe9YOqJ5xJuYWfyVTYby1sGpi+Wfp9qdppjvo1DRmawlCrtkGDx kOHg== X-Gm-Message-State: AOJu0YwdMnynrgJtO7YjWjr77pWXwRvXaFIb9d31kxDZpIUrKOHyEsC6 WDVBTJKOauffp98pzTKEEmKEaQ== X-Google-Smtp-Source: AGHT+IHR5gJkl+2hmALdgeC6Mspb2OIhkdCFo9rAzDT7R89li8y2sxi8zbkov4fwyI0EL6o8dg+K/w== X-Received: by 2002:a17:902:f542:b0:1c5:a7b7:291c with SMTP id h2-20020a170902f54200b001c5a7b7291cmr8203243plf.12.1698534836728; Sat, 28 Oct 2023 16:13:56 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:56 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 11/11] riscv: mm: Always use ASID to flush MM contexts Date: Sat, 28 Oct 2023 16:12:09 -0700 Message-ID: <20231028231339.3116618-12-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D9D44100002 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: x19zccd3iafk4krjar9rgfc9qaoyi4wu X-HE-Tag: 1698534837-606712 X-HE-Meta: U2FsdGVkX18szvFvj1OSAuLE570FNOsDueyq1NWtjF2vOCBhvYURzpi72c1RsE/eEInzmhpU9SAFZZMs6yUlnZN/h3+XMDb/DlWKSqkxPSmo09I4xDj/IGe3I+BnO8sUSrbDUJJTLl6kg9g6E7OseBSS2rIOa4AfVU6N+iHu2v6K1DufJrhlj17m60cVgzHt2lw5RfJ6KNE9QK9lWXxpVcn6N5+y5Nj+nTc8PmMtKhzGDbU4KmKl7Mxtzq3SI1xKQ8CrkHmWz2DDoGZz5+Hzy6BBz31/9SiNsUFssNkzrFcB+e9Seuv7OI5SasGYbxeYQcuiU3c/bfO+ppMTTHPkUD7qcnAAPK1E44K9jbfBnIR9QYlF/w4QJ6zr94vuxu4N4BHnx8UOStSlxX9184/WbYVwmRXfqQ6WI7lPBMWj4SnvfZzuk37UBx5ZEWpy/ub1Db7LCJwWKj4wYPSDknP0V6YkHH7aBR8yJn3FSeyNQh1+mIIePq4K2dKNlOYhSN2USB/ijBZwWoWV0xzjST6RkEBrJclOMYBhExL0WLXk4jk89L4eyCvm8q/DertYApg4d112HGnqJT1VGLxhYWmul+risFxft9mLmu5JkNC04Zq36tCg2p0PfJKDtVyw1TwQugGdGnyDcA0O51CHrbP1twvuZabihpcekW26NFLWLvAu7B2iDoOZH5ZxGe8csGLAwnxLWcF4W6juUPnG9WWVK70ylgS86e4Ju4feltytfZ0OGmj62WjxhvWYXoOvaC7Al9cFG4GOVO4W2o6/4yX+OUvbciMUbjikJp/FPZoKi4NmTmH3nfZvoXtk43f8RLbvtxwSP9wvB/jJdGqjGqljTH0iUlAmFTEYrm8hGbquUTUzpvqAvQR+2QWeKngFEt1rpf5WzN3gzOvsb930OSdhlgjY5Fa3YS8mN9ELXNHyHLs2arFFSVzegImeezpy/EIZHxpltnoMcNEZjJdLOJB hwjfNmQ5 x8qXB5JSAuAjTptOfU2cLwXR604gSatn0EZH51Qw1rs5Nd/jY/78xiVQgeh4eTDIU0Vp0W5jsZoHXz5NUcE8OIoYPT1NuP5C9Rdhdyapzkf+zbTS/ulgRdAGx/49iLN/cru72sowX1O+etIUM2caPjtSCvoZKxCv3gp9beBR4e3ntnTb7VOCYl9e1zk5/WTGhCSzLu11RcKkn9OvzyauoAPppbVUwIp7YykX8bt+tWmGrgIsWc3a6TKwiq/F09XG5Tc3E5+6j7X75dD4ubh/twAw42/3GRGdFMkPaya8eXTetoq8PU+WU3LO0bzuvqkmk7XLcRLETOKdx7/Fu/NZXqsG/R0CHTH9ddX0cUi2lndobytBfHB0zJAxFnBSftLl0uG4kdRQnsXLoDiGIP57qR71K+NuX+2lmpTgBNPXJAt+3j/X1ycF0/JgvEQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000022, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always most efficient to use the single-ASID code path. Signed-off-by: Samuel Holland --- Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/include/asm/mmu_context.h | 2 -- arch/riscv/mm/context.c | 3 +-- arch/riscv/mm/tlbflush.c | 5 ++--- 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h index 7030837adc1a..b0659413a080 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -33,8 +33,6 @@ static inline int init_new_context(struct task_struct *tsk, return 0; } -DECLARE_STATIC_KEY_FALSE(use_asid_allocator); - #include #endif /* _ASM_RISCV_MMU_CONTEXT_H */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3ca9b653df7d..20057085ab8a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -18,8 +18,7 @@ #ifdef CONFIG_MMU -DEFINE_STATIC_KEY_FALSE(use_asid_allocator); - +static DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long num_asids; static atomic_long_t current_version; diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 1cfac683bda4..9d06a3e9d330 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -90,8 +90,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, /* check if the tlbflush needs to be sent to other CPUs */ broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) - asid = cntx2asid(atomic_long_read(&mm->context.id)); + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; broadcast = true; @@ -122,7 +121,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, { unsigned long asid = FLUSH_TLB_NO_ASID; - if (mm && static_branch_unlikely(&use_asid_allocator)) + if (mm) asid = cntx2asid(atomic_long_read(&mm->context.id)); local_flush_tlb_range_asid(start, size, stride, asid);