From patchwork Sat Mar 9 14:52:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gary Guo X-Patchwork-Id: 10845953 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 110674B65 for ; Sat, 9 Mar 2019 14:52:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E92C32BBEB for ; Sat, 9 Mar 2019 14:52:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DD6D92BC2B; Sat, 9 Mar 2019 14:52:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAD_ENC_HEADER,BAYES_00, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,MISSING_HEADERS,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 37F352BC0E for ; Sat, 9 Mar 2019 14:52:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Date:Subject:From:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=uoYLO/ZOO91r02c2zNMfb7tQyCFpXSREDDLmxAJ+R40=; b=RlZxAlm+dkCEI8 rk3WfVcv+nhO7V934bRaEWQssWuo1AX6uAo+Pk4q8nvdMKkL3Y2zXFv4EdEQLChRyvLZgzba8Bybw rD0MbG12NueeCtRLvIHFmZC7TBKid4FxH9+NRXuG6MqPJi8vGl9yRHfpHqSjSvbRbLg5wLYtnzamu SyoXK9cc6fP6+im7Bzlm4e8MKjV3liNtJh2a+gNRgy5y92RrOO1RmAJzYxB8hZA7FUxpU9QcMgR3y qfnc/BQKd7D8EcHADub8eSRXCbn+zwLMfpjiJaS5BH4GijulqOfgktvMiSDlrC5Cid/dcQpiuLOuA Ws6W490q07H9MDqZBkpw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2dKe-0006A4-Pk; Sat, 09 Mar 2019 14:52:12 +0000 Received: from mail-eopbgr110109.outbound.protection.outlook.com ([40.107.11.109] helo=GBR01-CWL-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2dKc-00068V-CA for linux-riscv@lists.infradead.org; Sat, 09 Mar 2019 14:52:11 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=garyguo.net; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=73GXrscOF7oDNb7BvZksg+VnZfN+o+ZWacdQz5M6dk8=; b=LKzhnYajUUNSF8b07//yCxNFgdix2B3/NbMQOjKbTfRePFqTT5+4NlNMiFZ7fDbjDVHbLn1kvWOVwEN72YuegRKYewckZltDpEzTJ2ypxRjylxZNEgBnZrIDSuKV+UDNUNmMdAp3tNIJSAOiDktnGnMLjjp0nYd5nUYf/9GAlB8= Received: from LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM (20.176.139.20) by LO2P265MB0158.GBRP265.PROD.OUTLOOK.COM (10.166.98.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1686.18; Sat, 9 Mar 2019 14:52:04 +0000 Received: from LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM ([fe80::ed34:1290:4306:3157]) by LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM ([fe80::ed34:1290:4306:3157%3]) with mapi id 15.20.1686.021; Sat, 9 Mar 2019 14:52:04 +0000 From: Gary Guo Subject: [PATCH v3 1/4] riscv: move flush_icache_{all,mm} to cacheflush.c Thread-Topic: [PATCH v3 1/4] riscv: move flush_icache_{all, mm} to cacheflush.c Thread-Index: AQHU1oeoVwi1k3J/IEOIYrEWiVDqZw== Date: Sat, 9 Mar 2019 14:52:04 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: LNXP123CA0015.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:d2::27) To LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8c::20) x-mailer: git-send-email 2.17.1 authentication-results: spf=none (sender IP is ) smtp.mailfrom=gary@garyguo.net; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [2001:630:212:238:3697:f6ff:fe55:55b1] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 79a80851-4df4-4f34-b34b-08d6a49ecb19 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(7021145)(8989299)(4534185)(7022145)(4603075)(4627221)(201702281549075)(8990200)(7048125)(7024125)(7027125)(7023125)(5600127)(711020)(4605104)(2017052603328)(7153060)(7193020); SRVR:LO2P265MB0158; x-ms-traffictypediagnostic: LO2P265MB0158: x-microsoft-exchange-diagnostics: =?iso-8859-1?Q?1; LO2P265MB0158; 23:UTRbM9qcNUMGuKv3aSe5FKmnvFw0EAAmwaNbNz0?= =?iso-8859-1?q?8XuQGomPAM4MuVQ?= =?iso-8859-1?q?t60xV8T5B65k7QuM7w+ILBhHHo1Q0XdWqiIRtm/Yl767g4pmHNxBt2Ve8mmN?= =?iso-8859-1?q?lJ6FAIhhl4tfTnEEqIK9wq1sjeYbvADh939XNEgMQ1+AfhT7d2YeXHlcQqfZ?= =?iso-8859-1?q?LlyyRkEmM5Rvb+sMEkVEJMbFWA+nnUubqtDxb8b83E4WxCMa767bFOelf2xd?= =?iso-8859-1?q?NMxpa5NlqzUCOHn1QK2y3tSF+v+ayaMPi/TVS2PZ86uO9DqBNhzQCEPOjOfa?= =?iso-8859-1?q?o2tKgBbQroeh2vYBcEduGqAOCyHVlXNqtmiIZxqS39k4gerBZTOxz5I6SeRa?= =?iso-8859-1?q?f/hPfXek/dtuu4OLTBDgCqhspn9iN2R1SSZ+feujbFWDmF+5DLMi+IheS4pf?= =?iso-8859-1?q?VgLA5SA+0FRgDX8MDeaLqeEI5pF6FOPHEwLCW54ypnavghby52Y5Zk4PJp6B?= =?iso-8859-1?q?6B8/BfJ5hNpFDnCxVK32kt39oksWMhW02anhy1E8CMkbjZU0z6eC1K7fLkRm?= =?iso-8859-1?q?Gr/GzqNEZS7ybI1ochk3vZWTjQln21NMQA1KS52NSkr+XRO+5xUsPqV1aj8x?= =?iso-8859-1?q?JeZYViEhZ+mPKwPZvkYRr0VCqDndcHrW4EiXwWxdPMTCt4vPkciNRC7b/V08?= =?iso-8859-1?q?0F0V6cLilISUAnA6fqGHeMcJtw/kv1XpP91tz5g776wy/j/jbd3H4JD8VtAN?= =?iso-8859-1?q?Ax3dpKW+/6ZODCEpk2NBUZyVOVZ4LjKWXqAjzSlV2Vo8sKhqTGzQ+8bNRPAA?= =?iso-8859-1?q?ASm7PKarsgchdy3Hzu+HKU3DePYgt2Wm0IPKz6IQc/wN1f0ug/rf/UpGoEua?= =?iso-8859-1?q?6LGRBs6UdNp55nRHDNRWAMkwJ9+rD8ZmWZsDzPQg/tgLMTiMIyTTsCGvWqfL?= =?iso-8859-1?q?H9ADaZ/xhAwuYqGtDcaSbZlUbQZwGOWtTMot8aADRYg4g5BY5ghRx16pRVOe?= =?iso-8859-1?q?oSoCIZgSSv6qpGGqs/0gcd4XBs09kYtkZh0a3sWBLrJ+DUppDd1RPlct2iAp?= =?iso-8859-1?q?SUUwxrwf3RH9643U52s30XY4nLFw5uyGuiOvvzCs56Xz00Pffeyr4b7B1C1h?= =?iso-8859-1?q?ppyIHdKM6bZ2eITqDjG9ivkNsz04p/kI110UgKLEERgFMsGDKjXF8ViQGJeu?= =?iso-8859-1?q?aQ8loTEGJbszfvVN/7m0GwM7AsMb7SQFK2JBucwAVldEs/RDkoNKc2/ICDUO?= =?iso-8859-1?q?cS4Pl1OE4TKwAOhdAZB7Rh1AewfXREBN/g6fDVidUnrsRM3IttJqrGFxut3q?= =?iso-8859-1?q?b4lALYXt+GeqrtMOwmys64wliO?= x-microsoft-antispam-prvs: x-forefront-prvs: 0971922F40 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(376002)(39830400003)(346002)(366004)(136003)(396003)(199004)(189003)(305945005)(53936002)(6116002)(8676002)(1671002)(102836004)(86362001)(508600001)(97736004)(52116002)(36756003)(14454004)(76176011)(2906002)(99286004)(386003)(6506007)(7736002)(25786009)(81156014)(81166006)(4326008)(6436002)(6486002)(6512007)(446003)(11346002)(118296001)(2616005)(476003)(105586002)(68736007)(106356001)(71200400001)(71190400001)(186003)(486006)(8936002)(50226002)(5660300002)(14444005)(256004)(109986005)(107886003)(54906003)(59246006)(316002)(46003)(142933001); DIR:OUT; SFP:1102; SCL:1; SRVR:LO2P265MB0158; H:LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: garyguo.net does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: nl9Wt2rwE/P9aw2JSNxuU0qt1MqguIvDkqBLXtto8YebR+TP5zfPe+zm4xsMu+5mlfkILGDv+ur9+rRugDU4ZF1wof2lwJb2AARlmQamZks6Lte5IyVEbu8CHjii2SShYEptm0vTFKyVkfoTk4hR+RSLHWPhptiHJhMZg+RiGkOOqCTEkhFAzVO1ZgVUqfBqhcfZEyG/9fLXiNr4n265FVvDUlv99AbSqeB9wSLDKZfDrHjY9qk2VWlFdsVZcDtgCABMa2RL53jn9syHisqqmqzZhlBi/4ntLEfFxIr9ODzylKLo6GtOOdoIjePAsaVMIZ8Ha0wj4UeGOC4qV81OF4tvj2rC8qjT1JN+IVIut5e4ev1bYmcWw0P0ytZIGiQCjbWOHhbjjuJmT7BpmJvwrRPHrmE6X63Itck1Jmocg6M= MIME-Version: 1.0 X-OriginatorOrg: garyguo.net X-MS-Exchange-CrossTenant-Network-Message-Id: 79a80851-4df4-4f34-b34b-08d6a49ecb19 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Mar 2019 14:52:04.0922 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bbc898ad-b10f-4e10-8552-d9377b823d45 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: LO2P265MB0158 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190309_065210_412161_9E3D26A8 X-CRM114-Status: GOOD ( 15.06 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Palmer Dabbelt , Anup Patel , Christoph Hellwig , Atish Patra , Albert Ou , Gary Guo , "linux-riscv@lists.infradead.org" Sender: "linux-riscv" Errors-To: linux-riscv-bounces+patchwork-linux-riscv=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Gary Guo Currently, flush_icache_all is macro-expanded into a SBI call, yet no asm/sbi.h is included in asm/cacheflush.h. This could be moved to mm/cacheflush.c instead (SBI call will dominate performance-wise and there is no worry to not have it inlined. Currently, flush_icache_mm stays in kernel/smp.c, which looks like a hack to prevent it from being compiled when CONFIG_SMP=n. It should also be in mm/cacheflush.c. Signed-off-by: Gary Guo --- arch/riscv/include/asm/cacheflush.h | 2 +- arch/riscv/kernel/smp.c | 49 ----------------------- arch/riscv/mm/cacheflush.c | 61 +++++++++++++++++++++++++++++ 3 files changed, 62 insertions(+), 50 deletions(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index 8f13074413a7..1f4ba68ab9aa 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -47,7 +47,7 @@ static inline void flush_dcache_page(struct page *page) #else /* CONFIG_SMP */ -#define flush_icache_all() sbi_remote_fence_i(NULL) +void flush_icache_all(void); void flush_icache_mm(struct mm_struct *mm, bool local); #endif /* CONFIG_SMP */ diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 0c41d07ec281..17f491e8ed0a 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -199,52 +199,3 @@ void smp_send_reschedule(int cpu) send_ipi_message(cpumask_of(cpu), IPI_RESCHEDULE); } -/* - * Performs an icache flush for the given MM context. RISC-V has no direct - * mechanism for instruction cache shoot downs, so instead we send an IPI that - * informs the remote harts they need to flush their local instruction caches. - * To avoid pathologically slow behavior in a common case (a bunch of - * single-hart processes on a many-hart machine, ie 'make -j') we avoid the - * IPIs for harts that are not currently executing a MM context and instead - * schedule a deferred local instruction cache flush to be performed before - * execution resumes on each hart. - */ -void flush_icache_mm(struct mm_struct *mm, bool local) -{ - unsigned int cpu; - cpumask_t others, hmask, *mask; - - preempt_disable(); - - /* Mark every hart's icache as needing a flush for this MM. */ - mask = &mm->context.icache_stale_mask; - cpumask_setall(mask); - /* Flush this hart's I$ now, and mark it as flushed. */ - cpu = smp_processor_id(); - cpumask_clear_cpu(cpu, mask); - local_flush_icache_all(); - - /* - * Flush the I$ of other harts concurrently executing, and mark them as - * flushed. - */ - cpumask_andnot(&others, mm_cpumask(mm), cpumask_of(cpu)); - local |= cpumask_empty(&others); - if (mm != current->active_mm || !local) { - cpumask_clear(&hmask); - riscv_cpuid_to_hartid_mask(&others, &hmask); - sbi_remote_fence_i(hmask.bits); - } else { - /* - * It's assumed that at least one strongly ordered operation is - * performed on this hart between setting a hart's cpumask bit - * and scheduling this MM context on that hart. Sending an SBI - * remote message will do this, but in the case where no - * messages are sent we still need to order this hart's writes - * with flush_icache_deferred(). - */ - smp_mb(); - } - - preempt_enable(); -} diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 498c0a0814fe..497b7d07af0c 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -14,6 +14,67 @@ #include #include +#ifdef CONFIG_SMP + +#include + +void flush_icache_all(void) +{ + sbi_remote_fence_i(NULL); +} + +/* + * Performs an icache flush for the given MM context. RISC-V has no direct + * mechanism for instruction cache shoot downs, so instead we send an IPI that + * informs the remote harts they need to flush their local instruction caches. + * To avoid pathologically slow behavior in a common case (a bunch of + * single-hart processes on a many-hart machine, ie 'make -j') we avoid the + * IPIs for harts that are not currently executing a MM context and instead + * schedule a deferred local instruction cache flush to be performed before + * execution resumes on each hart. + */ +void flush_icache_mm(struct mm_struct *mm, bool local) +{ + unsigned int cpu; + cpumask_t others, hmask, *mask; + + preempt_disable(); + + /* Mark every hart's icache as needing a flush for this MM. */ + mask = &mm->context.icache_stale_mask; + cpumask_setall(mask); + /* Flush this hart's I$ now, and mark it as flushed. */ + cpu = smp_processor_id(); + cpumask_clear_cpu(cpu, mask); + local_flush_icache_all(); + + /* + * Flush the I$ of other harts concurrently executing, and mark them as + * flushed. + */ + cpumask_andnot(&others, mm_cpumask(mm), cpumask_of(cpu)); + local |= cpumask_empty(&others); + if (mm != current->active_mm || !local) { + cpumask_clear(&hmask); + riscv_cpuid_to_hartid_mask(&others, &hmask); + sbi_remote_fence_i(hmask.bits); + } else { + /* + * It's assumed that at least one strongly ordered operation is + * performed on this hart between setting a hart's cpumask bit + * and scheduling this MM context on that hart. Sending an SBI + * remote message will do this, but in the case where no + * messages are sent we still need to order this hart's writes + * with flush_icache_deferred(). + */ + smp_mb(); + } + + preempt_enable(); +} + +#endif /* CONFIG_SMP */ + void flush_icache_pte(pte_t pte) { struct page *page = pte_page(pte); From patchwork Sat Mar 9 14:52:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gary Guo X-Patchwork-Id: 10845955 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5D0A139A for ; Sat, 9 Mar 2019 14:52:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BAF802BBEB for ; Sat, 9 Mar 2019 14:52:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ACB642BC2B; Sat, 9 Mar 2019 14:52:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAD_ENC_HEADER,BAYES_00, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,MISSING_HEADERS,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1F5112BBEB for ; Sat, 9 Mar 2019 14:52:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Date:Subject:From:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SNMGKk15nHUCs0lpfUSW8nxYSU33+1csVezXgv/cAKc=; b=XlXNuoFVOd/4n/ UQabY6Nhss5HVqYBc1Ijc5qHGiCJc4zHOknwH78nI5wTWLlBDHD/R2E8/nbSuOWoX1jTjW2z8wS+v ITLs4ad7uBF06sV3PUx+loEngweJMobktiEP/9FJxiMaHkxoQKkwQJIRhLgRFvqzUZpe7bZ4NfWS2 uJuXczoIZEwd5FqyNfKB+GxvZKbW5DkUQ8iwDANAcoz9+wLjuym5+2Nq3r7tNun9yyw15BV8CifYF szuOmz7H9Bj8exCL3NB9x5oeCyAqak4XUeZlsqztPbb5givf8pq/+TJB5F+ZZpF647JpEXJG1XNDF kp4XBlUtt0I2Zvg2Kt9g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2dKh-0006D4-Fg; Sat, 09 Mar 2019 14:52:15 +0000 Received: from mail-eopbgr110109.outbound.protection.outlook.com ([40.107.11.109] helo=GBR01-CWL-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2dKe-00068V-7E for linux-riscv@lists.infradead.org; Sat, 09 Mar 2019 14:52:14 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=garyguo.net; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=U12VmDqp5PDY/bRQP1bMyCfddobhmO3WImVlkJZ+iZI=; b=zOUOj07Sbk1y9CN/QKNsVTz/s27bVI9OqoEBKQvneD3BAmFQhpe+SdTaQOLWZlXfW+lnaeP4YXvl1r2SuRMm/T91ueHyS0UsEvkmFZQ/6lVbtvhexWl+gq4uNGd5dzTHRigGzc9sB/2dOnqRVcz9RvjC57jsG4UyvJjngI7gL+8= Received: from LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM (20.176.139.20) by LO2P265MB0158.GBRP265.PROD.OUTLOOK.COM (10.166.98.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1686.18; Sat, 9 Mar 2019 14:52:04 +0000 Received: from LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM ([fe80::ed34:1290:4306:3157]) by LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM ([fe80::ed34:1290:4306:3157%3]) with mapi id 15.20.1686.021; Sat, 9 Mar 2019 14:52:04 +0000 From: Gary Guo Subject: [PATCH v3 2/4] riscv: move switch_mm to its own file Thread-Topic: [PATCH v3 2/4] riscv: move switch_mm to its own file Thread-Index: AQHU1oeoCrSBm6dJgk6uLg+buz3wTg== Date: Sat, 9 Mar 2019 14:52:04 +0000 Message-ID: <862300448c34e225f3fbb4da36ccf18d10753b0b.1552142593.git.gary@garyguo.net> References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: LNXP123CA0015.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:d2::27) To LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8c::20) x-mailer: git-send-email 2.17.1 authentication-results: spf=none (sender IP is ) smtp.mailfrom=gary@garyguo.net; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [2001:630:212:238:3697:f6ff:fe55:55b1] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 83cd0413-d915-49b5-c866-08d6a49ecb4b x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(7021145)(8989299)(4534185)(7022145)(4603075)(4627221)(201702281549075)(8990200)(7048125)(7024125)(7027125)(7023125)(5600127)(711020)(4605104)(2017052603328)(7153060)(7193020); SRVR:LO2P265MB0158; x-ms-traffictypediagnostic: LO2P265MB0158: x-microsoft-exchange-diagnostics: =?iso-8859-1?Q?1; LO2P265MB0158; 23:nRZ7QAfmTtRr56ltZmW/lShMTHoMJrEB79WHf1U?= =?iso-8859-1?q?1PQJstIqlgpo8qF?= =?iso-8859-1?q?E16eA0h0qjqzUqbAwzVtbhjVrKr2dJK9Oo1TDrzartgb2TPT4O/P08cNpA2Q?= =?iso-8859-1?q?Zy4yhsgUa7OQQTbYyP8nPuTNnTL8Xj5x4Qou0Vv4JHJJIBZFVWJANWzullAc?= =?iso-8859-1?q?quMu3IPM2zo0Ii0PQyINGycIzl2FadpxtKCsJQkC2LHDsz/z9B+Fe/T8DDHf?= =?iso-8859-1?q?TprYCJJWvcL0+TsM+/B6PRHrwSzFam16LDH1IeIYpRstnJ4FICNnxiQrpUUR?= =?iso-8859-1?q?5mmCLh9rK0eo4mcHOO6tSjDtzRqFUlW0C0zqTCPHuLeLDT2lG9JWyVhyQDT+?= =?iso-8859-1?q?TqhH4l7Z61L0D/h6LT/Myp4AN78YyTxgjH/vnbYK5Bm+y2snDFOUixlBgiyQ?= =?iso-8859-1?q?Kxx+hJiHViCZ7h6ZQBVVY+QsN7WlxWfyKsEHrbfpxJR7OFtmPekzCd/jGe7I?= =?iso-8859-1?q?+nC32OfVTapGnXpC6BSo0K5hsEw1pmwJhwHyPlfv6i/KDPhO2GxUstSkdbM+?= =?iso-8859-1?q?tPzqXQW99FkbhoFYiMOlIg9Kb3I80wuaKsEskruF5D32CKcqZitWYDJ3i37e?= =?iso-8859-1?q?ht3OGV+byyYJPqyp6oTxcDS9VHWSUDdgR/kDhCgFe86O6MRHk8UOlXVp5fEO?= =?iso-8859-1?q?MWOGlSZV2L+imI41jp3owuh+Rm06XeL/EXLeEU/Xce6VhkiOdxZlOp+EwaP+?= =?iso-8859-1?q?6SOk7cfpL7AanE7TLFHShykIC2Djd2qQdnqL0X7QUSDJRaZAuNwv5rgr1R7i?= =?iso-8859-1?q?9O9wStEyGd9ypH6yGT62QujGvv9lUg4MUc0doNsHYU5Unc9vi1l0uZRM8x3W?= =?iso-8859-1?q?1OAOd8qBiDKL06ozgtkEcZIj62EBl10y1XrAVW2PhNqSEjGnKqF6xHk/WK1r?= =?iso-8859-1?q?vN8agrruJIIkN95wCe90LbfIxP1tYRmgP/XWq83CMlZc6FT3ACTVOkeiWQJU?= =?iso-8859-1?q?6eJbV5tgBbNyqYVtva7GZufX0tKdhcu7g9/k5Dd7dd8r+LQVE2nX6//h5UFq?= =?iso-8859-1?q?coms+aDawYgaCdRRu6Lod9Ofprkvj+fWgoPqFQ8YksT4q03bu9SRMpWOGNcA?= =?iso-8859-1?q?ARgUWu3PQsOzWld9DpQ41xqV8ES34uL7DHnKmBTtKBlt9OCSwMinFltgJPNu?= =?iso-8859-1?q?n79j7JQS7a1qhURkf0RloPIiFaE86pgJqSAropBAVLN5M0f9aqT9ck+Tb08m?= =?iso-8859-1?q?zPcE9GKsAGZPM6IRIwGZZnsrtxJDmcJ/eOGHQlMaEP+ro2Z0O53cGjA6WBei?= =?iso-8859-1?q?eT7q4=3D?= x-microsoft-antispam-prvs: x-forefront-prvs: 0971922F40 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(376002)(39830400003)(346002)(366004)(136003)(396003)(199004)(189003)(305945005)(53936002)(6116002)(8676002)(1671002)(102836004)(86362001)(508600001)(97736004)(52116002)(36756003)(14454004)(76176011)(2906002)(99286004)(386003)(6506007)(7736002)(25786009)(81156014)(81166006)(4326008)(6436002)(6486002)(6512007)(446003)(11346002)(118296001)(2616005)(476003)(105586002)(68736007)(106356001)(71200400001)(71190400001)(186003)(486006)(8936002)(50226002)(5660300002)(14444005)(256004)(109986005)(107886003)(54906003)(59246006)(316002)(46003); DIR:OUT; SFP:1102; SCL:1; SRVR:LO2P265MB0158; H:LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: garyguo.net does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: x4We4L4K8HaGxialtwNgebAn1gEfweG/bg+oiHicmmdXIiUIOY92KMH+8QSgCxeE8xgHoGdM5ZT+5G35sf1oMzSeiTtn9efPD6BvZg1BGaR54Ks9A0ZxUWogiIwhQvIdlhmrxqXJ3PyAJ7zTqFHCmdY+G0bVljXUrNurgT7n8CYCYXBhTSa/4uVvARBfpOA8AWhW319H8Ozor80DeHAd0WWL/cSuWQKjea05zrqwCf/KF3adMZGeuFRmqNRc/+mHTSm8r+C9ETZJkcuO70GPCfIQg1Dn1gbBRGlMhzg010kLeIFhW56PZFOoeEEBBTIrU0JcbgrgNRQ9ySTCfymoX+vtFV+IVrS25FS3OXF1OvaDLnXX8FTAGPLGlvIKLbjGOvlnoIRjJz1htHWqzIe9UmX79X17rtqR/um0p6GCR8Q= MIME-Version: 1.0 X-OriginatorOrg: garyguo.net X-MS-Exchange-CrossTenant-Network-Message-Id: 83cd0413-d915-49b5-c866-08d6a49ecb4b X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Mar 2019 14:52:04.4424 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bbc898ad-b10f-4e10-8552-d9377b823d45 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: LO2P265MB0158 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190309_065212_285264_DB989477 X-CRM114-Status: GOOD ( 17.73 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Palmer Dabbelt , Anup Patel , Christoph Hellwig , Atish Patra , Albert Ou , Gary Guo , "linux-riscv@lists.infradead.org" Sender: "linux-riscv" Errors-To: linux-riscv-bounces+patchwork-linux-riscv=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Gary Guo switch_mm is an expensive operations that has two users. flush_icache_deferred is only called within switch_mm and can be moved together. The function is expected to be more complicated when ASID support is added, so clean up eagerly. By moving them to a separate file we also removes some excessive dependency of tlbflush.h and cacheflush.h. Signed-off-by: Gary Guo --- arch/riscv/include/asm/mmu_context.h | 59 +---------------------- arch/riscv/mm/Makefile | 1 + arch/riscv/mm/context.c | 71 ++++++++++++++++++++++++++++ 3 files changed, 74 insertions(+), 57 deletions(-) create mode 100644 arch/riscv/mm/context.c diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h index 336d60ec5698..bf4f097a9051 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -20,8 +20,6 @@ #include #include -#include -#include static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *task) @@ -39,61 +37,8 @@ static inline void destroy_context(struct mm_struct *mm) { } -/* - * When necessary, performs a deferred icache flush for the given MM context, - * on the local CPU. RISC-V has no direct mechanism for instruction cache - * shoot downs, so instead we send an IPI that informs the remote harts they - * need to flush their local instruction caches. To avoid pathologically slow - * behavior in a common case (a bunch of single-hart processes on a many-hart - * machine, ie 'make -j') we avoid the IPIs for harts that are not currently - * executing a MM context and instead schedule a deferred local instruction - * cache flush to be performed before execution resumes on each hart. This - * actually performs that local instruction cache flush, which implicitly only - * refers to the current hart. - */ -static inline void flush_icache_deferred(struct mm_struct *mm) -{ -#ifdef CONFIG_SMP - unsigned int cpu = smp_processor_id(); - cpumask_t *mask = &mm->context.icache_stale_mask; - - if (cpumask_test_cpu(cpu, mask)) { - cpumask_clear_cpu(cpu, mask); - /* - * Ensure the remote hart's writes are visible to this hart. - * This pairs with a barrier in flush_icache_mm. - */ - smp_mb(); - local_flush_icache_all(); - } -#endif -} - -static inline void switch_mm(struct mm_struct *prev, - struct mm_struct *next, struct task_struct *task) -{ - if (likely(prev != next)) { - /* - * Mark the current MM context as inactive, and the next as - * active. This is at least used by the icache flushing - * routines in order to determine who should - */ - unsigned int cpu = smp_processor_id(); - - cpumask_clear_cpu(cpu, mm_cpumask(prev)); - cpumask_set_cpu(cpu, mm_cpumask(next)); - - /* - * Use the old spbtr name instead of using the current satp - * name to support binutils 2.29 which doesn't know about the - * privileged ISA 1.10 yet. - */ - csr_write(sptbr, virt_to_pfn(next->pgd) | SATP_MODE); - local_flush_tlb_all(); - - flush_icache_deferred(next); - } -} +void switch_mm(struct mm_struct *prev, struct mm_struct *next, + struct task_struct *task); static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next) diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index eb22ab49b3e0..d75b035786d6 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -3,3 +3,4 @@ obj-y += fault.o obj-y += extable.o obj-y += ioremap.o obj-y += cacheflush.o +obj-y += context.o diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c new file mode 100644 index 000000000000..fbb1cfe80267 --- /dev/null +++ b/arch/riscv/mm/context.c @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2012 Regents of the University of California + * Copyright (C) 2017 SiFive + */ + +#include + +#include +#include + +/* + * When necessary, performs a deferred icache flush for the given MM context, + * on the local CPU. RISC-V has no direct mechanism for instruction cache + * shoot downs, so instead we send an IPI that informs the remote harts they + * need to flush their local instruction caches. To avoid pathologically slow + * behavior in a common case (a bunch of single-hart processes on a many-hart + * machine, ie 'make -j') we avoid the IPIs for harts that are not currently + * executing a MM context and instead schedule a deferred local instruction + * cache flush to be performed before execution resumes on each hart. This + * actually performs that local instruction cache flush, which implicitly only + * refers to the current hart. + */ +static inline void flush_icache_deferred(struct mm_struct *mm) +{ +#ifdef CONFIG_SMP + unsigned int cpu = smp_processor_id(); + cpumask_t *mask = &mm->context.icache_stale_mask; + + if (cpumask_test_cpu(cpu, mask)) { + cpumask_clear_cpu(cpu, mask); + /* + * Ensure the remote hart's writes are visible to this hart. + * This pairs with a barrier in flush_icache_mm. + */ + smp_mb(); + local_flush_icache_all(); + } + +#endif +} + +void switch_mm(struct mm_struct *prev, struct mm_struct *next, + struct task_struct *task) +{ + unsigned int cpu; + + if (unlikely(prev == next)) + return; + + /* + * Mark the current MM context as inactive, and the next as + * active. This is at least used by the icache flushing + * routines in order to determine who should be flushed. + */ + cpu = smp_processor_id(); + + cpumask_clear_cpu(cpu, mm_cpumask(prev)); + cpumask_set_cpu(cpu, mm_cpumask(next)); + + /* + * Use the old spbtr name instead of using the current satp + * name to support binutils 2.29 which doesn't know about the + * privileged ISA 1.10 yet. + */ + csr_write(sptbr, virt_to_pfn(next->pgd) | SATP_MODE); + local_flush_tlb_all(); + + flush_icache_deferred(next); +} + From patchwork Sat Mar 9 14:52:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gary Guo X-Patchwork-Id: 10845957 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0BF9B139A for ; Sat, 9 Mar 2019 14:52:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E92542BBEB for ; Sat, 9 Mar 2019 14:52:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DD8832BC2B; Sat, 9 Mar 2019 14:52:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAD_ENC_HEADER,BAYES_00, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,MISSING_HEADERS,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7D2A92BBEB for ; Sat, 9 Mar 2019 14:52:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Date:Subject:From:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fdiZekw2Au2SmSabFnh60eocFlGSmhjqygZjmsFMScU=; b=ErD6i0FScGjMKb mX3/paqesVynKkUWukj7abcZYJ2XnAF89OhdX/OBtRBs9XlXP9GT63SP9IJKmwjpaT6EWodChwQA8 Al3Hll0bhW3qU8+53eD2k15GllN3GMIQIrDSEpITQKNfXWOBFuxg40BPngwl5zmVh4GewiWbGPQm4 D0G3/tbaMkeQLV9Nibxkpl5sA3+xUf2dPxe6mPs3ldxJiRkuIdegr3ANzSu2Tvwo4Biju77a6yysr 5LN+qgjB1WKiad/a6tTknChMPgFn2AY9M+eCFZZ/y1f9CCC62dTdv1i71HrKrn4IS6hSMXIuQp5Ni iLujxqN7WxoGcX4sYqbw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2dKk-0006GA-57; Sat, 09 Mar 2019 14:52:18 +0000 Received: from mail-eopbgr110109.outbound.protection.outlook.com ([40.107.11.109] helo=GBR01-CWL-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2dKh-00068V-38 for linux-riscv@lists.infradead.org; Sat, 09 Mar 2019 14:52:16 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=garyguo.net; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zj/uHTSPRMiIS73g46NB5MYWGGCzJXJJvWiKMnQXqts=; b=Sx5DR1z/4Jc2f1sVl5nX9poZDx6FyyAfYeW05XmoNT/0lu0cBaPWVCLz0lqLCiBYTrtiFdaA6ePRDwRCp71qdVY2dmltQ9bD6OACJWTKENf7NIOiS83I2i3pExKMf2IHlMNGsLUKRF6eMyR61pHJJ7mZV9mQv0K4nNQaJm2YXx8= Received: from LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM (20.176.139.20) by LO2P265MB0158.GBRP265.PROD.OUTLOOK.COM (10.166.98.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1686.18; Sat, 9 Mar 2019 14:52:05 +0000 Received: from LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM ([fe80::ed34:1290:4306:3157]) by LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM ([fe80::ed34:1290:4306:3157%3]) with mapi id 15.20.1686.021; Sat, 9 Mar 2019 14:52:05 +0000 From: Gary Guo Subject: [PATCH v3 3/4] riscv: fix sbi_remote_sfence_vma{,_asid}. Thread-Topic: [PATCH v3 3/4] riscv: fix sbi_remote_sfence_vma{,_asid}. Thread-Index: AQHU1oep0uclkkgrWE6ZQOO7bzDQbQ== Date: Sat, 9 Mar 2019 14:52:04 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: LNXP123CA0015.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:d2::27) To LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8c::20) x-mailer: git-send-email 2.17.1 authentication-results: spf=none (sender IP is ) smtp.mailfrom=gary@garyguo.net; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [2001:630:212:238:3697:f6ff:fe55:55b1] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 648a48e1-bb04-4d86-79e0-08d6a49ecb7e x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(7021145)(8989299)(4534185)(7022145)(4603075)(4627221)(201702281549075)(8990200)(7048125)(7024125)(7027125)(7023125)(5600127)(711020)(4605104)(2017052603328)(7153060)(7193020); SRVR:LO2P265MB0158; x-ms-traffictypediagnostic: LO2P265MB0158: x-microsoft-exchange-diagnostics: =?iso-8859-1?Q?1; LO2P265MB0158; 23:YcTOZE1yUY9ZXw48KhMugPQab7mnHmMmrNGPsxv?= =?iso-8859-1?q?L7F7pMclrD23nqL?= =?iso-8859-1?q?aeRyXzrYzh0tjNAENzw8RBF5ImDmnWy3V7na2YmSJepm0LcxHCuo5qvPk/tc?= =?iso-8859-1?q?VACsMXQb5J4pe4v4LpQWj7pmfddf9z6HoaVnpluOZEZx0mT5IZ5BmP4pgOIl?= =?iso-8859-1?q?aQ4sjqYOSMAFxQG1EaSPtP79lDF9hWhInySWj9xI21LKD0HR0c2MoAjAjEbd?= =?iso-8859-1?q?QPWmes5xakYVom+ZYi+Pb3RKDUjSCFF2WoAA8BmZt0A1ftM5IXp2AIXyGkWp?= =?iso-8859-1?q?nK8cEhFns7bs7oKkE2wgbAm6JUMzRfbLqlAIkBZALGU5hgvXESjFXoWTOIUp?= =?iso-8859-1?q?En84800tuspwFyvm8X7jaHPqJ1P8FbrZnjeF5O8lz0AGzVWQeA0rQh9mSFcB?= =?iso-8859-1?q?EZ70PiPROlCLvXxW76UTSuoPEthKZhAmZh4sRaUO5V9ejgVvlIshH023ahW8?= =?iso-8859-1?q?RRjigTVM4nuY+gwfDbWhSuGLK4WlOb0hZ79oQS7SjJoe49D3E4Jr5WIL3r5I?= =?iso-8859-1?q?rAsy2rhR8pyzdxYBHqvUjPsTTak7IdrqbEiw8M3NUU+NvdMe1rHufghNWZpx?= =?iso-8859-1?q?6bxtJdMejc6RiudZolj8+yfsVhc2Nge3+ssXs2xLmyBIJ4Mcv+pWQULwoIZv?= =?iso-8859-1?q?Td5KBqGgkiSzTrJb8LKJSqsDBADMNxWHyzBRDc/kN11P90Nj6d0f27yhVR9P?= =?iso-8859-1?q?YxMx+y8jfvyfFM66OP9gzhF/zmIL5jikfFRsFxmuS1SVL40sRE272So4XAgX?= =?iso-8859-1?q?fRg2ctMgtg+/0eQqKV0bSZeAUtSquQi37JHk6mqFkw8BdITmWZsqedf7iKa9?= =?iso-8859-1?q?MkS5tFmylmOA1KfdhmlSNakMsA3e/nn0LltSj23qEy71qv25gIyX7TfeSJQu?= =?iso-8859-1?q?1qz6iGXrUNcFntwMzrUnnJVSF0qWlxmcIpvwmqewMU9kC4E8w8aSvh0W9np2?= =?iso-8859-1?q?j/9MO1Zk99aIrnfEd5zBfTiriLDGJBM66BKtjXr5mBiqVqRiRA3W3eCpNvr6?= =?iso-8859-1?q?azinmzxdyAhNKSoqn8eBDwJAiVfQUmtuVkYzJgoQWUDaMnEC87LPUw2hG9AJ?= =?iso-8859-1?q?tM2fYxwzF45NdJiM+nGTGPY9jcFc1wXXbK9NgNjwS3zksufYEL3xfTwRi88z?= =?iso-8859-1?q?wQ7RxrW/t436X4JUVCYxww27yq8Ie8+d3SrPSybXZ8h0eGl9u95lkKKFHa5n?= =?iso-8859-1?q?gqkcoVXj/iPoC4MdK4KiQZ4z1QPu+Pps1blIEadjbF0g=3D=3D?= x-microsoft-antispam-prvs: x-forefront-prvs: 0971922F40 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(376002)(39830400003)(346002)(366004)(136003)(396003)(199004)(189003)(305945005)(53936002)(6116002)(8676002)(1671002)(102836004)(86362001)(508600001)(97736004)(52116002)(36756003)(14454004)(76176011)(2906002)(99286004)(386003)(6506007)(7736002)(25786009)(81156014)(81166006)(4326008)(6436002)(6486002)(6512007)(446003)(11346002)(118296001)(2616005)(476003)(105586002)(68736007)(106356001)(71200400001)(71190400001)(186003)(486006)(8936002)(50226002)(5660300002)(256004)(109986005)(107886003)(54906003)(59246006)(316002)(46003); DIR:OUT; SFP:1102; SCL:1; SRVR:LO2P265MB0158; H:LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: garyguo.net does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: e4/AKhbGWnVae9PZR8YUnn49RlUqwSsDyRlM5JribZ2B7uG0nHE34iUwpWljK4IyDavLVoTKbjWXAfNjCLob/Dfuy+Ji+OC5l17eT28KIKeXbM5gkJOHVejeSz1Qf52bd8QMnTWk/Lrwjw5Cs3upBZdZf5/nJs1zmKFiLzVGQnq1bsNwJcwlcM4se65EGyJfxLOYYWhfVx9WQIHnhcFuRe+Mwg70Mg+73MSEPYmlKaRexe4zWlcxM63AG3WLVyjCX3FLny03VejcR82eyrL3YyOp3ZpPmmBigKYLvOd4Pq2vUkFz5yj+Llkns30XZB3UyNCgkVF1y2gOPUDtcwu5BTy88twzsUX6lvx/X2R2vQg4v14fJ+aVNazQzbHo1wefz3wMiY0y45wn6DUcZne2AsmJHlh4PGwk2RYuOeMNGbg= MIME-Version: 1.0 X-OriginatorOrg: garyguo.net X-MS-Exchange-CrossTenant-Network-Message-Id: 648a48e1-bb04-4d86-79e0-08d6a49ecb7e X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Mar 2019 14:52:04.7276 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bbc898ad-b10f-4e10-8552-d9377b823d45 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: LO2P265MB0158 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190309_065215_138258_4C329350 X-CRM114-Status: UNSURE ( 9.02 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Palmer Dabbelt , Anup Patel , Christoph Hellwig , Atish Patra , Albert Ou , Gary Guo , "linux-riscv@lists.infradead.org" Sender: "linux-riscv" Errors-To: linux-riscv-bounces+patchwork-linux-riscv=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Gary Guo Currently sbi_remote_sfence_vma{,_asid} does not pass their arguments to SBI at all, which is semantically incorrect. Neither BBL nor OpenSBI is using these arguments at the moment, and they just do a global flush instead. However we still need to provide correct arguments. Signed-off-by: Gary Guo --- arch/riscv/include/asm/sbi.h | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index b6bb10b92fe2..19f231615510 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -26,22 +26,27 @@ #define SBI_REMOTE_SFENCE_VMA_ASID 7 #define SBI_SHUTDOWN 8 -#define SBI_CALL(which, arg0, arg1, arg2) ({ \ +#define SBI_CALL(which, arg0, arg1, arg2, arg3) ({ \ register uintptr_t a0 asm ("a0") = (uintptr_t)(arg0); \ register uintptr_t a1 asm ("a1") = (uintptr_t)(arg1); \ register uintptr_t a2 asm ("a2") = (uintptr_t)(arg2); \ + register uintptr_t a3 asm ("a3") = (uintptr_t)(arg3); \ register uintptr_t a7 asm ("a7") = (uintptr_t)(which); \ asm volatile ("ecall" \ : "+r" (a0) \ - : "r" (a1), "r" (a2), "r" (a7) \ + : "r" (a1), "r" (a2), "r" (a3), "r" (a7) \ : "memory"); \ a0; \ }) /* Lazy implementations until SBI is finalized */ -#define SBI_CALL_0(which) SBI_CALL(which, 0, 0, 0) -#define SBI_CALL_1(which, arg0) SBI_CALL(which, arg0, 0, 0) -#define SBI_CALL_2(which, arg0, arg1) SBI_CALL(which, arg0, arg1, 0) +#define SBI_CALL_0(which) SBI_CALL(which, 0, 0, 0, 0) +#define SBI_CALL_1(which, arg0) SBI_CALL(which, arg0, 0, 0, 0) +#define SBI_CALL_2(which, arg0, arg1) SBI_CALL(which, arg0, arg1, 0, 0) +#define SBI_CALL_3(which, arg0, arg1, arg2) \ + SBI_CALL(which, arg0, arg1, arg2, 0) +#define SBI_CALL_4(which, arg0, arg1, arg2, arg3) \ + SBI_CALL(which, arg0, arg1, arg2, arg3) static inline void sbi_console_putchar(int ch) { @@ -86,7 +91,7 @@ static inline void sbi_remote_sfence_vma(const unsigned long *hart_mask, unsigned long start, unsigned long size) { - SBI_CALL_1(SBI_REMOTE_SFENCE_VMA, hart_mask); + SBI_CALL_3(SBI_REMOTE_SFENCE_VMA, hart_mask, start, size); } static inline void sbi_remote_sfence_vma_asid(const unsigned long *hart_mask, @@ -94,7 +99,7 @@ static inline void sbi_remote_sfence_vma_asid(const unsigned long *hart_mask, unsigned long size, unsigned long asid) { - SBI_CALL_1(SBI_REMOTE_SFENCE_VMA_ASID, hart_mask); + SBI_CALL_4(SBI_REMOTE_SFENCE_VMA_ASID, hart_mask, start, size, asid); } #endif From patchwork Sat Mar 9 14:52:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gary Guo X-Patchwork-Id: 10845959 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A49AB1669 for ; Sat, 9 Mar 2019 14:52:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8748D2BBEB for ; Sat, 9 Mar 2019 14:52:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7B2E92BC2B; Sat, 9 Mar 2019 14:52:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAD_ENC_HEADER,BAYES_00, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,MISSING_HEADERS,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8695C2BBEB for ; Sat, 9 Mar 2019 14:52:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Date:Subject:From:Reply-To:To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9V5frpEzAKhvL9xakJnxznZOmtc8LTKstR323gxVywI=; b=Z3aW6u1AHbwFY7 CJRh7ctOHOZoVAhj0PuQburReJT2QgNNe2ud/NRmV9vn/28GWgx4R2JUaymO6SABuzd5j5vIU10XC jk49rjCAEPqz/0FF9OjSn3qeauMZR4CFmQXZ3KETyC0pixzy0GZI5PF7Fbanek7jUz/RzbMqzGL/D wTURsCvCbQ9Q6lpWUVTSXMAZYNnzbOpWOacpGPAFbYCBpKKlyhi0jfeaY3uUnTsAQdgTNApfmnqTT zcvM5y3aXFfbYnTBJQZQy0GGY60WcrUjCjGi5CTGBNvwCsEfFlpMayy+Oq19whQmUjx0NKrWFrABw FzSvITcuo//H1i20Brag==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2dKm-0006Ir-Pz; Sat, 09 Mar 2019 14:52:20 +0000 Received: from mail-eopbgr110109.outbound.protection.outlook.com ([40.107.11.109] helo=GBR01-CWL-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h2dKi-00068V-Rc for linux-riscv@lists.infradead.org; Sat, 09 Mar 2019 14:52:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=garyguo.net; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mjUe+gjQOe7WLa11XdOFvSIAIfBZyhb/Mizds14Br9Q=; b=FVdc7WW2rS7LyGA4J5rNK0TBV7e1BAAxXhl0Nma+c23rnOcEv7xamHHVmKLydHqe8d4H0D2W1KbBoCxTy7gqcV7sovB7Y+oeZ4LtK6zMvrwFxEmzK07MlLHE8h1avfxwH1TRcYjtq9bUex2O6J08/hJ2mdf31x7e7JZI2EmW39g= Received: from LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM (20.176.139.20) by LO2P265MB0158.GBRP265.PROD.OUTLOOK.COM (10.166.98.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1686.18; Sat, 9 Mar 2019 14:52:05 +0000 Received: from LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM ([fe80::ed34:1290:4306:3157]) by LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM ([fe80::ed34:1290:4306:3157%3]) with mapi id 15.20.1686.021; Sat, 9 Mar 2019 14:52:05 +0000 From: Gary Guo Subject: [PATCH v3 4/4] riscv: rewrite tlb flush for performance Thread-Topic: [PATCH v3 4/4] riscv: rewrite tlb flush for performance Thread-Index: AQHU1oepEgAqnc5LGUKKinYCltp1SA== Date: Sat, 9 Mar 2019 14:52:04 +0000 Message-ID: <5f387ef7b800d5626f20e413090b30ee66d1c8d5.1552142593.git.gary@garyguo.net> References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: LNXP123CA0015.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:d2::27) To LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8c::20) x-mailer: git-send-email 2.17.1 authentication-results: spf=none (sender IP is ) smtp.mailfrom=gary@garyguo.net; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [2001:630:212:238:3697:f6ff:fe55:55b1] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: de66869a-16af-4d33-d717-08d6a49ecba9 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(7021145)(8989299)(4534185)(7022145)(4603075)(4627221)(201702281549075)(8990200)(7048125)(7024125)(7027125)(7023125)(5600127)(711020)(4605104)(2017052603328)(7153060)(7193020); SRVR:LO2P265MB0158; x-ms-traffictypediagnostic: LO2P265MB0158: x-microsoft-exchange-diagnostics: =?iso-8859-1?Q?1; LO2P265MB0158; 23:cyd/OuUuX9VOsWR8skm8ZWMJN/4RIocH2FAFzX5?= =?iso-8859-1?q?P8T6hmW5EowKMlg?= =?iso-8859-1?q?mTZs2OlQaX8NCWOSQDjXpxS7PBQuk820P7hYBeHZPLywfJt15HtJWd4L8OJr?= =?iso-8859-1?q?E8Y7wG+5LJeHzGD+AgqSW08AuTS1xJamug3y2YGsfj94aw+I+SE630vFH3iz?= =?iso-8859-1?q?MhNhIqOjMNeu7Vxgo7BKHz7fgk8mnoUciDahEEIhYaL81jZnWspa37gVlHDf?= =?iso-8859-1?q?F3wb+BjwXhrOsu3+69dsu4zGpBz8LIEdJcoHE5a/Rc+4ExjfDBHC8U3oR/Kv?= =?iso-8859-1?q?5fl4WqNb9WyDh4ek+feiN2aKViOfVnq56RTpUCz96k2QK1bjIguaot++58n+?= =?iso-8859-1?q?r3qXPVpvMkxVTHFP06GHIy1GHQltmL/ekekH1LFWVJTpIlsj9YTl19qdfYr6?= =?iso-8859-1?q?g5z4lj99RL1Y61KSHQC2YeWOIrQqquaVI4I3EXvMCdfa/khbAGYgnO/7OJ0Q?= =?iso-8859-1?q?8cHmvvzxiM+XNhouiRphriTeLA0hPc4XedOLFG6OX21uUvvgUBQM0ViGLkk8?= =?iso-8859-1?q?JubI7x1ya5cvHxtA+1u0FyO9SNxkz2F2k0VX8tVd4dET8d4fFul4Y3odLvtB?= =?iso-8859-1?q?s1ORz52v6EWTCbliH+ZIa+ZxQlpb1oG3CG1i6OCpJLazfRdbXGo4Ug4cCkml?= =?iso-8859-1?q?ArlawPWBbylTiTno9Cjepq/efweNE3feW80c75ZpLIaIojych0IRJZXdW4DI?= =?iso-8859-1?q?u7wMeZQ7xsoyjw5eOwx9CXcu7FzLylfBAQgamxgjAarSM8YuKBoxxyYEEQDi?= =?iso-8859-1?q?T0jTlwysGcLkntUeQEF3t5UAcOQJpBZdyvX8cmU1FnUXN1Rgpmnh2cv2nia7?= =?iso-8859-1?q?TybtCojCat1syq75Ixm+i2N9Wr/rs/4vp4DvCn3JEhUlRTmOyFvFMhOnY4rm?= =?iso-8859-1?q?oxGARu9CvdBnphCa07Oi4sIB/tU52KpR0togdgnambaVdEgg9o3EzMwrVyZN?= =?iso-8859-1?q?SzwzKJcgoZDBTWqlEV3SylNKKiqZEbLfL8FrUzYnwpmuJVZWTc5GYng/wlYU?= =?iso-8859-1?q?m3L386HKUqnYclx7z5DySOgvVZJ+9HJ12+Jddp/PbXw/+LOnAAGqWLE94ORH?= =?iso-8859-1?q?4dSYuT67n+yRxFAt6dCHeeOoXluH9JvcQLUdX94Wy4zBI9mibhPJBXW/7Wsr?= =?iso-8859-1?q?Kg27W97sQtu9UFXMxerpkVMnCP52lhB94Q60COOoY3fDSMQXYS5TVhkP1Sl6?= =?iso-8859-1?q?fGMaoFZveS+KR5jAfhNHn4xnm+twqEfTUFSha3KSD2+Lo761JtBiYesxUOfC?= =?iso-8859-1?q?Lj4XFto6u+o2z148EWZB6FRgxyp0LCNgBBCGl6XTWJbGh6NA=3D=3D?= x-microsoft-antispam-prvs: x-forefront-prvs: 0971922F40 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(376002)(39830400003)(346002)(366004)(136003)(396003)(199004)(189003)(305945005)(53936002)(6116002)(8676002)(1671002)(102836004)(86362001)(508600001)(97736004)(52116002)(36756003)(14454004)(76176011)(2906002)(99286004)(386003)(6506007)(7736002)(25786009)(81156014)(81166006)(4326008)(6436002)(6486002)(6512007)(446003)(11346002)(118296001)(2616005)(476003)(105586002)(68736007)(106356001)(71200400001)(71190400001)(186003)(486006)(8936002)(50226002)(5660300002)(14444005)(256004)(30864003)(109986005)(107886003)(54906003)(59246006)(316002)(46003)(2004002); DIR:OUT; SFP:1102; SCL:1; SRVR:LO2P265MB0158; H:LO2P265MB0847.GBRP265.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: garyguo.net does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: y2Q7sYbYH3V6og9zB/7eftsw2mR29GTlh5tkseZmsm3o655cFCx6NQk9kArXk5+7ziy4xxjuAB32/XCt1bpr1wsyp6hR+Bz3FvWHOuH1OKhBRCbjo8oFTLPH/T8tcgjXpI7vI3UvLIKydxaRFxGmkT/eQNtS0UMBmV2J+eIleL6GDFn5rl7ZDfmCXYjruH0P8c+yYKgNuTLApYKti3RM8TWgjcjK1AU53GqUXWxfvmd/hcb/1SYrYDq2Yptos4U+H2uinJzkvKU0fxPMenwV+X9DDmcJv82rLfqvuDl6vlSI3Peqk+kdvNwNMOcr5DKGyCFQSLQkDKSu1R6i9vEuXWdHa01HAJYkWdF6YMaxfrzA0pf1amPAMWdzMWxmW7fIpuTPz+jMM+VRo8wHcAXBTCZPIEMC0ka95co+vY3v/k8= MIME-Version: 1.0 X-OriginatorOrg: garyguo.net X-MS-Exchange-CrossTenant-Network-Message-Id: de66869a-16af-4d33-d717-08d6a49ecba9 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Mar 2019 14:52:04.9988 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bbc898ad-b10f-4e10-8552-d9377b823d45 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: LO2P265MB0158 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190309_065216_917961_6183275F X-CRM114-Status: GOOD ( 20.50 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Palmer Dabbelt , Anup Patel , Christoph Hellwig , Atish Patra , Albert Ou , Gary Guo , "linux-riscv@lists.infradead.org" Sender: "linux-riscv" Errors-To: linux-riscv-bounces+patchwork-linux-riscv=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Gary Guo This patch rewrites the logic related to TLB flushing, both to cleanup the code and to improve performance. We now use sfence.vma variant with specified ASID and virtual address whenever possible. Even though only ASID 0 is used, it still improves performance by preventing global mappings from being flushed from TLB. This patch also includes a IPI-based remote TLB shootdown, which is useful at this stage for testing because BBL/OpenSBI ignores operands of sbi_remote_sfence_vma_asid and always perform a global TLB flush. The SBI-based remote TLB shootdown can still be opt-in using boot cmdline "tlbi_method=sbi". Signed-off-by: Gary Guo --- arch/riscv/include/asm/pgtable.h | 2 +- arch/riscv/include/asm/tlbflush.h | 76 +++++------ arch/riscv/mm/Makefile | 1 + arch/riscv/mm/context.c | 8 +- arch/riscv/mm/init.c | 2 +- arch/riscv/mm/tlbflush.c | 216 ++++++++++++++++++++++++++++++ 6 files changed, 257 insertions(+), 48 deletions(-) create mode 100644 arch/riscv/mm/tlbflush.c diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 1141364d990e..19d1aeb059da 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -279,7 +279,7 @@ static inline void update_mmu_cache(struct vm_area_struct *vma, * Relying on flush_tlb_fix_spurious_fault would suffice, but * the extra traps reduce performance. So, eagerly SFENCE.VMA. */ - local_flush_tlb_page(address); + local_flush_tlb_page(vma, address); } #define __HAVE_ARCH_PTE_SAME diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 54fee0cadb1e..8c3086c6084c 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -1,22 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (C) 2009 Chen Liqin * Copyright (C) 2012 Regents of the University of California - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation, version 2. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. + * Copyright (C) 2019 Gary Guo, University of Cambridge */ #ifndef _ASM_RISCV_TLBFLUSH_H #define _ASM_RISCV_TLBFLUSH_H #include -#include /* * Flush entire local TLB. 'sfence.vma' implicitly fences with the instruction @@ -27,53 +19,47 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } -/* Flush one page from local TLB */ -static inline void local_flush_tlb_page(unsigned long addr) +static inline void local_flush_tlb_mm(struct mm_struct *mm) { - __asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory"); + /* Flush ASID 0 so that global mappings are not affected */ + __asm__ __volatile__ ("sfence.vma x0, %0" : : "r" (0) : "memory"); } -#ifndef CONFIG_SMP - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) +static inline void local_flush_tlb_page(struct vm_area_struct *vma, + unsigned long addr) { - local_flush_tlb_all(); + __asm__ __volatile__ ("sfence.vma %0, %1" + : : "r" (addr), "r" (0) + : "memory"); } -#define flush_tlb_mm(mm) flush_tlb_all() +static inline void local_flush_tlb_kernel_page(unsigned long addr) +{ + __asm__ __volatile ("sfence.vma %0" : : "r" (addr) : "memory"); +} -#else /* CONFIG_SMP */ +void local_flush_tlb_range(struct vm_area_struct *vma, unsigned long start, + unsigned long end); +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end); -#include +#ifdef CONFIG_SMP -static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start, - unsigned long size) -{ - struct cpumask hmask; +void flush_tlb_all(void); +void flush_tlb_mm(struct mm_struct *mm); +void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); +void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, + unsigned long end); +void flush_tlb_kernel_range(unsigned long start, unsigned long end); - cpumask_clear(&hmask); - riscv_cpuid_to_hartid_mask(cmask, &hmask); - sbi_remote_sfence_vma(hmask.bits, start, size); -} +#else /* CONFIG_SMP */ -#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1) -#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0) -#define flush_tlb_range(vma, start, end) \ - remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) - (start)) -#define flush_tlb_mm(mm) \ - remote_sfence_vma(mm_cpumask(mm), 0, -1) +#define flush_tlb_all() local_flush_tlb_all() +#define flush_tlb_mm(mm) local_flush_tlb_mm(mm) +#define flush_tlb_page(vma, addr) local_flush_tlb_page(vma, addr) +#define flush_tlb_range(vma, start, end) local_flush_tlb_range(vma, start, end) +#define flush_tlb_kernel_range(start, end) \ + local_flush_tlb_kernel_range(start, end) #endif /* CONFIG_SMP */ -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - flush_tlb_all(); -} - #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index d75b035786d6..53b68fd3cb45 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -4,3 +4,4 @@ obj-y += extable.o obj-y += ioremap.o obj-y += cacheflush.o obj-y += context.o +obj-y += tlbflush.o diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index fbb1cfe80267..0f787bcd3a7a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -64,7 +64,13 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next, * privileged ISA 1.10 yet. */ csr_write(sptbr, virt_to_pfn(next->pgd) | SATP_MODE); - local_flush_tlb_all(); + + /* + * sfence.vma after SATP write. We call it on MM context instead of + * calling local_flush_tlb_all to prevent global mappings from being + * affected. + */ + local_flush_tlb_mm(next); flush_icache_deferred(next); } diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index b379a75ac6a6..858f55e8b219 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -168,7 +168,7 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot) set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); } else { pte_clear(&init_mm, addr, ptep); - local_flush_tlb_page(addr); + local_flush_tlb_kernel_page(addr); } } diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c new file mode 100644 index 000000000000..9099dc037cbd --- /dev/null +++ b/arch/riscv/mm/tlbflush.c @@ -0,0 +1,216 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2019 Gary Guo, University of Cambridge + */ + +#include +#include + +#define SFENCE_VMA_FLUSH_ALL ((unsigned long) -1) + +/* + * This controls the maximum amount of page-level sfence.vma that the kernel + * can issue when the kernel needs to flush a range from the TLB. If the size + * of range goes beyond this threshold, a full sfence.vma is issued. + * + * Increase this number can negatively impact performance on implementations + * where sfence.vma's address operand is ignored and always perform a global + * TLB flush. On the other hand, implementations with page-level TLB flush + * support can benefit from a larger number. + */ +static unsigned long tlbi_range_threshold = PAGE_SIZE; + +static int __init setup_tlbi_max_ops(char *str) +{ + int value = 0; + + get_option(&str, &value); + + /* + * This value cannot be greater or equal to PTRS_PER_PTE, as we need + * to full flush for any non-leaf page table change. The value has also + * be at least 1. + */ + if (value >= PTRS_PER_PTE || value < 1) + return -EINVAL; + + tlbi_range_threshold = value * PAGE_SIZE; + return 0; +} +early_param("tlbi_max_ops", setup_tlbi_max_ops); + +void local_flush_tlb_range(struct vm_area_struct *vma, unsigned long start, + unsigned long end) +{ + if (end - start > tlbi_range_threshold) { + local_flush_tlb_mm(vma->vm_mm); + return; + } + + while (start < end) { + __asm__ __volatile__ ("sfence.vma %0, %1" + : : "r" (start), "r" (0) + : "memory"); + start += PAGE_SIZE; + } +} + +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + if (end - start > tlbi_range_threshold) { + local_flush_tlb_all(); + return; + } + + while (start < end) { + __asm__ __volatile__ ("sfence.vma %0" + : : "r" (start) + : "memory"); + start += PAGE_SIZE; + } +} + +#ifdef CONFIG_SMP + +/* + * BBL/OpenSBI are currently ignoring ASID and address range provided + * by SBI call argument, and do a full TLB flush instead. This may + * negatively impact performance on implementations with page-level + * sfence.vma support. + * + * We provide an IPI-based remote shootdown implementation to improve + * performance on implementations with page-level sfence.vma, and also to + * allow testing of these implementations. + * + * This parameter allows the approach (IPI/SBI) to be specified using boot + * cmdline. + */ +static bool tlbi_ipi = true; + +static int __init setup_tlbi_method(char *str) +{ + if (strcmp(str, "ipi") == 0) + tlbi_ipi = true; + else if (strcmp(str, "sbi") == 0) + tlbi_ipi = false; + else + return -EINVAL; + + return 0; +} +early_param("tlbi_method", setup_tlbi_method); + + +struct tlbi { + unsigned long start; + unsigned long size; + unsigned long asid; +}; + +static void ipi_remote_sfence_vma(void *info) +{ + struct tlbi *data = info; + unsigned long start = data->start; + unsigned long size = data->size; + unsigned long i; + + for (i = 0; i < size; i += PAGE_SIZE) { + __asm__ __volatile__ ("sfence.vma %0" + : : "r" (start + i) + : "memory"); + } +} + +static void ipi_remote_sfence_vma_asid(void *info) +{ + struct tlbi *data = info; + unsigned long asid = data->asid; + unsigned long start = data->start; + unsigned long size = data->size; + unsigned long i; + + /* Flush entire MM context */ + if (size == SFENCE_VMA_FLUSH_ALL) { + __asm__ __volatile__ ("sfence.vma x0, %0" + : : "r" (asid) + : "memory"); + return; + } + + for (i = 0; i < size; i += PAGE_SIZE) { + __asm__ __volatile__ ("sfence.vma %0, %1" + : : "r" (start + i), "r" (asid) + : "memory"); + } +} + +static void remote_sfence_vma(unsigned long start, unsigned long size) +{ + if (tlbi_ipi) { + struct tlbi info = { + .start = start, + .size = size, + }; + on_each_cpu(ipi_remote_sfence_vma, &info, 1); + } else + sbi_remote_sfence_vma(NULL, start, size); +} + +static void remote_sfence_vma_asid(cpumask_t *mask, unsigned long start, + unsigned long size, unsigned long asid) +{ + if (tlbi_ipi) { + struct tlbi info = { + .start = start, + .size = size, + .asid = asid, + }; + on_each_cpu_mask(mask, ipi_remote_sfence_vma_asid, &info, 1); + } else { + cpumask_t hmask; + + cpumask_clear(&hmask); + riscv_cpuid_to_hartid_mask(mask, &hmask); + sbi_remote_sfence_vma_asid(hmask.bits, start, size, asid); + } +} + + +void flush_tlb_all(void) +{ + sbi_remote_sfence_vma(NULL, 0, SFENCE_VMA_FLUSH_ALL); +} + +void flush_tlb_mm(struct mm_struct *mm) +{ + remote_sfence_vma_asid(mm_cpumask(mm), 0, SFENCE_VMA_FLUSH_ALL, 0); +} + +void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) +{ + remote_sfence_vma_asid(mm_cpumask(vma->vm_mm), addr, PAGE_SIZE, 0); +} + + +void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, + unsigned long end) +{ + if (end - start > tlbi_range_threshold) { + flush_tlb_mm(vma->vm_mm); + return; + } + + remote_sfence_vma_asid(mm_cpumask(vma->vm_mm), start, end - start, 0); +} + +void flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + if (end - start > tlbi_range_threshold) { + flush_tlb_all(); + return; + } + + remote_sfence_vma(start, end - start); +} + +#endif /* CONFIG_SMP */