From patchwork Tue Feb 21 14:21:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shanker Donthineni X-Patchwork-Id: 9584729 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id AD45C602A7 for ; Tue, 21 Feb 2017 14:32:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A082A2867B for ; Tue, 21 Feb 2017 14:32:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 955A0286B1; Tue, 21 Feb 2017 14:32:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2EDC82867B for ; Tue, 21 Feb 2017 14:32:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=7cREj2u6J23A623GehREcp+ldNA26eK70PxdyJpSxw4=; b=Ip+ uBykw3FZ067sQd+/UA0ShQ5D04m5HphjHgZb1T//Sy6GDVA9rb9K39vuJaCHe25Rh8jt1LyMHdzL/ DqrgdGXvA/0d1fWwiOTFWayUDTqYln51+r7LFzIXfeYLhAhfVKKwIPRidgzJMzjZaL2Xs0PNRHY6J JoM3cA1XiHMiyJR0zpCcFSLQi3a6hfHWp0UpBT0mu6pcHYLofVUzufBIN/6m3LaaOrlRzq4DbmgDd 93438qtZkswkvaQke2ui5WtUurDWRzUlUOrTv/6VPPdqPNqJB0nNtMR4XGv3UNTup3umgv+/ZYkgZ XzLB4HNoeEJ4ijV9b5Fc426il3NsiKQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1cgBUY-00071U-P0; Tue, 21 Feb 2017 14:32:34 +0000 Received: from smtp.codeaurora.org ([198.145.29.96]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1cgBKM-0002IJ-Lp for linux-arm-kernel@lists.infradead.org; Tue, 21 Feb 2017 14:22:04 +0000 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id 5580160C52; Tue, 21 Feb 2017 14:21:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1487686902; bh=5BD3NT0Bnx7VXr3f1L9jEc0TATAxlYBCreoT/yqf7+E=; h=From:To:Cc:Subject:Date:From; b=XGcvzk/szqfI438XTcue2D5vfbpdyYMqLIQCl4hdZ2a/twrzi6I3n7I5T3/TBH2EX Odcs5nwz0o7X4/TmKqS4F0hbZO6W4dPXgp6wqMPzU6FMFPhKfp92TNGiDHfot2giJ3 Jvlo3rn3FcJUHb0GycHwTVZG+ICNQIvvwEftoShY= Received: from shankerd-ubuntu.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: shankerd@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id 554EA60A7F; Tue, 21 Feb 2017 14:21:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1487686901; bh=5BD3NT0Bnx7VXr3f1L9jEc0TATAxlYBCreoT/yqf7+E=; h=From:To:Cc:Subject:Date:From; b=JS8vRIR9sgkKup1xPMMXRQgjHemJpk/82SqgZUPNkmA+RpvhO+GebIF6NdHdTeamG kAQrPjjbbo4S1fdJmZJ7zUvAkFuEYnh64TpQQfo3btREd+6seaU7knfktCb5NQWPL4 M26ewJ87VPh306gQgVhl019uKLwUkpxSbKbzN9mU= DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 554EA60A7F Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=shankerd@codeaurora.org From: Shanker Donthineni To: Catalin Marinas Subject: [RESEND PATCH v2] arm64: cache: Skip an unnecessary data cache clean PoU operation Date: Tue, 21 Feb 2017 08:21:35 -0600 Message-Id: <1487686895-18065-1-git-send-email-shankerd@codeaurora.org> X-Mailer: git-send-email 1.9.1 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170221_062202_770331_700AAFA4 X-CRM114-Status: GOOD ( 17.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Vikram Sethi , Suzuki K Poulose , Marc Zyngier , Will Deacon , linux-kernel , James Morse , Shanker Donthineni , Anna-Maria Gleixner , linux-arm-kernel MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP The cache management functions always do the data cache PoU (point of unification) operations even though it is not required on some systems. No need to clean data cache till PoU if all the cache levels below PoUIS are WT (Write-Through) caches. It causes a huge performance degradation when operating on a larger memory area, especially THP with 64K page size kernel. For each online CPU, check the need of 'dc cvau' instruction and update a global variable __dcache_flags. The two functions __flush_cache_user_range() and __clean_dcache_area_pou() are modified to skip an unnecessary code execution based on flags. It won't change the existing behavior if any one of the online CPU is capable of WB cache below PoUIS level. Signed-off-by: Shanker Donthineni --- Changes since v1: handle skipping a dcache clean POU operation by checking the global variable __dcache_flags in cache.S instead of patching the code segment. arch/arm64/include/asm/cachetype.h | 8 ++++++++ arch/arm64/kernel/cpuinfo.c | 30 ++++++++++++++++++++++++++++++ arch/arm64/mm/cache.S | 8 +++++++- 3 files changed, 45 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/cachetype.h b/arch/arm64/include/asm/cachetype.h index f558869..352fb23 100644 --- a/arch/arm64/include/asm/cachetype.h +++ b/arch/arm64/include/asm/cachetype.h @@ -28,6 +28,8 @@ #define ICACHE_POLICY_VIPT 2 #define ICACHE_POLICY_PIPT 3 +#define DCACHE_SKIP_POU 0 + #ifndef __ASSEMBLY__ #include @@ -39,6 +41,12 @@ extern unsigned long __icache_flags; +extern unsigned long __dcache_flags; + +#define CLIDR_LOUIS_SHIFT (21) +#define CLIDR_LOUIS_MASK (0x7) +#define CLIDR_LOUIS(x) (((x) >> CLIDR_LOUIS_SHIFT) & CLIDR_LOUIS_MASK) + /* * NumSets, bits[27:13] - (Number of sets in cache) - 1 * Associativity, bits[12:3] - (Associativity of cache) - 1 diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c index 7b7be71..0e1a30a 100644 --- a/arch/arm64/kernel/cpuinfo.c +++ b/arch/arm64/kernel/cpuinfo.c @@ -50,6 +50,7 @@ }; unsigned long __icache_flags; +unsigned long __dcache_flags; static const char *const hwcap_str[] = { "fp", @@ -305,6 +306,33 @@ static void cpuinfo_detect_icache_policy(struct cpuinfo_arm64 *info) pr_info("Detected %s I-cache on CPU%d\n", icache_policy_str[l1ip], cpu); } +/* + * Check if all the data cache levels below LoUIS doesn't support WB. + * The flag DCACHE_SKIP_POU set to 0 if any one of the online CPU + * doesn't support WB cache below LoUIS. + */ +static void cpuinfo_ckeck_dcache_pou(struct cpuinfo_arm64 *info) +{ + u32 louis = CLIDR_LOUIS(read_sysreg(clidr_el1)); + static bool update_pou_once; + u32 lvl, csidr; + + /* Set the DCACHE_SKIP_POU flag only first time */ + if (!update_pou_once) { + set_bit(DCACHE_SKIP_POU, &__dcache_flags); + update_pou_once = true; + } + + /* Go through all the cache level below LoUIS */ + for (lvl = 0; lvl < louis; lvl++) { + csidr = cache_get_ccsidr(lvl << 1); + if (csidr & CCSIDR_EL1_WRITE_BACK) { + clear_bit(DCACHE_SKIP_POU, &__dcache_flags); + break; + } + } +} + static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) { info->reg_cntfrq = arch_timer_get_cntfrq(); @@ -345,6 +373,8 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) } cpuinfo_detect_icache_policy(info); + cpuinfo_ckeck_dcache_pou(info); + } void cpuinfo_store_cpu(void) diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 83c27b6e..1884da2 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -24,6 +24,7 @@ #include #include #include +#include /* * flush_icache_range(start,end) @@ -50,6 +51,8 @@ ENTRY(flush_icache_range) */ ENTRY(__flush_cache_user_range) uaccess_ttbr0_enable x2, x3 + ldr_l x4, __dcache_flags + tbnz x4, #DCACHE_SKIP_POU, 2f dcache_line_size x2, x3 sub x3, x2, #1 bic x4, x0, x3 @@ -60,6 +63,7 @@ user_alt 9f, "dc cvau, x4", "dc civac, x4", ARM64_WORKAROUND_CLEAN_CACHE b.lo 1b dsb ish +2: icache_line_size x2, x3 sub x3, x2, #1 bic x4, x0, x3 @@ -104,8 +108,10 @@ ENDPIPROC(__flush_dcache_area) * - size - size in question */ ENTRY(__clean_dcache_area_pou) + ldr_l x2, __dcache_flags + tbnz x2, #DCACHE_SKIP_POU, 1f dcache_by_line_op cvau, ish, x0, x1, x2, x3 - ret +1: ret ENDPROC(__clean_dcache_area_pou) /*