From patchwork Tue May 19 20:19:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 11558721 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9AB98618 for ; Tue, 19 May 2020 20:20:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 707C220872 for ; Tue, 19 May 2020 20:20:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 707C220872 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9E66A80015; Tue, 19 May 2020 16:20:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9963A900003; Tue, 19 May 2020 16:20:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D40080015; Tue, 19 May 2020 16:20:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0071.hostedemail.com [216.40.44.71]) by kanga.kvack.org (Postfix) with ESMTP id 76573900003 for ; Tue, 19 May 2020 16:20:49 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 16B5C2A87 for ; Tue, 19 May 2020 20:20:49 +0000 (UTC) X-FDA: 76834587018.22.bit38_5e62c81c9ce62 X-Spam-Summary: 2,0,0,4e65d4b2d210e4a6,d41d8cd98f00b204,bigeasy@linutronix.de,,RULES_HIT:41:355:379:541:800:960:966:973:982:988:989:1260:1261:1311:1314:1345:1359:1431:1437:1513:1515:1521:1535:1543:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2901:3138:3139:3140:3141:3142:3354:3865:3867:3868:3871:3872:4250:4321:4385:5007:6117:6119:6261:6742:7576:7903:10004:11026:11473:11658:11914:12043:12160:12296:12297:12438:12555:12679:12683:12895:12986:13846:14096:14181:14394:14721:21080:21451:21627:21966:21990:30012:30054:30070,0,RBL:193.142.43.55:@linutronix.de:.lbl8.mailshell.net-62.8.6.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: bit38_5e62c81c9ce62 X-Filterd-Recvd-Size: 5471 Received: from Galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Tue, 19 May 2020 20:20:48 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=flow.W.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1jb8ii-00012c-4u; Tue, 19 May 2020 22:20:12 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: Peter Zijlstra , Ingo Molnar , Steven Rostedt , Will Deacon , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , "Luis Claudio R. Goncalves" , Seth Jennings , Dan Streetman , Vitaly Wool , Andrew Morton , linux-mm@kvack.org, Sebastian Andrzej Siewior Subject: [PATCH 8/8] mm/zswap: Use local lock to protect per-CPU data Date: Tue, 19 May 2020 22:19:12 +0200 Message-Id: <20200519201912.1564477-9-bigeasy@linutronix.de> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200519201912.1564477-1-bigeasy@linutronix.de> References: <20200519201912.1564477-1-bigeasy@linutronix.de> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Luis Claudio R. Goncalves" zwap uses per-CPU compression. The per-CPU data pointer is acquired with get_cpu_ptr() which implicitly disables preemption. It allocates memory inside the preempt disabled region which conflicts with the PREEMPT_RT semantics. Replace the implicit preemption control with an explicit local lock. This allows RT kernels to substitute it with a real per CPU lock, which serializes the access but keeps the code section preemptible. On non RT kernels this maps to preempt_disable() as before, i.e. no functional change. [bigeasy: Use local_lock(), additional hunks, patch description] Cc: Seth Jennings Cc: Dan Streetman Cc: Vitaly Wool Cc: Andrew Morton Cc: linux-mm@kvack.org Signed-off-by: Luis Claudio R. Goncalves Signed-off-by: Sebastian Andrzej Siewior --- mm/zswap.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index fbb782924ccc5..1db2ad941e501 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -388,6 +389,8 @@ static struct zswap_entry *zswap_entry_find_get(struct rb_root *root, * per-cpu code **********************************/ static DEFINE_PER_CPU(u8 *, zswap_dstmem); +/* Used for zswap_dstmem and tfm */ +static DEFINE_LOCAL_LOCK(zswap_cpu_lock); static int zswap_dstmem_prepare(unsigned int cpu) { @@ -919,10 +922,11 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) dlen = PAGE_SIZE; src = (u8 *)zhdr + sizeof(struct zswap_header); dst = kmap_atomic(page); - tfm = *get_cpu_ptr(entry->pool->tfm); + local_lock(zswap_cpu_lock); + tfm = *this_cpu_ptr(entry->pool->tfm); ret = crypto_comp_decompress(tfm, src, entry->length, dst, &dlen); - put_cpu_ptr(entry->pool->tfm); + local_unlock(zswap_cpu_lock); kunmap_atomic(dst); BUG_ON(ret); BUG_ON(dlen != PAGE_SIZE); @@ -1074,12 +1078,12 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset, } /* compress */ - dst = get_cpu_var(zswap_dstmem); - tfm = *get_cpu_ptr(entry->pool->tfm); + local_lock(zswap_cpu_lock); + dst = *this_cpu_ptr(&zswap_dstmem); + tfm = *this_cpu_ptr(entry->pool->tfm); src = kmap_atomic(page); ret = crypto_comp_compress(tfm, src, PAGE_SIZE, dst, &dlen); kunmap_atomic(src); - put_cpu_ptr(entry->pool->tfm); if (ret) { ret = -EINVAL; goto put_dstmem; @@ -1103,7 +1107,7 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset, memcpy(buf, &zhdr, hlen); memcpy(buf + hlen, dst, dlen); zpool_unmap_handle(entry->pool->zpool, handle); - put_cpu_var(zswap_dstmem); + local_unlock(zswap_cpu_lock); /* populate entry */ entry->offset = offset; @@ -1131,7 +1135,7 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset, return 0; put_dstmem: - put_cpu_var(zswap_dstmem); + local_unlock(zswap_cpu_lock); zswap_pool_put(entry->pool); freepage: zswap_entry_cache_free(entry); @@ -1176,9 +1180,10 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset, if (zpool_evictable(entry->pool->zpool)) src += sizeof(struct zswap_header); dst = kmap_atomic(page); - tfm = *get_cpu_ptr(entry->pool->tfm); + local_lock(zswap_cpu_lock); + tfm = *this_cpu_ptr(entry->pool->tfm); ret = crypto_comp_decompress(tfm, src, entry->length, dst, &dlen); - put_cpu_ptr(entry->pool->tfm); + local_unlock(zswap_cpu_lock); kunmap_atomic(dst); zpool_unmap_handle(entry->pool->zpool, entry->handle); BUG_ON(ret);