From patchwork Tue Feb 4 14:25:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 11364707 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C293513A4 for ; Tue, 4 Feb 2020 14:25:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 99E4B2087E for ; Tue, 4 Feb 2020 14:25:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 99E4B2087E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4FDF16B000C; Tue, 4 Feb 2020 09:25:20 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 45D956B000D; Tue, 4 Feb 2020 09:25:20 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2AFD86B000A; Tue, 4 Feb 2020 09:25:20 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0117.hostedemail.com [216.40.44.117]) by kanga.kvack.org (Postfix) with ESMTP id 020206B0007 for ; Tue, 4 Feb 2020 09:25:19 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A273521F0 for ; Tue, 4 Feb 2020 14:25:19 +0000 (UTC) X-FDA: 76452667158.01.edge68_2e6f19aae8c04 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,jack@suse.cz,:willy@infradead.org:linux-fsdevel@vger.kernel.org::jack@suse.cz:stable@vger.kernel.org,RULES_HIT:30034:30045:30054:30090,0,RBL:195.135.220.15:@suse.cz:.lbl8.mailshell.net-62.2.6.2 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:32,LUA_SUMMARY:none X-HE-Tag: edge68_2e6f19aae8c04 X-Filterd-Recvd-Size: 5612 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Tue, 4 Feb 2020 14:25:18 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id BD2AFAEDE; Tue, 4 Feb 2020 14:25:17 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id E38C01E0B00; Tue, 4 Feb 2020 15:25:15 +0100 (CET) From: Jan Kara To: Matthew Wilcox Cc: , , Jan Kara , stable@vger.kernel.org Subject: [PATCH 1/8] xarray: Fix premature termination of xas_for_each_marked() Date: Tue, 4 Feb 2020 15:25:07 +0100 Message-Id: <20200204142514.15826-2-jack@suse.cz> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20200204142514.15826-1-jack@suse.cz> References: <20200204142514.15826-1-jack@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: xas_for_each_marked() is using entry == NULL as a termination condition of the iteration. When xas_for_each_marked() is used protected only by RCU, this can however race with xas_store(xas, NULL) in the following way: TASK1 TASK2 page_cache_delete() find_get_pages_range_tag() xas_for_each_marked() xas_find_marked() off = xas_find_chunk() xas_store(&xas, NULL) xas_init_marks(&xas); ... rcu_assign_pointer(*slot, NULL); entry = xa_entry(off); And thus xas_for_each_marked() terminates prematurely possibly leading to missed entries in the iteration (translating to missing writeback of some pages or a similar problem). Fix the problem by creating a special version of xas_find_marked() - xas_find_valid_marked() - that does not return NULL marked entries and changing xas_next_marked() in the same way. CC: stable@vger.kernel.org Fixes: ef8e5717db01 "page cache: Convert delete_batch to XArray" Signed-off-by: Jan Kara Reported-by: Jan Kara Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Jan Kara --- include/linux/xarray.h | 64 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 47 insertions(+), 17 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index f73e1775ded0..5370716d7010 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1633,33 +1633,63 @@ static inline unsigned int xas_find_chunk(struct xa_state *xas, bool advance, } /** - * xas_next_marked() - Advance iterator to next marked entry. + * xas_find_valid_marked() - Find the next marked valid entry in the XArray. + * @xas: XArray operation state. + * @max: Highest index to return. + * @mark: Mark number to search for. + * + * This is like xas_find_marked() except that we also skip over all %NULL + * marked entries. + * + * Return: The entry, if found, otherwise %NULL. + */ +static inline void *xas_find_valid_marked(struct xa_state *xas, + unsigned long max, xa_mark_t mark) +{ + void *entry; + + do { + entry = xas_find_marked(xas, max, mark); + } while (unlikely(entry == NULL) && xas_valid(xas)); + + return entry; +} + +/** + * xas_next_valid_marked() - Advance iterator to next valid marked entry. * @xas: XArray operation state. * @max: Highest index to return. * @mark: Mark to search for. * - * xas_next_marked() is an inline function to optimise xarray traversal for - * speed. It is equivalent to calling xas_find_marked(), and will call - * xas_find_marked() for all the hard cases. + * xas_next_valid_marked() is an inline function to optimise xarray traversal + * for speed. It is equivalent to calling xas_find_valid_marked(), and will + * call xas_find_marked() for all the hard cases. The function skips over %NULL + * marked entries. * * Return: The next marked entry after the one currently referred to by @xas. */ -static inline void *xas_next_marked(struct xa_state *xas, unsigned long max, - xa_mark_t mark) +static inline void *xas_next_valid_marked(struct xa_state *xas, + unsigned long max, xa_mark_t mark) { struct xa_node *node = xas->xa_node; unsigned int offset; + void *entry; if (unlikely(xas_not_node(node) || node->shift)) - return xas_find_marked(xas, max, mark); - offset = xas_find_chunk(xas, true, mark); - xas->xa_offset = offset; - xas->xa_index = (xas->xa_index & ~XA_CHUNK_MASK) + offset; - if (xas->xa_index > max) - return NULL; - if (offset == XA_CHUNK_SIZE) - return xas_find_marked(xas, max, mark); - return xa_entry(xas->xa, node, offset); + return xas_find_valid_marked(xas, max, mark); + + do { + offset = xas_find_chunk(xas, true, mark); + xas->xa_offset = offset; + xas->xa_index = (xas->xa_index & ~XA_CHUNK_MASK) + offset; + if (xas->xa_index > max) + return NULL; + if (offset == XA_CHUNK_SIZE) + return xas_find_valid_marked(xas, max, mark); + entry = xa_entry(xas->xa, node, offset); + } while (unlikely(!entry)); + + return entry; } /* @@ -1702,8 +1732,8 @@ enum { * xas_pause() first. */ #define xas_for_each_marked(xas, entry, max, mark) \ - for (entry = xas_find_marked(xas, max, mark); entry; \ - entry = xas_next_marked(xas, max, mark)) + for (entry = xas_find_valid_marked(xas, max, mark); entry; \ + entry = xas_next_valid_marked(xas, max, mark)) /** * xas_for_each_conflict() - Iterate over a range of an XArray. From patchwork Tue Feb 4 14:25:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 11364721 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4EF3714B4 for ; Tue, 4 Feb 2020 14:25:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 26BE4217BA for ; Tue, 4 Feb 2020 14:25:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 26BE4217BA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5C03C6B0008; Tue, 4 Feb 2020 09:25:21 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 56FDD6B000A; Tue, 4 Feb 2020 09:25:21 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 485A76B000D; Tue, 4 Feb 2020 09:25:21 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2F0676B0008 for ; Tue, 4 Feb 2020 09:25:21 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id D794C2466 for ; Tue, 4 Feb 2020 14:25:20 +0000 (UTC) X-FDA: 76452667200.23.toad87_2ea6a96a1731f X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,jack@suse.cz,:willy@infradead.org:linux-fsdevel@vger.kernel.org::jack@suse.cz,RULES_HIT:30045:30054:30056:30090,0,RBL:195.135.220.15:@suse.cz:.lbl8.mailshell.net-62.2.6.2 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: toad87_2ea6a96a1731f X-Filterd-Recvd-Size: 3954 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Tue, 4 Feb 2020 14:25:20 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 801A9AF37; Tue, 4 Feb 2020 14:25:18 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id E754E1E0BB3; Tue, 4 Feb 2020 15:25:15 +0100 (CET) From: Jan Kara To: Matthew Wilcox Cc: , , Jan Kara Subject: [PATCH 2/8] xarray: Provide xas_erase() helper Date: Tue, 4 Feb 2020 15:25:08 +0100 Message-Id: <20200204142514.15826-3-jack@suse.cz> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20200204142514.15826-1-jack@suse.cz> References: <20200204142514.15826-1-jack@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently xas_store() clears marks when stored value is NULL. This is somewhat counter-intuitive and also causes measurable performance impact when mark clearing is not needed (e.g. because marks are already clear). So provide xas_erase() helper (similarly to existing xa_erase()) which stores NULL at given index and also takes care of clearing marks. Use this helper from __xa_erase() and item_kill_tree() in tools/testing. In the following patches, callers that use the mark-clearing property of xas_store() will be converted to xas_erase() and remaining users can enjoy better performance. Signed-off-by: Jan Kara --- include/linux/xarray.h | 1 + lib/xarray.c | 24 +++++++++++++++++++++++- tools/testing/radix-tree/test.c | 2 +- 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index 5370716d7010..be6c6950837e 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -1491,6 +1491,7 @@ static inline bool xas_retry(struct xa_state *xas, const void *entry) void *xas_load(struct xa_state *); void *xas_store(struct xa_state *, void *entry); +void *xas_erase(struct xa_state *); void *xas_find(struct xa_state *, unsigned long max); void *xas_find_conflict(struct xa_state *); diff --git a/lib/xarray.c b/lib/xarray.c index 1d9fab7db8da..ae8b7070e82c 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1319,6 +1319,28 @@ static void *xas_result(struct xa_state *xas, void *curr) return curr; } +/** + * xas_erase() - Erase this entry from the XArray + * @xas: XArray operation state. + * + * After this function returns, loading from @index will return %NULL. The + * function also clears all marks associated with the @index. If the index is + * part of a multi-index entry, all indices will be erased and none of the + * entries will be part of a multi-index entry. + * + * Return: The entry which used to be at this index. + */ +void *xas_erase(struct xa_state *xas) +{ + void *entry; + + entry = xas_store(xas, NULL); + xas_init_marks(xas); + + return entry; +} +EXPORT_SYMBOL(xas_erase); + /** * __xa_erase() - Erase this entry from the XArray while locked. * @xa: XArray. @@ -1334,7 +1356,7 @@ static void *xas_result(struct xa_state *xas, void *curr) void *__xa_erase(struct xarray *xa, unsigned long index) { XA_STATE(xas, xa, index); - return xas_result(&xas, xas_store(&xas, NULL)); + return xas_result(&xas, xas_erase(&xas)); } EXPORT_SYMBOL(__xa_erase); diff --git a/tools/testing/radix-tree/test.c b/tools/testing/radix-tree/test.c index a15d0512e633..07dc2b4dc587 100644 --- a/tools/testing/radix-tree/test.c +++ b/tools/testing/radix-tree/test.c @@ -261,7 +261,7 @@ void item_kill_tree(struct xarray *xa) if (!xa_is_value(entry)) { item_free(entry, xas.xa_index); } - xas_store(&xas, NULL); + xas_erase(&xas); } assert(xa_empty(xa)); From patchwork Tue Feb 4 14:25:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 11364725 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BB54714B4 for ; Tue, 4 Feb 2020 14:25:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9EEB72166E for ; Tue, 4 Feb 2020 14:25:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9EEB72166E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CF9386B000A; Tue, 4 Feb 2020 09:25:21 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C76DC6B0266; Tue, 4 Feb 2020 09:25:21 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 98EBF6B000A; Tue, 4 Feb 2020 09:25:21 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id 751766B000E for ; Tue, 4 Feb 2020 09:25:21 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3D59633CD for ; Tue, 4 Feb 2020 14:25:21 +0000 (UTC) X-FDA: 76452667242.11.bomb05_2eb07242b6761 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,jack@suse.cz,:willy@infradead.org:linux-fsdevel@vger.kernel.org::jack@suse.cz,RULES_HIT:30054:30090,0,RBL:195.135.220.15:@suse.cz:.lbl8.mailshell.net-62.2.6.2 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:17,LUA_SUMMARY:none X-HE-Tag: bomb05_2eb07242b6761 X-Filterd-Recvd-Size: 1898 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Tue, 4 Feb 2020 14:25:20 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 7ECF5B035; Tue, 4 Feb 2020 14:25:19 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id EBCCA1E0CF7; Tue, 4 Feb 2020 15:25:15 +0100 (CET) From: Jan Kara To: Matthew Wilcox Cc: , , Jan Kara Subject: [PATCH 3/8] xarray: Explicitely set XA_FREE_MARK in __xa_cmpxchg() Date: Tue, 4 Feb 2020 15:25:09 +0100 Message-Id: <20200204142514.15826-4-jack@suse.cz> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20200204142514.15826-1-jack@suse.cz> References: <20200204142514.15826-1-jack@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __xa_cmpxchg() relies on xas_store() to set XA_FREE_MARK when storing NULL into xarray that has free tracking enabled. Make the setting of XA_FREE_MARK explicit similarly as its clearing currently it. Signed-off-by: Jan Kara --- lib/xarray.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/xarray.c b/lib/xarray.c index ae8b7070e82c..4e32497c51bd 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1477,8 +1477,12 @@ void *__xa_cmpxchg(struct xarray *xa, unsigned long index, curr = xas_load(&xas); if (curr == old) { xas_store(&xas, entry); - if (xa_track_free(xa) && entry && !curr) - xas_clear_mark(&xas, XA_FREE_MARK); + if (xa_track_free(xa)) { + if (entry && !curr) + xas_clear_mark(&xas, XA_FREE_MARK); + else if (!entry && curr) + xas_set_mark(&xas, XA_FREE_MARK); + } } } while (__xas_nomem(&xas, gfp)); From patchwork Tue Feb 4 14:25:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 11364697 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1406417E0 for ; Tue, 4 Feb 2020 14:25:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ECA1A2084E for ; Tue, 4 Feb 2020 14:25:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ECA1A2084E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F34856B0003; Tue, 4 Feb 2020 09:25:19 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EBDBF6B0005; Tue, 4 Feb 2020 09:25:19 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAC306B0006; Tue, 4 Feb 2020 09:25:19 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id C0AAE6B0003 for ; Tue, 4 Feb 2020 09:25:19 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6EF7933CD for ; Tue, 4 Feb 2020 14:25:19 +0000 (UTC) X-FDA: 76452667158.22.base92_2e6d22bccd526 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,jack@suse.cz,:willy@infradead.org:linux-fsdevel@vger.kernel.org::jack@suse.cz,RULES_HIT:30054:30090,0,RBL:195.135.220.15:@suse.cz:.lbl8.mailshell.net-62.2.6.2 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:38,LUA_SUMMARY:none X-HE-Tag: base92_2e6d22bccd526 X-Filterd-Recvd-Size: 1656 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Tue, 4 Feb 2020 14:25:18 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 94BA5AE52; Tue, 4 Feb 2020 14:25:17 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id F0B9D1E0D09; Tue, 4 Feb 2020 15:25:15 +0100 (CET) From: Jan Kara To: Matthew Wilcox Cc: , , Jan Kara Subject: [PATCH 4/8] mm: Use xas_erase() in page_cache_delete_batch() Date: Tue, 4 Feb 2020 15:25:10 +0100 Message-Id: <20200204142514.15826-5-jack@suse.cz> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20200204142514.15826-1-jack@suse.cz> References: <20200204142514.15826-1-jack@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We need to clear marks when removing a page from xarray since there could be DIRTY or TOWRITE tags left for the page. Use xas_erase() to explicitely request mark clearing. Signed-off-by: Jan Kara --- mm/filemap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index bf6aa30be58d..ca7eeb067a23 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -333,7 +333,7 @@ static void page_cache_delete_batch(struct address_space *mapping, */ if (page->index + compound_nr(page) - 1 == xas.xa_index) i++; - xas_store(&xas, NULL); + xas_erase(&xas); total_pages++; } mapping->nrpages -= total_pages; From patchwork Tue Feb 4 14:25:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 11364717 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8DD6314B4 for ; Tue, 4 Feb 2020 14:25:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 726E020730 for ; Tue, 4 Feb 2020 14:25:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 726E020730 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A0F036B0005; Tue, 4 Feb 2020 09:25:20 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 633596B000A; Tue, 4 Feb 2020 09:25:20 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FA216B0006; Tue, 4 Feb 2020 09:25:20 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0196.hostedemail.com [216.40.44.196]) by kanga.kvack.org (Postfix) with ESMTP id 0AA396B0008 for ; Tue, 4 Feb 2020 09:25:20 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A4650824805A for ; Tue, 4 Feb 2020 14:25:19 +0000 (UTC) X-FDA: 76452667158.05.basin81_2e7667d4fea04 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,jack@suse.cz,:willy@infradead.org:linux-fsdevel@vger.kernel.org::jack@suse.cz,RULES_HIT:30054:30090,0,RBL:195.135.220.15:@suse.cz:.lbl8.mailshell.net-62.2.6.2 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:31,LUA_SUMMARY:none X-HE-Tag: basin81_2e7667d4fea04 X-Filterd-Recvd-Size: 1609 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Tue, 4 Feb 2020 14:25:19 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 18B33AEE7; Tue, 4 Feb 2020 14:25:18 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 018E71E0D0B; Tue, 4 Feb 2020 15:25:15 +0100 (CET) From: Jan Kara To: Matthew Wilcox Cc: , , Jan Kara Subject: [PATCH 5/8] dax: Use xas_erase() in __dax_invalidate_entry() Date: Tue, 4 Feb 2020 15:25:11 +0100 Message-Id: <20200204142514.15826-6-jack@suse.cz> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20200204142514.15826-1-jack@suse.cz> References: <20200204142514.15826-1-jack@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When truncating DAX entry, we need to clear all the outstanding marks for the entry. Use dax_erase() instead of dax_store(). Signed-off-by: Jan Kara --- fs/dax.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/dax.c b/fs/dax.c index 1f1f0201cad1..848215fcd1aa 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -643,7 +643,7 @@ static int __dax_invalidate_entry(struct address_space *mapping, xas_get_mark(&xas, PAGECACHE_TAG_TOWRITE))) goto out; dax_disassociate_entry(entry, mapping, trunc); - xas_store(&xas, NULL); + xas_erase(&xas); mapping->nrexceptional--; ret = 1; out: From patchwork Tue Feb 4 14:25:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 11364713 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3776D13A4 for ; Tue, 4 Feb 2020 14:25:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1BB3920730 for ; Tue, 4 Feb 2020 14:25:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1BB3920730 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7F0A76B0006; Tue, 4 Feb 2020 09:25:20 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 595DF6B0005; Tue, 4 Feb 2020 09:25:20 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34C146B0005; Tue, 4 Feb 2020 09:25:20 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0120.hostedemail.com [216.40.44.120]) by kanga.kvack.org (Postfix) with ESMTP id EE4926B0006 for ; Tue, 4 Feb 2020 09:25:19 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A30EC180AD802 for ; Tue, 4 Feb 2020 14:25:19 +0000 (UTC) X-FDA: 76452667158.29.vest02_2e7660e89252b X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,jack@suse.cz,:willy@infradead.org:linux-fsdevel@vger.kernel.org::jack@suse.cz,RULES_HIT:30054:30090,0,RBL:195.135.220.15:@suse.cz:.lbl8.mailshell.net-62.2.6.2 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: vest02_2e7660e89252b X-Filterd-Recvd-Size: 1575 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Tue, 4 Feb 2020 14:25:19 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 18C1CAEEC; Tue, 4 Feb 2020 14:25:18 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 060921E0D13; Tue, 4 Feb 2020 15:25:16 +0100 (CET) From: Jan Kara To: Matthew Wilcox Cc: , , Jan Kara Subject: [PATCH 6/8] idr: Use xas_erase() in ida_destroy() Date: Tue, 4 Feb 2020 15:25:12 +0100 Message-Id: <20200204142514.15826-7-jack@suse.cz> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20200204142514.15826-1-jack@suse.cz> References: <20200204142514.15826-1-jack@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Explicitely clear marks (and set XA_MARK_FREE) in ida_destroy() by calling xas_erase() instead of relying on xas_store() on implicitely doing this. Signed-off-by: Jan Kara --- lib/idr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/idr.c b/lib/idr.c index c2cf2c52bbde..fd4877fef06d 100644 --- a/lib/idr.c +++ b/lib/idr.c @@ -543,7 +543,7 @@ void ida_destroy(struct ida *ida) xas_for_each(&xas, bitmap, ULONG_MAX) { if (!xa_is_value(bitmap)) kfree(bitmap); - xas_store(&xas, NULL); + xas_erase(&xas); } xas_unlock_irqrestore(&xas, flags); } From patchwork Tue Feb 4 14:25:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 11364723 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0A8B214B4 for ; Tue, 4 Feb 2020 14:25:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E321221744 for ; Tue, 4 Feb 2020 14:25:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E321221744 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A7A4F6B000D; Tue, 4 Feb 2020 09:25:21 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A2A6B6B000E; Tue, 4 Feb 2020 09:25:21 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F2686B0010; Tue, 4 Feb 2020 09:25:21 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0045.hostedemail.com [216.40.44.45]) by kanga.kvack.org (Postfix) with ESMTP id 661786B000D for ; Tue, 4 Feb 2020 09:25:21 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 077D4181AEF1A for ; Tue, 4 Feb 2020 14:25:21 +0000 (UTC) X-FDA: 76452667242.08.shoes58_2ea7c260b712a X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,jack@suse.cz,:willy@infradead.org:linux-fsdevel@vger.kernel.org::jack@suse.cz,RULES_HIT:30054:30070:30090,0,RBL:195.135.220.15:@suse.cz:.lbl8.mailshell.net-62.2.6.2 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:32,LUA_SUMMARY:none X-HE-Tag: shoes58_2ea7c260b712a X-Filterd-Recvd-Size: 1592 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Tue, 4 Feb 2020 14:25:20 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 67B2DAF32; Tue, 4 Feb 2020 14:25:18 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 0A8ED1E0D14; Tue, 4 Feb 2020 15:25:16 +0100 (CET) From: Jan Kara To: Matthew Wilcox Cc: , , Jan Kara Subject: [PATCH 7/8] mm: Use xas_erase() in collapse_file() Date: Tue, 4 Feb 2020 15:25:13 +0100 Message-Id: <20200204142514.15826-8-jack@suse.cz> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20200204142514.15826-1-jack@suse.cz> References: <20200204142514.15826-1-jack@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When undoing failed collapse of ordinary pages into a huge page, use xas_erase() to explicitly clear any xarray marks that may have been added to entries. Signed-off-by: Jan Kara --- mm/khugepaged.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b679908743cb..13e061581cd0 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1800,7 +1800,7 @@ static void collapse_file(struct mm_struct *mm, break; nr_none--; /* Put holes back where they were */ - xas_store(&xas, NULL); + xas_erase(&xas); continue; } From patchwork Tue Feb 4 14:25:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Kara X-Patchwork-Id: 11364727 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7FB7314B4 for ; Tue, 4 Feb 2020 14:25:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 567EE20730 for ; Tue, 4 Feb 2020 14:25:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 567EE20730 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7E3C16B000E; Tue, 4 Feb 2020 09:25:24 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 76D026B0010; Tue, 4 Feb 2020 09:25:24 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6ABB56B0266; Tue, 4 Feb 2020 09:25:24 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4AE2E6B000E for ; Tue, 4 Feb 2020 09:25:24 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 06682181AC9BF for ; Tue, 4 Feb 2020 14:25:24 +0000 (UTC) X-FDA: 76452667368.08.toy27_2f1b2a395c110 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,jack@suse.cz,:willy@infradead.org:linux-fsdevel@vger.kernel.org::jack@suse.cz,RULES_HIT:30034:30051:30054:30090,0,RBL:195.135.220.15:@suse.cz:.lbl8.mailshell.net-62.2.6.2 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: toy27_2f1b2a395c110 X-Filterd-Recvd-Size: 2518 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Tue, 4 Feb 2020 14:25:23 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id B4E16ACF0; Tue, 4 Feb 2020 14:25:22 +0000 (UTC) Received: by quack2.suse.cz (Postfix, from userid 1000) id 0F8351E0D1A; Tue, 4 Feb 2020 15:25:16 +0100 (CET) From: Jan Kara To: Matthew Wilcox Cc: , , Jan Kara Subject: [PATCH 8/8] xarray: Don't clear marks in xas_store() Date: Tue, 4 Feb 2020 15:25:14 +0100 Message-Id: <20200204142514.15826-9-jack@suse.cz> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20200204142514.15826-1-jack@suse.cz> References: <20200204142514.15826-1-jack@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When storing NULL in xarray, xas_store() has been clearing all marks because it could otherwise confuse xas_for_each_marked(). That is however no longer true and no current user relies on this behavior. Furthermore it seems as a cleaner API to not do clearing behind caller's back in case we store NULL. This provides a nice boost to truncate numbers due to saving unnecessary tag initialization when clearing shadow entries. Sample benchmark showing time to truncate 128 files 1GB each on machine with 64GB of RAM (so about half of entries are shadow entries): AVG STDDEV Vanilla 4.825s 0.036 Patched 4.516s 0.014 So we can see about 6% reduction in overall truncate time. Signed-off-by: Jan Kara --- lib/xarray.c | 9 --------- 1 file changed, 9 deletions(-) diff --git a/lib/xarray.c b/lib/xarray.c index 4e32497c51bd..f165e83652f1 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -799,17 +799,8 @@ void *xas_store(struct xa_state *xas, void *entry) if (xas->xa_sibs) xas_squash_marks(xas); } - if (!entry) - xas_init_marks(xas); for (;;) { - /* - * Must clear the marks before setting the entry to NULL, - * otherwise xas_for_each_marked may find a NULL entry and - * stop early. rcu_assign_pointer contains a release barrier - * so the mark clearing will appear to happen before the - * entry is set to NULL. - */ rcu_assign_pointer(*slot, entry); if (xa_is_node(next) && (!node || node->shift)) xas_free_nodes(xas, xa_to_node(next));