From patchwork Fri Sep 30 19:15:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Layton X-Patchwork-Id: 12995910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5ADEAC43219 for ; Fri, 30 Sep 2022 19:15:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229539AbiI3TP6 (ORCPT ); Fri, 30 Sep 2022 15:15:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229531AbiI3TP5 (ORCPT ); Fri, 30 Sep 2022 15:15:57 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B6C5153133 for ; Fri, 30 Sep 2022 12:15:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 60882CE2664 for ; Fri, 30 Sep 2022 19:15:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5690EC433C1; Fri, 30 Sep 2022 19:15:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1664565352; bh=sVL8/mHcxst89Ak9xV8H6IlaBJfZIWW4Jyd0gQCnPVg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AJSYprMjTf4HlST0naPQa46VuR7cWPsk6bjEO7cdzjegMxvM3Diz21KyDrkKUAu1W ZdBra1juTit00nu1341oBfavQuW3vC+6+K6QS3LT87XDqpNX2TnVUT7fUaj4WnfRfz CWzSpcKgEdjX1jfn8x0O+Ef3mIyIHQDoFes267rZmIUMjIMWO86NWUVtFYCUvHADMS nWR6r2y2LoFpQqgidbSuuAhsXo/IckouJcSOn/dCoGEAfU/aKROmJgACcPku5bbzaQ mZEcglrrosWT8CQwmsyhhqhdCpnMVP3i7/Rp+HjOy3m9teeY6DumdXLnUFgGvsFaaC /l/n14+Y13QDA== From: Jeff Layton To: chuck.lever@oracle.com Cc: linux-nfs@vger.kernel.org Subject: [PATCH 1/3] nfsd: nfsd_do_file_acquire should hold rcu_read_lock while getting refs Date: Fri, 30 Sep 2022 15:15:48 -0400 Message-Id: <20220930191550.172087-2-jlayton@kernel.org> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220930191550.172087-1-jlayton@kernel.org> References: <20220930191550.172087-1-jlayton@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org nfsd_file is RCU-freed, so it's possible that one could be found that's in the process of being freed and the memory recycled. Ensure we hold the rcu_read_lock while attempting to get a reference on the object. Signed-off-by: Jeff Layton --- fs/nfsd/filecache.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index d5c57360b418..6237715bd23e 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -1077,10 +1077,12 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp, retry: /* Avoid allocation if the item is already in cache */ + rcu_read_lock(); nf = rhashtable_lookup_fast(&nfsd_file_rhash_tbl, &key, nfsd_file_rhash_params); if (nf) nf = nfsd_file_get(nf); + rcu_read_unlock(); if (nf) goto wait_for_construction; @@ -1090,16 +1092,21 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp, goto out_status; } + rcu_read_lock(); nf = rhashtable_lookup_get_insert_key(&nfsd_file_rhash_tbl, &key, &new->nf_rhash, nfsd_file_rhash_params); if (!nf) { + rcu_read_unlock(); nf = new; goto open_file; } - if (IS_ERR(nf)) + if (IS_ERR(nf)) { + rcu_read_unlock(); goto insert_err; + } nf = nfsd_file_get(nf); + rcu_read_unlock(); if (nf == NULL) { nf = new; goto open_file; From patchwork Fri Sep 30 19:15:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Layton X-Patchwork-Id: 12995907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 096DCC433F5 for ; Fri, 30 Sep 2022 19:15:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229613AbiI3TP4 (ORCPT ); Fri, 30 Sep 2022 15:15:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229531AbiI3TPz (ORCPT ); Fri, 30 Sep 2022 15:15:55 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75D72153133 for ; Fri, 30 Sep 2022 12:15:54 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C19246242D for ; Fri, 30 Sep 2022 19:15:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DAD05C433B5; Fri, 30 Sep 2022 19:15:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1664565353; bh=olAXaLNvcVyHbeX5iwRpk3Tf0FgMvTKXQEEez5T0iH8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VHGkT3e4MSL4tkFF7eJdpaVwP89B10ToVRWMCNS+0TXyLWGQ3Nit/VZ/y2bp/qhPR zT7Kl9UQs+YOHD00DDGpRp5nPJZtWYug3Cc3+rkhtjzUuiiEahFCNC0gvVu9dfLXGG AZb+d9PPxuZ0dyC6fTM0LcXxKiKn6Q/UZkYCsmLcUEEB6nqa8cgywjevnBuA78cPsY hnIgNjsFg3nnXiD4e45jzLpUQm4pM0r71uIBPvSwbMP/0DRQETxoPivd3wne267CGE x565GBRXvD+DQ834Yb7J2YyQ6OJdcWzhIFCBIZFtF8Nu/S0MQE4CUsssX+q2S08kgn qs/usv7k7FAEA== From: Jeff Layton To: chuck.lever@oracle.com Cc: linux-nfs@vger.kernel.org Subject: [PATCH 2/3] nfsd: fix potential race in nfsd_file_close Date: Fri, 30 Sep 2022 15:15:49 -0400 Message-Id: <20220930191550.172087-3-jlayton@kernel.org> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220930191550.172087-1-jlayton@kernel.org> References: <20220930191550.172087-1-jlayton@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Once we call nfsd_file_put, there is no guarantee that "nf" can still be safely accessed. That may have been the last reference. Change the code to instead check for whether nf_ref is 2 and then unhash it and put the reference if we're successful. We might occasionally race with another lookup and end up unhashing it when it probably shouldn't have been, but that should hopefully be rare and will just result in the competing lookup having to create a new nfsd_file. Signed-off-by: Jeff Layton --- fs/nfsd/filecache.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 6237715bd23e..58f4d9267f4a 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -461,12 +461,14 @@ nfsd_file_put(struct nfsd_file *nf) */ void nfsd_file_close(struct nfsd_file *nf) { - nfsd_file_put(nf); - if (refcount_dec_if_one(&nf->nf_ref)) { - nfsd_file_unhash(nf); - nfsd_file_lru_remove(nf); - nfsd_file_free(nf); + /* One for the reference being put, and one for the hash */ + if (refcount_read(&nf->nf_ref) == 2) { + if (nfsd_file_unhash(nf)) + nfsd_file_put_noref(nf); } + /* put the ref for the stateid */ + nfsd_file_put(nf); + } struct nfsd_file * From patchwork Fri Sep 30 19:15:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Layton X-Patchwork-Id: 12995908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7752CC433FE for ; Fri, 30 Sep 2022 19:15:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230378AbiI3TP5 (ORCPT ); Fri, 30 Sep 2022 15:15:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229594AbiI3TPz (ORCPT ); Fri, 30 Sep 2022 15:15:55 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3FBD156FA4 for ; Fri, 30 Sep 2022 12:15:54 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4C70062402 for ; Fri, 30 Sep 2022 19:15:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6AFF0C433D7; Fri, 30 Sep 2022 19:15:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1664565353; bh=mmEhW4HUdcoctlbmIXqvjeTkFOe307mvMkIbIf+PRpE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ts4P59ACseCen4g68pQzViJoK0hzU87hWm9ORtgv0s0CMq25h7aFXTHWKc2G+XUxI hwWVgr3r3uDMIHZ4lJSxWFPwSj9VLTAtCXQRNRIPRxwWOvB2KIGW2yPxWETiZ329pD 4pSjiQdl5UN3lFpVZnCCeke/x9sANQ96k+DDTsWYR9pg8r/YI6qOCMwfjbC9eqsJZt 2DzNmK+gBJRbgNo7hS3Se4m8+mil++3rF0LkGl7o8Dd8/8VM59NG2NWQpmRKL1PVzg PHPUsl9ui8tb2m056zskoy232bfvj+NUkj/w2SrohvdRdVmMYWKyHvyabUscw0gNJK 52e25Jdunkjng== From: Jeff Layton To: chuck.lever@oracle.com Cc: linux-nfs@vger.kernel.org Subject: [PATCH 3/3] nfsd: fix nfsd_file_unhash_and_dispose Date: Fri, 30 Sep 2022 15:15:50 -0400 Message-Id: <20220930191550.172087-4-jlayton@kernel.org> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220930191550.172087-1-jlayton@kernel.org> References: <20220930191550.172087-1-jlayton@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org This function is called two reasons: We're either shutting down and purging the filecache, or we've gotten a notification about a file delete, so we want to go ahead and unhash it so that it'll get cleaned up when we close. We're either walking the hashtable or doing a lookup in it and we don't take a reference in either case. What we want to do in both cases is to try and unhash the object and put it on the dispose list if that was successful. If it's no longer hashed, then we don't want to touch it, with the assumption being that something else is already cleaning up the sentinel reference. Instead of trying to selectively decrement the refcount in this function, just unhash it, and if that was successful, move it to the dispose list. Then, the disposal routine will just clean that up as usual. Also, just make this a void function, drop the WARN_ON_ONCE, and the comments about deadlocking since the nature of the purported deadlock is no longer clear. Signed-off-by: Jeff Layton --- fs/nfsd/filecache.c | 32 ++++++-------------------------- 1 file changed, 6 insertions(+), 26 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 58f4d9267f4a..16bd71a3894e 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -408,19 +408,14 @@ nfsd_file_unhash(struct nfsd_file *nf) /* * Return true if the file was unhashed. */ -static bool +static void nfsd_file_unhash_and_dispose(struct nfsd_file *nf, struct list_head *dispose) { trace_nfsd_file_unhash_and_dispose(nf); - if (!nfsd_file_unhash(nf)) - return false; - /* keep final reference for nfsd_file_lru_dispose */ - if (refcount_dec_not_one(&nf->nf_ref)) - return true; - - nfsd_file_lru_remove(nf); - list_add(&nf->nf_lru, dispose); - return true; + if (nfsd_file_unhash(nf)) { + nfsd_file_lru_remove(nf); + list_add(&nf->nf_lru, dispose); + } } static void @@ -564,8 +559,6 @@ nfsd_file_dispose_list_delayed(struct list_head *dispose) * @lock: LRU list lock (unused) * @arg: dispose list * - * Note this can deadlock with nfsd_file_cache_purge. - * * Return values: * %LRU_REMOVED: @item was removed from the LRU * %LRU_ROTATE: @item is to be moved to the LRU tail @@ -750,8 +743,6 @@ nfsd_file_close_inode(struct inode *inode) * * Walk the LRU list and close any entries that have not been used since * the last scan. - * - * Note this can deadlock with nfsd_file_cache_purge. */ static void nfsd_file_delayed_close(struct work_struct *work) @@ -893,16 +884,12 @@ nfsd_file_cache_init(void) goto out; } -/* - * Note this can deadlock with nfsd_file_lru_cb. - */ static void __nfsd_file_cache_purge(struct net *net) { struct rhashtable_iter iter; struct nfsd_file *nf; LIST_HEAD(dispose); - bool del; rhashtable_walk_enter(&nfsd_file_rhash_tbl, &iter); do { @@ -912,14 +899,7 @@ __nfsd_file_cache_purge(struct net *net) while (!IS_ERR_OR_NULL(nf)) { if (net && nf->nf_net != net) continue; - del = nfsd_file_unhash_and_dispose(nf, &dispose); - - /* - * Deadlock detected! Something marked this entry as - * unhased, but hasn't removed it from the hash list. - */ - WARN_ON_ONCE(!del); - + nfsd_file_unhash_and_dispose(nf, &dispose); nf = rhashtable_walk_next(&iter); }