From patchwork Wed Jun 19 17:39:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13704356 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DB201E87B for ; Wed, 19 Jun 2024 17:40:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718818805; cv=none; b=G4GMd8rLAAv6ahAz4KjX6wvdcp8HnQytdH+a/6hdAdszqrIqSZFthuG1aMQEcZ7d9vuM2BPXqXN8WFMgKLQiUVnPkcsKx1EKcivaGqkFfNx6ZAkhQB8t7EyGATSyFwe6IgbIKmX5vjLdUxh03KSpKwEItU+TELeyscSM07IZRUo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718818805; c=relaxed/simple; bh=Kct/wd8r1HaJT4Rkwjc2aXykFnAqUtzCO0o/F62p/W4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OdhomG4FpfdwngrtGmgKFpMITCn0KlXYjX+sGNxhOjaZuuOvQScPmuy1BQ/SrxzRfPjVAhjfE0TnxoyhJelzmMKGqxMQfLk6kUIFkKp5EsnRamK7XjCu5fxsoay7jfnP74f4Z3LgNSgsayUjqFo75KjiHP6AGEjY6XE+ZKJXAH4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gF12xJt+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gF12xJt+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0774BC32786; Wed, 19 Jun 2024 17:40:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718818805; bh=Kct/wd8r1HaJT4Rkwjc2aXykFnAqUtzCO0o/F62p/W4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gF12xJt+7Zww/VKxPC0TSx1ubUCLY+UVnVJf7qz+/J8T92Infi3ZQ3xb1+0FoE7EI /0Eo9L+JO0jhDn/LMs+BwLA8dF0xVIwJfCVwqmT6pnBhHboq5+/pZyFymH8VDeIKa0 lIWxPSLp6tMO0pzUTOzT3UqPntl4UTC/JtedCow0GdiBHaZsmzU2LOF64wAd60sLG7 CHgnFVd6F8BahYIyDVrZUXSRHEUg4dYhM1tqb+ueTLJf8p3vOIsRbyasTRz5pawRi8 wiTk163A3pv1pRwgj2boazK1wLkvJLNt4AMcbZ2iUQ1Z5ZRw+v5aRooLe6Tj28YTyy ETdlB486E/xuw== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever Subject: [RFC PATCH 1/4] nfs/blocklayout: SCSI layout trace points for reservation key reg/unreg Date: Wed, 19 Jun 2024 13:39:31 -0400 Message-ID: <20240619173929.177818-7-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240619173929.177818-6-cel@kernel.org> References: <20240619173929.177818-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6090; i=chuck.lever@oracle.com; h=from:subject; bh=IA+LOlwZoh9GveC01VQSVy969wWvp+xX60t7Vq8LkNc=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmcxfXQjOqO3x2PG1cQ931au3KKXNX5De4+/3Sg hNC9Mlw5CSJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnMX1wAKCRAzarMzb2Z/ l2NREACNMnqIFrGmmTFKqn8dhISN3ry/1bbBSk6y6v10PDd90U59dj0Ptrigvdyc83NFnDqYCO2 xb+Yw7U5s1oOFRWJjTkwOE964xFpTP85FJGtDpNMo8J6VV/I8F0r24VN3RdlnNCbF6GDh04aoU5 JKsSOtSlUk8pNOBK8kljHIih14hiLLWVQOil7XfZQ8mnPGlJL8NVVEovh+KqzDRRLI5gjYhJHsB N7pvlgkEaQF4ipnstKcjV5BYydSO6E2xQ4yWWbAoB2miwvrPiHP03kJN25rQwge2cS5HMWYw9GB bAlIGLrd6chbrgpOwpZfHsazCs7aNk86I5pxlapQOj0D+AYKcxADHWy4a94OK+K1L/Jhd8kqWzN vg6zcNeACP0+WrABusToVhUzrV65CJjOc6FsH2xjue/PF3pPUWwJRkBRmLjs+QYyaw4GetIj0Ky fkEFQxuXgi8buaVDfqnBxJPPHSZ3vg56fLwVEcu43ROXig/QkSqoM8PLlW38tlivhLsGnF1NBsG Sy4rxy/rGdhktbUtEhSh9qQdfmf4W4y0POlWplWNZrHW2nz4AZ2erXVsO3E1Bk87shMQR7XqvqK XhHsk2YzxhdJg4HPSUPxbHzrNJzz6Tw8G65akLknR/7c3Gd8AryP+cz24uoWzADKsGwDzodn/4Q PBzRrp2xNo4VVaQ== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever An administrator cannot take action on these messages, but the reported errors might be helpful for troubleshooting. Transition them to trace points so these events appear in the trace log and can be easily lined up with other traced NFS client operations. Examples: append_writer-6147 [000] 80.247393: bl_pr_key_reg: device=sdb key=0x666dcdabf29514fe append_writer-6147 [000] 80.247842: bl_pr_key_unreg: device=sdb key=0x666dcdabf29514fe umount.nfs4-6172 [002] 84.950409: bl_pr_key_unreg_err: device=sdb key=0x666dcdabf29514fe error=24 Signed-off-by: Chuck Lever --- fs/nfs/blocklayout/dev.c | 33 +++++++++++---------- fs/nfs/nfs4trace.c | 5 ++++ fs/nfs/nfs4trace.h | 62 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+), 15 deletions(-) diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index 519c310c745d..b3828e5ee079 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -10,6 +10,7 @@ #include #include "blocklayout.h" +#include "../nfs4trace.h" #define NFSDBG_FACILITY NFSDBG_PNFS_LD @@ -24,14 +25,17 @@ bl_free_device(struct pnfs_block_dev *dev) kfree(dev->children); } else { if (dev->pr_registered) { - const struct pr_ops *ops = - file_bdev(dev->bdev_file)->bd_disk->fops->pr_ops; + struct block_device *bdev = file_bdev(dev->bdev_file); + const struct pr_ops *ops = bdev->bd_disk->fops->pr_ops; int error; - error = ops->pr_register(file_bdev(dev->bdev_file), - dev->pr_key, 0, false); + error = ops->pr_register(bdev, dev->pr_key, 0, false); if (error) - pr_err("failed to unregister PR key.\n"); + trace_bl_pr_key_unreg_err(bdev->bd_disk->disk_name, + dev->pr_key, error); + else + trace_bl_pr_key_unreg(bdev->bd_disk->disk_name, + dev->pr_key); } if (dev->bdev_file) @@ -327,8 +331,9 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, struct pnfs_block_volume *volumes, int idx, gfp_t gfp_mask) { struct pnfs_block_volume *v = &volumes[idx]; - struct file *bdev_file; + struct block_device *bdev; const struct pr_ops *ops; + struct file *bdev_file; int error; if (!bl_validate_designator(v)) @@ -346,8 +351,9 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, if (IS_ERR(bdev_file)) return PTR_ERR(bdev_file); d->bdev_file = bdev_file; + bdev = file_bdev(bdev_file); - d->len = bdev_nr_bytes(file_bdev(d->bdev_file)); + d->len = bdev_nr_bytes(bdev); d->map = bl_map_simple; d->pr_key = v->scsi.pr_key; @@ -356,23 +362,20 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, goto out_blkdev_put; } - pr_info("pNFS: using block device %s (reservation key 0x%llx)\n", - file_bdev(d->bdev_file)->bd_disk->disk_name, d->pr_key); - - ops = file_bdev(d->bdev_file)->bd_disk->fops->pr_ops; + ops = bdev->bd_disk->fops->pr_ops; if (!ops) { pr_err("pNFS: block device %s does not support reservations.", - file_bdev(d->bdev_file)->bd_disk->disk_name); + bdev->bd_disk->disk_name); error = -EINVAL; goto out_blkdev_put; } - error = ops->pr_register(file_bdev(d->bdev_file), 0, d->pr_key, true); + error = ops->pr_register(bdev, 0, d->pr_key, true); if (error) { - pr_err("pNFS: failed to register key for block device %s.", - file_bdev(d->bdev_file)->bd_disk->disk_name); + trace_bl_pr_key_reg_err(bdev->bd_disk->disk_name, d->pr_key, error); goto out_blkdev_put; } + trace_bl_pr_key_reg(bdev->bd_disk->disk_name, d->pr_key); d->pr_registered = true; return 0; diff --git a/fs/nfs/nfs4trace.c b/fs/nfs/nfs4trace.c index d22c6670f770..74eff35fbc90 100644 --- a/fs/nfs/nfs4trace.c +++ b/fs/nfs/nfs4trace.c @@ -29,5 +29,10 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_read_error); EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_write_error); EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_commit_error); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_reg); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_reg_err); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_unreg); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_unreg_err); + EXPORT_TRACEPOINT_SYMBOL_GPL(fl_getdevinfo); #endif diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h index 4de8780a7c48..2af75b8e018d 100644 --- a/fs/nfs/nfs4trace.h +++ b/fs/nfs/nfs4trace.h @@ -2153,6 +2153,68 @@ TRACE_EVENT(ff_layout_commit_error, ) ); +DECLARE_EVENT_CLASS(pnfs_bl_pr_key_class, + TP_PROTO( + const char *name, + u64 key + ), + TP_ARGS(name, key), + TP_STRUCT__entry( + __string(name, name) + __field(u64, key) + ), + TP_fast_assign( + __assign_str(name); + __entry->key = key; + ), + TP_printk("device=%s key=0x%llx", + __get_str(name), __entry->key + ) +); + +#define DEFINE_NFS4_BLOCK_PRKEY_EVENT(name) \ + DEFINE_EVENT(pnfs_bl_pr_key_class, name, \ + TP_PROTO( \ + const char *name, \ + u64 key \ + ), \ + TP_ARGS(name, key)) +DEFINE_NFS4_BLOCK_PRKEY_EVENT(bl_pr_key_reg); +DEFINE_NFS4_BLOCK_PRKEY_EVENT(bl_pr_key_unreg); + +DECLARE_EVENT_CLASS(pnfs_bl_pr_key_err_class, + TP_PROTO( + const char *name, + u64 key, + int error + ), + TP_ARGS(name, key, error), + TP_STRUCT__entry( + __string(name, name) + __field(u64, key) + __field(int, error) + ), + TP_fast_assign( + __assign_str(name); + __entry->key = key; + __entry->error = error; + ), + TP_printk("device=%s key=0x%llx error=%d", + __get_str(name), __entry->key, __entry->error + ) +); + +#define DEFINE_NFS4_BLOCK_PRKEY_ERR_EVENT(name) \ + DEFINE_EVENT(pnfs_bl_pr_key_err_class, name, \ + TP_PROTO( \ + const char *name, \ + u64 key, \ + int error \ + ), \ + TP_ARGS(name, key, error)) +DEFINE_NFS4_BLOCK_PRKEY_ERR_EVENT(bl_pr_key_reg_err); +DEFINE_NFS4_BLOCK_PRKEY_ERR_EVENT(bl_pr_key_unreg_err); + #ifdef CONFIG_NFS_V4_2 TRACE_DEFINE_ENUM(NFS4_CONTENT_DATA); TRACE_DEFINE_ENUM(NFS4_CONTENT_HOLE); From patchwork Wed Jun 19 17:39:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13704357 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 828AB1E87B for ; Wed, 19 Jun 2024 17:40:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718818810; cv=none; b=RLbo3VqO3mYGv/6GLP3IcczG1zZpIyHemjAAQtsChz4Lj+azrqtaoKNtAtz6RY7xjfMfRDl6eT3d2HMNcYXUOb0f9UYZC7f/zeFcdkFuROwE5BHazMpnChP2OiJFPHNF1A4mk3lS6/nHChtj6BO2mi8Qol1bpSsPfyynlAccfrA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718818810; c=relaxed/simple; bh=tG7rzn/GBGXbKgCKoY1Mye3wCqijmdFwhRvKQNJvujg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i0QEX14kvV1Mi52xksYtVHapZzDMTUwx1HsPJIwpkgjfAC2glr2axtR7ISqmUawjYY1mP9E7hn/N8ksOdCTlflur7cSNa3yDroUKF08NedYqQJFEslrL/L2yNvueRpVtfVIKaiX5GaLGwtZExNlDBaYggLtpYnL3i6IRpFt3mXI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=X+AJsI+E; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="X+AJsI+E" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7FB07C2BBFC; Wed, 19 Jun 2024 17:40:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718818809; bh=tG7rzn/GBGXbKgCKoY1Mye3wCqijmdFwhRvKQNJvujg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=X+AJsI+EgZj3o53kuU8vuaxWrXNYjHKp/7ng+1DXB881S85xbe6ofOgCahDVB9Esr IEvHa+swUnOQUJk6rfYizANzV5UD1Yuo8OM6d1/XyK+psuBlQ+LYlcgmy1Tn+50vmy htmTWI4UjPFMIZuNIPzQVbtZH9l1r7Rh+735hV+eRBkzfKtj3D97fnLfZd9xubf0Bz R7JENWdXuWzdWfZGb8Om7jAqMMb+x3/5ymhJZ0y0nHbiH27gMINsqAHI5jkjoywvEk XIYQjAF/CV1Cs4SxsDWfyRgBwscw1J8UcZqRQ4vFoDyqUhXfUP3EHvabSqAgN0fNkq uO/kzn13m0AmQ== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever Subject: [RFC PATCH 2/4] nfs/blocklayout: Report only when /no/ device is found Date: Wed, 19 Jun 2024 13:39:32 -0400 Message-ID: <20240619173929.177818-8-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240619173929.177818-6-cel@kernel.org> References: <20240619173929.177818-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1693; i=chuck.lever@oracle.com; h=from:subject; bh=/RPBC7X+492DhdOrkWhyByy9G4vje6JR+0B9kohoCOw=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmcxfXIsWHC1AlQRUZNkR+feuLDq0k5xzBiGyyr 2WnGeiRk6CJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnMX1wAKCRAzarMzb2Z/ l3EIEACMoffOm2Y2bDf+1zaVntlAkoH1YpUA3UuE3rX0u4p+lEYzb67wvGUaimOewhsUVJqweIG x9I17R2NpksJRHJRi0Qy1/HtxQXAuA+uyhQt+pw4XX7sI7UFKYOkrT6R4QTFvlrQrgma9YjXdvr 7H/iwbtZBWuvFIC0U79ulWJ20hiLYX3DKX1fX+zqIdAOVkABGOXOZDrH1VCgx5zG4wmIocYWe4e LxdebtNBxvObNGctnp2KKOeea29YzwvACLd8fqGv0N4YInsrxw4/ZYTw+XaKs4qctB88n28zb9I CeWSwHA68sx2SXpgvFf0vUeWg+ziLvcrfQJK6CHDXY2osE7A/X9n+gs2//fR13XqbC1aUutUlNx BP/1kjLJ9HjYx81eETwDhn5qfibOgUG7EDg1Xx3djSaMc8mxcyMBnhC/H+n62qfAFyCBhTylepI MWL4HSbSvQnKyWuzZ5z5dBDmSDPJvm+bNBjaNlEjbvZh5SvRNvkmDMQ3xUAGx78uyg+yZK3iH5x pXA0WzcrZPKwid5NE7RkH6Vbgzdq4WZw6TbYM2mx2PLmHQBQVyuOYaRZqKWUm/4sJobgX+TGoQH 2FWvjU64dNAeqFV68ZQ53Xeqi0eSCzLvQbQORyI3bKbDP6CTsNC3AE+zhTRSsrFjrZE83ylyWto mvYdiXhe7DpcU3g== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever Since f931d8374cad ("nfs/blocklayout: refactor block device opening"), an error is reported when no multi-path device is found. But this isn't a fatal error if the subsequent device open is successful. On systems without multi-path devices, this message always appears whether there is a problem or not. Instead, generate less system journal noise by reporting an error only when both open attempts fail. The new error message is more actionable since it indicates that there is a real configuration issue to be addressed. Signed-off-by: Chuck Lever Reviewed-by: Christoph Hellwig --- fs/nfs/blocklayout/dev.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index b3828e5ee079..356bc967fb5d 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -318,7 +318,7 @@ bl_open_path(struct pnfs_block_volume *v, const char *prefix) bdev_file = bdev_file_open_by_path(devname, BLK_OPEN_READ | BLK_OPEN_WRITE, NULL, NULL); if (IS_ERR(bdev_file)) { - pr_warn("pNFS: failed to open device %s (%ld)\n", + dprintk("failed to open device %s (%ld)\n", devname, PTR_ERR(bdev_file)); } @@ -348,8 +348,11 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, bdev_file = bl_open_path(v, "dm-uuid-mpath-0x"); if (IS_ERR(bdev_file)) bdev_file = bl_open_path(v, "wwn-0x"); - if (IS_ERR(bdev_file)) + if (IS_ERR(bdev_file)) { + pr_err("pNFS: no device found for volume %*phN\n", + v->scsi.designator_len, v->scsi.designator); return PTR_ERR(bdev_file); + } d->bdev_file = bdev_file; bdev = file_bdev(bdev_file); From patchwork Wed Jun 19 17:39:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13704358 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5834F1E87B for ; Wed, 19 Jun 2024 17:40:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718818812; cv=none; b=M1Ya/96CRT01U72yASp62PnEobb14UYlW0O4t186yjcz3LiWnMeVepR7PStTYWNpZefgDFYvhw1vv1BI7iJ4wciBfJLzXazX7EbJDZcR+ZJ3oLWmtsf0AZtCup+oM3ZFP4ogNolrO7Rps3o8UOg/i88CxEWpw6zSzvxRwLuckdU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718818812; c=relaxed/simple; bh=7Cr0kcvmCok79YFMgU73r9NZ7nXhqcCcx2+MstZoL34=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h7O2fUgw4GMK8YU4LLYxvwBebms0xqElIqH494jU7krTuGFSYzjrkaEMUyMHxBS6NGi6lcEy0Q3pfWdDsJ3EYlplgOobfyuh5Dkp29oGPjEaw9HBAkVf9E6PyrndR+iXn/qR4oJzIFutIWrF2/OumjNXnMz//e8Eag9QVdHo/7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=uCYRoBji; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="uCYRoBji" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C91BDC32786; Wed, 19 Jun 2024 17:40:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718818812; bh=7Cr0kcvmCok79YFMgU73r9NZ7nXhqcCcx2+MstZoL34=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uCYRoBjiaElkD4j3zdiENHmRi2Uv8EXAnL+wP9WpWJkh9dOpROfVzibFpPohiikW+ wpAlFM942oKoLixJwEogwop7WmZGv66RqgHcpnrOQvp0nj1J3d2ZsAqor6AC5IE3pl yo1O8ddBAvq1inB7kycOfK6qCixKybxacmDEmk5CJ/Ie1DmBgnR9fHVKcFza1FowYh auovSyDOoB5FgkQ77GmYTkOF64UQDqkSds6rCxcIcQQ9OZE3JLRP86kfRxw9OppHsZ ZsywH5JxJbFSUi8Zsm1p9Ivxno2p/txITEWpxKqY47IlGXJVdSB0ZuvCx8KURob7mx WjoM+AUspI6bA== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever Subject: [RFC PATCH 3/4] nfs/blocklayout: Fix premature PR key unregistration Date: Wed, 19 Jun 2024 13:39:33 -0400 Message-ID: <20240619173929.177818-9-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240619173929.177818-6-cel@kernel.org> References: <20240619173929.177818-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5925; i=chuck.lever@oracle.com; h=from:subject; bh=dHe9H/hu2rXJ7p/feuLdWXoTXyCzRf6rlvqCZYCN73k=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmcxfXS51l0M8Yz9dM4lQ1EfrNWuINzGTV6Kvrc z5K3PZ62saJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnMX1wAKCRAzarMzb2Z/ l8ozD/92bsxP+C6MRkIJsNlroELGvxzspdShy0gUKNQDKiFy/tfC2AXNz39654SIjpTkApcwcr+ REGjyUzw3VAXUvmbeFGikFaIIgv0AAXiVDu2zUK+lgabk7X8RfMycWHoBJXyftSCdjNDorq74Dz jpjKl2Y4nPlFwpOaTpwx+gLxV1FwWqNQQbBBNFFzwRrUqqRHJ4So4Ipf0SW3NhtN7g6p0WiuiRZ DIt7X3OQDbLMGeUItNNYs1zs0Cq9O+RaPF7QgEZDX++Pw5bFr+9DpTk1nK1PjJFa4oiaI4jGaxv SY1vEmTQNbCJDX/Z+8B1UxJdmxNAiEaCq8ZmMRuU9r12CDf0ZpIJ8PqH4itIY/OxejBqgd40QUn AK728XgZF8AtA+dbHj89/rZ6DjA+Atu0wt+LpFW9zK+OKEAtynYO1gOq9FTCEC1TCokVc5ZEZrI KuDBPWClVIkvJaHbCF5UEtbSG0Pto5I0nqFvt8L/4BDh3F0ExIxpl/HbhKUfxwkyV4NJf1cWNZz 9Au3E7FqSjFheRDp46hk8FOHKrTqxyjtcZWG6AbPC4TxdKVrjUOYIxkAheefSTSqSbbjKa2JI0D BWf43OaG/1A+4Vrb/aJ/XpJsMm9i2Kpjyqks3EFZ0BCF67VmroumMaTY9FPD7vG9R88Js1uL7cw PUypTKX6WeJkJTA== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever During generic/069 runs with pNFS SCSI layouts, the NFS client emits the following in the system journal: kernel: pNFS: failed to open device /dev/disk/by-id/dm-uuid-mpath-0x6001405e3366f045b7949eb8e4540b51 (-2) kernel: pNFS: using block device sdb (reservation key 0x666b60901e7b26b3) kernel: pNFS: failed to open device /dev/disk/by-id/dm-uuid-mpath-0x6001405e3366f045b7949eb8e4540b51 (-2) kernel: pNFS: using block device sdb (reservation key 0x666b60901e7b26b3) kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: [sdb] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#16 CDB: Write(10) 2a 00 00 00 00 50 00 00 08 00 kernel: reservation conflict error, dev sdb, sector 80 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: [sdb] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#18 CDB: Write(10) 2a 00 00 00 00 60 00 00 08 00 kernel: sd 6:0:0:1: [sdb] tag#17 CDB: Write(10) 2a 00 00 00 00 58 00 00 08 00 kernel: reservation conflict error, dev sdb, sector 96 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 kernel: reservation conflict error, dev sdb, sector 88 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 systemd[1]: fstests-generic-069.scope: Deactivated successfully. systemd[1]: fstests-generic-069.scope: Consumed 5.092s CPU time. systemd[1]: media-test.mount: Deactivated successfully. systemd[1]: media-scratch.mount: Deactivated successfully. kernel: sd 6:0:0:1: reservation conflict kernel: failed to unregister PR key. This appears to be due to a race. bl_alloc_lseg() calls this: 561 static struct nfs4_deviceid_node * 562 bl_find_get_deviceid(struct nfs_server *server, 563 const struct nfs4_deviceid *id, const struct cred *cred, 564 gfp_t gfp_mask) 565 { 566 struct nfs4_deviceid_node *node; 567 unsigned long start, end; 568 569 retry: 570 node = nfs4_find_get_deviceid(server, id, cred, gfp_mask); 571 if (!node) 572 return ERR_PTR(-ENODEV); nfs4_find_get_deviceid() does a lookup without the spin lock first. If it can't find a matching deviceid, it creates a new device_info (which calls bl_alloc_deviceid_node, and that registers the device's PR key). Then it takes the nfs4_deviceid_lock and looks up the deviceid again. If it finds it this time, bl_find_get_deviceid() frees the spare (new) device_info, which unregisters the PR key for the same device. Any subsequent I/O from this client on that device gets EBADE. The umount later unregisters the device's PR key again. To prevent this problem, register the PR key after the deviceid_node lookup. Signed-off-by: Chuck Lever --- fs/nfs/blocklayout/blocklayout.c | 9 ++++++++- fs/nfs/blocklayout/blocklayout.h | 1 + fs/nfs/blocklayout/dev.c | 29 +++++++++++++++++++++-------- 3 files changed, 30 insertions(+), 9 deletions(-) diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c index 6be13e0ec170..75cc5e50bd37 100644 --- a/fs/nfs/blocklayout/blocklayout.c +++ b/fs/nfs/blocklayout/blocklayout.c @@ -571,8 +571,14 @@ bl_find_get_deviceid(struct nfs_server *server, if (!node) return ERR_PTR(-ENODEV); - if (test_bit(NFS_DEVICEID_UNAVAILABLE, &node->flags) == 0) + if (test_bit(NFS_DEVICEID_UNAVAILABLE, &node->flags) == 0) { + struct pnfs_block_dev *d = + container_of(node, struct pnfs_block_dev, node); + if (d->pr_reg) + if (d->pr_reg(d) < 0) + goto out_put; return node; + } end = jiffies; start = end - PNFS_DEVICE_RETRY_TIMEOUT; @@ -581,6 +587,7 @@ bl_find_get_deviceid(struct nfs_server *server, goto retry; } +out_put: nfs4_put_deviceid_node(node); return ERR_PTR(-ENODEV); } diff --git a/fs/nfs/blocklayout/blocklayout.h b/fs/nfs/blocklayout/blocklayout.h index f1eeb4914199..8aabaf5218b8 100644 --- a/fs/nfs/blocklayout/blocklayout.h +++ b/fs/nfs/blocklayout/blocklayout.h @@ -116,6 +116,7 @@ struct pnfs_block_dev { bool (*map)(struct pnfs_block_dev *dev, u64 offset, struct pnfs_block_dev_map *map); + int (*pr_reg)(struct pnfs_block_dev *dev); }; /* sector_t fields are all in 512-byte sectors */ diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index 356bc967fb5d..3d2401820ef4 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -230,6 +230,26 @@ static bool bl_map_stripe(struct pnfs_block_dev *dev, u64 offset, return true; } +static int bl_register_scsi(struct pnfs_block_dev *d) +{ + struct block_device *bdev = file_bdev(d->bdev_file); + const struct pr_ops *ops = bdev->bd_disk->fops->pr_ops; + int error; + + if (d->pr_registered) + return 0; + + error = ops->pr_register(bdev, 0, d->pr_key, true); + if (error) { + trace_bl_pr_key_reg_err(bdev->bd_disk->disk_name, d->pr_key, error); + return -error; + } + + trace_bl_pr_key_reg(bdev->bd_disk->disk_name, d->pr_key); + d->pr_registered = true; + return 0; +} + static int bl_parse_deviceid(struct nfs_server *server, struct pnfs_block_dev *d, struct pnfs_block_volume *volumes, int idx, gfp_t gfp_mask); @@ -373,14 +393,7 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, goto out_blkdev_put; } - error = ops->pr_register(bdev, 0, d->pr_key, true); - if (error) { - trace_bl_pr_key_reg_err(bdev->bd_disk->disk_name, d->pr_key, error); - goto out_blkdev_put; - } - trace_bl_pr_key_reg(bdev->bd_disk->disk_name, d->pr_key); - - d->pr_registered = true; + d->pr_reg = bl_register_scsi; return 0; out_blkdev_put: From patchwork Wed Jun 19 17:39:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13704359 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AE301E87B for ; Wed, 19 Jun 2024 17:40:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718818815; cv=none; b=dWOSRrVXHsawKynekVgIPo3c9kIw7YOAOP2M/9orBiaf3+eZGm86weFjWCU3geah+IOK1bvEzE2wSyi6oqwx9bhHaMXkaOKdRY+OGTT1AKPU3VoUTcWBW72RbTTNUikLc7vVcWQS3Oi9gpa8b+ODbtl+9FswXoXjGdScmdEIPxQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718818815; c=relaxed/simple; bh=zJ2XQCK0MP3WAtBTnxZaYNqZ+n/5Y3zGhBDNbYIwEn4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YqgxnsTHximj8NDmv7Egm0YcXgr2viJJW8afRO+cDM47+cBt6pdj1d7aVtYsGk5lR4thHX0XlYUd30yKPOAdekKZJHf0MD+YoFR93ZH8sa7NJ9sf0SzfhhApK+ltqOSqX0TMMTlgN7PXGBp1sLQt3aD0u8Kdh+hUwc4zT3L40+0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TQ1NlUMt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TQ1NlUMt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A0E5CC32786; Wed, 19 Jun 2024 17:40:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718818815; bh=zJ2XQCK0MP3WAtBTnxZaYNqZ+n/5Y3zGhBDNbYIwEn4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TQ1NlUMtp1CQFlqLPkvKxd+1m66Ojodlb8VLZ7l79gI4UparijWcSjYZdrUNw3eOL WimgwBzI91pu46F36uGu4U41ldhCpmpBNvnR7VMFZprdv/wSbQWelHDEbKCY/NvMBI WAFENZga9KTfnr0ngELPPLmRk44uOihc0l4r1iickUtVxTp1ghaIez3m+8R6DGmwg9 1zIAXGGkqOAcudOxOln2hQJh1s8u7MUzQWEvUNCdkhuYeoUGE1UWQDsnDG4gQ6ABic o8luBOmz02tMU6wV/8O8ermWhLUKJPVyID9q2CuQDmv4e/WQQZVDKyxw/WUEP+qVlG I+BQYoLi1lvpA== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever Subject: [RFC PATCH 4/4] nfs/blocklayout: Use bulk page allocation APIs Date: Wed, 19 Jun 2024 13:39:34 -0400 Message-ID: <20240619173929.177818-10-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240619173929.177818-6-cel@kernel.org> References: <20240619173929.177818-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1730; i=chuck.lever@oracle.com; h=from:subject; bh=64CPDNNydmmWUps9bRG2EqBzhLBVti4LBhnx7kukaOQ=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmcxfYifVBaxsBLvS44MUM7iy6e1jrCUVyxCgw8 ZVuIdrS49aJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnMX2AAKCRAzarMzb2Z/ l3NWD/9tVSTni7pWmae6kLIDJ3XRz1pA4jTYI1tcx4ziYC64/jluqvzKqShzu7n7IbcXOyEQZ+s VLU0o8HMojix9UsZW+ffFve1bgyFckqdzpAJmA/fyjfIN1O9A7TUh6VjBZVHIGyMFCcTDJwOTd7 5cUzoNAjuJalYAV980G70xP5XOwouZcdjtxa/kjZOHxwOIAor5jxwNk7EBiyC1BRBX6TntidXRt lEQXOJVIWlD1QOK4Ryvm4tnMGqZeKTQ5dPZFP6GRzYNiBi5tAl3gcl1L/7ENecuVKCLXR0XJlK8 YHn7/N7NoHsbhjcU879Hv7xHVjgt/fhmFyrNdQhZ/4R1wNUFR3jk4O9YJhq9jywBSed9IX2BTRq LDc68NcTX+T2A43C3k7nv7PMuhK0a5XDZjJl0WprzHXiQTMs2JqgUSH5tT4KmNGZb8AtfLoPOKh FdljbKpQGLkSluR+1SOMaGHtpIlZ2B1CZMw2BfXBi+P/FsqfLPyNflwo3DoSxxipbKleZ1YiuJx XIn4rWwhwpHtUKwhRysRmlJZ4EEeT6A9VlxC9NPv8aaIa80m0ZMUilplUjGafiuJJWlWUOcJYR5 EpmOKYbmgjQ2OduqCYIZBfvbWbsNpSiyWGPTY4IBP/fUDpDYal89SL/DxJ7FwAAY4sIH8JxVYXh D75fDPzMSnJsJzQ== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever nfs4_get_device_info() frequently requests more than a few pages when provisioning a nfs4_deviceid_node object. Make this more efficient by using alloc_pages_bulk_array(). This API is known to be several times faster than an open-coded loop around alloc_page(). release_pages() is folio-enabled so it is also more efficient than repeatedly invoking __free_pages(). Signed-off-by: Chuck Lever --- fs/nfs/pnfs_dev.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/fs/nfs/pnfs_dev.c b/fs/nfs/pnfs_dev.c index 178001c90156..26a78d69acab 100644 --- a/fs/nfs/pnfs_dev.c +++ b/fs/nfs/pnfs_dev.c @@ -101,9 +101,8 @@ nfs4_get_device_info(struct nfs_server *server, struct nfs4_deviceid_node *d = NULL; struct pnfs_device *pdev = NULL; struct page **pages = NULL; + int rc, i, max_pages; u32 max_resp_sz; - int max_pages; - int rc, i; /* * Use the session max response size as the basis for setting @@ -125,11 +124,9 @@ nfs4_get_device_info(struct nfs_server *server, if (!pages) goto out_free_pdev; - for (i = 0; i < max_pages; i++) { - pages[i] = alloc_page(gfp_flags); - if (!pages[i]) - goto out_free_pages; - } + i = alloc_pages_bulk_array(GFP_KERNEL, max_pages, pages); + if (i != max_pages) + goto out_free_pages; memcpy(&pdev->dev_id, dev_id, sizeof(*dev_id)); pdev->layout_type = server->pnfs_curr_ld->id; @@ -154,8 +151,8 @@ nfs4_get_device_info(struct nfs_server *server, set_bit(NFS_DEVICEID_NOCACHE, &d->flags); out_free_pages: - while (--i >= 0) - __free_page(pages[i]); + if (i) + release_pages(pages, i); kfree(pages); out_free_pdev: kfree(pdev);