From patchwork Tue Jun 25 20:02:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13712004 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC05B17D88A for ; Tue, 25 Jun 2024 20:02:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719345736; cv=none; b=AUcrw7ibCeNPuecXdikAYnbda05LNFo1Tu1zLe8xXL2Zeq5Ukb5eYiicZsYs+FCz3dRVk9vV/i9KF9j6jwToxvmUQY2gBEuYT1k1ZKAGPdsnTbzaAu+iHk32GTu5w9G/+C31E9gZKOFVtYg0mySy4pbPkh9K+8s1FvyEUzWkDfw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719345736; c=relaxed/simple; bh=87tOrnHzg8Y5Vy1Jc3JNDHlpcKVtZ9nE6wxT+vUhn00=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gPqbR2QlGajeiiSX8vQOL4n3Hh/+wJ6Ig1Yx46javqfyE0+mrI3bUIkzFlVX2Oo80h5rV0ndHQjP7OqDcSMqnfAI+nJD2Fdmq4+gXwHa/kXIwXTYzxwgvKpjGFeZxtfT1OrPuFus1XXX9b6WemBY9j91mvhM0vE592l3qQuBYac= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lWZt6wS9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lWZt6wS9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 29123C32781; Tue, 25 Jun 2024 20:02:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719345736; bh=87tOrnHzg8Y5Vy1Jc3JNDHlpcKVtZ9nE6wxT+vUhn00=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lWZt6wS9zX8AvFS5Lv4LCXKdZL1DLKauYgtfSKQhtu8h+5mKyPXJy5udikJUTQP12 mdKi3jxoqhhuODFInCrpYj6+T++iPDcbB+9h0B+uvyHiY+QtJDIu2hVlqfl7dg3IdG AD/Eng1uTyRA9iUW9F9a80dXep4vbpk0tn5tM/DT4swAtL85xf0YtO1r7nPDuiqiEV eH53gzvcsqPFZuvFiGHLmTvisXM9dek588OJss1tWhPyjHPJ89aIJEbHZeefqpKSmh UBnlHIOPkX592aENTEmySZqQWnA0iW4eVtl7v+OQJK/KaVZXwkFF8jgHNCtirgl55E hmEXMQ7L0pqgA== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever Subject: [PATCH v3 1/3] nfs/blocklayout: Fix premature PR key unregistration Date: Tue, 25 Jun 2024 16:02:06 -0400 Message-ID: <20240625200204.276770-6-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240625200204.276770-5-cel@kernel.org> References: <20240625200204.276770-5-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9397; i=chuck.lever@oracle.com; h=from:subject; bh=qJ3IwM8sZzntNZUTMYpHV59s+1ZH+Ys+1D3DUW/Ik7A=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmeyJCnoIp8tn3fL27E0Uor36SDnPBUD70UBcXO lBCn0L3OJyJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnsiQgAKCRAzarMzb2Z/ l730D/9FG2VTicLV2LX/cXMtM0j8nAioNdYiDAOPH0Gn7/KIzFHIbwVxV/VT4QYnqEJaxl5VjMc T/KTTX0lqn6NieIzxASw8ok16rkqTgW5tE3mhz4j6whuKTeEov45fBmIEMelXWnVOvuFayddLG4 tt9VYasdIkLrygJCI1zDYcK4KRl8E+As2uQjwSY3sxScfdkZ+YHYCei604o1C83tzQ9AllT/HLP qQeR2uzUyn7NMxCOxEuQqZoFwzWhzOHoMztfjYGwdDwQ2Ntg/hFDBGJ4xkIqvoofLGTAs4Ix8P2 LWjG8k0lxtZwcTEAKoVe+g4coOgbTNFYomV3YdxWgxs2CVSPEnRvXGqpPPTgIvHLBq4rnS6Sqbe kVLa7MBKQhTreZGpzc3YJnRROhRwBvGhXofYBn2tPFJnFu+G84hC67+h/vowAE5oXeG963lYkyG iwSYXTrRDP+OOn363ahXNUzT4HLw8ajTeD4UGK75VBHvvCBFyw4wfGj/L/mqVi73S0acojpKAWy ygMo7cIfdOsopSAP+094tGsvaxDYN/bY5Amr8Gkq9kWeG7fNWvvVy/7j83+iPKWRo/BDrmCj/D7 cqd6otZ0HqcKWRE3vFVCl2+o2WiRw0cWscm6cFOw/Uc/8XBmFLn5VNxCDBRyK1kse3KQZtafUMa 3N0Mhdj1Cll8vbA== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever During generic/069 runs with pNFS SCSI layouts, the NFS client emits the following in the system journal: kernel: pNFS: failed to open device /dev/disk/by-id/dm-uuid-mpath-0x6001405e3366f045b7949eb8e4540b51 (-2) kernel: pNFS: using block device sdb (reservation key 0x666b60901e7b26b3) kernel: pNFS: failed to open device /dev/disk/by-id/dm-uuid-mpath-0x6001405e3366f045b7949eb8e4540b51 (-2) kernel: pNFS: using block device sdb (reservation key 0x666b60901e7b26b3) kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: [sdb] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#16 CDB: Write(10) 2a 00 00 00 00 50 00 00 08 00 kernel: reservation conflict error, dev sdb, sector 80 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: [sdb] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#18 CDB: Write(10) 2a 00 00 00 00 60 00 00 08 00 kernel: sd 6:0:0:1: [sdb] tag#17 CDB: Write(10) 2a 00 00 00 00 58 00 00 08 00 kernel: reservation conflict error, dev sdb, sector 96 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 kernel: reservation conflict error, dev sdb, sector 88 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 systemd[1]: fstests-generic-069.scope: Deactivated successfully. systemd[1]: fstests-generic-069.scope: Consumed 5.092s CPU time. systemd[1]: media-test.mount: Deactivated successfully. systemd[1]: media-scratch.mount: Deactivated successfully. kernel: sd 6:0:0:1: reservation conflict kernel: failed to unregister PR key. This appears to be due to a race. bl_alloc_lseg() calls this: 561 static struct nfs4_deviceid_node * 562 bl_find_get_deviceid(struct nfs_server *server, 563 const struct nfs4_deviceid *id, const struct cred *cred, 564 gfp_t gfp_mask) 565 { 566 struct nfs4_deviceid_node *node; 567 unsigned long start, end; 568 569 retry: 570 node = nfs4_find_get_deviceid(server, id, cred, gfp_mask); 571 if (!node) 572 return ERR_PTR(-ENODEV); nfs4_find_get_deviceid() does a lookup without the spin lock first. If it can't find a matching deviceid, it creates a new device_info (which calls bl_alloc_deviceid_node, and that registers the device's PR key). Then it takes the nfs4_deviceid_lock and looks up the deviceid again. If it finds it this time, bl_find_get_deviceid() frees the spare (new) device_info, which unregisters the PR key for the same device. Any subsequent I/O from this client on that device gets EBADE. The umount later unregisters the device's PR key again. To prevent this problem, register the PR key after the deviceid_node lookup. Signed-off-by: Christoph Hellwig Signed-off-by: Chuck Lever Reviewed-by: Christoph Hellwig Reviewed-by: Benjamin Coddington --- fs/nfs/blocklayout/blocklayout.c | 25 +++++---- fs/nfs/blocklayout/blocklayout.h | 9 +++- fs/nfs/blocklayout/dev.c | 91 ++++++++++++++++++++++++-------- 3 files changed, 94 insertions(+), 31 deletions(-) diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c index 6be13e0ec170..0becdec12970 100644 --- a/fs/nfs/blocklayout/blocklayout.c +++ b/fs/nfs/blocklayout/blocklayout.c @@ -564,25 +564,32 @@ bl_find_get_deviceid(struct nfs_server *server, gfp_t gfp_mask) { struct nfs4_deviceid_node *node; - unsigned long start, end; + int err = -ENODEV; retry: node = nfs4_find_get_deviceid(server, id, cred, gfp_mask); if (!node) return ERR_PTR(-ENODEV); - if (test_bit(NFS_DEVICEID_UNAVAILABLE, &node->flags) == 0) - return node; + if (test_bit(NFS_DEVICEID_UNAVAILABLE, &node->flags)) { + unsigned long end = jiffies; + unsigned long start = end - PNFS_DEVICE_RETRY_TIMEOUT; - end = jiffies; - start = end - PNFS_DEVICE_RETRY_TIMEOUT; - if (!time_in_range(node->timestamp_unavailable, start, end)) { - nfs4_delete_deviceid(node->ld, node->nfs_client, id); - goto retry; + if (!time_in_range(node->timestamp_unavailable, start, end)) { + nfs4_delete_deviceid(node->ld, node->nfs_client, id); + goto retry; + } + goto out_put; } + if (!bl_register_dev(container_of(node, struct pnfs_block_dev, node))) + goto out_put; + + return node; + +out_put: nfs4_put_deviceid_node(node); - return ERR_PTR(-ENODEV); + return ERR_PTR(err); } static int diff --git a/fs/nfs/blocklayout/blocklayout.h b/fs/nfs/blocklayout/blocklayout.h index f1eeb4914199..6da40ca19570 100644 --- a/fs/nfs/blocklayout/blocklayout.h +++ b/fs/nfs/blocklayout/blocklayout.h @@ -104,20 +104,26 @@ struct pnfs_block_dev { u64 start; u64 len; + enum pnfs_block_volume_type type; u32 nr_children; struct pnfs_block_dev *children; u64 chunk_size; struct file *bdev_file; u64 disk_offset; + unsigned long flags; u64 pr_key; - bool pr_registered; bool (*map)(struct pnfs_block_dev *dev, u64 offset, struct pnfs_block_dev_map *map); }; +/* pnfs_block_dev flag bits */ +enum { + PNFS_BDEV_REGISTERED = 0, +}; + /* sector_t fields are all in 512-byte sectors */ struct pnfs_block_extent { union { @@ -172,6 +178,7 @@ struct bl_msg_hdr { #define BL_DEVICE_REQUEST_ERR 0x2 /* User level process fails */ /* dev.c */ +bool bl_register_dev(struct pnfs_block_dev *d); struct nfs4_deviceid_node *bl_alloc_deviceid_node(struct nfs_server *server, struct pnfs_device *pdev, gfp_t gfp_mask); void bl_free_deviceid_node(struct nfs4_deviceid_node *d); diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index 519c310c745d..fc4eeb7bbf05 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -13,9 +13,75 @@ #define NFSDBG_FACILITY NFSDBG_PNFS_LD +static void bl_unregister_scsi(struct pnfs_block_dev *dev) +{ + struct block_device *bdev = file_bdev(dev->bdev_file); + const struct pr_ops *ops = bdev->bd_disk->fops->pr_ops; + + if (!test_and_clear_bit(PNFS_BDEV_REGISTERED, &dev->flags)) + return; + + if (ops->pr_register(bdev, dev->pr_key, 0, false)) + pr_err("failed to unregister PR key.\n"); +} + +static bool bl_register_scsi(struct pnfs_block_dev *dev) +{ + struct block_device *bdev = file_bdev(dev->bdev_file); + const struct pr_ops *ops = bdev->bd_disk->fops->pr_ops; + int status; + + if (test_and_set_bit(PNFS_BDEV_REGISTERED, &dev->flags)) + return true; + + status = ops->pr_register(bdev, 0, dev->pr_key, true); + if (status) { + pr_err("pNFS: failed to register key for block device %s.", + bdev->bd_disk->disk_name); + return false; + } + return true; +} + +static void bl_unregister_dev(struct pnfs_block_dev *dev) +{ + u32 i; + + if (dev->nr_children) { + for (i = 0; i < dev->nr_children; i++) + bl_unregister_dev(&dev->children[i]); + return; + } + + if (dev->type == PNFS_BLOCK_VOLUME_SCSI) + bl_unregister_scsi(dev); +} + +bool bl_register_dev(struct pnfs_block_dev *dev) +{ + u32 i; + + if (dev->nr_children) { + for (i = 0; i < dev->nr_children; i++) { + if (!bl_register_dev(&dev->children[i])) { + while (i > 0) + bl_unregister_dev(&dev->children[--i]); + return false; + } + } + return true; + } + + if (dev->type == PNFS_BLOCK_VOLUME_SCSI) + return bl_register_scsi(dev); + return true; +} + static void bl_free_device(struct pnfs_block_dev *dev) { + bl_unregister_dev(dev); + if (dev->nr_children) { int i; @@ -23,17 +89,6 @@ bl_free_device(struct pnfs_block_dev *dev) bl_free_device(&dev->children[i]); kfree(dev->children); } else { - if (dev->pr_registered) { - const struct pr_ops *ops = - file_bdev(dev->bdev_file)->bd_disk->fops->pr_ops; - int error; - - error = ops->pr_register(file_bdev(dev->bdev_file), - dev->pr_key, 0, false); - if (error) - pr_err("failed to unregister PR key.\n"); - } - if (dev->bdev_file) fput(dev->bdev_file); } @@ -367,14 +422,6 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, goto out_blkdev_put; } - error = ops->pr_register(file_bdev(d->bdev_file), 0, d->pr_key, true); - if (error) { - pr_err("pNFS: failed to register key for block device %s.", - file_bdev(d->bdev_file)->bd_disk->disk_name); - goto out_blkdev_put; - } - - d->pr_registered = true; return 0; out_blkdev_put: @@ -460,7 +507,9 @@ static int bl_parse_deviceid(struct nfs_server *server, struct pnfs_block_dev *d, struct pnfs_block_volume *volumes, int idx, gfp_t gfp_mask) { - switch (volumes[idx].type) { + d->type = volumes[idx].type; + + switch (d->type) { case PNFS_BLOCK_VOLUME_SIMPLE: return bl_parse_simple(server, d, volumes, idx, gfp_mask); case PNFS_BLOCK_VOLUME_SLICE: @@ -472,7 +521,7 @@ bl_parse_deviceid(struct nfs_server *server, struct pnfs_block_dev *d, case PNFS_BLOCK_VOLUME_SCSI: return bl_parse_scsi(server, d, volumes, idx, gfp_mask); default: - dprintk("unsupported volume type: %d\n", volumes[idx].type); + dprintk("unsupported volume type: %d\n", d->type); return -EIO; } } From patchwork Tue Jun 25 20:02:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13712005 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22C0017D357 for ; Tue, 25 Jun 2024 20:02:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719345739; cv=none; b=n185Y0MH+gvzXDlF5xyfu4+Q2eC/xoxzw19fd8zgKfPIdbtOz85pduTMXLYfn50dSD0+B+Z8Kcqbvu3kExQN2DgLFrR2bNxIOcnh3IDZ/4x4kshnNnfxYyaONT+S4uAA8HF+Hv5FWFPrwz4YlqeVqjRHa3ZvnASIZZif3pR/sMs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719345739; c=relaxed/simple; bh=qVZsrmhLADFrwXeGquzGJpCWMKhlBsnWs7ue5Mux2s4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uhmmx4dfDIgNVWFXxCaHfDkG3Td85K3pLPkov7ZM/aARLg2T1JE3AsoW2248BSor8FqUc4j26OVZAqFsk0DEB7xfb5Lit1Y94ewnlfm/ga0Q2Y5jTSHW21/V+N/bvNCLut/qmpYsW1f+sxE0XsnoiN0+UkEZrW7AhD5qIY8Ur/g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DENlNKhx; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DENlNKhx" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4C6A6C32782; Tue, 25 Jun 2024 20:02:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719345738; bh=qVZsrmhLADFrwXeGquzGJpCWMKhlBsnWs7ue5Mux2s4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DENlNKhxks6mbIPbkwzS5cucgBuEEwIW/otLAbXg6KnpD1frVd2iJvCkAh5eklNHt vQyZJWs65yzG4KmwKJnlX2tSNqj1edh4TLkoA6sv/lA/911I+yHMRct82Pou3ElaVU CSYW8VvCHi49xrRqFCCaclhxfUOm8b/0AkUuYADAdy7R0N80LZbwAKh5BRY2I1nAZR 6JSTCUlvFeIrzxuNmal8CPtqnTIYtdnJH4RX77+C195UeBVa3USLqfo+iEwRetKSRY Xtw5uG5nCZCTJQcCvH7sdhv+Y4C9yO7tsTkIFnqZ3a9mGJH/9QuoRYx3AQ4W1KT84/ N8eWMPX/LnZlQ== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever , Benjamin Coddington Subject: [PATCH v3 2/3] nfs/blocklayout: Report only when /no/ device is found Date: Tue, 25 Jun 2024 16:02:07 -0400 Message-ID: <20240625200204.276770-7-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240625200204.276770-5-cel@kernel.org> References: <20240625200204.276770-5-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1822; i=chuck.lever@oracle.com; h=from:subject; bh=zehYuD43oBNIoiNM4I3Q5BqGsVBXQrmexrDvvzgyFAA=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmeyJCBuVh6FeSZ6txr7WxlkR1hxPjN5JR6xPjZ NcNDjaubjGJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnsiQgAKCRAzarMzb2Z/ lx8JD/wNPi8APv/Mt1C7O70rluI3Vh21WFSQIcEFVUQhxkzkppwAjbO3vfq29+vjuzYDYI6AdJe svrblBOO0p9eQx8GlgBCbUhGgAKnJCjGWj+HoFtVO2GyeSfMm+p3ZrSfAMBy8+rBFDfsfWIBsN0 /kx3VchzL6Ov8ZcbBrb+P8JrzcnzbBDDONxt6Hyhi49H0Nuk8iH1CAO4dwlIaGOsT5W2Hw+aurf DeaGfvbhE2WQ3y1/fODkwH595a3VLxelfT15KT7wZHNXZwTYuVJ4+uEmBPGV22BfhIzZhVRdaw2 p7rM9lWxCw0cddhuYN5E9asygqW1sJ43xT3WONfdD6121w5g3aczZjUQDWxluz8JWag/Q5Qrrf0 ekJlW0XV41wVHox2rCgjX1oLLPAO3yvfC+rGGyQAiUZWdvK4oJxmCj0HHANz1lIXQie988+1g2d yb5L/UMKa+k5vcRHLJx56gu5lAgN3JZFBeXC7tc/dT5BOBfd6OIGpzypuWf5BITtdet4aw3lCw9 t/CAG4RwpIE1Jf0yDTqiZ+BHrEG7ziJFt3Sna45c1JqXH7b0muagtys9T2lGTHd0jLHaQh/BOGv zoS2s51ymVzuuSXq2Kb1SgVGkiOdBGuJRYXqrbOImPMNd5h89mQxxlJZQU23RMWEMlyST/tvdDq NKWp1Gfz/BHxB2w== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever Since commit f931d8374cad ("nfs/blocklayout: refactor block device opening"), an error is reported when no multi-path device is found. But this isn't a fatal error if the subsequent device open is successful. On systems without multi-path devices, this message always appears whether there is a problem or not. Instead, generate less system journal noise by reporting an error only when both open attempts fail. The new error message is more actionable since it indicates that there is a real configuration issue to be addressed. Reviewed-by: Christoph Hellwig Reviewed-by: Benjamin Coddington Signed-off-by: Chuck Lever --- fs/nfs/blocklayout/dev.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index fc4eeb7bbf05..ebd268f81419 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -369,7 +369,7 @@ bl_open_path(struct pnfs_block_volume *v, const char *prefix) bdev_file = bdev_file_open_by_path(devname, BLK_OPEN_READ | BLK_OPEN_WRITE, NULL, NULL); if (IS_ERR(bdev_file)) { - pr_warn("pNFS: failed to open device %s (%ld)\n", + dprintk("failed to open device %s (%ld)\n", devname, PTR_ERR(bdev_file)); } @@ -398,8 +398,11 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, bdev_file = bl_open_path(v, "dm-uuid-mpath-0x"); if (IS_ERR(bdev_file)) bdev_file = bl_open_path(v, "wwn-0x"); - if (IS_ERR(bdev_file)) + if (IS_ERR(bdev_file)) { + pr_warn("pNFS: no device found for volume %*phN\n", + v->scsi.designator_len, v->scsi.designator); return PTR_ERR(bdev_file); + } d->bdev_file = bdev_file; d->len = bdev_nr_bytes(file_bdev(d->bdev_file)); From patchwork Tue Jun 25 20:02:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13712006 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1175917E465 for ; Tue, 25 Jun 2024 20:02:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719345741; cv=none; b=GnEoxZ+x7tzEKwAcRjMu1VQfqD/Yze1j2kekORwoZO4dvw1vm44wSCUNYk9Sjw8bMGOeQYM8eLQi7WVmRVjd0t9bxCBnSroaWg6RpHfNn29DinlPkAXcc/TBMGjze/qirpJouctJ3KlktTwE07X2nsKcyWto5HHssFosqbV6ZVg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1719345741; c=relaxed/simple; bh=EAaIHlUIwTFb5CHbZWEXJEdyeqA+wEA0ROyhdt1uI5o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uVJ8uVOaPjVSqXK/5s0mGMbEOtqBsztwUtFHIrc8ZRwukZc9gfUXsTqtFWhr9KqcU+4B7+oJ5GMb9z0xwewiFyIxueAd/y/XFjN0KKEE2sHB2CGsTHgT4+m56Kjwx1356Io0t6GlWzpArLMrCO2eoawEI9+h+NNlkHhbUna8yvA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FoANmJrB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FoANmJrB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 40962C32782; Tue, 25 Jun 2024 20:02:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719345740; bh=EAaIHlUIwTFb5CHbZWEXJEdyeqA+wEA0ROyhdt1uI5o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FoANmJrBBJI9LlYnPZv0OJfmoych7WZ+MLw08QmnEHVT8NC2v+nARMiDSLGpUwJAL CSIpDy2Eg9x0bz01IPE4jfr8b6jC7hgtI2tyLB+o0eFnsaH5y1aq1XpK0snWDdJJL8 xBcBBqId/wC/RLTNzTQHrO7IM1eeQkhnmx9pcUBxUG1c1FMCOanUh9ZlqEXfzlTqzj 8XO0k5gi5u05T5ZGSFMuJQpOEL/zRTYiIFinZHFyMq8dG49PWIwyikEhBl8F/5RTc+ p6NXHehtRjhtTBvPFKdrs6GjkogW6SYJ+r5TNPQkoGAs6Iq8Ydt1Kbw+a0viu//Mhp UuQVW1oRiqQgA== From: cel@kernel.org To: Cc: Christoph Hellwig , Chuck Lever , Benjamin Coddington Subject: [PATCH v3 3/3] nfs/blocklayout: SCSI layout trace points for reservation key reg/unreg Date: Tue, 25 Jun 2024 16:02:08 -0400 Message-ID: <20240625200204.276770-8-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240625200204.276770-5-cel@kernel.org> References: <20240625200204.276770-5-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7234; i=chuck.lever@oracle.com; h=from:subject; bh=sNUeZwjSUztiv1v3e0awYykOAMbyowzM4UaXe2ccEiM=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmeyJCVkGHYIei9dCcAjdMTxp4Is0cf4SPAFV66 lR5YEJF5uKJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZnsiQgAKCRAzarMzb2Z/ lwCwD/4lj1qGK6YOKs9v4EoTnbRWVP8rD7lip+ka6sa5NCKNnAlEJxiTimWs+Q0knfroQKIx8jB l9MyC79LBixtj5yzwoHi5VT3thSVmi+xsIp+I+aMZKbcUsQS/JRTvDzvOyFYoGTW6NsVYbZxuRH XovuQBHGOyD+Vz2VN+Q8fZAj9vQ2szZb6ENgkgtGBdr4t8twWprlMCT+BrKd2YV5fPr0KNXlmdC /nzX0o2wE0fzbUqJkDsUxRsTr4j5pys8yxQ+ycVf10aKGDT07zWdG3lL8sX+pU9hGa0RhLXBydS 1x65SurfaisbPjnVnuEkaUFxgVFQJTwX+VcC19L7WRGlgDbWlHX0q7qFrnYu/NvNfh3+PkKRUrd MbK0BZLfDMFmQEVHWiVt0k/dLZIzLrjN1Cpt7cu21I9erkpQeqVkQYtFKvywNnwww9BcFmGV11y FTjs6Q9BGZH7BM7IseztZ5J3EtqDCDQuACF64YFLnENVvPYllwmL16GNBcRFFz+m8TZ7B/VF45k lzqMFccyuAa5PQSx2J4JmBIt0oUP8cZ3Y+wjziAblsY7LpOOi/uBiwGty82d3WZ1oYIxzjMH2IU 9yU8Tydc1Z1UUpdxI45A0w/r9998m9pt7zbMYIm0J2DrL2naw4XiT3VVKJKVZsqmMeVj/R/bvj1 7U8rJ4O0flJVjhA== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever An administrator cannot take action on these messages, but the reported errors might be helpful for troubleshooting. Transition them to trace points so these events appear in the trace log and can be easily lined up with other traced NFS client operations. Examples: append_writer-6147 [000] 80.247393: bl_pr_key_reg: dev=8,0 (sda) key=0x6675bfcf59112e98 append_writer-6147 [000] 80.247842: bl_pr_key_unreg: dev=8,0 (sda) key=0x6675bfcf59112e98 umount.nfs4-6172 [002] 84.950409: bl_pr_key_unreg_err: dev=8,0 (sda) key=0x6675bfcf59112e98 status=RESERVATION_CONFLICT Reviewed-by: Benjamin Coddington Reviewed-by: Christoph Hellwig --- fs/nfs/blocklayout/dev.c | 26 +++++++----- fs/nfs/nfs4trace.c | 7 ++++ fs/nfs/nfs4trace.h | 88 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+), 11 deletions(-) diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index ebd268f81419..87f47e6e7181 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -10,6 +10,7 @@ #include #include "blocklayout.h" +#include "../nfs4trace.h" #define NFSDBG_FACILITY NFSDBG_PNFS_LD @@ -17,12 +18,16 @@ static void bl_unregister_scsi(struct pnfs_block_dev *dev) { struct block_device *bdev = file_bdev(dev->bdev_file); const struct pr_ops *ops = bdev->bd_disk->fops->pr_ops; + int status; if (!test_and_clear_bit(PNFS_BDEV_REGISTERED, &dev->flags)) return; - if (ops->pr_register(bdev, dev->pr_key, 0, false)) - pr_err("failed to unregister PR key.\n"); + status = ops->pr_register(bdev, dev->pr_key, 0, false); + if (status) + trace_bl_pr_key_unreg_err(bdev, dev->pr_key, status); + else + trace_bl_pr_key_unreg(bdev, dev->pr_key); } static bool bl_register_scsi(struct pnfs_block_dev *dev) @@ -36,10 +41,10 @@ static bool bl_register_scsi(struct pnfs_block_dev *dev) status = ops->pr_register(bdev, 0, dev->pr_key, true); if (status) { - pr_err("pNFS: failed to register key for block device %s.", - bdev->bd_disk->disk_name); + trace_bl_pr_key_reg_err(bdev, dev->pr_key, status); return false; } + trace_bl_pr_key_reg(bdev, dev->pr_key); return true; } @@ -382,8 +387,9 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, struct pnfs_block_volume *volumes, int idx, gfp_t gfp_mask) { struct pnfs_block_volume *v = &volumes[idx]; - struct file *bdev_file; + struct block_device *bdev; const struct pr_ops *ops; + struct file *bdev_file; int error; if (!bl_validate_designator(v)) @@ -404,8 +410,9 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, return PTR_ERR(bdev_file); } d->bdev_file = bdev_file; + bdev = file_bdev(bdev_file); - d->len = bdev_nr_bytes(file_bdev(d->bdev_file)); + d->len = bdev_nr_bytes(bdev); d->map = bl_map_simple; d->pr_key = v->scsi.pr_key; @@ -414,13 +421,10 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, goto out_blkdev_put; } - pr_info("pNFS: using block device %s (reservation key 0x%llx)\n", - file_bdev(d->bdev_file)->bd_disk->disk_name, d->pr_key); - - ops = file_bdev(d->bdev_file)->bd_disk->fops->pr_ops; + ops = bdev->bd_disk->fops->pr_ops; if (!ops) { pr_err("pNFS: block device %s does not support reservations.", - file_bdev(d->bdev_file)->bd_disk->disk_name); + bdev->bd_disk->disk_name); error = -EINVAL; goto out_blkdev_put; } diff --git a/fs/nfs/nfs4trace.c b/fs/nfs/nfs4trace.c index d22c6670f770..389941ccc9c9 100644 --- a/fs/nfs/nfs4trace.c +++ b/fs/nfs/nfs4trace.c @@ -2,6 +2,8 @@ /* * Copyright (c) 2013 Trond Myklebust */ +#include +#include #include #include "nfs4_fs.h" #include "internal.h" @@ -29,5 +31,10 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_read_error); EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_write_error); EXPORT_TRACEPOINT_SYMBOL_GPL(ff_layout_commit_error); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_reg); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_reg_err); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_unreg); +EXPORT_TRACEPOINT_SYMBOL_GPL(bl_pr_key_unreg_err); + EXPORT_TRACEPOINT_SYMBOL_GPL(fl_getdevinfo); #endif diff --git a/fs/nfs/nfs4trace.h b/fs/nfs/nfs4trace.h index 4de8780a7c48..22c973316f0b 100644 --- a/fs/nfs/nfs4trace.h +++ b/fs/nfs/nfs4trace.h @@ -2153,6 +2153,94 @@ TRACE_EVENT(ff_layout_commit_error, ) ); +DECLARE_EVENT_CLASS(pnfs_bl_pr_key_class, + TP_PROTO( + const struct block_device *bdev, + u64 key + ), + TP_ARGS(bdev, key), + TP_STRUCT__entry( + __field(u64, key) + __field(dev_t, dev) + __string(device, bdev->bd_disk->disk_name) + ), + TP_fast_assign( + __entry->key = key; + __entry->dev = bdev->bd_dev; + __assign_str(device); + ), + TP_printk("dev=%d,%d (%s) key=0x%016llx", + MAJOR(__entry->dev), MINOR(__entry->dev), + __get_str(device), __entry->key + ) +); + +#define DEFINE_NFS4_BLOCK_PRKEY_EVENT(name) \ + DEFINE_EVENT(pnfs_bl_pr_key_class, name, \ + TP_PROTO( \ + const struct block_device *bdev, \ + u64 key \ + ), \ + TP_ARGS(bdev, key)) +DEFINE_NFS4_BLOCK_PRKEY_EVENT(bl_pr_key_reg); +DEFINE_NFS4_BLOCK_PRKEY_EVENT(bl_pr_key_unreg); + +/* + * From uapi/linux/pr.h + */ +TRACE_DEFINE_ENUM(PR_STS_SUCCESS); +TRACE_DEFINE_ENUM(PR_STS_IOERR); +TRACE_DEFINE_ENUM(PR_STS_RESERVATION_CONFLICT); +TRACE_DEFINE_ENUM(PR_STS_RETRY_PATH_FAILURE); +TRACE_DEFINE_ENUM(PR_STS_PATH_FAST_FAILED); +TRACE_DEFINE_ENUM(PR_STS_PATH_FAILED); + +#define show_pr_status(x) \ + __print_symbolic(x, \ + { PR_STS_SUCCESS, "SUCCESS" }, \ + { PR_STS_IOERR, "IOERR" }, \ + { PR_STS_RESERVATION_CONFLICT, "RESERVATION_CONFLICT" }, \ + { PR_STS_RETRY_PATH_FAILURE, "RETRY_PATH_FAILURE" }, \ + { PR_STS_PATH_FAST_FAILED, "PATH_FAST_FAILED" }, \ + { PR_STS_PATH_FAILED, "PATH_FAILED" }) + +DECLARE_EVENT_CLASS(pnfs_bl_pr_key_err_class, + TP_PROTO( + const struct block_device *bdev, + u64 key, + int status + ), + TP_ARGS(bdev, key, status), + TP_STRUCT__entry( + __field(u64, key) + __field(dev_t, dev) + __field(unsigned long, status) + __string(device, bdev->bd_disk->disk_name) + ), + TP_fast_assign( + __entry->key = key; + __entry->dev = bdev->bd_dev; + __entry->status = status; + __assign_str(device); + ), + TP_printk("dev=%d,%d (%s) key=0x%016llx status=%s", + MAJOR(__entry->dev), MINOR(__entry->dev), + __get_str(device), __entry->key, + show_pr_status(__entry->status) + ) +); + +#define DEFINE_NFS4_BLOCK_PRKEY_ERR_EVENT(name) \ + DEFINE_EVENT(pnfs_bl_pr_key_err_class, name, \ + TP_PROTO( \ + const struct block_device *bdev, \ + u64 key, \ + int status \ + ), \ + TP_ARGS(bdev, key, status)) +DEFINE_NFS4_BLOCK_PRKEY_ERR_EVENT(bl_pr_key_reg_err); +DEFINE_NFS4_BLOCK_PRKEY_ERR_EVENT(bl_pr_key_unreg_err); + #ifdef CONFIG_NFS_V4_2 TRACE_DEFINE_ENUM(NFS4_CONTENT_DATA); TRACE_DEFINE_ENUM(NFS4_CONTENT_HOLE);