From patchwork Fri Feb 3 21:18:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 9555129 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2F4E5602B7 for ; Fri, 3 Feb 2017 21:18:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1CAB727F94 for ; Fri, 3 Feb 2017 21:18:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1091D28425; Fri, 3 Feb 2017 21:18:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 43E4F27F94 for ; Fri, 3 Feb 2017 21:18:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752542AbdBCVSl (ORCPT ); Fri, 3 Feb 2017 16:18:41 -0500 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:37203 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752331AbdBCVSk (ORCPT ); Fri, 3 Feb 2017 16:18:40 -0500 Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v13LEn43006575; Fri, 3 Feb 2017 13:18:32 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : subject : date : message-id : mime-version : content-type; s=facebook; bh=uZizn2ARq5/MR49c83QyrSO6uEoXVCgIf+UCZdCRJoI=; b=hN6k9Mv72Io/7mXzdTjJhR2cF99C3WigAw0kYuyJaYsyR22VA5wQFLTNEdoCirZVw3ri gQ66XeEqfYnMkrctOzz+v3ZQJz9TFNC/r5aXJHLYwCaqG6omKgBAbDtoB7cTB4iJYvQj AV90eIE/CX9Oxx0vwClMhxvb/FlLWMegj5M= Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 28cxv5rmhv-1 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 03 Feb 2017 13:18:32 -0800 Received: from NAM03-DM3-obe.outbound.protection.outlook.com (192.168.183.28) by o365-in.thefacebook.com (192.168.177.25) with Microsoft SMTP Server (TLS) id 14.3.294.0; Fri, 3 Feb 2017 16:18:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector1-fb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=uZizn2ARq5/MR49c83QyrSO6uEoXVCgIf+UCZdCRJoI=; b=Ifwwrgp+mwOO6qYUp16Dg7rcL9CnXuoiYG019TNUKNGwIQp5+naYWRjRS89HTYeBwXY2aSw51wMrSgTnRCWoXHeptroDY2fzgG+3OYc7eYur5jC5oz1VvAi+XL4dv3CaRzk8TXRtERBoclJw1D+0Nufj/NRWFdMR0KXKHKARsgo= Received: from localhost (2606:a000:4381:1201:225:22ff:feb3:e51a) by CY4PR15MB1318.namprd15.prod.outlook.com (10.172.181.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.888.16; Fri, 3 Feb 2017 21:18:28 +0000 From: Josef Bacik To: , , Subject: [PATCH] nbd: handle single path failures gracefully Date: Fri, 3 Feb 2017 16:18:22 -0500 Message-ID: <1486156702-9417-1-git-send-email-jbacik@fb.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Originating-IP: [2606:a000:4381:1201:225:22ff:feb3:e51a] X-ClientProxiedBy: DM5PR18CA0012.namprd18.prod.outlook.com (10.175.218.150) To CY4PR15MB1318.namprd15.prod.outlook.com (10.172.181.136) X-MS-Office365-Filtering-Correlation-Id: d3c6fd62-534f-4bdc-748f-08d44c7a32b4 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001); SRVR:CY4PR15MB1318; X-Microsoft-Exchange-Diagnostics: 1; CY4PR15MB1318; 3:d/mra7cDyLW6kP50b3J2CDdia/7F2sYIPdnvxgz2R9a0qw/DZdVGqsoaYVgvsakOa+78xt2i5pEIxHD5jNqwdHinkDeQs3Jt8/i3gJAr+Clot2Snl2LlKoNwqw8sO94YBzRn0N/KtwwpQO9bG/79+4tkzo2xFM6gK4Ea7KkN4RLYD/w0nN8uaCWRNqwfMW37Q2U1KCk88vG+BFuPVP+7LLjM+PTK9cguOMvwSkA/s/RDHAuheq5vv8q3AOaLkmTN7O7saal708/EtfTlF86g7Q==; 25:BZYbTKye3mLoewZCTlJTYTlHQYO3hxKaSkho4mXsO+0Ai4MDZmby2ZGAC7dAI8UqlKkjleCVSEbztAwyFlXJy+yQ0QjpIifk0qUxDLOPpaIJPAS5aRaMr0vca/Irh3UK2tUK/8Pe4dwJbsGnANs5YGPyUu86oqQquEcAei0IU2AJDPv+sJf+ZltW81Tjic0Haessj8Keu7jAQrMKZ1cQ1Vm6HKGIkn8xmBPO4BsFlRxbvqXL3XhwXlygiBECFj+MCl+T3S9MGLkCu1hKtFp+RY9kT6yoRf2QLYXnVpG+LyJsERNimobx4J8nTdTReGxQitBwLagi+9M0zIBUT+7cjf7h8a50FoK5cZH+chty09BoqsBbRAfnjhtKiWtHy4bUTVaOO6voWrITxQLDivNN2c79Au08rY+ta8lKvg/bnVEfEZcnYzbs8VfsJ0IqEMBLlkkQmoFsf2WrpsnC7dWDlQ== X-Microsoft-Exchange-Diagnostics: 1; CY4PR15MB1318; 31:pH9UQBg3RkxXWp5n1/7UKSHlTHs5YljjZVEW33UwONDlw1E8qHkHyQNKpjwKvFxSW85hd4nwxOwf/RliUJo1EVfd3sT1cgCBKxJfMyCu6ju7XHfxald4WJiXj7gOSDm/KtrXVnSF3dCezOYyMdzUJEQr3dioHjawRfouOQayJUOEHmpEW3LKtB8ZHl8juPZRSMuRIRjEh1OILsEStS6q1Jipx3DzWtMUcbCtxxSTgY/xF0G5lhn+WY89uAiOms9K; 20:MSZEVpR9cnJEJdybJ5Ol0Iysoefi5nYAWQHX9Tb4rjjnEX3gjWzH2KtpS/J1MqVIclbdKazH+KmvezOc8TOPSOwbLQzbpk7ExcF2DxLWuuR217V38zzgUOQfMTmJ1EQKZnLsPQlVdOnmzvKz5kvpAKT6KbcMwujo8OeAMB9fVGvuJjqbH6/xrElANAlVE+mUwbKFmoQtvH/S6aY7km8POVrBcGOXZm5+RVxewT70e4DG0tmowxb4r0z90lel0cEUxfmmf+UcF4CGTInMcJDtK6D7xQtW4rgYHcV1P0Dd0EJnsHXjqh2djty0P6qmLVMp7T7Wy72xM5aF7Pbt0yM+0LJtVpbu3HxABu4mvfjY2PTK0tNSWzbO0OPSz8sIuB+kPBaJclKbvqbPF9WCZsyh+KOXRlltuDaKO8aIXbNDdcc5Kp9uZOHCvmhD8pmBlC5fHLsfth+7bSiwkNMsX7Wqf3CuG0sR+RYuBdwnJD5F0CzToyStgCIEGPHALqMWPT0Z X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(67672495146484); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6041248)(20161123555025)(20161123560025)(20161123558025)(20161123562025)(20161123564025)(6072148); SRVR:CY4PR15MB1318; BCL:0; PCL:0; RULEID:; SRVR:CY4PR15MB1318; X-Microsoft-Exchange-Diagnostics: 1; CY4PR15MB1318; 4:7Ua+KyUHFDaouIXvRWmU+dIfdN6raVMSMwhQ2kBYgeUaM3DZo52h+L0KqdaqiUo+OlP5nmPJ0MCKYa2s+JK9ccYOTrPErXdspAd2GImmTME6Uh3uZKYOrviuC9Qo11n9UUKU9jrdVOx+HbK+3kcHWQGpocUL9AkJCtcDlPuiLIRYNvOY2mTOTAG1cYpHDeCVzVqKGY2K80Id7REZT8kWklzJDocS69+o4fSRb7V9oCyZ55mP1c1j3bxk0tVynKqltms9Kg/AwqP07oaTp3oZ7mYWhOGVpcjFk9JIewHTxp1PtJjFafQ7/Lv1SCbusnf4L/fSFe76UoSoRuqabZIUiHc1uF1D6ufx/7vIeKTHcJ4Ul6kYDzWSjLY2TnSIzimLmRO22eSeg6bGNxika+QUY6Ooc3+zkcVb/9Jit/edhgV2eliLYj/K7sepM9ZNQwEl3ZghIYemQ1U31RY7TKv3luQQBga2y01wAQyx2qLwE0qe2WbAorA6vHMoot2Em+SaWQklI39MpmRYnMxeR2TlTwGaZknmbcqwmFkuNRXcNrhcyyfgnpw9IeC4YeSlNLQ7m8iYBeA6dqOSgRMiZj7mz0RIa5VdSmWqxk8AVFEcAc7o1mR2jydcNux13by0oBpRMdkh/Gmjg4e0eaD1AOYaRA== X-Forefront-PRVS: 02070414A1 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(4630300001)(6069001)(6009001)(7916002)(39450400003)(189002)(199003)(45904002)(107886002)(97736004)(5001770100001)(305945005)(68736007)(6116002)(6496005)(189998001)(6486002)(38730400001)(47776003)(5660300001)(6666003)(7736002)(25786008)(2906002)(5003940100001)(50466002)(86362001)(101416001)(36756003)(48376002)(92566002)(42186005)(50226002)(53936002)(76506005)(105586002)(106356001)(33646002)(81156014)(81166006)(2201001)(8676002)(50986999); DIR:OUT; SFP:1102; SCL:1; SRVR:CY4PR15MB1318; H:localhost; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: fb.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR15MB1318; 23:DHL0EIdLD/93lQg5jULWKv8lltSPy35vkDnEUtUiQ?= =?us-ascii?Q?KHZV5MrIldZ9fBQRVA73ny5P62zTSGXY2ZsPbh+f2KCqNpmCJ6tV7oTJAoiX?= =?us-ascii?Q?DEqeRexM+wTvU305O5a6n6IDaw3i9TKdGk8EwuqRvoYwVSCybcHZYOOLaD6J?= =?us-ascii?Q?dn/P29UDnPYlGUN/n4SxXUgNjlkjmrYpcqD22cLa3HGAo9t1YpvT8nbCu7gO?= =?us-ascii?Q?PG7LyPrvdQE/saqF2tkha6WDzkAhagYz+BGjZKvr1RFjebco/lFLE+Su8fLr?= =?us-ascii?Q?Pt0CDQSZ7zd7icullVcMe9xiHDhYRCEqLci5BWS8njlO+jVx79tT0jKW2YyQ?= =?us-ascii?Q?mn9VVMzLNqGqMhlJkbf1v5QahUfsLlOR6gtaTdA1aLwGvruK/PmY3W50GLHr?= =?us-ascii?Q?kpU9S5qsdQTzaPI3h5AFbKg3wm/2XKntFcgFiZmp2U/r3fzI7McCmogNuWk3?= =?us-ascii?Q?/kzEx5DIiAuKhD7wB3QLAxRdKrMS3wfKlQybwFkovZopnDiC9NJ/4hqO908O?= =?us-ascii?Q?ZxbzC0ctigZkPL2HtACn1hzlcJbEs+9Qk11F6gvZqHarZfaSzfBaZp4/tBra?= =?us-ascii?Q?PVWdQTlYThGYgboCmPR0CwqI8I/ez4ucIkCTaHVaImtj9EX6k6Na9HAU1uk7?= =?us-ascii?Q?+59F5HlQi4JR+Zszvcw7qUnyZiWBehKHqDc98PdaYY1NNNThRGfuwiDkBrWW?= =?us-ascii?Q?8xVotmqLuDk/Oi8gfxyFYj1k43ijYVJcDFU7jn9pl/ZSCVGXxjk8QG5UPh88?= =?us-ascii?Q?7xEkS0DFktuloGPxTGUoyL8Vns5k5Qw46KYBGm5hKmLm4nf6tT4ocNc78A0Q?= =?us-ascii?Q?UMUx7jTpyoYqBT/2Bwd5tF+BwAZrfpfY72lQMeZDnuf/HHPLAE6mc+j9K17o?= =?us-ascii?Q?g9WQzHdg5CfIa6gT9q1EmUxXilmg2EWALS/tWG+sjJNx3Hz7nJg3F5B4Sksk?= =?us-ascii?Q?Nt6JfXAw6H+J411Ynlxub4IbSvkwezgZFQgs2yY74AGrzgrCltN2P+fTOdeZ?= =?us-ascii?Q?ntFad1zHhveGfr2KDYKK8sPRa146HZ1AwN701cP9GqDCB4TJkme6Im5cNyqD?= =?us-ascii?Q?xOLCnjXxH17Ph8WGYxE0Yt9xQfN?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR15MB1318; 6:vMWdNI4OR04le+jKOuAd797SnaH2EQSnSY/YgvAKeCW5+piaKcAAuw5TKdsYOjLT2ZVio2GB11OfwI6MGJJATzkSr9G8QXgP3uGo8xbj2K4Mprk9wfE2tP1GRnuPqb3DN5Pz4Qu99ChlBFeqGiUQUzNiQvcIc0t+BdByuSiwZatXShhRMK0z4YfA4wuE/Fbtt5ODgPIYHQEE2IR6Oh1V/3IxopthI+ZfxPD6DUkl4OMVtuM5AM0APM30lAf/5dNHwlOekQr7OVb1ixQgEP0h9/mQJi/LT3ZnWMT4X7YINnetO389OCPryJnOlJEPRvfqz9d84/uHLdbBqZEAVhCD7e6F5LDcF/Kw5W4T0+ARCoqw1qdt/3xrGuiglHs4sDtttckASOhSg5IyVNkbNtpvqw==; 5:OyGRwQH/sB5O7IJujhKL8S8+xPWoauSmuFTzA5nAjSvMtuP4GqCzXvSbC4nZxiphBWhPY4eEmypdMh3HqxScYHPwoE6bAZcz3YWfgpWD+8k5F5wtBa/Tm06N1T5Ib4YbglVnSSpPjAdNgXeiCGCJow==; 24:GKBP2qhpUoJmXjHsjen70y0HIz68P4CwYrFD9gSLykckEQS3rgK1AIxB4NML1As2a18bSzFpSDxzcv/qQIbQnjeWfLuZjPdqsdZiRtmC2z4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR15MB1318; 7:pbvr26EGi9WOouHo2zhvlnC9pK+pTYaIi/FPsB+fB75fhBk9jipdRvuWC3H0dgahCWAQqgQ4BFAq9vhBdOi3u2B1Pegq6AeGdhImnrW5e+dTA6Aq5TvrepJZ5ZeHjFPquumkEv8dpmLM+UzpYghn7NydTYTORMX9vlGOuN1/HY3YVT4zXhlk2UlS5t+7i1kPVAvKMCJjEXHzamIjwQoVdtW8dtnT1zzy5iyRThNBQ/K3c4x3pVlCklTTEIX9qmlmA9yHoBZSpdFEqyMKCL+MD3dzRHH/srq/FNmSihKrJgVEmwgXhaoC9rqFgmmnDJ6nP7s/P3XgFskQ2Oefrdro+H+35UqvZPtxI+OtD2hlNv8W7lZKUF2OBckXPVgLM5qvmmOkVb27DXNwViy6MeEwq+uF+2G72iSlkTagCcUPb/cJ2JZsFv1Vj/CYETu1i6gI3cQ9rwmgo80WhWu/lRSLvTA6baF2QO++J+ZkKSoZsupPy/zIZ7xWcbo79Kd9RtTNCoSPaJxpfB7Dc9HkLtmmEw==; 20:UGUFb3/hMoE6ayAhVgOcZYXm3MZAb7Kn6zkYJRxFSMYUD7Y1PZP5s3MX2R6Vqmvun6lkkQ3B12y0xQKEGzRLKrBrnWjA4CmJEhHdpY/+mT1yiQnz8QjrrqdpwqW6/i+cqM4TxfBap92R2zpZOgukziAcF0YN61yR+ePeZzsp9+U= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Feb 2017 21:18:28.4298 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR15MB1318 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-02-03_13:, , signatures=0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently if we have multiple connections and one of them goes down we will tear down the whole device. However there's no reason we need to do this as we could have other connections that are working fine. Deal with this by keeping track of the state of the different connections, and if we lose one we mark it as dead and send all IO destined for that socket to one of the other healthy sockets. Any outstanding requests that were on the dead socket will timeout and be re-submitted properly. Signed-off-by: Josef Bacik --- drivers/block/nbd.c | 165 +++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 124 insertions(+), 41 deletions(-) diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index 0be84a3..164a548 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -47,6 +47,8 @@ static DEFINE_MUTEX(nbd_index_mutex); struct nbd_sock { struct socket *sock; struct mutex tx_lock; + bool dead; + int fallback_index; }; #define NBD_TIMEDOUT 0 @@ -80,6 +82,7 @@ struct nbd_device { struct nbd_cmd { struct nbd_device *nbd; + int index; struct completion send_complete; }; @@ -193,7 +196,32 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, struct nbd_cmd *cmd = blk_mq_rq_to_pdu(req); struct nbd_device *nbd = cmd->nbd; - dev_err(nbd_to_dev(nbd), "Connection timed out, shutting down connection\n"); + if (nbd->num_connections > 1) { + dev_err_ratelimited(nbd_to_dev(nbd), + "Connection timed out, retrying\n"); + mutex_lock(&nbd->config_lock); + /* + * Hooray we have more connections, requeue this IO, the submit + * path will put it on a real connection. + */ + if (nbd->socks && nbd->num_connections > 1) { + if (cmd->index < nbd->num_connections) { + struct nbd_sock *nsock = + nbd->socks[cmd->index]; + mutex_lock(&nsock->tx_lock); + nsock->dead = true; + kernel_sock_shutdown(nsock->sock, SHUT_RDWR); + mutex_unlock(&nsock->tx_lock); + } + mutex_unlock(&nbd->config_lock); + blk_mq_requeue_request(req, true); + return BLK_EH_RESET_TIMER; + } + mutex_unlock(&nbd->config_lock); + } else { + dev_err_ratelimited(nbd_to_dev(nbd), + "Connection timed out\n"); + } set_bit(NBD_TIMEDOUT, &nbd->runtime_flags); req->errors++; @@ -299,6 +327,7 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index) return -EIO; } + cmd->index = index; memset(&request, 0, sizeof(request)); request.magic = htonl(NBD_REQUEST_MAGIC); request.type = htonl(type); @@ -316,7 +345,7 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index) if (result <= 0) { dev_err_ratelimited(disk_to_dev(nbd->disk), "Send control failed (result %d)\n", result); - return -EIO; + return -EAGAIN; } if (type != NBD_CMD_WRITE) @@ -339,7 +368,7 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index) dev_err(disk_to_dev(nbd->disk), "Send data failed (result %d)\n", result); - return -EIO; + return -EAGAIN; } /* * The completion might already have come in, @@ -366,6 +395,12 @@ static inline int sock_recv_bvec(struct nbd_device *nbd, int index, return result; } +static int nbd_disconnected(struct nbd_device *nbd) +{ + return test_bit(NBD_DISCONNECTED, &nbd->runtime_flags) || + test_bit(NBD_DISCONNECT_REQUESTED, &nbd->runtime_flags); +} + /* NULL returned = something went wrong, inform userspace */ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index) { @@ -379,8 +414,7 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index) reply.magic = 0; result = sock_xmit(nbd, index, 0, &reply, sizeof(reply), MSG_WAITALL); if (result <= 0) { - if (!test_bit(NBD_DISCONNECTED, &nbd->runtime_flags) && - !test_bit(NBD_DISCONNECT_REQUESTED, &nbd->runtime_flags)) + if (!nbd_disconnected(nbd)) dev_err(disk_to_dev(nbd->disk), "Receive control failed (result %d)\n", result); return ERR_PTR(result); @@ -421,8 +455,19 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index) if (result <= 0) { dev_err(disk_to_dev(nbd->disk), "Receive data failed (result %d)\n", result); - req->errors++; - return cmd; + /* + * If we've disconnected or we only have 1 + * connection then we need to make sure we + * complete this request, otherwise error out + * and let the timeout stuff handle resubmitting + * this request onto another connection. + */ + if (nbd_disconnected(nbd) || + nbd->num_connections <= 1) { + req->errors++; + return cmd; + } + return ERR_PTR(-EIO); } dev_dbg(nbd_to_dev(nbd), "request %p: got %d bytes data\n", cmd, bvec.bv_len); @@ -467,19 +512,13 @@ static void recv_work(struct work_struct *work) while (1) { cmd = nbd_read_stat(nbd, args->index); if (IS_ERR(cmd)) { + nbd->socks[args->index]->dead = true; ret = PTR_ERR(cmd); break; } nbd_end_request(cmd); } - - /* - * We got an error, shut everybody down if this wasn't the result of a - * disconnect request. - */ - if (ret && !test_bit(NBD_DISCONNECT_REQUESTED, &nbd->runtime_flags)) - sock_shutdown(nbd); atomic_dec(&nbd->recv_threads); wake_up(&nbd->recv_wq); } @@ -503,50 +542,89 @@ static void nbd_clear_que(struct nbd_device *nbd) dev_dbg(disk_to_dev(nbd->disk), "queue cleared\n"); } +static int find_fallback(struct nbd_device *nbd, int index) +{ + int new_index = -1; + struct nbd_sock *nsock = nbd->socks[index]; + int fallback = nsock->fallback_index; + + if (test_bit(NBD_DISCONNECTED, &nbd->runtime_flags)) + return new_index; + + if (nbd->num_connections <= 1) { + dev_err_ratelimited(disk_to_dev(nbd->disk), + "Attempted send on invalid socket\n"); + return new_index; + } + + if (fallback >= 0 && fallback < nbd->num_connections && + !nbd->socks[fallback]->dead) + return fallback; -static void nbd_handle_cmd(struct nbd_cmd *cmd, int index) + mutex_lock(&nsock->tx_lock); + if (nsock->fallback_index < 0 || + nsock->fallback_index >= nbd->num_connections || + nbd->socks[nsock->fallback_index]->dead) { + int i; + for (i = 0; i < nbd->num_connections; i++) { + if (i == index) + continue; + if (!nbd->socks[i]->dead) { + new_index = i; + break; + } + } + if (new_index < 0) { + mutex_unlock(&nsock->tx_lock); + dev_err_ratelimited(disk_to_dev(nbd->disk), + "Dead connection, failed to find a fallback\n"); + return new_index; + } + nsock->fallback_index = new_index; + } + new_index = nsock->fallback_index; + mutex_unlock(&nsock->tx_lock); + return new_index; +} + +static int nbd_handle_cmd(struct nbd_cmd *cmd, int index) { struct request *req = blk_mq_rq_from_pdu(cmd); struct nbd_device *nbd = cmd->nbd; struct nbd_sock *nsock; + int ret; if (index >= nbd->num_connections) { dev_err_ratelimited(disk_to_dev(nbd->disk), "Attempted send on invalid socket\n"); - goto error_out; - } - - if (test_bit(NBD_DISCONNECTED, &nbd->runtime_flags)) { - dev_err_ratelimited(disk_to_dev(nbd->disk), - "Attempted send on closed socket\n"); - goto error_out; + return -EINVAL; } - req->errors = 0; - +again: nsock = nbd->socks[index]; - mutex_lock(&nsock->tx_lock); - if (unlikely(!nsock->sock)) { - mutex_unlock(&nsock->tx_lock); - dev_err_ratelimited(disk_to_dev(nbd->disk), - "Attempted send on closed socket\n"); - goto error_out; + if (nsock->dead) { + index = find_fallback(nbd, index); + if (index < 0) + return -EIO; + nsock = nbd->socks[index]; } - if (nbd_send_cmd(nbd, cmd, index) != 0) { + /* + * Some failures are related to the link going down, so anything that + * returns EAGAIN can be retried on a different socket. + */ + mutex_lock(&nsock->tx_lock); + ret = nbd_send_cmd(nbd, cmd, index); + if (ret == -EAGAIN) { dev_err_ratelimited(disk_to_dev(nbd->disk), - "Request send failed\n"); - req->errors++; - nbd_end_request(cmd); + "Request send failed trying another connection\n"); + nsock->dead = true; + mutex_unlock(&nsock->tx_lock); + goto again; } - mutex_unlock(&nsock->tx_lock); - return; - -error_out: - req->errors++; - nbd_end_request(cmd); + return ret; } static int nbd_queue_rq(struct blk_mq_hw_ctx *hctx, @@ -565,7 +643,10 @@ static int nbd_queue_rq(struct blk_mq_hw_ctx *hctx, */ init_completion(&cmd->send_complete); blk_mq_start_request(bd->rq); - nbd_handle_cmd(cmd, hctx->queue_num); + if (nbd_handle_cmd(cmd, hctx->queue_num) != 0) { + bd->rq->errors++; + nbd_end_request(cmd); + } complete(&cmd->send_complete); return BLK_MQ_RQ_QUEUE_OK; @@ -594,6 +675,8 @@ static int nbd_add_socket(struct nbd_device *nbd, struct socket *sock) nbd->socks = socks; + nsock->fallback_index = -1; + nsock->dead = false; mutex_init(&nsock->tx_lock); nsock->sock = sock; socks[nbd->num_connections++] = nsock;