From patchwork Tue Nov 22 19:04:40 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 9441949 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id BF983605EE for ; Tue, 22 Nov 2016 19:06:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B04D1262F2 for ; Tue, 22 Nov 2016 19:06:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A484A285B4; Tue, 22 Nov 2016 19:06:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D125284C0 for ; Tue, 22 Nov 2016 19:06:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756243AbcKVTGW (ORCPT ); Tue, 22 Nov 2016 14:06:22 -0500 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:46693 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754789AbcKVTGU (ORCPT ); Tue, 22 Nov 2016 14:06:20 -0500 Received: from pps.filterd (m0044008.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.17/8.16.0.17) with SMTP id uAMJ3wHX014576; Tue, 22 Nov 2016 11:04:50 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fb.com; h=from : to : subject : date : message-id : mime-version : content-type; s=facebook; bh=r727x9h3ziDdujeUdsSvHGQHlicQqfL9tqYxEc2dvts=; b=Acn5NkztU5fSEFcogl/V64ValNQ3l16EasfomBK3KQ9Zt7xe0PCBWFW5BqyX2nedc5pG rhAOTXdTQzMyhbPOnV2yAb0X+hx+mdYK5sy49ucqlYHhSEwQ9cAXe3gtGqYrkHOENEa9 jMgWjeKdTx0OHhz6G+PrPFCitIAwZUleoF0= Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 26vumur5w8-1 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 22 Nov 2016 11:04:50 -0800 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (192.168.183.28) by o365-in.thefacebook.com (192.168.177.23) with Microsoft SMTP Server (TLS) id 14.3.294.0; Tue, 22 Nov 2016 14:04:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector1-fb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=oEaPzOv1TdBG+j+CnduUEB59/Ni3RUPm4tjqiD3yIRM=; b=i5kYl7R9KMxGs58fp63FawBFIrpNT5sobMaCfDkth2YjadG38B9s5zjn3jQnl13gsyC78UfHRtgmldfGcmj5bu1C3JdsdDtI3e4aeYcc+fE6is1OxUo0pR3gkxTS0y033F/hSreHZLbbr9W4f6HqEiOu8e6cP3+UWQxjZYmDSNE= Received: from localhost (2620:10d:c091:180::596c) by BN6PR15MB1315.namprd15.prod.outlook.com (10.172.206.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.721.10; Tue, 22 Nov 2016 19:04:45 +0000 From: Josef Bacik To: , , , , , , Subject: [PATCH][V5] nbd: add multi-connection support Date: Tue, 22 Nov 2016 14:04:40 -0500 Message-ID: <1479841480-18094-1-git-send-email-jbacik@fb.com> X-Mailer: git-send-email 2.5.5 MIME-Version: 1.0 X-Originating-IP: [2620:10d:c091:180::596c] X-ClientProxiedBy: DM5PR17CA0017.namprd17.prod.outlook.com (10.168.112.155) To BN6PR15MB1315.namprd15.prod.outlook.com (10.172.206.141) X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1315; 2:3IIlBMimONGAakOrhyCZQ+GASYL4ltxrOcKw/npF7pIRVbhSD52bGY8wGpM7whvb99avQfWPoXc3pYuwvy4Xb8A9A8UuKLuoJCKubJuwJZfbQY/Pi0u6GFLw+vylpo9C8qQmc1/Ad50kNnOuqWWJ+4kaq+XFPc+3T05s1l6MD2w=; 3:VwP1OYEk+kgDD6jWqLhuNafhLNJo95iQd7rcVa6KqQo8Jrhu98Wz+JXYKRdUrhB43ihQGIegXwnTCHEYjYy+8Vm66cKeMZgS9CoPNzJy1FO4G7ftuMgyG1N9z1hEvtAvR5mFzkmUUEu9CaFSkac8LxuWm6EUQbyWqp7LbHIQhiY= X-MS-Office365-Filtering-Correlation-Id: 444d70d3-935b-4a2a-a337-08d4130a6caa X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001); SRVR:BN6PR15MB1315; X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1315; 25:MEJAngXvKhj1Ho9IRThSamYUCHYwSvt43NAnO9hYwq1O5e0SI4fnb8N3DoNuiuxRm3hwh0BGP/vPEQZbiGXfSF8bpAyqhwGAm7jfEm66yPYuCI0KcuNFu8auW5KdGWG/3dg9LLn6hejRFLO3rcSgi8JTHJEh4ubvEPD7T7gprCxyRMQ5Mf0yyIO4o55Et3e8MXVMHlTR1OqyFARaMPyYmO82rRrlI9UhIebOCAXJjPCzeplpzGHeEOsaNpIhMjnQzX/e6wNhP0b3DRhvTSR18no2f4XXZZLRWGZKBvTiW0YXd6Qh6TPt3XMUwt6VTJE1Ch7CkztlE33ce1VphhqbvqmBnLgYTDPPRKRJIbfJc0/3ud/iRwmiZku7YdjvIiiILcMrhMt+0OFIcWNW+Lss705ox4NszPs9xS1qbTwtV2FYnebJli4MyVjrLTTBTa89BZOzb0PQubs5Gc1ESJArWIaUH2U+apXMXIPY5erClo/xK7EyNA3hRcFbrk0a5gb2ESJSgz/NVoScqpjClr7DQhrdXuBU63wOkrNcZ+yEpP1tXs8d8ofZUHXtpP0h+Rqv8WXJHbT3R7KuBsLV6rOeoHJv9nVl07hMbMVunVh+uO3s6tUU67kNn7bjBSNu93k3hptaSDfdIvRSyu+kZ3bhK27TZh9doQppFdkRM3brhGuS1vFFzuegkBzJ26bP83YgXBxTTsAeHrS3tYDIyR90fn2qnY93KK9YRevVTe0ipyv3lx+sFjv8bxDAMGGEWOURw7bOwvTAb+jiWrD+CIhNzGyLo7vh2GDS/Hw4JG4OrTzdgkpwCZum5/UUEKxHTBPa X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1315; 31:0qD0nv6LT7IB5TkajnxCBBmLBf3+VptEdtY2eWOqckj9xErMa/9b+gXsUlEFjbR7OuOGh+m0IwqcKCgb+u2FYGrbVvxZx6YQyfkf34dxaHsoblWwxFWH+oHfFYhTqCPeapTwJEUp/YHS6Rk0WB0qq64KxtmxhmUEKkCDQQIjiVuM5+f2+oAkamhPfEXArF034e/4wLG/pO9139vPdR30hptVc3WU6rbrEFf60J1hYKEzRffL1eQdmZbH/n7JZN38Sn18I23q0BVdSB4oRlE8AQ==; 20:pThNwjEDMnylVNZMc9d1JAKxAlV7ItJ0ovd2maFlAFe4ZZvlP7lDb0kP7EHqtY0dJyravJEMqo11SoW7XsqET2riVxoH6ptrAxFgBYyfeZPPJFZt0fjN/A2mIcY94mjLYcZU5w1LO3ecz1pPFPUmBaULQSf1Gf+bU7W2BHJYWNtflBLLa3HGnisxpztQbdYu8WGYVLJyt5t7RMtCHD9v2tASkoZiqufMXujbhsjxSkfnA7c49eqq2o13+FSY68i8 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(158342451672863)(67672495146484); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040307)(6045199)(6060326)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6061324)(6041248)(6042181); SRVR:BN6PR15MB1315; BCL:0; PCL:0; RULEID:; SRVR:BN6PR15MB1315; X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1315; 4:CUmEhKp5ftI2RIbQ1zFyf+yP5bUK85MQAYv5BzzbqbBONXGil6R6NnEdYLMm7EHHfvHkTu1NlRUPK02coZjkAXac0kiusytZebddNTmR8JaiQURZz7SBMye7pKzbyc2Ps6iXLsYxqdBqVx+oBZeh0ID+Pff/PodJlCbdZAw+Pm9y35psAM6QXC9vF0cd2FZmjWvIviItOk6IMVhjy8fX5q2h8so9YmszU4FNEYenWJqifFdSCOTi5p4Rl66uNTGVLBvfj9NtIjpU3NhwEFi4pe573C6A3lrDkj/qYTl44dlYTU4+II2jQTmx6We93O+HHjMawCPlY10w6jKvXwgYAJoMb0jmjRkOyWJLkt+raApIrOenl7ZOd3cLCZB5h8eErhIa5BTDmUpsHoPfXhIneXCj41NSJ0g3J82TSeYtB+vbOG6lOryL6eoyiF8jAtXYYy6Mxb7iK+shcsblOg6MsNNk0KQndmE7nwHbZhaeSHRrpyjU2jsLBfA2Ti95hQeD+J9tmkzatCL8MC7IrJrU8uDZefulika03GxUUBATuvRniQDI0Peq0LgiAm+6pE/r X-Forefront-PRVS: 0134AD334F X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(4630300001)(6069001)(6009001)(7916002)(189002)(45904002)(199003)(33646002)(101416001)(50986999)(86362001)(76506005)(50226002)(5003940100001)(48376002)(106356001)(42186005)(105586002)(50466002)(68736007)(77096005)(92566002)(97736004)(36756003)(5660300001)(6666003)(189998001)(5001770100001)(107886002)(7846002)(8676002)(305945005)(47776003)(7736002)(6116002)(81166006)(2906002)(81156014)(2201001)(38730400001)(781001); DIR:OUT; SFP:1102; SCL:1; SRVR:BN6PR15MB1315; H:localhost; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: fb.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN6PR15MB1315; 23:6KjCaAnqS7ufI2UvP6dybezJU0TC7CE5tnsVhJU6k?= =?us-ascii?Q?NMdOdemg7yCV5FjHQ6ecdZDOu6skjwlTvGiD2wyjgoJU8eA8Sjq7Zuoh4ymO?= =?us-ascii?Q?+x6KMHrOn3aY6hKGSp2RAfdVIQAD5g8ivyV7nRrkoWTY8D8TLNku2du3cjMz?= =?us-ascii?Q?wgbs6bm7+Ri/fNOps/UgGkDAtXbQTRRKYEGUddETnrYFMjN7gWrH1veFB1HK?= =?us-ascii?Q?c4z4/AMwniBzbHCmh7lC7SDokpu+KCTyurwVl2/mh63mYIv3ZYNtHN7Ox6lJ?= =?us-ascii?Q?QN1LkN9VSxLY2YJB0XjLisvzPdhby0vmenvSiGIopB61buvkgO6+Ap3XpB6h?= =?us-ascii?Q?OJLhsy1ijQ4aGpQnNvIFrxBnR9uoEdVdejo6O5P5T9V2ZGANfYtQaOpKEJMi?= =?us-ascii?Q?DCtnaacWT7eKAjgMrYXnpBmCnzu6RWPbsl3tvAnrNRrA0/AdQyiJN49lRxFR?= =?us-ascii?Q?M8vTqLPl8XLwJjBYm71KbyuOVfXhNW7MR5e2QvTOyt6FFK3ohGIF/SjxTfQZ?= =?us-ascii?Q?3Yyl8+F9oC8NUNqPu34fyD8eCyPjkEyVuUacpKO4Jqms24oYov8m3ZYX5BHq?= =?us-ascii?Q?gxfQW4g7FQDsTLRspPqyIxZtsKboWMfXEF94xJU2RweltphaXEbo2zrfea28?= =?us-ascii?Q?IM85RYlL+v5lg9JVBdJKZ//J2R1e/vrx6zn0M+Cn9a9V6ooba9GwJbTKrMzf?= =?us-ascii?Q?9N38V1K5q5XaKVKHAfOdzBFvV94st2O6/KYoI1Ak3kIF0SwDXraN70gzRBnK?= =?us-ascii?Q?W8p4xdiSFenPGUNEqq4gBBPugFUgYfMiHf8zSJrokB4DVGupvNmLt12Ptsx3?= =?us-ascii?Q?aQ6JUpkkhZmrPNdVe1wOvICygiU1OxadQvhPxaJuLOepazJ+QkKtzioykk7W?= =?us-ascii?Q?XbIXKN5FZbfM/COna/hZAIad0981D1oCRdeJq5MV0fevJSUG+aax+p2wNidW?= =?us-ascii?Q?yq1nXt0eiYP+3K3RE35Vdfl1q2HKoizSDy6hY+ukv9ryFm0/JHV5JL8wKhBw?= =?us-ascii?Q?tEJ0zs483Mk6LuC6BMMJKssKestz+aTTLMPz9bMpQcwMtwvdSCV4f8/wGdcI?= =?us-ascii?Q?e2/E7E=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1315; 6:BE2w7UGYd5F/eT8K8r0AGDR50ueEkcbUwqn4X+kMEZ2uCOdMEDwjTe7enZSJ30CL9TXSSAZK1o0lFOWgcs0sVbOBlvUnE0kCtnokhhqeyKCv5IV87jHdL3FwFjRkv3KaGXXCTHFL74z3aqq86X36WQ6uKN4+CFVJOkPthK0CIEOfDzcuXAK95IVEJbMWIH7iAhrmPlekl3OhEQstvmYGd+v5++BMhGE3IZgU+2AScpJWpIyyhDrPyTsb7FrGz6a6VVIj0hsMwX6k7+TBxo9t38suJP456KZ1UHi+V4hafPoIsvETrfhQEhkAJs6Y3G1MDOpZh1kV7X34K3u5wBrNkqJSVydDLr6K0Fdeg3rUoQ0=; 5:Ev48CwIaN964Z/jzoVztlcjchUtapSOplVQFjzZX+zim+lUkTF1pnESA9E5Lmwe+FrVrGddeMDMyZ94c73M5dQVrHiZvk0KVD6M/M1BgNG5awBouY2EmGHkZc6rHClarnMkm5I0RcME2+YO4J/Jmzg==; 24:txJX+QnLsYJcLJ7mK/mDG4wwqRT8EUHAV6fRfMvVyICC12VZqqCDqY1vfuFR5ymHqSCuUaktPFAxkDdBKRqu119ED/FylFuDbMOAuecP2i4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BN6PR15MB1315; 7:s/UAe60sGWkch6Ipz7/pfL/WHBEueWLvSXPLAo+YQFmyxL0fVYLYIqKCftranTzKKiftLWE5FO0EOUNc6zqOJREyc801cfILgBCtjLmdDXEkwWRsDrheI3XzihIp2Q8wDxCl5FsGtHW5brPDOkefCNe5GvfaX6jNwnV8rC3aNy4lEvFEO5oC1LH9zSW1BOvhUZW/QDZEd2ZyM7YG+nT2PG3Y9D7pQZRI8e5NJLx1XVIcE0WwmlwIXXb50V5ymuVB6LUomuT97rL7cptHtgfCxTV+wZSs/ZN/OhqoWTDqPk04b2tKJZrpdRiaT8GibxDYveYJPDv9e7q0ZTsKHH4kjG5NTujclnQ9qrq5oUB+8JQ=; 20:5TtEAh9d8yBcqWt/u5A3GhXivsfsno8ty8AlvUGEWylvPM+McxtV02DOj6MqF8EcNqFF7xh5JR5WIHAmJlwPGa8lozZNgrBb9woYq+Dw3FbanzN3rZkCoPEMQYVifEARiS8AlPTva1WksvU2cm3u4nnaQevq4tyGeBz/KFLyf1k= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2016 19:04:45.5501 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR15MB1315 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2016-11-22_11:, , signatures=0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP NBD can become contended on its single connection. We have to serialize all writes and we can only process one read response at a time. Fix this by allowing userspace to provide multiple connections to a single nbd device. This coupled with block-mq drastically increases performance in multi-process cases. Thanks, Signed-off-by: Josef Bacik --- V4->V5: -Updated the multi-connection flag to be an actual unused value. V3->V4: -Fixed a problem where fast completes (or early completes) would crash because we were still accessing the bio's on the submit side. -Added a flag to disallow multi-connection support if the server doesn't explicitly allow for them. V2->V3: -Fixed a problem with the tag used for the requests. -Rebased onto the patch that enables async submit. V1->V2: -Dropped the index from nbd_cmd and just used the hctx->queue_num as HCH suggested -Added the pid attribute back to the /sys/block/nbd*/ directory for the recv pid. -Reworked the disconnect to simply send the command on all connections instead of sending a special command through the block layer. -Fixed some of the disconnect handling to be less verbose when we specifically request a disconnect. drivers/block/nbd.c | 382 +++++++++++++++++++++++++++++------------------ include/uapi/linux/nbd.h | 9 +- 2 files changed, 243 insertions(+), 148 deletions(-) diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index 9b33686..e683e22 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -41,26 +41,34 @@ #include +struct nbd_sock { + struct socket *sock; + struct mutex tx_lock; +}; + #define NBD_TIMEDOUT 0 #define NBD_DISCONNECT_REQUESTED 1 +#define NBD_DISCONNECTED 2 +#define NBD_RUNNING 3 struct nbd_device { u32 flags; unsigned long runtime_flags; - struct socket * sock; /* If == NULL, device is not ready, yet */ + struct nbd_sock **socks; int magic; struct blk_mq_tag_set tag_set; - struct mutex tx_lock; + struct mutex config_lock; struct gendisk *disk; + int num_connections; + atomic_t recv_threads; + wait_queue_head_t recv_wq; int blksize; loff_t bytesize; - /* protects initialization and shutdown of the socket */ - spinlock_t sock_lock; struct task_struct *task_recv; - struct task_struct *task_send; + struct task_struct *task_setup; #if IS_ENABLED(CONFIG_DEBUG_FS) struct dentry *dbg_dir; @@ -69,7 +77,7 @@ struct nbd_device { struct nbd_cmd { struct nbd_device *nbd; - struct list_head list; + struct completion send_complete; }; #if IS_ENABLED(CONFIG_DEBUG_FS) @@ -159,22 +167,20 @@ static void nbd_end_request(struct nbd_cmd *cmd) */ static void sock_shutdown(struct nbd_device *nbd) { - struct socket *sock; - - spin_lock(&nbd->sock_lock); + int i; - if (!nbd->sock) { - spin_unlock(&nbd->sock_lock); + if (nbd->num_connections == 0) + return; + if (test_and_set_bit(NBD_DISCONNECTED, &nbd->runtime_flags)) return; - } - - sock = nbd->sock; - dev_warn(disk_to_dev(nbd->disk), "shutting down socket\n"); - nbd->sock = NULL; - spin_unlock(&nbd->sock_lock); - kernel_sock_shutdown(sock, SHUT_RDWR); - sockfd_put(sock); + for (i = 0; i < nbd->num_connections; i++) { + struct nbd_sock *nsock = nbd->socks[i]; + mutex_lock(&nsock->tx_lock); + kernel_sock_shutdown(nsock->sock, SHUT_RDWR); + mutex_unlock(&nsock->tx_lock); + } + dev_warn(disk_to_dev(nbd->disk), "shutting down sockets\n"); } static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, @@ -182,35 +188,31 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, { struct nbd_cmd *cmd = blk_mq_rq_to_pdu(req); struct nbd_device *nbd = cmd->nbd; - struct socket *sock = NULL; - - spin_lock(&nbd->sock_lock); + dev_err(nbd_to_dev(nbd), "Connection timed out, shutting down connection\n"); set_bit(NBD_TIMEDOUT, &nbd->runtime_flags); - - if (nbd->sock) { - sock = nbd->sock; - get_file(sock->file); - } - - spin_unlock(&nbd->sock_lock); - if (sock) { - kernel_sock_shutdown(sock, SHUT_RDWR); - sockfd_put(sock); - } - req->errors++; - dev_err(nbd_to_dev(nbd), "Connection timed out, shutting down connection\n"); + + /* + * If our disconnect packet times out then we're already holding the + * config_lock and could deadlock here, so just set an error and return, + * we'll handle shutting everything down later. + */ + if (req->cmd_type == REQ_TYPE_DRV_PRIV) + return BLK_EH_HANDLED; + mutex_lock(&nbd->config_lock); + sock_shutdown(nbd); + mutex_unlock(&nbd->config_lock); return BLK_EH_HANDLED; } /* * Send or receive packet. */ -static int sock_xmit(struct nbd_device *nbd, int send, void *buf, int size, - int msg_flags) +static int sock_xmit(struct nbd_device *nbd, int index, int send, void *buf, + int size, int msg_flags) { - struct socket *sock = nbd->sock; + struct socket *sock = nbd->socks[index]->sock; int result; struct msghdr msg; struct kvec iov; @@ -254,19 +256,19 @@ static int sock_xmit(struct nbd_device *nbd, int send, void *buf, int size, return result; } -static inline int sock_send_bvec(struct nbd_device *nbd, struct bio_vec *bvec, - int flags) +static inline int sock_send_bvec(struct nbd_device *nbd, int index, + struct bio_vec *bvec, int flags) { int result; void *kaddr = kmap(bvec->bv_page); - result = sock_xmit(nbd, 1, kaddr + bvec->bv_offset, + result = sock_xmit(nbd, index, 1, kaddr + bvec->bv_offset, bvec->bv_len, flags); kunmap(bvec->bv_page); return result; } /* always call with the tx_lock held */ -static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd) +static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd, int index) { struct request *req = blk_mq_rq_from_pdu(cmd); int result, flags; @@ -274,10 +276,9 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd) unsigned long size = blk_rq_bytes(req); struct bio *bio; u32 type; + u32 tag = blk_mq_unique_tag(req); - if (req->cmd_type == REQ_TYPE_DRV_PRIV) - type = NBD_CMD_DISC; - else if (req_op(req) == REQ_OP_DISCARD) + if (req_op(req) == REQ_OP_DISCARD) type = NBD_CMD_TRIM; else if (req_op(req) == REQ_OP_FLUSH) type = NBD_CMD_FLUSH; @@ -289,16 +290,16 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd) memset(&request, 0, sizeof(request)); request.magic = htonl(NBD_REQUEST_MAGIC); request.type = htonl(type); - if (type != NBD_CMD_FLUSH && type != NBD_CMD_DISC) { + if (type != NBD_CMD_FLUSH) { request.from = cpu_to_be64((u64)blk_rq_pos(req) << 9); request.len = htonl(size); } - memcpy(request.handle, &req->tag, sizeof(req->tag)); + memcpy(request.handle, &tag, sizeof(tag)); dev_dbg(nbd_to_dev(nbd), "request %p: sending control (%s@%llu,%uB)\n", cmd, nbdcmd_to_ascii(type), (unsigned long long)blk_rq_pos(req) << 9, blk_rq_bytes(req)); - result = sock_xmit(nbd, 1, &request, sizeof(request), + result = sock_xmit(nbd, index, 1, &request, sizeof(request), (type == NBD_CMD_WRITE) ? MSG_MORE : 0); if (result <= 0) { dev_err(disk_to_dev(nbd->disk), @@ -323,7 +324,7 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd) flags = MSG_MORE; dev_dbg(nbd_to_dev(nbd), "request %p: sending %d bytes data\n", cmd, bvec.bv_len); - result = sock_send_bvec(nbd, &bvec, flags); + result = sock_send_bvec(nbd, index, &bvec, flags); if (result <= 0) { dev_err(disk_to_dev(nbd->disk), "Send data failed (result %d)\n", @@ -344,31 +345,34 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd) return 0; } -static inline int sock_recv_bvec(struct nbd_device *nbd, struct bio_vec *bvec) +static inline int sock_recv_bvec(struct nbd_device *nbd, int index, + struct bio_vec *bvec) { int result; void *kaddr = kmap(bvec->bv_page); - result = sock_xmit(nbd, 0, kaddr + bvec->bv_offset, bvec->bv_len, - MSG_WAITALL); + result = sock_xmit(nbd, index, 0, kaddr + bvec->bv_offset, + bvec->bv_len, MSG_WAITALL); kunmap(bvec->bv_page); return result; } /* NULL returned = something went wrong, inform userspace */ -static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd) +static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index) { int result; struct nbd_reply reply; struct nbd_cmd *cmd; struct request *req = NULL; u16 hwq; - int tag; + u32 tag; reply.magic = 0; - result = sock_xmit(nbd, 0, &reply, sizeof(reply), MSG_WAITALL); + result = sock_xmit(nbd, index, 0, &reply, sizeof(reply), MSG_WAITALL); if (result <= 0) { - dev_err(disk_to_dev(nbd->disk), - "Receive control failed (result %d)\n", result); + if (!test_bit(NBD_DISCONNECTED, &nbd->runtime_flags) && + !test_bit(NBD_DISCONNECT_REQUESTED, &nbd->runtime_flags)) + dev_err(disk_to_dev(nbd->disk), + "Receive control failed (result %d)\n", result); return ERR_PTR(result); } @@ -378,7 +382,7 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd) return ERR_PTR(-EPROTO); } - memcpy(&tag, reply.handle, sizeof(int)); + memcpy(&tag, reply.handle, sizeof(u32)); hwq = blk_mq_unique_tag_to_hwq(tag); if (hwq < nbd->tag_set.nr_hw_queues) @@ -390,7 +394,6 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd) return ERR_PTR(-ENOENT); } cmd = blk_mq_rq_to_pdu(req); - if (ntohl(reply.error)) { dev_err(disk_to_dev(nbd->disk), "Other side returned error (%d)\n", ntohl(reply.error)); @@ -404,7 +407,7 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd) struct bio_vec bvec; rq_for_each_segment(bvec, req, iter) { - result = sock_recv_bvec(nbd, &bvec); + result = sock_recv_bvec(nbd, index, &bvec); if (result <= 0) { dev_err(disk_to_dev(nbd->disk), "Receive data failed (result %d)\n", result); @@ -414,6 +417,9 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd) dev_dbg(nbd_to_dev(nbd), "request %p: got %d bytes data\n", cmd, bvec.bv_len); } + } else { + /* See the comment in nbd_queue_rq. */ + wait_for_completion(&cmd->send_complete); } return cmd; } @@ -432,25 +438,24 @@ static struct device_attribute pid_attr = { .show = pid_show, }; -static int nbd_thread_recv(struct nbd_device *nbd, struct block_device *bdev) +struct recv_thread_args { + struct work_struct work; + struct nbd_device *nbd; + int index; +}; + +static void recv_work(struct work_struct *work) { + struct recv_thread_args *args = container_of(work, + struct recv_thread_args, + work); + struct nbd_device *nbd = args->nbd; struct nbd_cmd *cmd; - int ret; + int ret = 0; BUG_ON(nbd->magic != NBD_MAGIC); - - sk_set_memalloc(nbd->sock->sk); - - ret = device_create_file(disk_to_dev(nbd->disk), &pid_attr); - if (ret) { - dev_err(disk_to_dev(nbd->disk), "device_create_file failed!\n"); - return ret; - } - - nbd_size_update(nbd, bdev); - while (1) { - cmd = nbd_read_stat(nbd); + cmd = nbd_read_stat(nbd, args->index); if (IS_ERR(cmd)) { ret = PTR_ERR(cmd); break; @@ -459,10 +464,14 @@ static int nbd_thread_recv(struct nbd_device *nbd, struct block_device *bdev) nbd_end_request(cmd); } - nbd_size_clear(nbd, bdev); - - device_remove_file(disk_to_dev(nbd->disk), &pid_attr); - return ret; + /* + * We got an error, shut everybody down if this wasn't the result of a + * disconnect request. + */ + if (ret && !test_bit(NBD_DISCONNECT_REQUESTED, &nbd->runtime_flags)) + sock_shutdown(nbd); + atomic_dec(&nbd->recv_threads); + wake_up(&nbd->recv_wq); } static void nbd_clear_req(struct request *req, void *data, bool reserved) @@ -480,26 +489,35 @@ static void nbd_clear_que(struct nbd_device *nbd) { BUG_ON(nbd->magic != NBD_MAGIC); - /* - * Because we have set nbd->sock to NULL under the tx_lock, all - * modifications to the list must have completed by now. - */ - BUG_ON(nbd->sock); - blk_mq_tagset_busy_iter(&nbd->tag_set, nbd_clear_req, NULL); dev_dbg(disk_to_dev(nbd->disk), "queue cleared\n"); } -static void nbd_handle_cmd(struct nbd_cmd *cmd) +static void nbd_handle_cmd(struct nbd_cmd *cmd, int index) { struct request *req = blk_mq_rq_from_pdu(cmd); struct nbd_device *nbd = cmd->nbd; + struct nbd_sock *nsock; + + if (index >= nbd->num_connections) { + dev_err(disk_to_dev(nbd->disk), + "Attempted send on invalid socket\n"); + goto error_out; + } + + if (test_bit(NBD_DISCONNECTED, &nbd->runtime_flags)) { + dev_err(disk_to_dev(nbd->disk), + "Attempted send on closed socket\n"); + goto error_out; + } - if (req->cmd_type != REQ_TYPE_FS) + if (req->cmd_type != REQ_TYPE_FS && + req->cmd_type != REQ_TYPE_DRV_PRIV) goto error_out; - if (rq_data_dir(req) == WRITE && + if (req->cmd_type == REQ_TYPE_FS && + rq_data_dir(req) == WRITE && (nbd->flags & NBD_FLAG_READ_ONLY)) { dev_err(disk_to_dev(nbd->disk), "Write on read-only\n"); @@ -508,23 +526,22 @@ static void nbd_handle_cmd(struct nbd_cmd *cmd) req->errors = 0; - mutex_lock(&nbd->tx_lock); - nbd->task_send = current; - if (unlikely(!nbd->sock)) { - mutex_unlock(&nbd->tx_lock); + nsock = nbd->socks[index]; + mutex_lock(&nsock->tx_lock); + if (unlikely(!nsock->sock)) { + mutex_unlock(&nsock->tx_lock); dev_err(disk_to_dev(nbd->disk), "Attempted send on closed socket\n"); goto error_out; } - if (nbd_send_cmd(nbd, cmd) != 0) { + if (nbd_send_cmd(nbd, cmd, index) != 0) { dev_err(disk_to_dev(nbd->disk), "Request send failed\n"); req->errors++; nbd_end_request(cmd); } - nbd->task_send = NULL; - mutex_unlock(&nbd->tx_lock); + mutex_unlock(&nsock->tx_lock); return; @@ -538,39 +555,70 @@ static int nbd_queue_rq(struct blk_mq_hw_ctx *hctx, { struct nbd_cmd *cmd = blk_mq_rq_to_pdu(bd->rq); + /* + * Since we look at the bio's to send the request over the network we + * need to make sure the completion work doesn't mark this request done + * before we are done doing our send. This keeps us from dereferencing + * freed data if we have particularly fast completions (ie we get the + * completion before we exit sock_xmit on the last bvec) or in the case + * that the server is misbehaving (or there was an error) before we're + * done sending everything over the wire. + */ + init_completion(&cmd->send_complete); blk_mq_start_request(bd->rq); - nbd_handle_cmd(cmd); + nbd_handle_cmd(cmd, hctx->queue_num); + complete(&cmd->send_complete); + return BLK_MQ_RQ_QUEUE_OK; } -static int nbd_set_socket(struct nbd_device *nbd, struct socket *sock) +static int nbd_add_socket(struct nbd_device *nbd, struct socket *sock) { - int ret = 0; - - spin_lock_irq(&nbd->sock_lock); + struct nbd_sock **socks; + struct nbd_sock *nsock; - if (nbd->sock) { - ret = -EBUSY; - goto out; + if (!nbd->task_setup) + nbd->task_setup = current; + if (nbd->task_setup != current) { + dev_err(disk_to_dev(nbd->disk), + "Device being setup by another task"); + return -EINVAL; } - nbd->sock = sock; + socks = krealloc(nbd->socks, (nbd->num_connections + 1) * + sizeof(struct nbd_sock *), GFP_KERNEL); + if (!socks) + return -ENOMEM; + nsock = kzalloc(sizeof(struct nbd_sock), GFP_KERNEL); + if (!nsock) + return -ENOMEM; -out: - spin_unlock_irq(&nbd->sock_lock); + nbd->socks = socks; - return ret; + mutex_init(&nsock->tx_lock); + nsock->sock = sock; + socks[nbd->num_connections++] = nsock; + + return 0; } /* Reset all properties of an NBD device */ static void nbd_reset(struct nbd_device *nbd) { + int i; + + for (i = 0; i < nbd->num_connections; i++) + kfree(nbd->socks[i]); + kfree(nbd->socks); + nbd->socks = NULL; nbd->runtime_flags = 0; nbd->blksize = 1024; nbd->bytesize = 0; set_capacity(nbd->disk, 0); nbd->flags = 0; nbd->tag_set.timeout = 0; + nbd->num_connections = 0; + nbd->task_setup = NULL; queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, nbd->disk->queue); } @@ -596,48 +644,67 @@ static void nbd_parse_flags(struct nbd_device *nbd, struct block_device *bdev) blk_queue_write_cache(nbd->disk->queue, false, false); } +static void send_disconnects(struct nbd_device *nbd) +{ + struct nbd_request request = {}; + int i, ret; + + request.magic = htonl(NBD_REQUEST_MAGIC); + request.type = htonl(NBD_CMD_DISC); + + for (i = 0; i < nbd->num_connections; i++) { + ret = sock_xmit(nbd, i, 1, &request, sizeof(request), 0); + if (ret <= 0) + dev_err(disk_to_dev(nbd->disk), + "Send disconnect failed %d\n", ret); + } +} + static int nbd_dev_dbg_init(struct nbd_device *nbd); static void nbd_dev_dbg_close(struct nbd_device *nbd); -/* Must be called with tx_lock held */ - +/* Must be called with config_lock held */ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, unsigned int cmd, unsigned long arg) { switch (cmd) { case NBD_DISCONNECT: { - struct request *sreq; - dev_info(disk_to_dev(nbd->disk), "NBD_DISCONNECT\n"); - if (!nbd->sock) + if (!nbd->socks) return -EINVAL; - sreq = blk_mq_alloc_request(bdev_get_queue(bdev), WRITE, 0); - if (!sreq) - return -ENOMEM; - - mutex_unlock(&nbd->tx_lock); + mutex_unlock(&nbd->config_lock); fsync_bdev(bdev); - mutex_lock(&nbd->tx_lock); - sreq->cmd_type = REQ_TYPE_DRV_PRIV; + mutex_lock(&nbd->config_lock); /* Check again after getting mutex back. */ - if (!nbd->sock) { - blk_mq_free_request(sreq); + if (!nbd->socks) return -EINVAL; - } - set_bit(NBD_DISCONNECT_REQUESTED, &nbd->runtime_flags); - - nbd_send_cmd(nbd, blk_mq_rq_to_pdu(sreq)); - blk_mq_free_request(sreq); + if (!test_and_set_bit(NBD_DISCONNECT_REQUESTED, + &nbd->runtime_flags)) + send_disconnects(nbd); return 0; } - + case NBD_CLEAR_SOCK: sock_shutdown(nbd); nbd_clear_que(nbd); kill_bdev(bdev); + nbd_bdev_reset(bdev); + /* + * We want to give the run thread a chance to wait for everybody + * to clean up and then do it's own cleanup. + */ + if (!test_bit(NBD_RUNNING, &nbd->runtime_flags)) { + int i; + + for (i = 0; i < nbd->num_connections; i++) + kfree(nbd->socks[i]); + kfree(nbd->socks); + nbd->socks = NULL; + nbd->num_connections = 0; + } return 0; case NBD_SET_SOCK: { @@ -647,7 +714,7 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, if (!sock) return err; - err = nbd_set_socket(nbd, sock); + err = nbd_add_socket(nbd, sock); if (!err && max_part) bdev->bd_invalidated = 1; @@ -676,26 +743,58 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, return 0; case NBD_DO_IT: { - int error; + struct recv_thread_args *args; + int num_connections = nbd->num_connections; + int error, i; if (nbd->task_recv) return -EBUSY; - if (!nbd->sock) + if (!nbd->socks) return -EINVAL; + if (num_connections > 1 && + !(nbd->flags & NBD_FLAG_CAN_MULTI_CONN)) { + dev_err(disk_to_dev(nbd->disk), "server does not support multiple connections per device.\n"); + goto out_err; + } - /* We have to claim the device under the lock */ + set_bit(NBD_RUNNING, &nbd->runtime_flags); + blk_mq_update_nr_hw_queues(&nbd->tag_set, nbd->num_connections); + args = kcalloc(num_connections, sizeof(*args), GFP_KERNEL); + if (!args) + goto out_err; nbd->task_recv = current; - mutex_unlock(&nbd->tx_lock); + mutex_unlock(&nbd->config_lock); nbd_parse_flags(nbd, bdev); + error = device_create_file(disk_to_dev(nbd->disk), &pid_attr); + if (error) { + dev_err(disk_to_dev(nbd->disk), "device_create_file failed!\n"); + goto out_recv; + } + + nbd_size_update(nbd, bdev); + nbd_dev_dbg_init(nbd); - error = nbd_thread_recv(nbd, bdev); + for (i = 0; i < num_connections; i++) { + sk_set_memalloc(nbd->socks[i]->sock->sk); + atomic_inc(&nbd->recv_threads); + INIT_WORK(&args[i].work, recv_work); + args[i].nbd = nbd; + args[i].index = i; + queue_work(system_long_wq, &args[i].work); + } + wait_event_interruptible(nbd->recv_wq, + atomic_read(&nbd->recv_threads) == 0); + for (i = 0; i < num_connections; i++) + flush_work(&args[i].work); nbd_dev_dbg_close(nbd); - - mutex_lock(&nbd->tx_lock); + nbd_size_clear(nbd, bdev); + device_remove_file(disk_to_dev(nbd->disk), &pid_attr); +out_recv: + mutex_lock(&nbd->config_lock); nbd->task_recv = NULL; - +out_err: sock_shutdown(nbd); nbd_clear_que(nbd); kill_bdev(bdev); @@ -708,7 +807,6 @@ static int __nbd_ioctl(struct block_device *bdev, struct nbd_device *nbd, error = -ETIMEDOUT; nbd_reset(nbd); - return error; } @@ -740,9 +838,9 @@ static int nbd_ioctl(struct block_device *bdev, fmode_t mode, BUG_ON(nbd->magic != NBD_MAGIC); - mutex_lock(&nbd->tx_lock); + mutex_lock(&nbd->config_lock); error = __nbd_ioctl(bdev, nbd, cmd, arg); - mutex_unlock(&nbd->tx_lock); + mutex_unlock(&nbd->config_lock); return error; } @@ -762,8 +860,6 @@ static int nbd_dbg_tasks_show(struct seq_file *s, void *unused) if (nbd->task_recv) seq_printf(s, "recv: %d\n", task_pid_nr(nbd->task_recv)); - if (nbd->task_send) - seq_printf(s, "send: %d\n", task_pid_nr(nbd->task_send)); return 0; } @@ -887,9 +983,7 @@ static int nbd_init_request(void *data, struct request *rq, unsigned int numa_node) { struct nbd_cmd *cmd = blk_mq_rq_to_pdu(rq); - cmd->nbd = data; - INIT_LIST_HEAD(&cmd->list); return 0; } @@ -999,13 +1093,13 @@ static int __init nbd_init(void) for (i = 0; i < nbds_max; i++) { struct gendisk *disk = nbd_dev[i].disk; nbd_dev[i].magic = NBD_MAGIC; - spin_lock_init(&nbd_dev[i].sock_lock); - mutex_init(&nbd_dev[i].tx_lock); + mutex_init(&nbd_dev[i].config_lock); disk->major = NBD_MAJOR; disk->first_minor = i << part_shift; disk->fops = &nbd_fops; disk->private_data = &nbd_dev[i]; sprintf(disk->disk_name, "nbd%d", i); + init_waitqueue_head(&nbd_dev[i].recv_wq); nbd_reset(&nbd_dev[i]); add_disk(disk); } diff --git a/include/uapi/linux/nbd.h b/include/uapi/linux/nbd.h index e08e413..f063cff 100644 --- a/include/uapi/linux/nbd.h +++ b/include/uapi/linux/nbd.h @@ -38,11 +38,12 @@ enum { }; /* values for flags field */ -#define NBD_FLAG_HAS_FLAGS (1 << 0) /* nbd-server supports flags */ -#define NBD_FLAG_READ_ONLY (1 << 1) /* device is read-only */ -#define NBD_FLAG_SEND_FLUSH (1 << 2) /* can flush writeback cache */ +#define NBD_FLAG_HAS_FLAGS (1 << 0) /* nbd-server supports flags */ +#define NBD_FLAG_READ_ONLY (1 << 1) /* device is read-only */ +#define NBD_FLAG_SEND_FLUSH (1 << 2) /* can flush writeback cache */ /* there is a gap here to match userspace */ -#define NBD_FLAG_SEND_TRIM (1 << 5) /* send trim/discard */ +#define NBD_FLAG_SEND_TRIM (1 << 5) /* send trim/discard */ +#define NBD_FLAG_CAN_MULTI_CONN (1 << 7) /* Server supports multiple connections per export. */ /* userspace doesn't need the nbd_device structure */