From patchwork Mon Jan 25 18:01:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vivek Goyal X-Patchwork-Id: 12043915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB602C433E0 for ; Mon, 25 Jan 2021 18:07:14 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0AAAB2251F for ; Mon, 25 Jan 2021 18:07:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0AAAB2251F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47280 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l46Gf-0005Ns-1Z for qemu-devel@archiver.kernel.org; Mon, 25 Jan 2021 13:07:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45244) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l46Be-0001Qt-Ug for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:08 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:32885) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1l46BZ-0006lG-Vu for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:02 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1611597717; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JdE/mjhcHwPCn5Y64DNoT5FJDKEiUtgq25W1YQzZvl0=; b=gwZWEi+B6wmnOFjHPvH34hMaNnzSo6sa8CPvQk6RjHMZYKZzunMA6gaNKcF+tkl9hAPb27 iTpyn6IvCTJl0G/zpndLXKiypPRTjIsO/wDzMnxB3fztesXgiWbQfF0kNgV1UtA/SeGw32 ptqGhPSLlLLR+otB1b0NtjKtrHIQ5AI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-520-o-Vo9qbIMG-50QFFs_HfQg-1; Mon, 25 Jan 2021 13:01:55 -0500 X-MC-Unique: o-Vo9qbIMG-50QFFs_HfQg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 93CA1107ACE3 for ; Mon, 25 Jan 2021 18:01:54 +0000 (UTC) Received: from horse.redhat.com (ovpn-116-119.rdu2.redhat.com [10.10.116.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id A12341002D43; Mon, 25 Jan 2021 18:01:38 +0000 (UTC) Received: by horse.redhat.com (Postfix, from userid 10451) id 3662E223D98; Mon, 25 Jan 2021 13:01:38 -0500 (EST) From: Vivek Goyal To: qemu-devel@nongnu.org, virtio-fs@redhat.com Subject: [PATCH 1/6] virtiofsd: Drop ->vu_dispatch_rwlock while waiting for thread to exit Date: Mon, 25 Jan 2021 13:01:10 -0500 Message-Id: <20210125180115.22936-2-vgoyal@redhat.com> In-Reply-To: <20210125180115.22936-1-vgoyal@redhat.com> References: <20210125180115.22936-1-vgoyal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=vgoyal@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=vgoyal@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.255, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, stefanha@redhat.com, dgilbert@redhat.com, vgoyal@redhat.com, mst@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When we are shutting down virtqueues, virtio_loop() receives a message VHOST_USER_GET_VRING_BASE from master. We acquire ->vu_dispatch_rwlock and get into the process of shutting down virtqueue. In one of the final steps, we are waiting for fv_queue_thread() to exit/finish and wait with ->vu_dispatch_rwlock held. But it is possible that fv_queue_thread() itself is waiting to get ->vu_dispatch_rwlock (With --thread-pool=0 option). If requests are being processed by fv_queue_worker(), then fv_queue_worker() can wait for the ->vu_dispatch_rwlock, and fv_queue_thread() will wait for fv_queue_worker() before thread pool can be stopped. IOW, if guest is shutdown uncleanly (some sort of emergency reboot), it is possible that virtiofsd is processing a fs request and qemu initiates device shutdown sequence. In that case there seem to be two options. Either abort the existing request completely or let existing request finish. This patch is taking second approach. That is drop the ->vu_dispatch_rwlock temporarily so that fv_queue_thread() can finish and deadlock does not happen. ->vu_dispatch_rwlock provides mutual exclusion between virtio_loop() (handling vhost-user protocol messages) and fv_queue_thread() (handling fuse filesystem requests). Rational seems to be that protocol message might change queue memory mappings, so we don't want both to proceed at the same time. In this case queue is shutting down, so I hope it is fine for fv_queue_thread() to send response back while virtio_loop() is still waiting (and not handling any further vho-user protocol messages). IOW, assumption here is that while virto_loop is blocked processing VHOST_USER_GET_VRING_BASE message, it is still ok to send back the response on vq by fv_queue_thread(). Signed-off-by: Vivek Goyal --- tools/virtiofsd/fuse_virtio.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tools/virtiofsd/fuse_virtio.c b/tools/virtiofsd/fuse_virtio.c index 9577eaa68d..6805d8ba01 100644 --- a/tools/virtiofsd/fuse_virtio.c +++ b/tools/virtiofsd/fuse_virtio.c @@ -813,11 +813,20 @@ static void fv_queue_cleanup_thread(struct fv_VuDev *vud, int qidx) fuse_log(FUSE_LOG_ERR, "Eventfd_write for queue %d: %s\n", qidx, strerror(errno)); } + + /* + * Drop ->vu_dispath_rwlock and reacquire. We are about to wait for + * for fv_queue_thread() and that might require ->vu_dispatch_rwlock + * to finish. + */ + pthread_rwlock_unlock(&vud->vu_dispatch_rwlock); ret = pthread_join(ourqi->thread, NULL); if (ret) { fuse_log(FUSE_LOG_ERR, "%s: Failed to join thread idx %d err %d\n", __func__, qidx, ret); } + pthread_rwlock_wrlock(&vud->vu_dispatch_rwlock); + pthread_mutex_destroy(&ourqi->vq_lock); close(ourqi->kill_fd); ourqi->kick_fd = -1; @@ -952,7 +961,11 @@ int virtio_loop(struct fuse_session *se) /* * Make sure all fv_queue_thread()s quit on exit, as we're about to * free virtio dev and fuse session, no one should access them anymore. + * Hold ->vu_dispatch_rwlock in write mode as fv_queue_cleanup_thread() + * assumes mutex is locked and unlocks/re-locks it. */ + + pthread_rwlock_wrlock(&se->virtio_dev->vu_dispatch_rwlock); for (int i = 0; i < se->virtio_dev->nqueues; i++) { if (!se->virtio_dev->qi[i]) { continue; @@ -961,6 +974,7 @@ int virtio_loop(struct fuse_session *se) fuse_log(FUSE_LOG_INFO, "%s: Stopping queue %d thread\n", __func__, i); fv_queue_cleanup_thread(se->virtio_dev, i); } + pthread_rwlock_unlock(&se->virtio_dev->vu_dispatch_rwlock); fuse_log(FUSE_LOG_INFO, "%s: Exit\n", __func__); From patchwork Mon Jan 25 18:01:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vivek Goyal X-Patchwork-Id: 12043909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0588C433DB for ; Mon, 25 Jan 2021 18:04:55 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3E59D2251F for ; Mon, 25 Jan 2021 18:04:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E59D2251F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:42334 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l46EQ-0003Gp-1t for qemu-devel@archiver.kernel.org; Mon, 25 Jan 2021 13:04:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45186) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l46Bc-0001OV-0R for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:00 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:57761) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1l46BY-0006km-Up for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:01:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1611597714; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Go5qmUqyPVhLjJ+A7W/OFRJFVNMBRxbvYLjmjXBhUUo=; b=dDt9MElGp4Nvy1J3SOUFgxbUKmOc/pAIJ3gBVpXubRQnOGTSInz6SgAERUYSDxFXO9KdmU qjMzdOLpci/7QuVsZY9sSaFck1K/uXbky8+35i3HdJWrCsAuKnFfGv14qSrLmMY2vkXUuU Ae9Fzmcs2kEvOCTkgWSkTPjNpK+WCsA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-532-EsoIWbSdO1K3EDcob8FgWQ-1; Mon, 25 Jan 2021 13:01:52 -0500 X-MC-Unique: EsoIWbSdO1K3EDcob8FgWQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C1C80100C606 for ; Mon, 25 Jan 2021 18:01:51 +0000 (UTC) Received: from horse.redhat.com (ovpn-116-119.rdu2.redhat.com [10.10.116.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id A16D26F983; Mon, 25 Jan 2021 18:01:38 +0000 (UTC) Received: by horse.redhat.com (Postfix, from userid 10451) id 39C61223D99; Mon, 25 Jan 2021 13:01:38 -0500 (EST) From: Vivek Goyal To: qemu-devel@nongnu.org, virtio-fs@redhat.com Subject: [PATCH 2/6] libvhost-user: Use slave_mutex in all slave messages Date: Mon, 25 Jan 2021 13:01:11 -0500 Message-Id: <20210125180115.22936-3-vgoyal@redhat.com> In-Reply-To: <20210125180115.22936-1-vgoyal@redhat.com> References: <20210125180115.22936-1-vgoyal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=vgoyal@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=vgoyal@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.255, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, stefanha@redhat.com, dgilbert@redhat.com, vgoyal@redhat.com, mst@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" dev->slave_mutex needs to be taken when sending messages on slave_fd. Currently _vu_queue_notify() does not do that. Introduce a helper vu_message_slave_send_receive() which sends as well as receive response. Use this helper in all the paths which send message on slave_fd channel. Signed-off-by: Vivek Goyal Reviewed-by: Greg Kurz --- subprojects/libvhost-user/libvhost-user.c | 50 ++++++++++++----------- 1 file changed, 27 insertions(+), 23 deletions(-) diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c index 4cf4aef63d..7a56c56dc8 100644 --- a/subprojects/libvhost-user/libvhost-user.c +++ b/subprojects/libvhost-user/libvhost-user.c @@ -403,7 +403,7 @@ vu_send_reply(VuDev *dev, int conn_fd, VhostUserMsg *vmsg) * Processes a reply on the slave channel. * Entered with slave_mutex held and releases it before exit. * Returns true on success. - * *payload is written on success + * *payload is written on success, if payload is not NULL. */ static bool vu_process_message_reply(VuDev *dev, const VhostUserMsg *vmsg, @@ -427,7 +427,9 @@ vu_process_message_reply(VuDev *dev, const VhostUserMsg *vmsg, goto out; } - *payload = msg_reply.payload.u64; + if (payload) { + *payload = msg_reply.payload.u64; + } result = true; out: @@ -435,6 +437,25 @@ out: return result; } +/* Returns true on success, false otherwise */ +static bool +vu_message_slave_send_receive(VuDev *dev, VhostUserMsg *vmsg, uint64_t *payload) +{ + pthread_mutex_lock(&dev->slave_mutex); + if (!vu_message_write(dev, dev->slave_fd, vmsg)) { + pthread_mutex_unlock(&dev->slave_mutex); + return false; + } + + if ((vmsg->flags & VHOST_USER_NEED_REPLY_MASK) == 0) { + pthread_mutex_unlock(&dev->slave_mutex); + return true; + } + + /* Also unlocks the slave_mutex */ + return vu_process_message_reply(dev, vmsg, payload); +} + /* Kick the log_call_fd if required. */ static void vu_log_kick(VuDev *dev) @@ -1340,16 +1361,8 @@ bool vu_set_queue_host_notifier(VuDev *dev, VuVirtq *vq, int fd, return false; } - pthread_mutex_lock(&dev->slave_mutex); - if (!vu_message_write(dev, dev->slave_fd, &vmsg)) { - pthread_mutex_unlock(&dev->slave_mutex); - return false; - } - - /* Also unlocks the slave_mutex */ - res = vu_process_message_reply(dev, &vmsg, &payload); + res = vu_message_slave_send_receive(dev, &vmsg, &payload); res = res && (payload == 0); - return res; } @@ -2395,10 +2408,7 @@ static void _vu_queue_notify(VuDev *dev, VuVirtq *vq, bool sync) vmsg.flags |= VHOST_USER_NEED_REPLY_MASK; } - vu_message_write(dev, dev->slave_fd, &vmsg); - if (ack) { - vu_message_read_default(dev, dev->slave_fd, &vmsg); - } + vu_message_slave_send_receive(dev, &vmsg, NULL); return; } @@ -2942,17 +2952,11 @@ int64_t vu_fs_cache_request(VuDev *dev, VhostUserSlaveRequest req, int fd, return -EINVAL; } - pthread_mutex_lock(&dev->slave_mutex); - if (!vu_message_write(dev, dev->slave_fd, &vmsg)) { - pthread_mutex_unlock(&dev->slave_mutex); - return -EIO; - } - - /* Also unlocks the slave_mutex */ - res = vu_process_message_reply(dev, &vmsg, &payload); + res = vu_message_slave_send_receive(dev, &vmsg, &payload); if (!res) { return -EIO; } + /* * Payload is delivered as uint64_t but is actually signed for * errors. From patchwork Mon Jan 25 18:01:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vivek Goyal X-Patchwork-Id: 12043921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20717C433DB for ; Mon, 25 Jan 2021 18:10:02 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BD64E22583 for ; Mon, 25 Jan 2021 18:10:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BD64E22583 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:55800 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l46JM-0000bV-UB for qemu-devel@archiver.kernel.org; Mon, 25 Jan 2021 13:10:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45262) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l46Bl-0001SV-3U for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:09 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:26250) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1l46BY-0006l6-V4 for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1611597716; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YmEUC9rCNqDqwD5zQ92uQmGlnwpe6P4xyRXIFapD23A=; b=UPQB+x2nxElTB4cwmsVJVMwVc2X3tuS3ArFFsSNaM5kq8F6+uukKkeoWUWw9HOygt4DDwC RUC7k9bPm+k5sdqW+VTlcb4IP8iAHe7jam3Wqc9vvNZKJRJhm1+hj3eO5uAwbEx1LaU+gq vvQa4l4ZWd3p1chcrDL2+zxNXThvceM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-572-2D5W2i-RM0mrobIlyaL-_Q-1; Mon, 25 Jan 2021 13:01:53 -0500 X-MC-Unique: 2D5W2i-RM0mrobIlyaL-_Q-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A78B91081B2B for ; Mon, 25 Jan 2021 18:01:52 +0000 (UTC) Received: from horse.redhat.com (ovpn-116-119.rdu2.redhat.com [10.10.116.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id AD59A5D6DC; Mon, 25 Jan 2021 18:01:38 +0000 (UTC) Received: by horse.redhat.com (Postfix, from userid 10451) id 3DD4E225FCD; Mon, 25 Jan 2021 13:01:38 -0500 (EST) From: Vivek Goyal To: qemu-devel@nongnu.org, virtio-fs@redhat.com Subject: [PATCH 3/6] vhost-user: Return error code from slave_read() Date: Mon, 25 Jan 2021 13:01:12 -0500 Message-Id: <20210125180115.22936-4-vgoyal@redhat.com> In-Reply-To: <20210125180115.22936-1-vgoyal@redhat.com> References: <20210125180115.22936-1-vgoyal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=vgoyal@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=vgoyal@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.255, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, stefanha@redhat.com, dgilbert@redhat.com, vgoyal@redhat.com, mst@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Right now slave_read() is called through main event loop and does not return error. In next few patches I want to call slave_read() from vhost device shutdown path as well and want to know if an error happened so that caller can give up and return error accordingly. Hence, create helper function do_slave_read(), which returns an integer. Success is 0 and negative number is error code. slave_read() calls do_slave_read() and ignores error code. Signed-off-by: Vivek Goyal --- hw/virtio/vhost-user.c | 43 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 37 insertions(+), 6 deletions(-) diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index d95dbc39e3..867cac034f 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -1401,7 +1401,7 @@ static uint64_t vhost_user_slave_handle_vring_host_notifier( return false; } -static void slave_read(void *opaque) +static int do_slave_read(void *opaque) { struct vhost_dev *dev = opaque; struct vhost_user *u = dev->opaque; @@ -1432,13 +1432,22 @@ static void slave_read(void *opaque) size = recvmsg(u->slave_fd, &msgh, 0); } while (size < 0 && (errno == EINTR || errno == EAGAIN)); - if (size != VHOST_USER_HDR_SIZE) { + if (size < 0) { + ret = -errno; error_report("Failed to read from slave."); goto err; } + if (size != VHOST_USER_HDR_SIZE) { + error_report("Failed to read %lu bytes from slave.", + VHOST_USER_HDR_SIZE); + ret = -EBADMSG; + goto err; + } + if (msgh.msg_flags & MSG_CTRUNC) { error_report("Truncated message."); + ret = -EBADMSG; goto err; } @@ -1456,6 +1465,7 @@ static void slave_read(void *opaque) error_report("Failed to read msg header." " Size %d exceeds the maximum %zu.", hdr.size, VHOST_USER_PAYLOAD_SIZE); + ret = -EBADMSG; goto err; } @@ -1464,8 +1474,15 @@ static void slave_read(void *opaque) size = read(u->slave_fd, &payload, hdr.size); } while (size < 0 && (errno == EINTR || errno == EAGAIN)); - if (size != hdr.size) { + if (size == -1) { error_report("Failed to read payload from slave."); + ret = errno; + goto err; + } + + if (size != hdr.size) { + error_report("Failed to read %d payload bytes from slave.", hdr.size); + ret = -EBADMSG; goto err; } @@ -1529,13 +1546,22 @@ static void slave_read(void *opaque) size = writev(u->slave_fd, iovec, ARRAY_SIZE(iovec)); } while (size < 0 && (errno == EINTR || errno == EAGAIN)); - if (size != VHOST_USER_HDR_SIZE + hdr.size) { + if (size == -1) { error_report("Failed to send msg reply to slave."); + ret = -errno; + goto err; + } + + if (size != VHOST_USER_HDR_SIZE + hdr.size) { + error_report("Failed to send msg reply to slave. Wrote %d bytes" + " expected %lu bytes.", size, + VHOST_USER_HDR_SIZE + hdr.size); + ret = -EIO; goto err; } } - return; + return 0; err: qemu_set_fd_handler(u->slave_fd, NULL, NULL, NULL); @@ -1546,7 +1572,12 @@ err: close(fd[i]); } } - return; + return ret; +} + +static void slave_read(void *opaque) +{ + do_slave_read(opaque); } static int vhost_setup_slave_channel(struct vhost_dev *dev) From patchwork Mon Jan 25 18:01:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vivek Goyal X-Patchwork-Id: 12043917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83FAFC433E0 for ; Mon, 25 Jan 2021 18:07:19 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EEEBD2251F for ; Mon, 25 Jan 2021 18:07:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EEEBD2251F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47826 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l46Gj-0005hC-To for qemu-devel@archiver.kernel.org; Mon, 25 Jan 2021 13:07:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45260) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l46Bl-0001SU-2c for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:09 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:44661) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1l46BY-0006kk-V4 for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1611597714; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DVU4Q6xMtVOhCV7Mcgnky2YsorHb7GhIScj9GdyYhiE=; b=K3xhMG1xXE/EFvMb3cND7amcacKnOVd5cXM4HHOcFMcclPkzozuTYetem9tPEu6T2TjRa4 EqJltnuRNpbbShF+CYfH2UP+OUFT4VA2gKui3Tap/jDQ7Sycxck2Hfn5yFiO67CNatVDvG /dNt9B9aJu9sXlmEJvobWfb1QCjJT2A= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-165-Sun8uxuYMC-1aNJlQhanOg-1; Mon, 25 Jan 2021 13:01:52 -0500 X-MC-Unique: Sun8uxuYMC-1aNJlQhanOg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A45E11842149 for ; Mon, 25 Jan 2021 18:01:51 +0000 (UTC) Received: from horse.redhat.com (ovpn-116-119.rdu2.redhat.com [10.10.116.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id B070B71D4F; Mon, 25 Jan 2021 18:01:38 +0000 (UTC) Received: by horse.redhat.com (Postfix, from userid 10451) id 44BF7225FCE; Mon, 25 Jan 2021 13:01:38 -0500 (EST) From: Vivek Goyal To: qemu-devel@nongnu.org, virtio-fs@redhat.com Subject: [PATCH 4/6] qemu, vhost-user: Extend protocol to start/stop/flush slave channel Date: Mon, 25 Jan 2021 13:01:13 -0500 Message-Id: <20210125180115.22936-5-vgoyal@redhat.com> In-Reply-To: <20210125180115.22936-1-vgoyal@redhat.com> References: <20210125180115.22936-1-vgoyal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=vgoyal@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=vgoyal@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.255, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, stefanha@redhat.com, dgilbert@redhat.com, vgoyal@redhat.com, mst@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Currently we don't have a mechanism to flush slave channel while shutting down vhost-user device and that can result a deadlock. Consider following scenario. 1. Slave gets a request from guest on virtqueue (say index 1, vq1), to map a portion of file in qemu address space. 2. Thread serving vq1 receives this request and sends a message to qemu on slave channel/fd and gets blocked in waiting for a response from qemu. 3. In the mean time, user does "echo b > /proc/sysrq-trigger" in guest. This leads to qemu reset and ultimately in main thread we end up in vhost_dev_stop() which is trying to shutdown virtqueues for the device. 4. Slave gets VHOST_USER_GET_VRING_BASE message to shutdown a virtqueue on unix socket being used for communication. 5. Slave tries to shutdown the thread serving vq1 and waits for it to terminate. But vq1 thread can't terminate because it is waiting for a response from qemu on slave_fd. And qemu is not processing slave_fd anymore as qemu is ressing (and not running main event loop anymore) and is waiting for a response to VHOST_USER_GET_VRING_BASE. So we have a deadlock. Qemu is waiting on slave to response to VHOST_USER_GET_VRING_BASE message and slave is waiting on qemu to respond to request it sent on slave_fd. I can reliably reproduce this race with virtio-fs. Hence here is the proposal to solve this problem. Enhance vhost-user protocol to properly shutdown slave_fd channel. And if there are pending requests, flush the channel completely before sending the request to shutdown virtqueues. New workflow to shutdown slave channel is. - Qemu sends VHOST_USER_STOP_SLAVE_CHANNEL request to slave. It waits for an reply from guest. - Then qemu sits in a tight loop waiting for VHOST_USER_SLAVE_STOP_CHANNEL_COMPLETE message from slave on slave_fd. And while waiting for this message, qemu continues to process requests on slave_fd to flush any pending requests. This will unblock threads in slave and allow slave to shutdown slave channel. - Once qemu gets VHOST_USER_SLAVE_STOP_CHANNEL_COMPLETE message, it knows no more requests will come on slave_fd and it can continue to shutdown virtqueues now. - Once device starts again, qemu will send VHOST_USER_START_SLAVE_CHANNEL message to slave to open the slave channel and receive requests. IOW, this allows for proper shutdown of slave channel, making sure no threads in slave are blocked on sending/receiving message. And this in-turn allows for shutting down of virtqueues, hence resolving the deadlock. Signed-off-by: Vivek Goyal --- hw/virtio/vhost-user.c | 108 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 867cac034f..56be61d4e2 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -80,6 +80,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_RESET_DEVICE = 13, /* Feature 14 reserved for VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS. */ VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS = 15, + VHOST_USER_PROTOCOL_F_SLAVE_CH_START_STOP = 16, VHOST_USER_PROTOCOL_F_MAX }; @@ -125,6 +126,8 @@ typedef enum VhostUserRequest { VHOST_USER_GET_MAX_MEM_SLOTS = 36, VHOST_USER_ADD_MEM_REG = 37, VHOST_USER_REM_MEM_REG = 38, + VHOST_USER_START_SLAVE_CHANNEL = 39, + VHOST_USER_STOP_SLAVE_CHANNEL = 40, VHOST_USER_MAX } VhostUserRequest; @@ -139,6 +142,7 @@ typedef enum VhostUserSlaveRequest { VHOST_USER_SLAVE_FS_UNMAP = 7, VHOST_USER_SLAVE_FS_SYNC = 8, VHOST_USER_SLAVE_FS_IO = 9, + VHOST_USER_SLAVE_STOP_CHANNEL_COMPLETE = 10, VHOST_USER_SLAVE_MAX } VhostUserSlaveRequest; @@ -246,6 +250,7 @@ struct vhost_user { /* Shared between vhost devs of the same virtio device */ VhostUserState *user; int slave_fd; + bool slave_channel_open; NotifierWithReturn postcopy_notifier; struct PostCopyFD postcopy_fd; uint64_t postcopy_client_bases[VHOST_USER_MAX_RAM_SLOTS]; @@ -1511,6 +1516,10 @@ static int do_slave_read(void *opaque) ret = vhost_user_fs_slave_io(dev, &payload.fs, fd[0]); break; #endif + case VHOST_USER_SLAVE_STOP_CHANNEL_COMPLETE: + u->slave_channel_open = false; + ret = 0; + break; default: error_report("Received unexpected msg type: %d.", hdr.request); ret = (uint64_t)-EINVAL; @@ -1580,6 +1589,70 @@ static void slave_read(void *opaque) do_slave_read(opaque); } +static int vhost_start_slave_channel(struct vhost_dev *dev) +{ + struct vhost_user *u = dev->opaque; + VhostUserMsg msg = { + .hdr.request = VHOST_USER_START_SLAVE_CHANNEL, + .hdr.flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK, + }; + int ret = 0; + + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_SLAVE_CH_START_STOP)) { + return 0; + } + + ret = vhost_user_write(dev, &msg, NULL, 0); + if (ret) { + return ret; + } + + ret = process_message_reply(dev, &msg); + if (ret) + return ret; + + u->slave_channel_open = true; + return ret; +} + +static int vhost_stop_slave_channel(struct vhost_dev *dev) +{ + struct vhost_user *u = dev->opaque; + VhostUserMsg msg = { + .hdr.request = VHOST_USER_STOP_SLAVE_CHANNEL, + .hdr.flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK, + }; + int ret = 0; + + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_SLAVE_CH_START_STOP)) { + return 0; + } + + ret = vhost_user_write(dev, &msg, NULL, 0); + if (ret) { + return ret; + } + + ret = process_message_reply(dev, &msg); + if (ret) { + return ret; + } + + /* + * Wait for flush finish message from slave. And continue to process + * slave messages till we get flush finish. + */ + while (u->slave_channel_open) { + ret = do_slave_read(dev); + if (ret) + break; + } + + return ret; +} + static int vhost_setup_slave_channel(struct vhost_dev *dev) { VhostUserMsg msg = { @@ -1860,6 +1933,7 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque) u = g_new0(struct vhost_user, 1); u->user = opaque; u->slave_fd = -1; + u->slave_channel_open = false; u->dev = dev; dev->opaque = u; @@ -1934,6 +2008,17 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque) u->user->memory_slots = MIN(ram_slots, VHOST_USER_MAX_RAM_SLOTS); } + + if (virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_SLAVE_CH_START_STOP) && + !(virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_SLAVE_REQ) && + virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_REPLY_ACK))) { + error_report("Slave channel start/stop support requires reply-ack" + " and slave-req protocol features."); + return -1; + } } if (dev->migration_blocker == NULL && @@ -1949,6 +2034,10 @@ static int vhost_user_backend_init(struct vhost_dev *dev, void *opaque) if (err < 0) { return err; } + err = vhost_start_slave_channel(dev); + if (err < 0) { + return err; + } } u->postcopy_notifier.notify = vhost_user_postcopy_notifier; @@ -2408,6 +2497,24 @@ void vhost_user_cleanup(VhostUserState *user) user->chr = NULL; } +static int vhost_user_dev_start(struct vhost_dev *dev, bool started) +{ + int ret; + + if (!started) { + ret = vhost_stop_slave_channel(dev); + if (ret < 0) { + return ret; + } + } else { + ret = vhost_start_slave_channel(dev); + if (ret < 0) { + return ret; + } + } + return 0; +} + const VhostOps user_ops = { .backend_type = VHOST_BACKEND_TYPE_USER, .vhost_backend_init = vhost_user_backend_init, @@ -2434,6 +2541,7 @@ const VhostOps user_ops = { .vhost_net_set_mtu = vhost_user_net_set_mtu, .vhost_set_iotlb_callback = vhost_user_set_iotlb_callback, .vhost_send_device_iotlb_msg = vhost_user_send_device_iotlb_msg, + .vhost_dev_start = vhost_user_dev_start, .vhost_get_config = vhost_user_get_config, .vhost_set_config = vhost_user_set_config, .vhost_crypto_create_session = vhost_user_crypto_create_session, From patchwork Mon Jan 25 18:01:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vivek Goyal X-Patchwork-Id: 12043923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58364C433E6 for ; Mon, 25 Jan 2021 18:10:29 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F3B7B22583 for ; Mon, 25 Jan 2021 18:10:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F3B7B22583 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:56410 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l46Jo-0000s3-2i for qemu-devel@archiver.kernel.org; Mon, 25 Jan 2021 13:10:28 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45322) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l46Bq-0001YH-Fo for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:15 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:42398) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1l46Bm-0006mf-Vo for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1611597730; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dlEMOdA1icJ2kgLpjxfqG5+bbZpxaLJO4PoxLpABNz0=; b=JwzxjdEwU+/PTwnhS9mjao8fJW2a6oy+Vm4sr4YcZiY5YY3KmAMs+h/ySbC3G/A2R2W0/d ZlfEr7CIHDXQpMoXdaHhSt92Lw1c6i9UgEGLhRtgFfEMTB+pocBSeEgb5M+RPKClRgr4VT Kb51dfCIniY4XujsmZhoHnkvkqajcVE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-79-G4zK6IvgOhGXF71cQhHEuQ-1; Mon, 25 Jan 2021 13:02:07 -0500 X-MC-Unique: G4zK6IvgOhGXF71cQhHEuQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 93912107ACE3 for ; Mon, 25 Jan 2021 18:02:06 +0000 (UTC) Received: from horse.redhat.com (ovpn-116-119.rdu2.redhat.com [10.10.116.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 23A3260C62; Mon, 25 Jan 2021 18:01:52 +0000 (UTC) Received: by horse.redhat.com (Postfix, from userid 10451) id 4931A225FCF; Mon, 25 Jan 2021 13:01:38 -0500 (EST) From: Vivek Goyal To: qemu-devel@nongnu.org, virtio-fs@redhat.com Subject: [PATCH 5/6] libvhost-user: Add support to start/stop/flush slave channel Date: Mon, 25 Jan 2021 13:01:14 -0500 Message-Id: <20210125180115.22936-6-vgoyal@redhat.com> In-Reply-To: <20210125180115.22936-1-vgoyal@redhat.com> References: <20210125180115.22936-1-vgoyal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=vgoyal@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=vgoyal@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.255, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, stefanha@redhat.com, dgilbert@redhat.com, vgoyal@redhat.com, mst@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This patch adds support to start/stop/flush slave channel functionality. Signed-off-by: Vivek Goyal --- subprojects/libvhost-user/libvhost-user.c | 103 ++++++++++++++++++++-- subprojects/libvhost-user/libvhost-user.h | 8 +- 2 files changed, 105 insertions(+), 6 deletions(-) diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c index 7a56c56dc8..b4c795c63e 100644 --- a/subprojects/libvhost-user/libvhost-user.c +++ b/subprojects/libvhost-user/libvhost-user.c @@ -140,6 +140,8 @@ vu_request_to_string(unsigned int req) REQ(VHOST_USER_GET_MAX_MEM_SLOTS), REQ(VHOST_USER_ADD_MEM_REG), REQ(VHOST_USER_REM_MEM_REG), + REQ(VHOST_USER_START_SLAVE_CHANNEL), + REQ(VHOST_USER_STOP_SLAVE_CHANNEL), REQ(VHOST_USER_MAX), }; #undef REQ @@ -437,11 +439,11 @@ out: return result; } -/* Returns true on success, false otherwise */ +/* slave mutex should be held. Will be unlocked upon return */ static bool -vu_message_slave_send_receive(VuDev *dev, VhostUserMsg *vmsg, uint64_t *payload) +vu_message_slave_send_receive_locked(VuDev *dev, VhostUserMsg *vmsg, + uint64_t *payload) { - pthread_mutex_lock(&dev->slave_mutex); if (!vu_message_write(dev, dev->slave_fd, vmsg)) { pthread_mutex_unlock(&dev->slave_mutex); return false; @@ -456,6 +458,46 @@ vu_message_slave_send_receive(VuDev *dev, VhostUserMsg *vmsg, uint64_t *payload) return vu_process_message_reply(dev, vmsg, payload); } +/* Returns true on success, false otherwise */ +static bool +vu_message_slave_send_receive(VuDev *dev, VhostUserMsg *vmsg, + uint64_t *payload) +{ + pthread_mutex_lock(&dev->slave_mutex); + if (!dev->slave_channel_open) { + pthread_mutex_unlock(&dev->slave_mutex); + return false; + } + return vu_message_slave_send_receive_locked(dev, vmsg, payload); +} + +static bool +vu_finish_stop_slave(VuDev *dev) +{ + bool res; + uint64_t payload = 0; + VhostUserMsg vmsg = { + .request = VHOST_USER_SLAVE_STOP_CHANNEL_COMPLETE, + .flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK, + .size = sizeof(vmsg.payload.u64), + .payload.u64 = 0, + }; + + /* + * Once we get slave_mutex, this should make sure no other caller is + * currently in the process of sending or receiving message on slave_fd. + * And setting slave_channel_open to false now will make sure any new + * callers will not send message and instead get error back. So it + * is now safe to send stop finished message to master. + */ + pthread_mutex_lock(&dev->slave_mutex); + dev->slave_channel_open = false; + /* This also drops slave_mutex */ + res = vu_message_slave_send_receive_locked(dev, &vmsg, &payload); + res = res && (payload == 0); + return res; +} + /* Kick the log_call_fd if required. */ static void vu_log_kick(VuDev *dev) @@ -1529,6 +1571,35 @@ vu_set_slave_req_fd(VuDev *dev, VhostUserMsg *vmsg) return false; } +static bool +vu_slave_channel_start(VuDev *dev, VhostUserMsg *vmsg) +{ + pthread_mutex_lock(&dev->slave_mutex); + dev->slave_channel_open = true; + pthread_mutex_unlock(&dev->slave_mutex); + /* Caller (vu_dispatch()) will send a reply */ + return false; +} + +static bool +vu_slave_channel_stop(VuDev *dev, VhostUserMsg *vmsg, bool *reply_sent, + bool *reply_status) +{ + vmsg_set_reply_u64(vmsg, 0); + *reply_sent = true; + *reply_status = false; + if (!vu_send_reply(dev, dev->sock, vmsg)) { + return false; + } + + if (!vu_finish_stop_slave(dev)) { + return false; + } + + *reply_status = true; + return false; +} + static bool vu_get_config(VuDev *dev, VhostUserMsg *vmsg) { @@ -1823,7 +1894,8 @@ static bool vu_handle_get_max_memslots(VuDev *dev, VhostUserMsg *vmsg) } static bool -vu_process_message(VuDev *dev, VhostUserMsg *vmsg) +vu_process_message(VuDev *dev, VhostUserMsg *vmsg, bool *reply_sent, + bool *reply_status) { int do_reply = 0; @@ -1843,6 +1915,14 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) DPRINT("\n"); } + if (reply_sent) { + *reply_sent = false; + } + + if (reply_status) { + *reply_status = false; + } + if (dev->iface->process_msg && dev->iface->process_msg(dev, vmsg, &do_reply)) { return do_reply; @@ -1912,6 +1992,10 @@ vu_process_message(VuDev *dev, VhostUserMsg *vmsg) return vu_add_mem_reg(dev, vmsg); case VHOST_USER_REM_MEM_REG: return vu_rem_mem_reg(dev, vmsg); + case VHOST_USER_START_SLAVE_CHANNEL: + return vu_slave_channel_start(dev, vmsg); + case VHOST_USER_STOP_SLAVE_CHANNEL: + return vu_slave_channel_stop(dev, vmsg, reply_sent, reply_status); default: vmsg_close_fds(vmsg); vu_panic(dev, "Unhandled request: %d", vmsg->request); @@ -1926,6 +2010,7 @@ vu_dispatch(VuDev *dev) VhostUserMsg vmsg = { 0, }; int reply_requested; bool need_reply, success = false; + bool reply_sent = false, reply_status = false; if (!dev->read_msg(dev, dev->sock, &vmsg)) { goto end; @@ -1933,7 +2018,14 @@ vu_dispatch(VuDev *dev) need_reply = vmsg.flags & VHOST_USER_NEED_REPLY_MASK; - reply_requested = vu_process_message(dev, &vmsg); + reply_requested = vu_process_message(dev, &vmsg, &reply_sent, + &reply_status); + /* reply has already been sent, if needed */ + if (reply_sent) { + success = reply_status; + goto end; + } + if (!reply_requested && need_reply) { vmsg_set_reply_u64(&vmsg, 0); reply_requested = 1; @@ -2051,6 +2143,7 @@ vu_init(VuDev *dev, dev->log_call_fd = -1; pthread_mutex_init(&dev->slave_mutex, NULL); dev->slave_fd = -1; + dev->slave_channel_open = false; dev->max_queues = max_queues; dev->vq = malloc(max_queues * sizeof(dev->vq[0])); diff --git a/subprojects/libvhost-user/libvhost-user.h b/subprojects/libvhost-user/libvhost-user.h index ee75d4931f..1d0ef54f69 100644 --- a/subprojects/libvhost-user/libvhost-user.h +++ b/subprojects/libvhost-user/libvhost-user.h @@ -64,6 +64,7 @@ enum VhostUserProtocolFeature { VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD = 12, VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS = 14, VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS = 15, + VHOST_USER_PROTOCOL_F_SLAVE_CH_START_STOP = 16, VHOST_USER_PROTOCOL_F_MAX }; @@ -109,6 +110,8 @@ typedef enum VhostUserRequest { VHOST_USER_GET_MAX_MEM_SLOTS = 36, VHOST_USER_ADD_MEM_REG = 37, VHOST_USER_REM_MEM_REG = 38, + VHOST_USER_START_SLAVE_CHANNEL = 39, + VHOST_USER_STOP_SLAVE_CHANNEL = 40, VHOST_USER_MAX } VhostUserRequest; @@ -123,6 +126,7 @@ typedef enum VhostUserSlaveRequest { VHOST_USER_SLAVE_FS_UNMAP = 7, VHOST_USER_SLAVE_FS_SYNC = 8, VHOST_USER_SLAVE_FS_IO = 9, + VHOST_USER_SLAVE_STOP_CHANNEL_COMPLETE = 10, VHOST_USER_SLAVE_MAX } VhostUserSlaveRequest; @@ -405,9 +409,11 @@ struct VuDev { VuVirtq *vq; VuDevInflightInfo inflight_info; int log_call_fd; - /* Must be held while using slave_fd */ + /* Must be held while using slave_fd, slave_channel_open */ pthread_mutex_t slave_mutex; int slave_fd; + /* If not set, do not send more requests on slave fd. */ + bool slave_channel_open; uint64_t log_size; uint8_t *log_table; uint64_t features; From patchwork Mon Jan 25 18:01:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vivek Goyal X-Patchwork-Id: 12043911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3136FC433E6 for ; Mon, 25 Jan 2021 18:04:57 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8BB9D2251F for ; Mon, 25 Jan 2021 18:04:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8BB9D2251F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:42424 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l46ER-0003JT-Ed for qemu-devel@archiver.kernel.org; Mon, 25 Jan 2021 13:04:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:45222) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l46Bd-0001Q0-2H for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:02 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:33897) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1l46BY-0006kv-Uq for qemu-devel@nongnu.org; Mon, 25 Jan 2021 13:02:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1611597715; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IfhOBGn+oEwzTYx5DwqV0KlLI7vT/PBfxDmUKVwxCSE=; b=DnIHMxyWg3/h3zJv3epp1XbdfPv16k1SlCdSAhrW/rxjglK6446k++9RmznPxk6vfaXNhE ysAuGZi1iOo3jLbq77Pr/n7Kj4FU4+0B4zLlXtR//SZf2KCPUgrOiNhMipbn6AufcF9gkh 1+frV2dbfqTAbVaqEVP8+ZgX2QhA/oA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-538-UBocEi2CNGy3K7NhZKAU0A-1; Mon, 25 Jan 2021 13:01:53 -0500 X-MC-Unique: UBocEi2CNGy3K7NhZKAU0A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5AA6C8144F3 for ; Mon, 25 Jan 2021 18:01:52 +0000 (UTC) Received: from horse.redhat.com (ovpn-116-119.rdu2.redhat.com [10.10.116.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2F40F6FEED; Mon, 25 Jan 2021 18:01:52 +0000 (UTC) Received: by horse.redhat.com (Postfix, from userid 10451) id 4D7CF225FD0; Mon, 25 Jan 2021 13:01:38 -0500 (EST) From: Vivek Goyal To: qemu-devel@nongnu.org, virtio-fs@redhat.com Subject: [PATCH 6/6] virtiofsd: Opt in for slave start/stop/shutdown functionality Date: Mon, 25 Jan 2021 13:01:15 -0500 Message-Id: <20210125180115.22936-7-vgoyal@redhat.com> In-Reply-To: <20210125180115.22936-1-vgoyal@redhat.com> References: <20210125180115.22936-1-vgoyal@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=vgoyal@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=vgoyal@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -30 X-Spam_score: -3.1 X-Spam_bar: --- X-Spam_report: (-3.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.255, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: marcandre.lureau@redhat.com, stefanha@redhat.com, dgilbert@redhat.com, vgoyal@redhat.com, mst@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is an opt-in feature not enabled by default. Enable it in protocol features by setting VHOST_USER_PROTOCOL_F_SLAVE_CH_START_STOP. Signed-off-by: Vivek Goyal --- tools/virtiofsd/fuse_virtio.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tools/virtiofsd/fuse_virtio.c b/tools/virtiofsd/fuse_virtio.c index 6805d8ba01..9328c8fac6 100644 --- a/tools/virtiofsd/fuse_virtio.c +++ b/tools/virtiofsd/fuse_virtio.c @@ -103,6 +103,11 @@ static void fv_set_features(VuDev *dev, uint64_t features) { } +static uint64_t fv_get_protocol_features(VuDev *dev) +{ + return 1ULL << VHOST_USER_PROTOCOL_F_SLAVE_CH_START_STOP; +} + /* * Callback from libvhost-user if there's a new fd we're supposed to listen * to, typically a queue kick? @@ -902,6 +907,7 @@ static const VuDevIface fv_iface = { .get_features = fv_get_features, .set_features = fv_set_features, + .get_protocol_features = fv_get_protocol_features, /* Don't need process message, we've not got any at vhost-user level */ .queue_set_started = fv_queue_set_started,