From patchwork Mon Apr 20 09:32:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cindy Lu X-Patchwork-Id: 11498991 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4049C6CA for ; Mon, 20 Apr 2020 13:42:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1698B20857 for ; Mon, 20 Apr 2020 13:42:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ZeCtimIr" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1698B20857 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35942 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jQWhO-0004HD-6r for patchwork-qemu-devel@patchwork.kernel.org; Mon, 20 Apr 2020 09:42:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44082 helo=eggs1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jQSpT-00040n-HV for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:35:03 -0400 Received: from Debian-exim by eggs1p.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jQSpS-0004Rs-Vx for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:35:03 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:27016 helo=us-smtp-delivery-1.mimecast.com) by eggs1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jQSpS-0004Qu-JE for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:35:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1587375302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=moQA5ztc3Yw+N49069KRQBO7QJ36CZ0pzA0Cdh1gfLI=; b=ZeCtimIrh4hD+TRZpNZtOFA9tlHYq/mPTC4+CeQNhLjfQdHTFR/Jq4hjtK1XKjaRM8Z91S comgChy83mdG+3Jh+J5CN3jGD7qHcXwl6cynvzRK3be9Jv6IzvhqPe2AE8M4s9hBXkudNr 9/bJ4glQdS/L9ej3MQ/x6jIALv4clsQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-13-f4RZ7gWQPvWfQGERoX3QSA-1; Mon, 20 Apr 2020 05:33:31 -0400 X-MC-Unique: f4RZ7gWQPvWfQGERoX3QSA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 29A3A1902EA2; Mon, 20 Apr 2020 09:33:29 +0000 (UTC) Received: from laptop.redhat.com (ovpn-12-243.pek2.redhat.com [10.72.12.243]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8D7E092FA1; Mon, 20 Apr 2020 09:33:11 +0000 (UTC) From: Cindy Lu To: mst@redhat.com, armbru@redhat.com, eblake@redhat.com, cohuck@redhat.com, jasowang@redhat.com Subject: [RFC v1 1/4] net: Introduce qemu_get_peer Date: Mon, 20 Apr 2020 17:32:38 +0800 Message-Id: <20200420093241.4238-2-lulu@redhat.com> In-Reply-To: <20200420093241.4238-1-lulu@redhat.com> References: <20200420093241.4238-1-lulu@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.81; envelope-from=lulu@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs1p.gnu.org: First seen = 2020/04/20 01:47:04 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-Mailman-Approved-At: Mon, 20 Apr 2020 09:36:04 -0400 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mhabets@solarflare.com, qemu-devel@nongnu.org, rob.miller@broadcom.com, saugatm@xilinx.com, lulu@redhat.com, hanand@xilinx.com, hch@infradead.org, eperezma@redhat.com, jgg@mellanox.com, shahafs@mellanox.com, kevin.tian@intel.com, parav@mellanox.com, vmireyno@marvell.com, cunming.liang@intel.com, gdawar@xilinx.com, jiri@mellanox.com, xiao.w.wang@intel.com, stefanha@redhat.com, zhihong.wang@intel.com, aadam@redhat.com, rdunlap@infradead.org, maxime.coquelin@redhat.com, lingshan.zhu@intel.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" This is a small function that can get the peer from given NetClientState and queue_index Signed-off-by: Cindy Lu --- hw/net/vhost_net.c | 16 ++++++++++------ include/net/net.h | 1 + net/net.c | 6 ++++++ 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c index 6b82803fa7..4096d64aaf 100644 --- a/hw/net/vhost_net.c +++ b/hw/net/vhost_net.c @@ -306,7 +306,9 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(dev))); VirtioBusState *vbus = VIRTIO_BUS(qbus); VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus); + struct vhost_net *net; int r, e, i; + NetClientState *peer; if (!k->set_guest_notifiers) { error_report("binding does not support guest notifiers"); @@ -314,9 +316,9 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, } for (i = 0; i < total_queues; i++) { - struct vhost_net *net; - net = get_vhost_net(ncs[i].peer); + peer = qemu_get_peer(ncs, i); + net = get_vhost_net(peer); vhost_net_set_vq_index(net, i * 2); /* Suppress the masking guest notifiers on vhost user @@ -335,15 +337,16 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, } for (i = 0; i < total_queues; i++) { - r = vhost_net_start_one(get_vhost_net(ncs[i].peer), dev); + peer = qemu_get_peer(ncs, i); + r = vhost_net_start_one(get_vhost_net(peer), dev); if (r < 0) { goto err_start; } - if (ncs[i].peer->vring_enable) { + if (peer->vring_enable) { /* restore vring enable state */ - r = vhost_set_vring_enable(ncs[i].peer, ncs[i].peer->vring_enable); + r = vhost_set_vring_enable(peer, peer->vring_enable); if (r < 0) { goto err_start; @@ -355,7 +358,8 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, err_start: while (--i >= 0) { - vhost_net_stop_one(get_vhost_net(ncs[i].peer), dev); + peer = qemu_get_peer(ncs , i); + vhost_net_stop_one(get_vhost_net(peer), dev); } e = k->set_guest_notifiers(qbus->parent, total_queues * 2, false); if (e < 0) { diff --git a/include/net/net.h b/include/net/net.h index e175ba9677..0a74324ccd 100644 --- a/include/net/net.h +++ b/include/net/net.h @@ -175,6 +175,7 @@ void hmp_info_network(Monitor *mon, const QDict *qdict); void net_socket_rs_init(SocketReadState *rs, SocketReadStateFinalize *finalize, bool vnet_hdr); +NetClientState *qemu_get_peer(NetClientState *nc, int queue_index); /* NIC info */ diff --git a/net/net.c b/net/net.c index 84aa6d8d00..ac5080dda1 100644 --- a/net/net.c +++ b/net/net.c @@ -324,6 +324,12 @@ void *qemu_get_nic_opaque(NetClientState *nc) return nic->opaque; } +NetClientState *qemu_get_peer(NetClientState *nc, int queue_index) +{ + NetClientState *ncs = nc + queue_index; + assert(ncs != NULL); + return ncs->peer; +} static void qemu_cleanup_net_client(NetClientState *nc) { From patchwork Mon Apr 20 09:32:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cindy Lu X-Patchwork-Id: 11498999 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 475FA14DD for ; Mon, 20 Apr 2020 13:45:27 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0AF10206E9 for ; Mon, 20 Apr 2020 13:45:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="N311CnBf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0AF10206E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:36002 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jQWjl-000059-53 for patchwork-qemu-devel@patchwork.kernel.org; Mon, 20 Apr 2020 09:45:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44086 helo=eggs1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jQSpT-000414-Sk for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:35:04 -0400 Received: from Debian-exim by eggs1p.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jQSpS-0004RH-SS for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:35:03 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:53476 helo=us-smtp-1.mimecast.com) by eggs1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jQSpS-0004Qs-EY for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:35:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1587375301; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bJFKinKLHaZGvttsOWXRknWuJtcBwLotFDvA1gN9HN0=; b=N311CnBfcbqdnWNTAHaMApVyE7M9xh2mC+f3uSdrnwxwmfrLv6IdK2SaPGTxanOp5u0t3I L4TwO4zPrOfqlAt6APiHaY+cYU7UNORcHaTV9CJjLgpLoplN0VZONhC1qBO9a2xm2B59MV cqn9io2KS37vvP6DJI+wq6tEQsE1tYM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-174-u-HPlV3NN022kKgYdGuDCA-1; Mon, 20 Apr 2020 05:33:53 -0400 X-MC-Unique: u-HPlV3NN022kKgYdGuDCA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 985331005513; Mon, 20 Apr 2020 09:33:50 +0000 (UTC) Received: from laptop.redhat.com (ovpn-12-243.pek2.redhat.com [10.72.12.243]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0CF0692FA1; Mon, 20 Apr 2020 09:33:31 +0000 (UTC) From: Cindy Lu To: mst@redhat.com, armbru@redhat.com, eblake@redhat.com, cohuck@redhat.com, jasowang@redhat.com Subject: [RFC v1 2/4] vhost-vdpa: introduce vhost-vdpa net client Date: Mon, 20 Apr 2020 17:32:39 +0800 Message-Id: <20200420093241.4238-3-lulu@redhat.com> In-Reply-To: <20200420093241.4238-1-lulu@redhat.com> References: <20200420093241.4238-1-lulu@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=lulu@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs1p.gnu.org: First seen = 2020/04/20 03:29:13 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.120 X-Mailman-Approved-At: Mon, 20 Apr 2020 09:36:04 -0400 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mhabets@solarflare.com, qemu-devel@nongnu.org, rob.miller@broadcom.com, saugatm@xilinx.com, lulu@redhat.com, hanand@xilinx.com, hch@infradead.org, eperezma@redhat.com, jgg@mellanox.com, shahafs@mellanox.com, kevin.tian@intel.com, parav@mellanox.com, vmireyno@marvell.com, cunming.liang@intel.com, gdawar@xilinx.com, jiri@mellanox.com, xiao.w.wang@intel.com, stefanha@redhat.com, zhihong.wang@intel.com, aadam@redhat.com, rdunlap@infradead.org, maxime.coquelin@redhat.com, lingshan.zhu@intel.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" This patch set introduces a new net client type: vhost-vdpa. vhost-vdpa net client will set up a vDPA device which is svhostdevpecified by a "vhostdev" parameter. Author: Tiwei Bie Signed-off-by: Cindy Lu --- include/net/vhost-vdpa.h | 18 ++++ include/net/vhost_net.h | 1 + net/Makefile.objs | 2 +- net/clients.h | 2 + net/net.c | 1 + net/vhost-vdpa.c | 211 +++++++++++++++++++++++++++++++++++++++ qapi/net.json | 21 +++- 7 files changed, 253 insertions(+), 3 deletions(-) create mode 100644 include/net/vhost-vdpa.h create mode 100644 net/vhost-vdpa.c diff --git a/include/net/vhost-vdpa.h b/include/net/vhost-vdpa.h new file mode 100644 index 0000000000..9ddd538dad --- /dev/null +++ b/include/net/vhost-vdpa.h @@ -0,0 +1,18 @@ +/* + * vhost-vdpa.h + * + * Copyright(c) 2017 Intel Corporation. All rights reserved. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef VHOST_VDPA_H +#define VHOST_VDPA_H + +struct vhost_net; +struct vhost_net *vhost_vdpa_get_vhost_net(NetClientState *nc); +uint64_t vhost_vdpa_get_acked_features(NetClientState *nc); + +#endif /* VHOST_VDPA_H */ diff --git a/include/net/vhost_net.h b/include/net/vhost_net.h index 77e47398c4..6f3a624cf7 100644 --- a/include/net/vhost_net.h +++ b/include/net/vhost_net.h @@ -39,5 +39,6 @@ int vhost_set_vring_enable(NetClientState * nc, int enable); uint64_t vhost_net_get_acked_features(VHostNetState *net); int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu); +int vhost_set_state(NetClientState *nc, int state); #endif diff --git a/net/Makefile.objs b/net/Makefile.objs index c5d076d19c..da459cfc19 100644 --- a/net/Makefile.objs +++ b/net/Makefile.objs @@ -26,7 +26,7 @@ tap-obj-$(CONFIG_SOLARIS) = tap-solaris.o tap-obj-y ?= tap-stub.o common-obj-$(CONFIG_POSIX) += tap.o $(tap-obj-y) common-obj-$(CONFIG_WIN32) += tap-win32.o - +common-obj-$(CONFIG_VHOST_KERNEL) += vhost-vdpa.o vde.o-libs = $(VDE_LIBS) common-obj-$(CONFIG_CAN_BUS) += can/ diff --git a/net/clients.h b/net/clients.h index a6ef267e19..92f9b59aed 100644 --- a/net/clients.h +++ b/net/clients.h @@ -61,4 +61,6 @@ int net_init_netmap(const Netdev *netdev, const char *name, int net_init_vhost_user(const Netdev *netdev, const char *name, NetClientState *peer, Error **errp); +int net_init_vhost_vdpa(const Netdev *netdev, const char *name, + NetClientState *peer, Error **errp); #endif /* QEMU_NET_CLIENTS_H */ diff --git a/net/net.c b/net/net.c index ac5080dda1..2beb95388a 100644 --- a/net/net.c +++ b/net/net.c @@ -964,6 +964,7 @@ static int (* const net_client_init_fun[NET_CLIENT_DRIVER__MAX])( [NET_CLIENT_DRIVER_HUBPORT] = net_init_hubport, #ifdef CONFIG_VHOST_NET_USER [NET_CLIENT_DRIVER_VHOST_USER] = net_init_vhost_user, + [NET_CLIENT_DRIVER_VHOST_VDPA] = net_init_vhost_vdpa, #endif #ifdef CONFIG_L2TPV3 [NET_CLIENT_DRIVER_L2TPV3] = net_init_l2tpv3, diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c new file mode 100644 index 0000000000..5daeba0b76 --- /dev/null +++ b/net/vhost-vdpa.c @@ -0,0 +1,211 @@ +/* + * vhost-vdpa.c + * + * Copyright(c) 2017-2018 Intel Corporation. All rights reserved. + * Copyright(c) 2020 Red Hat, Inc. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "clients.h" +#include "net/vhost_net.h" +#include "net/vhost-vdpa.h" +#include "hw/virtio/vhost-vdpa.h" +#include "chardev/char-fe.h" +#include "qemu/config-file.h" +#include "qemu/error-report.h" +#include "qemu/option.h" +#include "qapi/error.h" +#include "trace.h" +#include +#include +#include +#include + + +typedef struct VhostVDPAState { + NetClientState nc; + struct vhost_vdpa vhost_vdpa; + VHostNetState *vhost_net; + uint64_t acked_features; + bool started; +} VhostVDPAState; + +VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc) +{ + VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); + return s->vhost_net; +} + +uint64_t vhost_vdpa_get_acked_features(NetClientState *nc) +{ + VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); + return s->acked_features; +} + +static void vhost_vdpa_stop(NetClientState *ncs) +{ + VhostVDPAState *s; + + assert(ncs->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); + + s = DO_UPCAST(VhostVDPAState, nc, ncs); + + if (s->vhost_net) { + /* save acked features */ + uint64_t features = vhost_net_get_acked_features(s->vhost_net); + if (features) { + s->acked_features = features; + } + vhost_net_cleanup(s->vhost_net); + } +} + +static int vhost_vdpa_start(NetClientState *ncs, void *be) +{ + VhostNetOptions options; + struct vhost_net *net = NULL; + VhostVDPAState *s; + + options.backend_type = VHOST_BACKEND_TYPE_VDPA; + + assert(ncs->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); + + s = DO_UPCAST(VhostVDPAState, nc, ncs); + + options.net_backend = ncs; + options.opaque = be; + options.busyloop_timeout = 0; + net = vhost_net_init(&options); + if (!net) { + error_report("failed to init vhost_net for queue"); + goto err; + } + + if (s->vhost_net) { + vhost_net_cleanup(s->vhost_net); + g_free(s->vhost_net); + } + s->vhost_net = net; + + return 0; + +err: + if (net) { + vhost_net_cleanup(net); + } + vhost_vdpa_stop(ncs); + return -1; +} +static void vhost_vdpa_cleanup(NetClientState *nc) +{ + VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + + if (s->vhost_net) { + vhost_net_cleanup(s->vhost_net); + g_free(s->vhost_net); + s->vhost_net = NULL; + } + + qemu_purge_queued_packets(nc); +} + +static bool vhost_vdpa_has_vnet_hdr(NetClientState *nc) +{ + assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); + + return true; +} + +static bool vhost_vdpa_has_ufo(NetClientState *nc) +{ + assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA); + VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc); + uint64_t features = 0; + + features |= (1ULL << VIRTIO_NET_F_HOST_UFO); + features = vhost_net_get_features(s->vhost_net, features); + return !!(features & (1ULL << VIRTIO_NET_F_HOST_UFO)); + +} + +static NetClientInfo net_vhost_vdpa_info = { + .type = NET_CLIENT_DRIVER_VHOST_VDPA, + .size = sizeof(VhostVDPAState), + .cleanup = vhost_vdpa_cleanup, + .has_vnet_hdr = vhost_vdpa_has_vnet_hdr, + .has_ufo = vhost_vdpa_has_ufo, +}; + +static int net_vhost_vdpa_init(NetClientState *peer, const char *device, + const char *name, const char *vhostdev) +{ + NetClientState *nc = NULL; + VhostVDPAState *s; + int vdpa_device_fd; + assert(name); + + nc = qemu_new_net_client(&net_vhost_vdpa_info, peer, device, name); + snprintf(nc->info_str, sizeof(nc->info_str), "vhost-vdpa"); + nc->queue_index = 0; + + s = DO_UPCAST(VhostVDPAState, nc, nc); + + vdpa_device_fd = open(vhostdev, O_RDWR); + if (vdpa_device_fd == -1) { + return -errno; + } + s->vhost_vdpa.device_fd = vdpa_device_fd; + vhost_vdpa_start(nc, (void *)&s->vhost_vdpa); + + assert(s->vhost_net); + + return 0; +} + +static int net_vhost_check_net(void *opaque, QemuOpts *opts, Error **errp) +{ + const char *name = opaque; + const char *driver, *netdev; + + driver = qemu_opt_get(opts, "driver"); + netdev = qemu_opt_get(opts, "netdev"); + + if (!driver || !netdev) { + return 0; + } + + if (strcmp(netdev, name) == 0 && + !g_str_has_prefix(driver, "virtio-net-")) { + error_setg(errp, "vhost-vdpa requires frontend driver virtio-net-*"); + return -1; + } + + return 0; +} + +int net_init_vhost_vdpa(const Netdev *netdev, const char *name, + NetClientState *peer, Error **errp) +{ + const NetdevVhostVDPAOptions *vhost_vdpa_opts; + + assert(netdev->type == NET_CLIENT_DRIVER_VHOST_VDPA); + vhost_vdpa_opts = &netdev->u.vhost_vdpa; + + /* verify net frontend */ + if (qemu_opts_foreach(qemu_find_opts("device"), net_vhost_check_net, + (char *)name, errp)) { + return -1; + } + + + return net_vhost_vdpa_init(peer, "vhost_vdpa", name, + vhost_vdpa_opts->vhostdev); + + return 0; +} diff --git a/qapi/net.json b/qapi/net.json index 335295be50..35a5ccee39 100644 --- a/qapi/net.json +++ b/qapi/net.json @@ -441,6 +441,22 @@ '*queues': 'int' } } ## +# @NetdevVhostVDPAOptions: +# +# Vhost-vdpa network backend +# +# @vhostdev: name of a mdev dev path in sysfs +# +# @queues: number of queues to be created for multiqueue vhost-vdpa +# (default: 1) (Since 2.11) +# +# Since: 2.11 +## +{ 'struct': 'NetdevVhostVDPAOptions', + 'data': { + '*vhostdev': 'str', + '*queues': 'int' } } +## # @NetClientDriver: # # Available netdev drivers. @@ -451,7 +467,7 @@ ## { 'enum': 'NetClientDriver', 'data': [ 'none', 'nic', 'user', 'tap', 'l2tpv3', 'socket', 'vde', - 'bridge', 'hubport', 'netmap', 'vhost-user' ] } + 'bridge', 'hubport', 'netmap', 'vhost-user', 'vhost-vdpa' ] } ## # @Netdev: @@ -479,7 +495,8 @@ 'bridge': 'NetdevBridgeOptions', 'hubport': 'NetdevHubPortOptions', 'netmap': 'NetdevNetmapOptions', - 'vhost-user': 'NetdevVhostUserOptions' } } + 'vhost-user': 'NetdevVhostUserOptions', + 'vhost-vdpa': 'NetdevVhostVDPAOptions' } } ## # @NetLegacy: From patchwork Mon Apr 20 09:32:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cindy Lu X-Patchwork-Id: 11498989 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 86D8D6CA for ; Mon, 20 Apr 2020 13:42:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4AD9D20857 for ; Mon, 20 Apr 2020 13:42:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="QkSDJUJY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4AD9D20857 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35914 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jQWgo-00031w-EU for patchwork-qemu-devel@patchwork.kernel.org; Mon, 20 Apr 2020 09:42:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43878 helo=eggs1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jQSop-0003S5-KO for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:34:24 -0400 Received: from Debian-exim by eggs1p.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jQSon-0003l9-4S for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:34:23 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:30363 helo=us-smtp-delivery-1.mimecast.com) by eggs1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jQSom-0003kl-HQ for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:34:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1587375259; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EtYeccdo/i4pcw9ZES8IVfCQlKHuCERwzeJQj8b4NIM=; b=QkSDJUJYFam003/wlpG+qwb4Zh2kg2EQYAN3of4M7wL5A5Hy8OKW4QYiU96cQPWQnpxqdj h6Uk4ln8xNz0OEyjHjuM7RH7Kknqq8/NGtWSAqGTiXIv+pA4+ViCd83qztALKANgL01NER mnzwm4MRdGEScJRlHPfMVGnnR9OJqvY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-266-x33DMka0OuaGrUJub4Co4Q-1; Mon, 20 Apr 2020 05:34:14 -0400 X-MC-Unique: x33DMka0OuaGrUJub4Co4Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7882E1902EA1; Mon, 20 Apr 2020 09:34:12 +0000 (UTC) Received: from laptop.redhat.com (ovpn-12-243.pek2.redhat.com [10.72.12.243]) by smtp.corp.redhat.com (Postfix) with ESMTP id D32B592FA1; Mon, 20 Apr 2020 09:33:54 +0000 (UTC) From: Cindy Lu To: mst@redhat.com, armbru@redhat.com, eblake@redhat.com, cohuck@redhat.com, jasowang@redhat.com Subject: [RFC v1 3/4] vhost-vdpa: implement vhost-vdpa backend Date: Mon, 20 Apr 2020 17:32:40 +0800 Message-Id: <20200420093241.4238-4-lulu@redhat.com> In-Reply-To: <20200420093241.4238-1-lulu@redhat.com> References: <20200420093241.4238-1-lulu@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=205.139.110.61; envelope-from=lulu@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs1p.gnu.org: First seen = 2020/04/20 03:29:18 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-Mailman-Approved-At: Mon, 20 Apr 2020 09:36:04 -0400 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mhabets@solarflare.com, qemu-devel@nongnu.org, rob.miller@broadcom.com, saugatm@xilinx.com, lulu@redhat.com, hanand@xilinx.com, hch@infradead.org, eperezma@redhat.com, jgg@mellanox.com, shahafs@mellanox.com, kevin.tian@intel.com, parav@mellanox.com, vmireyno@marvell.com, cunming.liang@intel.com, gdawar@xilinx.com, jiri@mellanox.com, xiao.w.wang@intel.com, stefanha@redhat.com, zhihong.wang@intel.com, aadam@redhat.com, rdunlap@infradead.org, maxime.coquelin@redhat.com, lingshan.zhu@intel.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Currently we have 2 types of vhost backends in QEMU: vhost kernel and vhost-user. The above patch provides a generic device for vDPA purpose, this vDPA device exposes to user space a non-vendor-specific configuration interface for setting up a vhost HW accelerator, this patch set introduces a third vhost backend called vhost-vdpa based on the vDPA interface. Vhost-vdpa usage: qemu-system-x86_64 -cpu host -enable-kvm \ ...... -netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-id,id=vhost-vdpa0 \ -device virtio-net-pci,netdev=vhost-vdpa0,page-per-vq=on \ Author: Tiwei Bie Signed-off-by: Cindy Lu --- hw/net/vhost_net.c | 43 ++++ hw/net/virtio-net.c | 9 + hw/virtio/Makefile.objs | 2 +- hw/virtio/vhost-backend.c | 3 + hw/virtio/vhost-vdpa.c | 379 ++++++++++++++++++++++++++++++ hw/virtio/vhost.c | 5 + include/hw/virtio/vhost-backend.h | 6 +- include/hw/virtio/vhost-vdpa.h | 14 ++ 8 files changed, 459 insertions(+), 2 deletions(-) create mode 100644 hw/virtio/vhost-vdpa.c create mode 100644 include/hw/virtio/vhost-vdpa.h diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c index 4096d64aaf..0d13fda2fc 100644 --- a/hw/net/vhost_net.c +++ b/hw/net/vhost_net.c @@ -17,8 +17,10 @@ #include "net/net.h" #include "net/tap.h" #include "net/vhost-user.h" +#include "net/vhost-vdpa.h" #include "standard-headers/linux/vhost_types.h" +#include "linux-headers/linux/vhost.h" #include "hw/virtio/virtio-net.h" #include "net/vhost_net.h" #include "qemu/error-report.h" @@ -85,6 +87,29 @@ static const int user_feature_bits[] = { VHOST_INVALID_FEATURE_BIT }; +static const int vdpa_feature_bits[] = { + VIRTIO_F_NOTIFY_ON_EMPTY, + VIRTIO_RING_F_INDIRECT_DESC, + VIRTIO_RING_F_EVENT_IDX, + VIRTIO_F_ANY_LAYOUT, + VIRTIO_F_VERSION_1, + VIRTIO_NET_F_CSUM, + VIRTIO_NET_F_GUEST_CSUM, + VIRTIO_NET_F_GSO, + VIRTIO_NET_F_GUEST_TSO4, + VIRTIO_NET_F_GUEST_TSO6, + VIRTIO_NET_F_GUEST_ECN, + VIRTIO_NET_F_GUEST_UFO, + VIRTIO_NET_F_HOST_TSO4, + VIRTIO_NET_F_HOST_TSO6, + VIRTIO_NET_F_HOST_ECN, + VIRTIO_NET_F_HOST_UFO, + VIRTIO_NET_F_MRG_RXBUF, + VIRTIO_NET_F_MTU, + VIRTIO_F_IOMMU_PLATFORM, + VIRTIO_NET_F_GUEST_ANNOUNCE, + VHOST_INVALID_FEATURE_BIT +}; static const int *vhost_net_get_feature_bits(struct vhost_net *net) { const int *feature_bits = 0; @@ -96,6 +121,9 @@ static const int *vhost_net_get_feature_bits(struct vhost_net *net) case NET_CLIENT_DRIVER_VHOST_USER: feature_bits = user_feature_bits; break; + case NET_CLIENT_DRIVER_VHOST_VDPA: + feature_bits = vdpa_feature_bits; + break; default: error_report("Feature bits not defined for this type: %d", net->nc->info->type); @@ -434,6 +462,10 @@ VHostNetState *get_vhost_net(NetClientState *nc) assert(vhost_net); break; #endif + case NET_CLIENT_DRIVER_VHOST_VDPA: + vhost_net = vhost_vdpa_get_vhost_net(nc); + assert(vhost_net); + break; default: break; } @@ -465,3 +497,14 @@ int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu) return vhost_ops->vhost_net_set_mtu(&net->dev, mtu); } +int vhost_set_state(NetClientState *nc, int state) +{ + struct vhost_net *net = get_vhost_net(nc); + struct vhost_dev *hdev = &net->dev; + if (nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) { + if (hdev->vhost_ops->vhost_set_state) { + return hdev->vhost_ops->vhost_set_state(hdev, state); + } + } + return 0; +} diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index db3d7c38e6..bbecd7ab96 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -206,6 +206,9 @@ static void virtio_net_vhost_status(VirtIONet *n, uint8_t status) VirtIODevice *vdev = VIRTIO_DEVICE(n); NetClientState *nc = qemu_get_queue(n->nic); int queues = n->multiqueue ? n->max_queues : 1; + NetClientState *peer = nc->peer; + uint8_t status_set = vdev->status ; + uint8_t vhost_started_pre = n->vhost_started; if (!get_vhost_net(nc->peer)) { return; @@ -245,6 +248,7 @@ static void virtio_net_vhost_status(VirtIONet *n, uint8_t status) return; } } + status_set = status_set | VIRTIO_CONFIG_S_DRIVER_OK; n->vhost_started = 1; r = vhost_net_start(vdev, n->nic->ncs, queues); @@ -252,11 +256,16 @@ static void virtio_net_vhost_status(VirtIONet *n, uint8_t status) error_report("unable to start vhost net: %d: " "falling back on userspace virtio", -r); n->vhost_started = 0; + status_set = status_set & ~VIRTIO_CONFIG_S_DRIVER_OK; } } else { vhost_net_stop(vdev, n->nic->ncs, queues); + status_set = status_set & ~VIRTIO_CONFIG_S_DRIVER_OK; n->vhost_started = 0; } + if (vhost_started_pre != n->vhost_started) { + vhost_set_state(peer, status_set); + } } static int virtio_net_set_vnet_endian_one(VirtIODevice *vdev, diff --git a/hw/virtio/Makefile.objs b/hw/virtio/Makefile.objs index e2f70fbb89..17361d959e 100644 --- a/hw/virtio/Makefile.objs +++ b/hw/virtio/Makefile.objs @@ -2,7 +2,7 @@ ifeq ($(CONFIG_VIRTIO),y) common-obj-y += virtio-bus.o obj-y += virtio.o -obj-$(call lor,$(CONFIG_VHOST_USER),$(CONFIG_VHOST_KERNEL)) += vhost.o vhost-backend.o +obj-$(call lor,$(CONFIG_VHOST_USER),$(CONFIG_VHOST_KERNEL)) += vhost.o vhost-backend.o vhost-vdpa.o common-obj-$(call lnot,$(call lor,$(CONFIG_VHOST_USER),$(CONFIG_VHOST_KERNEL))) += vhost-stub.o obj-$(CONFIG_VHOST_USER) += vhost-user.o diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c index 48905383f8..935cd9e561 100644 --- a/hw/virtio/vhost-backend.c +++ b/hw/virtio/vhost-backend.c @@ -286,6 +286,9 @@ int vhost_set_backend_type(struct vhost_dev *dev, VhostBackendType backend_type) dev->vhost_ops = &user_ops; break; #endif + case VHOST_BACKEND_TYPE_VDPA: + dev->vhost_ops = &vdpa_ops; + break; default: error_report("Unknown vhost backend type"); r = -1; diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c new file mode 100644 index 0000000000..213b327600 --- /dev/null +++ b/hw/virtio/vhost-vdpa.c @@ -0,0 +1,379 @@ +/* + * vhost-vdpa + * + * Copyright(c) 2017-2018 Intel Corporation. All rights reserved. + * Copyright(c) 2020 Red Hat, Inc. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include +#include +#include +#include +#include "hw/virtio/vhost.h" +#include "hw/virtio/vhost-backend.h" +#include "hw/virtio/virtio-net.h" +#include "hw/virtio/vhost-vdpa.h" +#include "qemu/main-loop.h" +#include +#include "sysemu/kvm.h" + + +static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section) +{ + return (!memory_region_is_ram(section->mr) && + !memory_region_is_iommu(section->mr)) || + /* + * Sizing an enabled 64-bit BAR can cause spurious mappings to + * addresses in the upper part of the 64-bit address space. These + * are never accessed by the CPU and beyond the address width of + * some IOMMU hardware. TODO: VDPA should tell us the IOMMU width. + */ + section->offset_within_address_space & (1ULL << 63); +} + +static int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size, + void *vaddr, bool readonly) +{ + struct vhost_msg_v2 msg; + int fd = v->device_fd; + int ret = 0; + + msg.type = VHOST_IOTLB_MSG_V2; + msg.iotlb.iova = iova; + msg.iotlb.size = size; + msg.iotlb.uaddr = (uint64_t)vaddr; + msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW; + msg.iotlb.type = VHOST_IOTLB_UPDATE; + + if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) { + error_report("failed to write, fd=%d, errno=%d (%s)", + fd, errno, strerror(errno)); + return -EIO ; + } + + return ret; +} + +static int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova, + hwaddr size) +{ + struct vhost_msg_v2 msg; + int fd = v->device_fd; + int ret = 0; + + msg.type = VHOST_IOTLB_MSG_V2; + msg.iotlb.iova = iova; + msg.iotlb.size = size; + msg.iotlb.type = VHOST_IOTLB_INVALIDATE; + + if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) { + error_report("failed to write, fd=%d, errno=%d (%s)", + fd, errno, strerror(errno)); + return -EIO ; + } + + return ret; +} + +static void vhost_vdpa_listener_region_add(MemoryListener *listener, + MemoryRegionSection *section) +{ + struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener); + hwaddr iova; + Int128 llend, llsize; + void *vaddr; + int ret; + + if (vhost_vdpa_listener_skipped_section(section)) { + return; + } + + if (unlikely((section->offset_within_address_space & ~TARGET_PAGE_MASK) != + (section->offset_within_region & ~TARGET_PAGE_MASK))) { + error_report("%s received unaligned region", __func__); + return; + } + + iova = TARGET_PAGE_ALIGN(section->offset_within_address_space); + llend = int128_make64(section->offset_within_address_space); + llend = int128_add(llend, section->size); + llend = int128_and(llend, int128_exts64(TARGET_PAGE_MASK)); + + if (int128_ge(int128_make64(iova), llend)) { + return; + } + + memory_region_ref(section->mr); + + /* Here we assume that memory_region_is_ram(section->mr)==true */ + + vaddr = memory_region_get_ram_ptr(section->mr) + + section->offset_within_region + + (iova - section->offset_within_address_space); + + llsize = int128_sub(llend, int128_make64(iova)); + + ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize), + vaddr, section->readonly); + if (ret) { + error_report("vhost vdpa map fail!"); + if (memory_region_is_ram_device(section->mr)) { + /* Allow unexpected mappings not to be fatal for RAM devices */ + error_report("map ram fail!"); + return ; + } + goto fail; + } + + return; + +fail: + if (memory_region_is_ram_device(section->mr)) { + error_report("failed to vdpa_dma_map. pci p2p may not work"); + return; + + } + /* + * On the initfn path, store the first error in the container so we + * can gracefully fail. Runtime, there's not much we can do other + * than throw a hardware error. + */ + error_report("vhost-vdpa: DMA mapping failed, unable to continue"); + return; + +} + +static void vhost_vdpa_listener_region_del(MemoryListener *listener, + MemoryRegionSection *section) +{ + struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener); + hwaddr iova; + Int128 llend, llsize; + int ret; + bool try_unmap = true; + + if (vhost_vdpa_listener_skipped_section(section)) { + return; + } + + if (unlikely((section->offset_within_address_space & ~TARGET_PAGE_MASK) != + (section->offset_within_region & ~TARGET_PAGE_MASK))) { + error_report("%s received unaligned region", __func__); + return; + } + + iova = TARGET_PAGE_ALIGN(section->offset_within_address_space); + llend = int128_make64(section->offset_within_address_space); + llend = int128_add(llend, section->size); + llend = int128_and(llend, int128_exts64(TARGET_PAGE_MASK)); + + if (int128_ge(int128_make64(iova), llend)) { + return; + } + + llsize = int128_sub(llend, int128_make64(iova)); + + if (try_unmap) { + ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize)); + if (ret) { + error_report("vhost_vdpa dma unmap error!"); + } + } + + memory_region_unref(section->mr); +} + +static const MemoryListener vhost_vdpa_memory_listener = { + .region_add = vhost_vdpa_listener_region_add, + .region_del = vhost_vdpa_listener_region_del, +}; + + +static int vhost_vdpa_call(struct vhost_dev *dev, unsigned long int request, + void *arg) +{ + struct vhost_vdpa *v = dev->opaque; + int fd = v->device_fd; + + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA); + + return ioctl(fd, request, arg); +} + + + +static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque) +{ + struct vhost_vdpa *v; + + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA); + + v = opaque; + dev->opaque = opaque ; + + v->listener = vhost_vdpa_memory_listener; + memory_listener_register(&v->listener, &address_space_memory); + + return 0; +} + +static int vhost_vdpa_cleanup(struct vhost_dev *dev) +{ + struct vhost_vdpa *v; + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA); + + v = dev->opaque; + memory_listener_unregister(&v->listener); + + dev->opaque = NULL; + return 0; +} + +static int vhost_vdpa_memslots_limit(struct vhost_dev *dev) +{ + return INT_MAX; +} + +static int vhost_vdpa_set_log_base(struct vhost_dev *dev, uint64_t base, + struct vhost_log *log) +{ + return 0; +} + +static int vhost_vdpa_set_mem_table(struct vhost_dev *dev, + struct vhost_memory *mem) +{ + + if (mem->padding) { + return -1; + } + + return 0; +} + +static int vhost_vdpa_set_vring_addr(struct vhost_dev *dev, + struct vhost_vring_addr *addr) +{ + return vhost_vdpa_call(dev, VHOST_SET_VRING_ADDR, addr); +} + +static int vhost_vdpa_set_vring_num(struct vhost_dev *dev, + struct vhost_vring_state *ring) +{ + return vhost_vdpa_call(dev, VHOST_SET_VRING_NUM, ring); +} + +static int vhost_vdpa_set_vring_base(struct vhost_dev *dev, + struct vhost_vring_state *ring) +{ + return vhost_vdpa_call(dev, VHOST_GET_VRING_BASE, ring); +} + +static int vhost_vdpa_get_vring_base(struct vhost_dev *dev, + struct vhost_vring_state *ring) +{ + + return vhost_vdpa_call(dev, VHOST_GET_VRING_BASE, ring); +} + +static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev, + struct vhost_vring_file *file) +{ + return vhost_vdpa_call(dev, VHOST_SET_VRING_KICK, file); +} + +static int vhost_vdpa_set_vring_call(struct vhost_dev *dev, + struct vhost_vring_file *file) +{ + return vhost_vdpa_call(dev, VHOST_SET_VRING_CALL, file); +} + +static int vhost_vdpa_set_features(struct vhost_dev *dev, + uint64_t features) +{ + + features |= (1ULL << VIRTIO_F_IOMMU_PLATFORM); + return vhost_vdpa_call(dev, VHOST_SET_FEATURES, &features); + +} + +static int vhost_vdpa_get_features(struct vhost_dev *dev, + uint64_t *features) +{ + return vhost_vdpa_call(dev, VHOST_GET_FEATURES, features); +} + +static int vhost_vdpa_set_owner(struct vhost_dev *dev) +{ + return vhost_vdpa_call(dev, VHOST_SET_OWNER, NULL); +} + +static int vhost_vdpa_reset_device(struct vhost_dev *dev) +{ + return vhost_vdpa_call(dev, VHOST_RESET_OWNER, NULL); +} + +static int vhost_vdpa_get_vq_index(struct vhost_dev *dev, int idx) +{ + assert(idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs); + + return idx - dev->vq_index; +} + +static int vhost_vdpa_set_vring_enable(struct vhost_dev *dev, int enable) +{ + int i; + + for (i = 0; i < dev->nvqs; ++i) { + struct vhost_vring_state state = { + .index = dev->vq_index + i, + .num = enable, + }; + + state.num = 1; + + vhost_vdpa_call(dev, VHOST_VDPA_SET_VRING_ENABLE, &state); + } + + return 0; +} + +static int vhost_vdpa_set_state(struct vhost_dev *dev, int state) +{ + return vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &state); +} + + +const VhostOps vdpa_ops = { + .backend_type = VHOST_BACKEND_TYPE_VDPA, + .vhost_backend_init = vhost_vdpa_init, + .vhost_backend_cleanup = vhost_vdpa_cleanup, + .vhost_backend_memslots_limit = vhost_vdpa_memslots_limit, + .vhost_set_log_base = vhost_vdpa_set_log_base, + .vhost_set_mem_table = vhost_vdpa_set_mem_table, + .vhost_set_vring_addr = vhost_vdpa_set_vring_addr, + .vhost_set_vring_endian = NULL, + .vhost_set_vring_num = vhost_vdpa_set_vring_num, + .vhost_set_vring_base = vhost_vdpa_set_vring_base, + .vhost_get_vring_base = vhost_vdpa_get_vring_base, + .vhost_set_vring_kick = vhost_vdpa_set_vring_kick, + .vhost_set_vring_call = vhost_vdpa_set_vring_call, + .vhost_set_features = vhost_vdpa_set_features, + .vhost_get_features = vhost_vdpa_get_features, + .vhost_set_owner = vhost_vdpa_set_owner, + .vhost_reset_device = vhost_vdpa_reset_device, + .vhost_get_vq_index = vhost_vdpa_get_vq_index, + .vhost_set_vring_enable = vhost_vdpa_set_vring_enable, + .vhost_requires_shm_log = NULL, + .vhost_migration_done = NULL, + .vhost_backend_can_merge = NULL, + .vhost_net_set_mtu = NULL, + .vhost_set_iotlb_callback = NULL, + .vhost_send_device_iotlb_msg = NULL, + .vhost_set_state = vhost_vdpa_set_state, +}; diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index 4da0d5a6c5..d1f2c4add7 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -746,6 +746,11 @@ static int vhost_virtqueue_set_addr(struct vhost_dev *dev, .log_guest_addr = vq->used_phys, .flags = enable_log ? (1 << VHOST_VRING_F_LOG) : 0, }; + if (dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA) { + addr.desc_user_addr = (uint64_t)(unsigned long)vq->desc_phys; + addr.avail_user_addr = (uint64_t)(unsigned long)vq->avail_phys; + addr.used_user_addr = (uint64_t)(unsigned long)vq->used_phys; + } int r = dev->vhost_ops->vhost_set_vring_addr(dev, &addr); if (r < 0) { VHOST_OPS_DEBUG("vhost_set_vring_addr failed"); diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h index 6f6670783f..d81bd9885f 100644 --- a/include/hw/virtio/vhost-backend.h +++ b/include/hw/virtio/vhost-backend.h @@ -17,7 +17,8 @@ typedef enum VhostBackendType { VHOST_BACKEND_TYPE_NONE = 0, VHOST_BACKEND_TYPE_KERNEL = 1, VHOST_BACKEND_TYPE_USER = 2, - VHOST_BACKEND_TYPE_MAX = 3, + VHOST_BACKEND_TYPE_VDPA = 3, + VHOST_BACKEND_TYPE_MAX = 4, } VhostBackendType; typedef enum VhostSetConfigType { @@ -112,6 +113,7 @@ typedef int (*vhost_get_inflight_fd_op)(struct vhost_dev *dev, typedef int (*vhost_set_inflight_fd_op)(struct vhost_dev *dev, struct vhost_inflight *inflight); +typedef int (*vhost_set_state_op)(struct vhost_dev *dev, int state); typedef struct VhostOps { VhostBackendType backend_type; vhost_backend_init vhost_backend_init; @@ -152,9 +154,11 @@ typedef struct VhostOps { vhost_backend_mem_section_filter_op vhost_backend_mem_section_filter; vhost_get_inflight_fd_op vhost_get_inflight_fd; vhost_set_inflight_fd_op vhost_set_inflight_fd; + vhost_set_state_op vhost_set_state; } VhostOps; extern const VhostOps user_ops; +extern const VhostOps vdpa_ops; int vhost_set_backend_type(struct vhost_dev *dev, VhostBackendType backend_type); diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h new file mode 100644 index 0000000000..889c1a4410 --- /dev/null +++ b/include/hw/virtio/vhost-vdpa.h @@ -0,0 +1,14 @@ + +#ifndef HW_VIRTIO_VHOST_VDPA_H +#define HW_VIRTIO_VHOST_VDPA_H + +#include "hw/virtio/virtio.h" + +typedef struct vhost_vdpa { + int device_fd; + MemoryListener listener; +} VhostVDPA; + +extern AddressSpace address_space_memory; + +#endif From patchwork Mon Apr 20 09:32:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cindy Lu X-Patchwork-Id: 11498993 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F3E11392 for ; Mon, 20 Apr 2020 13:43:53 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E903F21473 for ; Mon, 20 Apr 2020 13:43:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="aBk24Cql" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E903F21473 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35954 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jQWiG-0005v9-2V for patchwork-qemu-devel@patchwork.kernel.org; Mon, 20 Apr 2020 09:43:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:43940 helo=eggs1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jQSp4-0003i3-6x for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:34:38 -0400 Received: from Debian-exim by eggs1p.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jQSp3-0003x4-FS for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:34:37 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:43624 helo=us-smtp-1.mimecast.com) by eggs1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jQSp3-0003wi-1b for qemu-devel@nongnu.org; Mon, 20 Apr 2020 05:34:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1587375276; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uvyWz9oNluUgLDVfQHYTG0prGOLJNzfYk5AQKmhXsR8=; b=aBk24CqlopDt7P9J23CgmKbgKT0QHF8GtIcosYbKdDbm9XD5+9baAbKB8ibmaaZIL1e88G SwHvajK+QZ/q+Lao4yhYcDemsY9ix31fMOHxV4pGDz9dyAQnQLHVVoc0PAkU7tPMqVq10d YVVMgEFfkQ5PDbmW++DsmMR0P/Mmc/g= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-171-43VqRM5FO0Oq8yZga9pcvw-1; Mon, 20 Apr 2020 05:34:34 -0400 X-MC-Unique: 43VqRM5FO0Oq8yZga9pcvw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E4878800D53; Mon, 20 Apr 2020 09:34:31 +0000 (UTC) Received: from laptop.redhat.com (ovpn-12-243.pek2.redhat.com [10.72.12.243]) by smtp.corp.redhat.com (Postfix) with ESMTP id BF3FB92FA2; Mon, 20 Apr 2020 09:34:14 +0000 (UTC) From: Cindy Lu To: mst@redhat.com, armbru@redhat.com, eblake@redhat.com, cohuck@redhat.com, jasowang@redhat.com Subject: [RFC v1 4/4] vhost: introduce vhost_set_vring_ready method Date: Mon, 20 Apr 2020 17:32:41 +0800 Message-Id: <20200420093241.4238-5-lulu@redhat.com> In-Reply-To: <20200420093241.4238-1-lulu@redhat.com> References: <20200420093241.4238-1-lulu@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=207.211.31.120; envelope-from=lulu@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs1p.gnu.org: First seen = 2020/04/20 03:29:13 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.120 X-Mailman-Approved-At: Mon, 20 Apr 2020 09:36:04 -0400 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mhabets@solarflare.com, qemu-devel@nongnu.org, rob.miller@broadcom.com, saugatm@xilinx.com, lulu@redhat.com, hanand@xilinx.com, hch@infradead.org, eperezma@redhat.com, jgg@mellanox.com, shahafs@mellanox.com, kevin.tian@intel.com, parav@mellanox.com, vmireyno@marvell.com, cunming.liang@intel.com, gdawar@xilinx.com, jiri@mellanox.com, xiao.w.wang@intel.com, stefanha@redhat.com, zhihong.wang@intel.com, aadam@redhat.com, rdunlap@infradead.org, maxime.coquelin@redhat.com, lingshan.zhu@intel.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" From: Jason Wang Vhost-vdpa introduces VHOST_VDPA_SET_VRING_ENABLE which complies the semantic of queue_enable defined in virtio spec. This method can be used for preventing device from executing request for a specific virtqueue. This patch introduces the vhost_ops for this. Note that, we've already had vhost_set_vring_enable which has different semantic which allows to enable or disable a specific virtqueue for some kinds of vhost backends. E.g vhost-user use this to changes the number of active queue pairs. Author: Jason Wang Signed-off-by: Jason Wang --- hw/net/vhost_net-stub.c | 5 +++++ hw/net/vhost_net.c | 16 ++++++++++++++++ hw/virtio/vhost-vdpa.c | 9 +++------ hw/virtio/virtio-pci.c | 13 +++++++++++++ hw/virtio/virtio.c | 6 ++++++ include/hw/virtio/vhost-backend.h | 2 ++ include/hw/virtio/virtio-bus.h | 4 ++++ include/net/vhost_net.h | 1 + 8 files changed, 50 insertions(+), 6 deletions(-) diff --git a/hw/net/vhost_net-stub.c b/hw/net/vhost_net-stub.c index aac0e98228..f5ef1e3055 100644 --- a/hw/net/vhost_net-stub.c +++ b/hw/net/vhost_net-stub.c @@ -86,6 +86,11 @@ int vhost_set_vring_enable(NetClientState *nc, int enable) return 0; } +int vhost_set_vring_ready(NetClientState *nc) +{ + return 0; +} + int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu) { return 0; diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c index 0d13fda2fc..463e333531 100644 --- a/hw/net/vhost_net.c +++ b/hw/net/vhost_net.c @@ -380,6 +380,10 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs, goto err_start; } } + + if (virtio_queue_enabled(dev, i)) { + vhost_set_vring_ready(peer); + } } return 0; @@ -487,6 +491,18 @@ int vhost_set_vring_enable(NetClientState *nc, int enable) return 0; } +int vhost_set_vring_ready(NetClientState *nc) +{ + VHostNetState *net = get_vhost_net(nc); + const VhostOps *vhost_ops = net->dev.vhost_ops; + + if (vhost_ops && vhost_ops->vhost_set_vring_ready) { + return vhost_ops->vhost_set_vring_ready(&net->dev); + } + + return 0; +} + int vhost_net_set_mtu(struct vhost_net *net, uint16_t mtu) { const VhostOps *vhost_ops = net->dev.vhost_ops; diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 213b327600..49224ef9f8 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -325,18 +325,15 @@ static int vhost_vdpa_get_vq_index(struct vhost_dev *dev, int idx) return idx - dev->vq_index; } -static int vhost_vdpa_set_vring_enable(struct vhost_dev *dev, int enable) +static int vhost_vdpa_set_vring_ready(struct vhost_dev *dev) { int i; for (i = 0; i < dev->nvqs; ++i) { struct vhost_vring_state state = { .index = dev->vq_index + i, - .num = enable, + .num = 1, }; - - state.num = 1; - vhost_vdpa_call(dev, VHOST_VDPA_SET_VRING_ENABLE, &state); } @@ -368,7 +365,7 @@ const VhostOps vdpa_ops = { .vhost_set_owner = vhost_vdpa_set_owner, .vhost_reset_device = vhost_vdpa_reset_device, .vhost_get_vq_index = vhost_vdpa_get_vq_index, - .vhost_set_vring_enable = vhost_vdpa_set_vring_enable, + .vhost_set_vring_ready = vhost_vdpa_set_vring_ready, .vhost_requires_shm_log = NULL, .vhost_migration_done = NULL, .vhost_backend_can_merge = NULL, diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index c6b47a9c73..4aaf5d953e 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -1103,6 +1103,18 @@ static AddressSpace *virtio_pci_get_dma_as(DeviceState *d) return pci_get_address_space(dev); } +static bool virtio_pci_queue_enabled(DeviceState *d, int n) +{ + VirtIOPCIProxy *proxy = VIRTIO_PCI(d); + VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus); + + if (virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1)) { + return proxy->vqs[vdev->queue_sel].enabled; + } + + return virtio_queue_get_desc_addr(vdev, n) != 0; +} + static int virtio_pci_add_mem_cap(VirtIOPCIProxy *proxy, struct virtio_pci_cap *cap) { @@ -2053,6 +2065,7 @@ static void virtio_pci_bus_class_init(ObjectClass *klass, void *data) k->ioeventfd_enabled = virtio_pci_ioeventfd_enabled; k->ioeventfd_assign = virtio_pci_ioeventfd_assign; k->get_dma_as = virtio_pci_get_dma_as; + k->queue_enabled = virtio_pci_queue_enabled; } static const TypeInfo virtio_pci_bus_info = { diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c index 04716b5f6c..09732a8836 100644 --- a/hw/virtio/virtio.c +++ b/hw/virtio/virtio.c @@ -3169,6 +3169,12 @@ hwaddr virtio_queue_get_desc_addr(VirtIODevice *vdev, int n) bool virtio_queue_enabled(VirtIODevice *vdev, int n) { + BusState *qbus = qdev_get_parent_bus(DEVICE(vdev)); + VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus); + + if (k->queue_enabled) + return k->queue_enabled(qbus->parent, n); + return virtio_queue_get_desc_addr(vdev, n) != 0; } diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h index d81bd9885f..ce8de6d308 100644 --- a/include/hw/virtio/vhost-backend.h +++ b/include/hw/virtio/vhost-backend.h @@ -78,6 +78,7 @@ typedef int (*vhost_reset_device_op)(struct vhost_dev *dev); typedef int (*vhost_get_vq_index_op)(struct vhost_dev *dev, int idx); typedef int (*vhost_set_vring_enable_op)(struct vhost_dev *dev, int enable); +typedef int (*vhost_set_vring_ready_op)(struct vhost_dev *dev); typedef bool (*vhost_requires_shm_log_op)(struct vhost_dev *dev); typedef int (*vhost_migration_done_op)(struct vhost_dev *dev, char *mac_addr); @@ -140,6 +141,7 @@ typedef struct VhostOps { vhost_reset_device_op vhost_reset_device; vhost_get_vq_index_op vhost_get_vq_index; vhost_set_vring_enable_op vhost_set_vring_enable; + vhost_set_vring_ready_op vhost_set_vring_ready; vhost_requires_shm_log_op vhost_requires_shm_log; vhost_migration_done_op vhost_migration_done; vhost_backend_can_merge_op vhost_backend_can_merge; diff --git a/include/hw/virtio/virtio-bus.h b/include/hw/virtio/virtio-bus.h index 38c9399cd4..0f6f215925 100644 --- a/include/hw/virtio/virtio-bus.h +++ b/include/hw/virtio/virtio-bus.h @@ -83,6 +83,10 @@ typedef struct VirtioBusClass { */ int (*ioeventfd_assign)(DeviceState *d, EventNotifier *notifier, int n, bool assign); + /* + * Whether queue number n is enabled. + */ + bool (*queue_enabled)(DeviceState *d, int n); /* * Does the transport have variable vring alignment? * (ie can it ever call virtio_queue_set_align()?) diff --git a/include/net/vhost_net.h b/include/net/vhost_net.h index 6f3a624cf7..db473ff4d2 100644 --- a/include/net/vhost_net.h +++ b/include/net/vhost_net.h @@ -35,6 +35,7 @@ int vhost_net_notify_migration_done(VHostNetState *net, char* mac_addr); VHostNetState *get_vhost_net(NetClientState *nc); int vhost_set_vring_enable(NetClientState * nc, int enable); +int vhost_set_vring_ready(NetClientState * nc); uint64_t vhost_net_get_acked_features(VHostNetState *net);