From patchwork Wed Aug 3 00:35:08 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Stabellini X-Patchwork-Id: 9260519 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 16CBE60865 for ; Wed, 3 Aug 2016 00:39:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00071284F2 for ; Wed, 3 Aug 2016 00:39:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E59D328508; Wed, 3 Aug 2016 00:39:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id ED344284F2 for ; Wed, 3 Aug 2016 00:38:57 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bUk9V-0000LW-Ug; Wed, 03 Aug 2016 00:35:17 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bUk9T-0000LQ-Lg for xen-devel@lists.xenproject.org; Wed, 03 Aug 2016 00:35:16 +0000 Received: from [85.158.137.68] by server-6.bemta-3.messagelabs.com id 2D/F1-28813-24C31A75; Wed, 03 Aug 2016 00:35:14 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrLKsWRWlGSWpSXmKPExsVyMfSOia6jzcJ wg6kLOS2+b5nM5MDocfjDFZYAxijWzLyk/IoE1oyOs9+ZCmZsYqromTmPsYHxyjfGLkYuDiGB mYwSp3YsZO1i5ORgEfjPIjHluQ5IQkJgNqvE9pZtbCAJCYEYiXnLljBB2NUSu9/NArOFBJQkN rb8YYOYtIBJYtvOo8wQk7Qkvm4+zwJiswnoSZz51w3UwAHUbCix5DMHSFgEqPfeqslMIL3MAk cYJa7/vwN2hbCAm8S5Q8/A5vAKeEu07DzNDmKLCuhKHPr3hw0iLihxcuYTsPnMAn4Sm/Z0s0P Y3hJ/OvYwTmAUmoWkbBaSsllIyiBsLYmzR5sZYeL3jl9ihrDtJLatfcMKYStKbD6wmxWmZuXR i1C2qMSKG3Oges0llq7vYF3AyLWKUb04tagstUjXTC+pKDM9oyQ3MTNH19DAWC83tbg4MT01J zGpWC85P3cTIzD66hkYGHcwXmlzPsQoycGkJMrr+nZBuBBfUn5KZUZicUZ8UWlOavEhRhkODi UJ3odWC8OFBItS01Mr0jJzgGkAJi3BwaMkwvvWEijNW1yQmFucmQ6ROsVoyXFo+rW1TBxbfoP IbVPvrWUSYsnLz0uVEuedBzJPAKQhozQPbhwsVV1ilJUS5mVkYGAQ4ilILcrNLEGVf8UozsGo JMz7CGQKT2ZeCdzWV0AHMQEddMJgAchBJYkIKakGRt5DbDEdhzRfblzd+31h0ZWd4reD1x+e8 vTQP1XOm8v0PCZNMV38J58ljHPdtaCr5pl3EicsPrE5KO6h+L0parpqyg+MKtfOCUnsWn/yqJ 1Hm+Dub8JxH1es1nzh3hLiVSBad2Wem9uB/g+tD/Ru1BuYPIll2Fi9IjHPz+PZpCfpc6JC5/b deKbEUpyRaKjFXFScCABiSpQwUAMAAA== X-Env-Sender: stefano@aporeto.com X-Msg-Ref: server-6.tower-31.messagelabs.com!1470184511!27435784!1 X-Originating-IP: [209.85.220.52] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 8.77; banners=-,-,- X-VirusChecked: Checked Received: (qmail 28444 invoked from network); 3 Aug 2016 00:35:12 -0000 Received: from mail-pa0-f52.google.com (HELO mail-pa0-f52.google.com) (209.85.220.52) by server-6.tower-31.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 3 Aug 2016 00:35:12 -0000 Received: by mail-pa0-f52.google.com with SMTP id iw10so67918066pac.2 for ; Tue, 02 Aug 2016 17:35:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aporeto.com; s=google; h=date:from:to:cc:subject:message-id:user-agent:mime-version :content-id; bh=j7O/fF6aGs8bIOldIouCtzP+EHRvqfmzjzJ68gdy+0Q=; b=dOQvOD1mOP7TRJNs/nUmHT4yFQ+c9KCTFM4ks5kPYxSkAJAWxGDKg7ncGJcVgIo9yI +QYWqltEwPQ0VctMHCTnk6jgsSOtEs92yTx1UlszDBlGtF5sc07RNrpDySBkNMbj9w+x 5PSgb+J0W3jL4hbS/CC3cviI4MnzTwhNfc3h2siJV02Sy9Z0tZ96N5IW32rgW+IhNvP4 1TpqLZnr8+dKe3z6hQ/KvJj/0ymM+RbCoiikQNj/Fzzdjl5Ybo8uTBBN+yEvufU4vjGD J7EhjJ9adNLX4qEEu3yc9EcMPOhLycvh8uGfpMhzP3r+QoIX/nju7FZ3iIWehqk4V5td BtYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:user-agent :mime-version:content-id; bh=j7O/fF6aGs8bIOldIouCtzP+EHRvqfmzjzJ68gdy+0Q=; b=dJike5ULGDqZDzhe64CdFtSfSD5U7qeV5EOD/hNRNteRCU9R7HoJHt98BMhLHH49Rt bcczUko3xDjSfeds2VsEi5ejJmv+RhfQaODqziGuHLnzw/AawcuBUkkUdKs7iPgkd/DM THeidOPDubGPa4Rj61j2G6WDzxistKF8G0+wQWELCtNqbBzPeQaM0EGO6vZjjC0OMwa/ 0dHtvfROB3k4w/urKdn1B1FpjJ9640QvzSjz5QO4prh+C6xck0ok+9qpzEmWn39yCRwH uBYAh4ndhFkLu8xx5pjUx5qUPDsnjYH/ix0vEdagmXbLYY3kqi8pw2mR3rRFm22BnRDn 9NDQ== X-Gm-Message-State: AEkoouuSo80T3lKM7KtnfHwlu6GZJLSKNCtJc58sC2CaSsd2ulbQ48yK4RMq1SqfIvnIbyuz X-Received: by 10.66.134.20 with SMTP id pg20mr109774250pab.98.1470184510507; Tue, 02 Aug 2016 17:35:10 -0700 (PDT) Received: from [2601:646:8181:5c00:19c0:d26e:babb:7b7] ([2601:646:8181:5c00:19c0:d26e:babb:7b7]) by smtp.gmail.com with ESMTPSA id cf1sm7496312pac.20.2016.08.02.17.35.09 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 02 Aug 2016 17:35:09 -0700 (PDT) Date: Tue, 2 Aug 2016 17:35:08 -0700 (PDT) From: Stefano Stabellini X-X-Sender: sstabellini@sstabellini-ThinkPad-X260 To: xen-devel@lists.xenproject.org Message-ID: User-Agent: Alpine 2.10 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-ID: Cc: jgross@suse.com, lars.kurth@citrix.com, wei.liu2@citrix.com, stefano@aporeto.com, Paul.Durrant@citrix.com, joao.m.martins@oracle.com, boris.ostrovsky@oracle.com, roger.pau@citrix.com Subject: [Xen-devel] [DRAFT v4] PV Calls protocol design document (former XenSock) X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Hi all, This is the design document of the PV Calls protocol. You can find prototypes of the Linux frontend and backend drivers here: git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git pvcalls-4 To use them, make sure to enable CONFIG_PVCALLS in your kernel config and add "pvcalls=1" to the command line of your DomU Linux kernel. You also need the toolstack to create the initial xenstore nodes for the protocol. To do that, please apply the attached patch to libxl (the patch is based on Xen 4.7.0-rc3) and add "pvcalls=1" to your DomU config file. Note that previous versions of the protocols were named XenSock. It has been renamed for clarity of scope and to avoid confusion with hv_sock and vsock, which are used for inter-VMs communications. Cheers, Stefano Changes in v4: - rename xensock to pvcalls Changes in v3: - add a dummy element to struct xen_xensock_request to make sure the size of the struct is the same on both x86_32 and x86_64 Changes in v2: - add max-dataring-page-order - add "Publish backend features and transport parameters" to backend xenbus workflow - update new cmd values - update xen_xensock_request - add backlog parameter to listen and binary layout - add description of new data ring format (interface+data) - modify connect and accept to reflect new data ring format - add link to POSIX docs - add error numbers - add address format section and relevant numeric definitions - add explicit mention of unimplemented commands - add protocol node name - add xenbus shutdown diagram - add socket operation --- # PV Calls Protocol ## Rationale PV Calls is a paravirtualized protocol for the POSIX socket API. The purpose of PV Calls is to allow the implementation of a specific set of POSIX functions to be done in a domain other than your own. It allows connect, accept, bind, release, listen, poll, recvmsg and sendmsg to be implemented in another domain. PV Calls provides the following benefits: * guest networking works out of the box with VPNs, wireless networks and any other complex configurations on the host * guest services listen on ports bound directly to the backend domain IP addresses * localhost becomes a secure namespace for inter-VMs communications * full visibility of the guest behavior on the backend domain, allowing for inexpensive filtering and manipulation of any guest calls * excellent performance ## Design ### Xenstore The frontend and the backend connect to each other exchanging information via xenstore. The toolstack creates front and back nodes with state XenbusStateInitialising. The protocol node name is **pvcalls**. There can only be one PV Calls frontend per domain. #### Frontend XenBus Nodes port Values: The identifier of the Xen event channel used to signal activity in the ring buffer. ring-ref Values: The Xen grant reference granting permission for the backend to map the sole page in a single page sized ring buffer. #### Backend XenBus Nodes max-dataring-page-order Values: The maximum supported size of the data ring in units of lb(machine pages). (e.g. 0 == 1 page, 1 = 2 pages, 2 == 4 pages, etc.). #### State Machine Initialization: *Front* *Back* XenbusStateInitialising XenbusStateInitialising - Query virtual device - Query backend device properties. identification data. - Setup OS device instance. - Publish backend features - Allocate and initialize the and transport parameters request ring. | - Publish transport parameters | that will be in effect during V this connection. XenbusStateInitWait | | V XenbusStateInitialised - Query frontend transport parameters. - Connect to the request ring and event channel. | | V XenbusStateConnected - Query backend device properties. - Finalize OS virtual device instance. | | V XenbusStateConnected Once frontend and backend are connected, they have a shared page, which will is used to exchange messages over a ring, and an event channel, which is used to send notifications. Shutdown: *Front* *Back* XenbusStateConnected XenbusStateConnected | | V XenbusStateClosing - Unmap grants - Unbind evtchns | | V XenbusStateClosing - Unbind evtchns - Free rings - Free data structures | | V XenbusStateClosed - Free remaining data structures | | V XenbusStateClosed ### Commands Ring The shared ring is used by the frontend to forward socket API calls to the backend. I'll refer to this ring as **commands ring** to distinguish it from other rings which will be created later in the lifecycle of the protocol (data rings). The ring format is defined using the familiar `DEFINE_RING_TYPES` macro (`xen/include/public/io/ring.h`). Frontend requests are allocated on the ring using the `RING_GET_REQUEST` macro. The format is defined as follows: #define PVCALLS_SOCKET 0 #define PVCALLS_CONNECT 1 #define PVCALLS_RELEASE 2 #define PVCALLS_BIND 3 #define PVCALLS_LISTEN 4 #define PVCALLS_ACCEPT 5 #define PVCALLS_POLL 6 struct xen_pvcalls_request { uint32_t id; /* private to guest, echoed in response */ uint32_t cmd; /* command to execute */ uint64_t sockid; union { struct xen_pvcalls_socket { uint32_t domain; uint32_t type; uint32_t protocol; } socket; struct xen_pvcalls_connect { uint8_t addr[28]; uint32_t len; uint32_t flags; grant_ref_t ref; uint32_t evtchn; } connect; struct xen_pvcalls_bind { uint8_t addr[28]; uint32_t len; } bind; struct xen_pvcalls_listen { uint32_t backlog; } listen; struct xen_pvcalls_accept { uint64_t sockid; grant_ref_t ref; uint32_t evtchn; } accept; /* dummy member to force sizeof(struct xen_pvcalls_request) to match across archs */ struct xen_pvcalls_dummy { uint8_t dummy[48]; } dummy; } u; }; The first three fields are common for every command. Their binary layout is: 0 4 8 12 16 +-------+-------+-------+-------+ | id | cmd | sockid | +-------+-------+-------+-------+ - **id** is generated by the frontend and identifies one specific request - **cmd** is the command requested by the frontend: - `PVCALLS_SOCKET`: 0 - `PVCALLS_CONNECT`: 1 - `PVCALLS_RELEASE`: 2 - `PVCALLS_BIND`: 3 - `PVCALLS_LISTEN`: 4 - `PVCALLS_ACCEPT`: 5 - `PVCALLS_POLL`: 6 - **sockid** is generated by the frontend and identifies the socket to connect, bind, etc. A new sockid is required on the `PVCALLS_SOCKET` command. A new sockid is also required on `PVCALLS_ACCEPT`, for the new socket. All three fields are echoed back by the backend. As for the other Xen ring based protocols, after writing a request to the ring, the frontend calls `RING_PUSH_REQUESTS_AND_CHECK_NOTIFY` and issues an event channel notification when a notification is required. Backend responses are allocated on the ring using the `RING_GET_RESPONSE` macro. The format is the following: struct xen_pvcalls_response { uint32_t id; uint32_t cmd; uint64_t sockid; int32_t ret; }; 0 4 8 12 16 20 +-------+-------+-------+-------+-------+ | id | cmd | sockid | ret | +-------+-------+-------+-------+-------+ - **id**: echoed back from request - **cmd**: echoed back from request - **sockid**: echoed back from request - **ret**: return value, identifies success (0) or failure (see error numbers below). If the **cmd** is not supported by the backend, ret is ENOTSUPP. After calling `RING_PUSH_RESPONSES_AND_CHECK_NOTIFY`, the backend checks whether it needs to notify the frontend and does so via event channel. A description of each command, their additional request fields and the expected responses follow. #### Socket The **socket** operation corresponds to the POSIX [socket][socket] function. It creates a new socket of the specified family, type and protocol. **sockid** is freely chosen by the frontend and references this specific socket from this point forward. See "Socket families and address format" below. Fields: - **cmd** value: 0 - additional fields: - **domain**: the communication domain - **type**: the socket type - **protocol**: the particular protocol to be used with the socket, usually 0 Binary layout: 16 20 24 28 +--------+--------+--------+ | domain | type |protocol| +--------+--------+--------+ Return value: - 0 on success - See the [POSIX socket function][connect] for error names; the corresponding error numbers are specified later in this document. #### Connect The **connect** operation corresponds to the POSIX [connect][connect] function. It connects a previously created socket (identified by **sockid**) to the specified address. The connect operation creates a new shared ring, which we'll call **data ring**. The data ring is used to send and receive data from the socket. The connect operation passes two additional parameters which are utilized to setup the new ring: **evtchn** and **ref**. **evtchn** is the port number of a new event channel which will be used for notifications of activity on the data ring. **ref** is the grant reference of a page which containes shared pointers to write and read data from the data ring and the full array of grant references for the ring buffers. It will be described in more detailed later. The data ring is unmapped and freed upon issuing a **release** command on the active socket identified by **sockid**. When the frontend issues a **connect** command, the backend: - finds its own internal socket corresponding to **sockid** - connects the socket to **addr** - maps the grant reference **ref**, the shared page contains the data ring interface (`struct pvcalls_data_intf`) - maps all the grant references listed in `struct pvcalls_data_intf` and uses them as shared memory for the ring buffers - bind the **evtchn** - replies to the frontend The data ring format will be described in the following section. Fields: - **cmd** value: 0 - additional fields: - **addr**: address to connect to, see the address format section for more information - **len**: address length - **flags**: flags for the connection, reserved for future usage - **ref**: grant reference of the page containing `struct pvcalls_data_intf` - **evtchn**: port number of the evtchn to signal activity on the data ring Binary layout: 16 20 24 28 32 36 40 44 48 +-------+-------+-------+-------+-------+-------+-------+-------+ | addr | len | +-------+-------+-------+-------+-------+-------+-------+-------+ | flags | ref |evtchn | +-------+-------+-------+ Return value: - 0 on success - See the [POSIX connect function][connect] for error names; the corresponding error numbers are specified later in this document. #### Release The **release** operation closes an existing active or a passive socket. When a release command is issued on a passive socket, the backend releases it and frees its internal mappings. When a release command is issued for an active socket, the data ring is also unmapped and freed: - frontend sends release command for an active socket - backend releases the socket - backend unmaps the data ring buffers - backend unmaps the data ring interface - backend unbinds the evtchn - backend replies to frontend - frontend frees ring and unbinds evtchn Fields: - **cmd** value: 1 - additional fields: none Return value: - 0 on success - See the [POSIX shutdown function][shutdown] for error names; the corresponding error numbers are specified later in this document. #### Bind The **bind** operation corresponds to the POSIX [bind][bind] function. It assigns the address passed as parameter to a previously created socket, identified by **sockid**. **Bind**, **listen** and **accept** are the three operations required to have fully working passive sockets and should be issued in this order. Fields: - **cmd** value: 2 - additional fields: - **addr**: address to connect to, see the address format section for more information - **len**: address length Binary layout: 16 20 24 28 32 36 40 44 48 +-------+-------+-------+-------+-------+-------+-------+-------+ | addr | len | +-------+-------+-------+-------+-------+-------+-------+-------+ Return value: - 0 on success - See the [POSIX bind function][bind] for error names; the corresponding error numbers are specified later in this document. #### Listen The **listen** operation marks the socket as a passive socket. It corresponds to the [POSIX listen function][listen]. Fields: - **cmd** value: 3 - additional fields: - **backlog**: the maximum length to which the queue of pending connections may grow Binary layout: 16 20 +-------+ |backlog| +-------+ Return value: - 0 on success - See the [POSIX listen function][listen] for error names; the corresponding error numbers are specified later in this document. #### Accept The **accept** operation extracts the first connection request on the queue of pending connections for the listening socket identified by **sockid** and creates a new connected socket. The **sockid** of the new socket is also chosen by the frontend and passed as an additional field of the accept request struct. See the [POSIX accept function][accept] as reference. Similarly to the **connect** operation, **accept** creates a new data ring. Information necessary to setup the new ring, such the grant table reference of the page containing the data ring interface (`struct pvcalls_data_intf`) and event channel port, are passed from the frontend to the backend as part of the request. The backend will reply to the request only when a new connection is successfully accepted, i.e. the backend does not return EAGAIN or EWOULDBLOCK. Example workflow: - frontend issues an **accept** request - backend waits for a connection to be available on the socket - a new connection becomes available - backend accepts the new connection - backend creates an internal mapping from **sockid** to the new socket - backend maps the grant reference **ref**, the shared page contains the data ring interface (`struct pvcalls_data_intf`) - backend maps all the grant references listed in `struct pvcalls_data_intf` and uses them as shared memory for the new data ring - backend binds the **evtchn** - backend replies to the frontend Fields: - **cmd** value: 4 - additional fields: - **sockid**: id of the new socket - **ref**: grant reference of the data ring interface (`struct pvcalls_data_intf`) - **evtchn**: port number of the evtchn to signal activity on the data ring Binary layout: 16 20 24 28 32 +-------+-------+-------+-------+ | sockid | ref |evtchn | +-------+-------+-------+-------+ Return value: - 0 on success - See the [POSIX accept function][accept] for error names; the corresponding error numbers are specified later in this document. #### Poll The **poll** operation is only valid for passive sockets. For active sockets, the frontend should look at the state of the data ring. When a new connection is available in the queue of the passive socket, the backend generates a response and notifies the frontend. Fields: - **cmd** value: 5 - additional fields: none Return value: - 0 on success - See the [POSIX poll function][poll] for error names; the corresponding error numbers are specified later in this document. #### Error numbers The numbers corresponding to the error names specified by POSIX are: [EPERM] -1 [ENOENT] -2 [ESRCH] -3 [EINTR] -4 [EIO] -5 [ENXIO] -6 [E2BIG] -7 [ENOEXEC] -8 [EBADF] -9 [ECHILD] -10 [EAGAIN] -11 [EWOULDBLOCK] -11 [ENOMEM] -12 [EACCES] -13 [EFAULT] -14 [EBUSY] -16 [EEXIST] -17 [EXDEV] -18 [ENODEV] -19 [EISDIR] -21 [EINVAL] -22 [ENFILE] -23 [EMFILE] -24 [ENOSPC] -28 [EROFS] -30 [EMLINK] -31 [EDOM] -33 [ERANGE] -34 [EDEADLK] -35 [EDEADLOCK] -35 [ENAMETOOLONG] -36 [ENOLCK] -37 [ENOTEMPTY] -39 [ENOSYS] -38 [ENODATA] -61 [ETIME] -62 [EBADMSG] -74 [EOVERFLOW] -75 [EILSEQ] -84 [ERESTART] -85 [ENOTSOCK] -88 [EOPNOTSUPP] -95 [EAFNOSUPPORT] -97 [EADDRINUSE] -98 [EADDRNOTAVAIL] -99 [ENOBUFS] -105 [EISCONN] -106 [ENOTCONN] -107 [ETIMEDOUT] -110 [ENOTSUPP] -524 #### Socket families and address format The following definitions and explicit sizes, together with POSIX [sys/socket.h][address] and [netinet/in.h][in] define socket families and address format. Please be aware that only the **domain** `AF_INET`, **type** `SOCK_STREAM` and **protocol** `0` are supported by this version of the spec. #define AF_UNSPEC 0 #define AF_UNIX 1 /* Unix domain sockets */ #define AF_LOCAL 1 /* POSIX name for AF_UNIX */ #define AF_INET 2 /* Internet IP Protocol */ #define AF_INET6 10 /* IP version 6 */ #define SOCK_STREAM 1 #define SOCK_DGRAM 2 #define SOCK_RAW 3 /* generic address format */ struct sockaddr { uint16_t sa_family_t; char sa_data[26]; }; struct in_addr { uint32_t s_addr; }; /* AF_INET address format */ struct sockaddr_in { uint16_t sa_family_t; uint16_t sin_port; struct in_addr sin_addr; char sin_zero[20]; }; ### Data ring Data rings are used for sending and receiving data over a connected socket. They are created upon a successful **accept** or **connect** command. A data ring is composed of two pieces: the interface and the **in** and **out** buffers. The interface, represented by `struct pvcalls_ring_intf` is shared first and resides on the page whose grant reference is passed by **accept** and **connect** as parameter. `struct pvcalls_ring_intf` contains the list of grant references which constitute the **in** and **out** data buffers. #### Data ring interface struct pvcalls_data_intf { PVCALLS_RING_IDX in_cons, in_prod; PVCALLS_RING_IDX out_cons, out_prod; int32_t in_error, out_error; uint32_t ring_order; grant_ref_t ref[]; }; /* not actually C compliant (ring_order changes from socket to socket) */ struct pvcalls_data { char in[((1< cons) size = prod - cons; else { size = ring_size - cons; size += prod; } return size; } The producer (the backend for **in**, the frontend for **out**) writes to the array in the following way: - read *cons*, *prod*, *error* from shared memory - memory barrier - return on *error* - write to array at position *prod* up to *cons*, wrapping around the circular buffer when necessary - memory barrier - increase *prod* - notify the other end via evtchn The consumer (the backend for **out**, the frontend for **in**) reads from the array in the following way: - read *prod*, *cons*, *error* from shared memory - memory barrier - return on *error* - read from array at position *cons* up to *prod*, wrapping around the circular buffer when necessary - memory barrier - increase *cons* - notify the other end via evtchn The producer takes care of writing only as many bytes as available in the buffer up to *cons*. The consumer takes care of reading only as many bytes as available in the buffer up to *prod*. *error* is set by the backend when an error occurs writing or reading from the socket. [address]: http://pubs.opengroup.org/onlinepubs/7908799/xns/syssocket.h.html [in]: http://pubs.opengroup.org/onlinepubs/000095399/basedefs/netinet/in.h.html [socket]: http://pubs.opengroup.org/onlinepubs/009695399/functions/socket.html [connect]: http://pubs.opengroup.org/onlinepubs/7908799/xns/connect.html [shutdown]: http://pubs.opengroup.org/onlinepubs/7908799/xns/shutdown.html [bind]: http://pubs.opengroup.org/onlinepubs/7908799/xns/bind.html [listen]: http://pubs.opengroup.org/onlinepubs/7908799/xns/listen.html [accept]: http://pubs.opengroup.org/onlinepubs/7908799/xns/accept.html [poll]: http://pubs.opengroup.org/onlinepubs/7908799/xsh/poll.html diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c index c39d745..d784a10 100644 --- a/tools/libxl/libxl.c +++ b/tools/libxl/libxl.c @@ -2299,6 +2299,70 @@ int libxl_devid_to_device_vtpm(libxl_ctx *ctx, return rc; } +/******************************************************************************/ + +int libxl__device_pvcalls_setdefault(libxl__gc *gc, libxl_device_pvcalls *pvcalls) +{ + int rc; + + rc = libxl__resolve_domid(gc, pvcalls->backend_domname, &pvcalls->backend_domid); + return rc; +} + +static int libxl__device_from_pvcalls(libxl__gc *gc, uint32_t domid, + libxl_device_pvcalls *pvcalls, + libxl__device *device) +{ + device->backend_devid = pvcalls->devid; + device->backend_domid = pvcalls->backend_domid; + device->backend_kind = LIBXL__DEVICE_KIND_PVCALLS; + device->devid = pvcalls->devid; + device->domid = domid; + device->kind = LIBXL__DEVICE_KIND_PVCALLS; + + return 0; +} + + +int libxl__device_pvcalls_add(libxl__gc *gc, uint32_t domid, + libxl_device_pvcalls *pvcalls) +{ + flexarray_t *front; + flexarray_t *back; + libxl__device device; + int rc; + + rc = libxl__device_pvcalls_setdefault(gc, pvcalls); + if (rc) goto out; + + front = flexarray_make(gc, 16, 1); + back = flexarray_make(gc, 16, 1); + + if (pvcalls->devid == -1) { + if ((pvcalls->devid = libxl__device_nextid(gc, domid, "pvcalls")) < 0) { + rc = ERROR_FAIL; + goto out; + } + } + + rc = libxl__device_from_pvcalls(gc, domid, pvcalls, &device); + if (rc != 0) goto out; + + flexarray_append_pair(back, "frontend-id", libxl__sprintf(gc, "%d", domid)); + flexarray_append_pair(back, "online", "1"); + flexarray_append_pair(back, "state", GCSPRINTF("%d", XenbusStateInitialising)); + flexarray_append_pair(front, "backend-id", + libxl__sprintf(gc, "%d", pvcalls->backend_domid)); + flexarray_append_pair(front, "state", GCSPRINTF("%d", XenbusStateInitialising)); + + libxl__device_generic_add(gc, XBT_NULL, &device, + libxl__xs_kvs_of_flexarray(gc, back, back->count), + libxl__xs_kvs_of_flexarray(gc, front, front->count), + NULL); + rc = 0; +out: + return rc; +} /******************************************************************************/ @@ -4250,6 +4314,8 @@ out: * libxl_device_vfb_destroy * libxl_device_usbctrl_remove * libxl_device_usbctrl_destroy + * libxl_device_pvcalls_remove + * libxl_device_pvcalls_destroy */ #define DEFINE_DEVICE_REMOVE_EXT(type, remtype, removedestroy, f) \ int libxl_device_##type##_##removedestroy(libxl_ctx *ctx, \ @@ -4311,6 +4377,11 @@ DEFINE_DEVICE_REMOVE(vtpm, destroy, 1) DEFINE_DEVICE_REMOVE_CUSTOM(usbctrl, remove, 0) DEFINE_DEVICE_REMOVE_CUSTOM(usbctrl, destroy, 1) +/* pvcalls */ + +DEFINE_DEVICE_REMOVE(pvcalls, remove, 0) +DEFINE_DEVICE_REMOVE(pvcalls, destroy, 1) + /* channel/console hotunplug is not implemented. There are 2 possibilities: * 1. add support for secondary consoles to xenconsoled * 2. dynamically add/remove qemu chardevs via qmp messages. */ diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h index 2c0f868..9358071 100644 --- a/tools/libxl/libxl.h +++ b/tools/libxl/libxl.h @@ -1753,6 +1753,16 @@ int libxl_device_vfb_destroy(libxl_ctx *ctx, uint32_t domid, const libxl_asyncop_how *ao_how) LIBXL_EXTERNAL_CALLERS_ONLY; +/* pvcalls */ +int libxl_device_pvcalls_remove(libxl_ctx *ctx, uint32_t domid, + libxl_device_pvcalls *pvcalls, + const libxl_asyncop_how *ao_how) + LIBXL_EXTERNAL_CALLERS_ONLY; +int libxl_device_pvcalls_destroy(libxl_ctx *ctx, uint32_t domid, + libxl_device_pvcalls *pvcalls, + const libxl_asyncop_how *ao_how) + LIBXL_EXTERNAL_CALLERS_ONLY; + /* PCI Passthrough */ int libxl_device_pci_add(libxl_ctx *ctx, uint32_t domid, libxl_device_pci *pcidev, diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c index 5000bd0..f019c37 100644 --- a/tools/libxl/libxl_create.c +++ b/tools/libxl/libxl_create.c @@ -1374,6 +1374,8 @@ static void domcreate_launch_dm(libxl__egc *egc, libxl__multidev *multidev, libxl__device_vfb_add(gc, domid, &d_config->vfbs[i]); libxl__device_vkb_add(gc, domid, &d_config->vkbs[i]); } + for (i = 0; i < d_config->num_pvcallss; i++) + libxl__device_pvcalls_add(gc, domid, &d_config->pvcallss[i]); init_console_info(gc, &console, 0); console.backend_domid = state->console_domid; diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h index c791418..063d926 100644 --- a/tools/libxl/libxl_internal.h +++ b/tools/libxl/libxl_internal.h @@ -1224,6 +1224,7 @@ _hidden int libxl__device_vkb_setdefault(libxl__gc *gc, libxl_device_vkb *vkb); _hidden int libxl__device_pci_setdefault(libxl__gc *gc, libxl_device_pci *pci); _hidden void libxl__rdm_setdefault(libxl__gc *gc, libxl_domain_build_info *b_info); +_hidden int libxl__device_pvcalls_setdefault(libxl__gc *gc, libxl_device_pvcalls *pvcalls); _hidden const char *libxl__device_nic_devname(libxl__gc *gc, uint32_t domid, @@ -2647,6 +2648,10 @@ _hidden int libxl__device_vkb_add(libxl__gc *gc, uint32_t domid, _hidden int libxl__device_vfb_add(libxl__gc *gc, uint32_t domid, libxl_device_vfb *vfb); +/* Internal function to connect a pvcalls device */ +_hidden int libxl__device_pvcalls_add(libxl__gc *gc, uint32_t domid, + libxl_device_pvcalls *pvcalls); + /* Waits for the passed device to reach state XenbusStateInitWait. * This is not really useful by itself, but is important when executing * hotplug scripts, since we need to be sure the device is in the correct diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl index 9840f3b..a99c7be 100644 --- a/tools/libxl/libxl_types.idl +++ b/tools/libxl/libxl_types.idl @@ -685,6 +685,12 @@ libxl_device_vtpm = Struct("device_vtpm", [ ("uuid", libxl_uuid), ]) +libxl_device_pvcalls = Struct("device_pvcalls", [ + ("backend_domid", libxl_domid), + ("backend_domname", string), + ("devid", libxl_devid), +]) + libxl_device_channel = Struct("device_channel", [ ("backend_domid", libxl_domid), ("backend_domname", string), @@ -709,6 +715,7 @@ libxl_domain_config = Struct("domain_config", [ ("vfbs", Array(libxl_device_vfb, "num_vfbs")), ("vkbs", Array(libxl_device_vkb, "num_vkbs")), ("vtpms", Array(libxl_device_vtpm, "num_vtpms")), + ("pvcallss", Array(libxl_device_pvcalls, "num_pvcallss")), # a channel manifests as a console with a name, # see docs/misc/channels.txt ("channels", Array(libxl_device_channel, "num_channels")), diff --git a/tools/libxl/libxl_types_internal.idl b/tools/libxl/libxl_types_internal.idl index 177f9b7..b41122b 100644 --- a/tools/libxl/libxl_types_internal.idl +++ b/tools/libxl/libxl_types_internal.idl @@ -24,6 +24,7 @@ libxl__device_kind = Enumeration("device_kind", [ (8, "VTPM"), (9, "VUSB"), (10, "QUSB"), + (11, "PVCALLS"), ]) libxl__console_backend = Enumeration("console_backend", [ diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c index 03ab644..23f9793 100644 --- a/tools/libxl/xl_cmdimpl.c +++ b/tools/libxl/xl_cmdimpl.c @@ -1898,6 +1898,18 @@ static void parse_config_data(const char *config_source, free(buf2); } } + + if (!xlu_cfg_get_long(config, "pvcalls", &l, 0)) { + libxl_device_pvcalls *pvcalls; + fprintf(stderr, "Creating pvcalls l=%lu\n", l); + d_config->num_pvcallss = 0; + d_config->pvcallss = NULL; + pvcalls = ARRAY_EXTEND_INIT(d_config->pvcallss, + d_config->num_pvcallss, + libxl_device_pvcalls_init); + libxl_device_pvcalls_init(pvcalls); + replace_string(&pvcalls->backend_domname, "0"); + } if (!xlu_cfg_get_list (config, "channel", &channels, 0, 0)) { d_config->num_channels = 0;