| Message ID | 1505516440-11111-2-git-send-email-sstabellini@kernel.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
> + > +struct pvcalls_bedata { > + struct xen_pvcalls_front_ring ring; > + grant_ref_t ref; > + int irq; > + > + struct list_head socket_mappings; > + struct list_head socketpass_mappings; > + spinlock_t socket_lock; > + > + wait_queue_head_t inflight_req; > + struct xen_pvcalls_response rsp[PVCALLS_NR_REQ_PER_RING]; > +}; > +static struct xenbus_device *pvcalls_front_dev; > +static atomic_t pvcalls_refcount; Should the refcount be per back/frontend? > + > +/* first increment refcount, then proceed */ > +#define pvcalls_enter { \ > + atomic_inc(&pvcalls_refcount); \ > + smp_mb(); \ > +} > + > +/* first complete other operations, then decrement refcount */ > +#define pvcalls_exit { \ > + smp_mb(); \ > + atomic_dec(&pvcalls_refcount); \ > +} I think atomic increment/decrement imply a barrier. -boris
> > static int pvcalls_front_remove(struct xenbus_device *dev) > { > + struct pvcalls_bedata *bedata; > + struct sock_mapping *map = NULL, *n; > + > + bedata = dev_get_drvdata(&pvcalls_front_dev->dev); > + dev_set_drvdata(&dev->dev, NULL); > + pvcalls_front_dev = NULL; One more comment on this patch: should this be the last thing to do? pvcalls_front_dev is what prevents pvcalls_front_probe() from proceeding and even though I am not sure a probe can be initiated while we are here perhaps moving this to the end would make things slightly safer. -boris > + if (bedata->irq >= 0) > + unbind_from_irqhandler(bedata->irq, dev); > + > + smp_mb(); > + while (atomic_read(&pvcalls_refcount) > 0) > + cpu_relax(); > + list_for_each_entry_safe(map, n, &bedata->socket_mappings, list) { > + pvcalls_front_free_map(bedata, map); > + kfree(map); > + } > + list_for_each_entry_safe(map, n, &bedata->socketpass_mappings, list) { > + spin_lock(&bedata->socket_lock); > + list_del_init(&map->list); > + spin_unlock(&bedata->socket_lock); > + kfree(map); > + } > + if (bedata->ref >= 0) > + gnttab_end_foreign_access(bedata->ref, 0, 0); > + kfree(bedata->ring.sring); > + kfree(bedata); > + xenbus_switch_state(dev, XenbusStateClosed); > return 0; > } >
On Wed, 20 Sep 2017, Boris Ostrovsky wrote: > > + > > +struct pvcalls_bedata { > > + struct xen_pvcalls_front_ring ring; > > + grant_ref_t ref; > > + int irq; > > + > > + struct list_head socket_mappings; > > + struct list_head socketpass_mappings; > > + spinlock_t socket_lock; > > + > > + wait_queue_head_t inflight_req; > > + struct xen_pvcalls_response rsp[PVCALLS_NR_REQ_PER_RING]; > > +}; > > +static struct xenbus_device *pvcalls_front_dev; > > +static atomic_t pvcalls_refcount; > > Should the refcount be per back/frontend? Yes it is, but only one back/frontend connection is supported by the frontend. I can add a comment. > > + > > +/* first increment refcount, then proceed */ > > +#define pvcalls_enter { \ > > + atomic_inc(&pvcalls_refcount); \ > > + smp_mb(); \ > > +} > > + > > +/* first complete other operations, then decrement refcount */ > > +#define pvcalls_exit { \ > > + smp_mb(); \ > > + atomic_dec(&pvcalls_refcount); \ > > +} > > I think atomic increment/decrement imply a barrier. You are right. From Documentation/core-api/atomic_ops.rst: One very important aspect of these two routines is that they DO NOT require any explicit memory barriers.
On 10/06/2017 01:51 PM, Stefano Stabellini wrote: > On Wed, 20 Sep 2017, Boris Ostrovsky wrote: >>> + >>> +struct pvcalls_bedata { >>> + struct xen_pvcalls_front_ring ring; >>> + grant_ref_t ref; >>> + int irq; >>> + >>> + struct list_head socket_mappings; >>> + struct list_head socketpass_mappings; >>> + spinlock_t socket_lock; >>> + >>> + wait_queue_head_t inflight_req; >>> + struct xen_pvcalls_response rsp[PVCALLS_NR_REQ_PER_RING]; >>> +}; >>> +static struct xenbus_device *pvcalls_front_dev; >>> +static atomic_t pvcalls_refcount; >> Should the refcount be per back/frontend? > Yes it is, but only one back/frontend connection is supported by the > frontend. I can add a comment. Since it's per backend connection --- shouldn't it be part of struct pvcalls_bedata? -boris
On Fri, 6 Oct 2017, Boris Ostrovsky wrote: > On 10/06/2017 01:51 PM, Stefano Stabellini wrote: > > On Wed, 20 Sep 2017, Boris Ostrovsky wrote: > >>> + > >>> +struct pvcalls_bedata { > >>> + struct xen_pvcalls_front_ring ring; > >>> + grant_ref_t ref; > >>> + int irq; > >>> + > >>> + struct list_head socket_mappings; > >>> + struct list_head socketpass_mappings; > >>> + spinlock_t socket_lock; > >>> + > >>> + wait_queue_head_t inflight_req; > >>> + struct xen_pvcalls_response rsp[PVCALLS_NR_REQ_PER_RING]; > >>> +}; > >>> +static struct xenbus_device *pvcalls_front_dev; > >>> +static atomic_t pvcalls_refcount; > >> Should the refcount be per back/frontend? > > Yes it is, but only one back/frontend connection is supported by the > > frontend. I can add a comment. > > Since it's per backend connection --- shouldn't it be part of struct > pvcalls_bedata? struct pvcalls_bedata is allocated with kzalloc. To protect accesses to it, pvcalls_refcount needs to be outside of it.
On 10/06/2017 04:29 PM, Stefano Stabellini wrote: > On Fri, 6 Oct 2017, Boris Ostrovsky wrote: >> On 10/06/2017 01:51 PM, Stefano Stabellini wrote: >>> On Wed, 20 Sep 2017, Boris Ostrovsky wrote: >>>>> + >>>>> +struct pvcalls_bedata { >>>>> + struct xen_pvcalls_front_ring ring; >>>>> + grant_ref_t ref; >>>>> + int irq; >>>>> + >>>>> + struct list_head socket_mappings; >>>>> + struct list_head socketpass_mappings; >>>>> + spinlock_t socket_lock; >>>>> + >>>>> + wait_queue_head_t inflight_req; >>>>> + struct xen_pvcalls_response rsp[PVCALLS_NR_REQ_PER_RING]; >>>>> +}; >>>>> +static struct xenbus_device *pvcalls_front_dev; >>>>> +static atomic_t pvcalls_refcount; >>>> Should the refcount be per back/frontend? >>> Yes it is, but only one back/frontend connection is supported by the >>> frontend. I can add a comment. >> Since it's per backend connection --- shouldn't it be part of struct >> pvcalls_bedata? > struct pvcalls_bedata is allocated with kzalloc. To protect accesses to > it, pvcalls_refcount needs to be outside of it. Oh, yes, of course. I think you also might be accessing it after the struct is freed. Nevermind then. -boris
On Wed, 20 Sep 2017, Boris Ostrovsky wrote: > > static int pvcalls_front_remove(struct xenbus_device *dev) > > { > > + struct pvcalls_bedata *bedata; > > + struct sock_mapping *map = NULL, *n; > > + > > + bedata = dev_get_drvdata(&pvcalls_front_dev->dev); > > + dev_set_drvdata(&dev->dev, NULL); > > + pvcalls_front_dev = NULL; > > One more comment on this patch: should this be the last thing to do? > pvcalls_front_dev is what prevents pvcalls_front_probe() from proceeding > and even though I am not sure a probe can be initiated while we are here > perhaps moving this to the end would make things slightly safer. pvcalls_front_dev = NULL is set early to prevent most other functions (connect, listen, etc) from proceeding. On the other end, pvcalls_front_probe can continue safely in parallel with pvcalls_front_remove to add a new frontend-backend connection (which is not supported today anyway). pvcalls_front_probe would be called with a different xenbus_device parameter. > > + if (bedata->irq >= 0) > > + unbind_from_irqhandler(bedata->irq, dev); > > + > > + smp_mb(); > > + while (atomic_read(&pvcalls_refcount) > 0) > > + cpu_relax(); > > + list_for_each_entry_safe(map, n, &bedata->socket_mappings, list) { > > + pvcalls_front_free_map(bedata, map); > > + kfree(map); > > + } > > + list_for_each_entry_safe(map, n, &bedata->socketpass_mappings, list) { > > + spin_lock(&bedata->socket_lock); > > + list_del_init(&map->list); > > + spin_unlock(&bedata->socket_lock); > > + kfree(map); > > + } > > + if (bedata->ref >= 0) > > + gnttab_end_foreign_access(bedata->ref, 0, 0); > > + kfree(bedata->ring.sring); > > + kfree(bedata); > > + xenbus_switch_state(dev, XenbusStateClosed); > > return 0; > > } > > >
diff --git a/drivers/xen/pvcalls-front.c b/drivers/xen/pvcalls-front.c index a8d38c2..67c8337 100644 --- a/drivers/xen/pvcalls-front.c +++ b/drivers/xen/pvcalls-front.c @@ -20,6 +20,42 @@ #include <xen/xenbus.h> #include <xen/interface/io/pvcalls.h> +#define PVCALLS_INVALID_ID UINT_MAX +#define PVCALLS_RING_ORDER XENBUS_MAX_RING_GRANT_ORDER +#define PVCALLS_NR_REQ_PER_RING __CONST_RING_SIZE(xen_pvcalls, XEN_PAGE_SIZE) + +struct pvcalls_bedata { + struct xen_pvcalls_front_ring ring; + grant_ref_t ref; + int irq; + + struct list_head socket_mappings; + struct list_head socketpass_mappings; + spinlock_t socket_lock; + + wait_queue_head_t inflight_req; + struct xen_pvcalls_response rsp[PVCALLS_NR_REQ_PER_RING]; +}; +static struct xenbus_device *pvcalls_front_dev; +static atomic_t pvcalls_refcount; + +/* first increment refcount, then proceed */ +#define pvcalls_enter { \ + atomic_inc(&pvcalls_refcount); \ + smp_mb(); \ +} + +/* first complete other operations, then decrement refcount */ +#define pvcalls_exit { \ + smp_mb(); \ + atomic_dec(&pvcalls_refcount); \ +} + +static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id) +{ + return IRQ_HANDLED; +} + static const struct xenbus_device_id pvcalls_front_ids[] = { { "pvcalls" }, { "" } @@ -27,6 +63,33 @@ static int pvcalls_front_remove(struct xenbus_device *dev) { + struct pvcalls_bedata *bedata; + struct sock_mapping *map = NULL, *n; + + bedata = dev_get_drvdata(&pvcalls_front_dev->dev); + dev_set_drvdata(&dev->dev, NULL); + pvcalls_front_dev = NULL; + if (bedata->irq >= 0) + unbind_from_irqhandler(bedata->irq, dev); + + smp_mb(); + while (atomic_read(&pvcalls_refcount) > 0) + cpu_relax(); + list_for_each_entry_safe(map, n, &bedata->socket_mappings, list) { + pvcalls_front_free_map(bedata, map); + kfree(map); + } + list_for_each_entry_safe(map, n, &bedata->socketpass_mappings, list) { + spin_lock(&bedata->socket_lock); + list_del_init(&map->list); + spin_unlock(&bedata->socket_lock); + kfree(map); + } + if (bedata->ref >= 0) + gnttab_end_foreign_access(bedata->ref, 0, 0); + kfree(bedata->ring.sring); + kfree(bedata); + xenbus_switch_state(dev, XenbusStateClosed); return 0; }
Introduce a data structure named pvcalls_bedata. It contains pointers to the command ring, the event channel, a list of active sockets and a list of passive sockets. Lists accesses are protected by a spin_lock. Introduce a waitqueue to allow waiting for a response on commands sent to the backend. Introduce an array of struct xen_pvcalls_response to store commands responses. pvcalls_refcount is used to keep count of the outstanding pvcalls users. Only remove connections once the refcount is zero. Implement pvcalls frontend removal function. Go through the list of active and passive sockets and free them all, one at a time. Signed-off-by: Stefano Stabellini <stefano@aporeto.com> CC: boris.ostrovsky@oracle.com CC: jgross@suse.com --- drivers/xen/pvcalls-front.c | 63 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+)