From patchwork Mon Mar 16 12:36:23 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hannes Reinecke X-Patchwork-Id: 6017821 X-Patchwork-Delegate: christophe.varoqui@free.fr Return-Path: X-Original-To: patchwork-dm-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id CE62CBFE6A for ; Mon, 16 Mar 2015 12:41:07 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id D0C1A204D1 for ; Mon, 16 Mar 2015 12:41:06 +0000 (UTC) Received: from mx6-phx2.redhat.com (mx6-phx2.redhat.com [209.132.183.39]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 759D3204EB for ; Mon, 16 Mar 2015 12:41:04 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx6-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t2GCcEvn005297; Mon, 16 Mar 2015 08:38:14 -0400 Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id t2GCbHOm028795 for ; Mon, 16 Mar 2015 08:37:17 -0400 Received: from mx1.redhat.com (ext-mx04.extmail.prod.ext.phx2.redhat.com [10.5.110.28]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t2GCbGpl004175 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 16 Mar 2015 08:37:16 -0400 Received: from mx2.suse.de (cantor2.suse.de [195.135.220.15]) by mx1.redhat.com (Postfix) with ESMTPS id DE5EDC4086; Mon, 16 Mar 2015 12:37:15 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 5C301ADDC; Mon, 16 Mar 2015 12:37:09 +0000 (UTC) From: Hannes Reinecke To: Christophe Varoqui Date: Mon, 16 Mar 2015 13:36:23 +0100 Message-Id: <1426509425-15978-37-git-send-email-hare@suse.de> In-Reply-To: <1426509425-15978-1-git-send-email-hare@suse.de> References: <1426509425-15978-1-git-send-email-hare@suse.de> X-RedHat-Spam-Score: -7.309 (BAYES_00, DCC_REPUT_00_12, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, URIBL_BLOCKED) 195.135.220.15 cantor2.suse.de 195.135.220.15 cantor2.suse.de X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Scanned-By: MIMEDefang 2.75 on 10.5.110.28 X-loop: dm-devel@redhat.com Cc: dm-devel@redhat.com Subject: [dm-devel] [PATCH 36/78] Use poll() when receiving uevents X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk Reply-To: device-mapper development List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently uevent_listen() will be adding each event individually to the internal queue. This leads to a lock contention on high load, as uevent_listen() has to grab the shared lock before doing so. This patch batches the uevent reception so that uevents will only ever be added to the internal queue if there are no more events pending. Signed-off-by: Hannes Reinecke --- libmultipath/list.h | 47 +++++++++++++++++++++++++------- libmultipath/uevent.c | 74 +++++++++++++++++++++++++++++++++++++++------------ 2 files changed, 95 insertions(+), 26 deletions(-) diff --git a/libmultipath/list.h b/libmultipath/list.h index 8626630..7a3cf16 100644 --- a/libmultipath/list.h +++ b/libmultipath/list.h @@ -161,18 +161,18 @@ static inline int list_empty(struct list_head *head) return head->next == head; } -static inline void __list_splice(struct list_head *list, - struct list_head *head) +static inline void __list_splice(const struct list_head *list, + struct list_head *prev, + struct list_head *next) { struct list_head *first = list->next; struct list_head *last = list->prev; - struct list_head *at = head->next; - first->prev = head; - head->next = first; + first->prev = prev; + prev->next = first; - last->next = at; - at->prev = last; + last->next = next; + next->prev = last; } /** @@ -183,7 +183,19 @@ static inline void __list_splice(struct list_head *list, static inline void list_splice(struct list_head *list, struct list_head *head) { if (!list_empty(list)) - __list_splice(list, head); + __list_splice(list, head, head->next); +} + +/** + * list_splice_tail - join two lists, each list being a queue + * @list: the new list to add. + * @head: the place to add it in the first list. + */ +static inline void list_splice_tail(struct list_head *list, + struct list_head *head) +{ + if (!list_empty(list)) + __list_splice(list, head->prev, head); } /** @@ -197,7 +209,24 @@ static inline void list_splice_init(struct list_head *list, struct list_head *head) { if (!list_empty(list)) { - __list_splice(list, head); + __list_splice(list, head, head->next); + INIT_LIST_HEAD(list); + } +} + +/** + * list_splice_tail_init - join two lists and reinitialise the emptied list + * @list: the new list to add. + * @head: the place to add it in the first list. + * + * Each of the lists is a queue. + * The list at @list is reinitialised + */ +static inline void list_splice_tail_init(struct list_head *list, + struct list_head *head) +{ + if (!list_empty(list)) { + __list_splice(list, head->prev, head); INIT_LIST_HEAD(list); } } diff --git a/libmultipath/uevent.c b/libmultipath/uevent.c index 7e41c6f..478c6ce 100644 --- a/libmultipath/uevent.c +++ b/libmultipath/uevent.c @@ -34,9 +34,11 @@ #include #include #include +#include #include #include #include +#include #include #include #include @@ -470,10 +472,14 @@ struct uevent *uevent_from_udev_device(struct udev_device *dev) int uevent_listen(struct udev *udev) { - int err; + int err = 2; struct udev_monitor *monitor = NULL; - int fd, socket_flags; + int fd, fd_ep = -1, socket_flags, events; int need_failback = 1; + int timeout = 30; + sigset_t mask; + LIST_HEAD(uevlisten_tmp); + /* * Queue uevents for service by dedicated thread so that the uevent * listening thread does not block on multipathd locks (vecs->lock) @@ -490,7 +496,6 @@ int uevent_listen(struct udev *udev) monitor = udev_monitor_new_from_netlink(udev, "udev"); if (!monitor) { condlog(2, "failed to create udev monitor"); - err = 2; goto out; } #ifdef LIBUDEV_API_RECVBUF @@ -522,28 +527,63 @@ int uevent_listen(struct udev *udev) condlog(2, "failed to enable receiving : %s", strerror(-err)); goto out; } + + pthread_sigmask(SIG_SETMASK, NULL, &mask); + sigdelset(&mask, SIGHUP); + sigdelset(&mask, SIGUSR1); + events = 0; while (1) { struct uevent *uev; struct udev_device *dev; - - dev = udev_monitor_receive_device(monitor); - if (!dev) { - condlog(0, "failed getting udev device"); + struct pollfd ev_poll; + struct timespec poll_timeout; + int fdcount; + + memset(&ev_poll, 0, sizeof(struct pollfd)); + ev_poll.fd = fd; + ev_poll.events = POLLIN; + memset(&poll_timeout, 0, sizeof(struct timespec)); + poll_timeout.tv_sec = timeout; + errno = 0; + fdcount = ppoll(&ev_poll, 1, &poll_timeout, &mask); + if (fdcount && ev_poll.revents & POLLIN) { + timeout = 0; + dev = udev_monitor_receive_device(monitor); + if (!dev) { + condlog(0, "failed getting udev device"); + continue; + } + uev = uevent_from_udev_device(dev); + if (!uev) + continue; + list_add_tail(&uev->node, &uevlisten_tmp); + events++; continue; } - uev = uevent_from_udev_device(dev); - if (!uev) - continue; - /* - * Queue uevent and poke service pthread. - */ - pthread_mutex_lock(uevq_lockp); - list_add_tail(&uev->node, &uevq); - pthread_cond_signal(uev_condp); - pthread_mutex_unlock(uevq_lockp); + if (fdcount < 0) { + if (errno != EINTR) + condlog(0, "error receiving " + "uevent message: %m"); + err = -errno; + break; + } + if (!list_empty(&uevlisten_tmp)) { + /* + * Queue uevents and poke service pthread. + */ + condlog(3, "Forwarding %d uevents", events); + pthread_mutex_lock(uevq_lockp); + list_splice_tail_init(&uevlisten_tmp, &uevq); + pthread_cond_signal(uev_condp); + pthread_mutex_unlock(uevq_lockp); + events = 0; + } + timeout = 30; } need_failback = 0; out: + if (fd_ep >= 0) + close(fd_ep); if (monitor) udev_monitor_unref(monitor); if (need_failback)