From patchwork Sun Mar 17 12:22:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikos Tsironis X-Patchwork-Id: 10856289 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A5D721515 for ; Sun, 17 Mar 2019 12:23:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B07929506 for ; Sun, 17 Mar 2019 12:23:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 450D42950E; Sun, 17 Mar 2019 12:23:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 120E629506 for ; Sun, 17 Mar 2019 12:23:09 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AD56C3091782; Sun, 17 Mar 2019 12:23:08 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7224160851; Sun, 17 Mar 2019 12:23:08 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 64565181A137; Sun, 17 Mar 2019 12:23:06 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x2HCN58n021919 for ; Sun, 17 Mar 2019 08:23:05 -0400 Received: by smtp.corp.redhat.com (Postfix) id 5F67960C70; Sun, 17 Mar 2019 12:23:05 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from mx1.redhat.com (ext-mx14.extmail.prod.ext.phx2.redhat.com [10.5.110.43]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5A3F66025C for ; Sun, 17 Mar 2019 12:23:03 +0000 (UTC) Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 38A8B3092647 for ; Sun, 17 Mar 2019 12:23:02 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id f65so10394028wma.2 for ; Sun, 17 Mar 2019 05:23:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arrikto-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id; bh=VjG0kRzlBwR8YGaPDB6PFGDHZQlihIoW/wLAoir9C6s=; b=h7t5XH2PM9T1GfnLSkLWRzEdRNSs47JxDFqrJG+d/OIH/8SyW+DoHFfzB6PdUSlzsx LW5/9fl/4W6U3k/wp6c/82PrDLiEPrST+sQcgX0VffsDZJEpZtpz/80NIrR9+pPcXlBJ caAyK2F0UvItWGPvGWJ1f1NoR9jYrMgJUXZJCYlQMn6QJUFAUkZMGtsuqpg4miIJAz4T HQ0hKzqbzeoa+nDuOZRT3Huhi6hUNXfGfjR6AxcoZdWiZEmoxsRH5Xh496Iwrmm/tVuz iuAZ65nVJJL+PXNZbes+HLoQ72GScTVuuBD6H0qlxCc3ERtnBDyhWB4EDka0DsVaZk0J LWLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=VjG0kRzlBwR8YGaPDB6PFGDHZQlihIoW/wLAoir9C6s=; b=hXsPckbrcjNnGKjv6pnViBBSoDrHEv6fs4JaQG6d/dnAkAWgkk1F+OycBo+xhzgG83 3rqYw0PtWAbGTwRadJ5M9KtcHmnsKQbLZxT4aXCPGmyEDx363SNFNntgzefYMJ0oV0D4 uRFX1lwBKhbhzXgiNwDbod0MrFvxV+Bo13upMnjFHdlKOpUvemqp1HrAuBmc1UociS9Q Cl5839CwWoCMGQSzuk1SXv0sPFqrFkJ/lZhQVaSOMCNAT6Hk5JMO55zI8LsKkGCUIy9n szkzG9xYfX0+2IpP77pxD3aMhwfczrXLtTcwzntHd31bDRVVqaD2/dAPfqt6Uk/LqDhE /N3Q== X-Gm-Message-State: APjAAAVT/qwSeMITDIb0bhs+oW6Qund685cX6/aD91bOIh6riZ8c0bLh CX3SSo6Qoblp5lMo1IKC5jwTtA== X-Google-Smtp-Source: APXvYqyMJyha9n/BqjuPYtl9H8kEeJRJ9AyYsISWHIBvbXB70exu3BpBG5W5C4NdH5XRKXYgEN9FSA== X-Received: by 2002:a1c:7306:: with SMTP id d6mr8175785wmb.40.1552825380974; Sun, 17 Mar 2019 05:23:00 -0700 (PDT) Received: from snf-864.vm.snf.arr ([31.177.62.212]) by smtp.gmail.com with ESMTPSA id z10sm5453292wrs.11.2019.03.17.05.22.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 17 Mar 2019 05:23:00 -0700 (PDT) From: Nikos Tsironis To: snitzer@redhat.com, agk@redhat.com, dm-devel@redhat.com Date: Sun, 17 Mar 2019 14:22:52 +0200 Message-Id: <20190317122258.21760-1-ntsironis@arrikto.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.43]); Sun, 17 Mar 2019 12:23:02 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.43]); Sun, 17 Mar 2019 12:23:02 +0000 (UTC) for IP:'209.85.128.65' DOMAIN:'mail-wm1-f65.google.com' HELO:'mail-wm1-f65.google.com' FROM:'ntsironis@arrikto.com' RCPT:'' X-RedHat-Spam-Score: -0.012 (DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_NONE, SPF_PASS) 209.85.128.65 mail-wm1-f65.google.com 209.85.128.65 mail-wm1-f65.google.com X-Scanned-By: MIMEDefang 2.84 on 10.5.110.43 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: dm-devel@redhat.com Cc: hch@infradead.org, paulmck@linux.ibm.com, mpatocka@redhat.com, linux-kernel@vger.kernel.org, iliastsi@arrikto.com Subject: [dm-devel] [PATCH v3 0/6] dm snapshot: Improve performance using a more fine-grained locking scheme X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.41]); Sun, 17 Mar 2019 12:23:09 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP dm-snapshot uses a single mutex to serialize every access to the snapshot state, including accesses to the exception hash tables. This mutex is a bottleneck preventing dm-snapshot to scale as the number of threads doing IO increases. The major contention points are __origin_write()/snapshot_map() and pending_complete(), i.e., the submission and completion of pending exceptions. This patchset substitutes the single mutex with: * A read-write semaphore, which protects the mostly read fields of the snapshot structure. * Per-bucket bit spinlocks, that protect accesses to the exception hash tables. fio benchmarks using the null_blk device show significant performance improvements as the number of worker processes increases. Write latency is almost halved and write IOPS are nearly doubled. The relevant patch provides detailed benchmark results. A summary of the patchset follows: 1. The first patch removes an unnecessary use of WRITE_ONCE() in hlist_add_behind(). 2. The second patch adds two helper functions to linux/list_bl.h, which is used to implement the per-bucket bit spinlocks in dm-snapshot. 3. The third patch removes the need to sleep holding the snapshot lock in pending_complete(), thus allowing us to replace the mutex with the per-bucket bit spinlocks. 4. Patches 4, 5 and 6 change the locking scheme, as described previously. Changes in v3: - Don't use WRITE_ONCE() in hlist_bl_add_behind(), as it's not needed. - Fix hlist_add_behind() to also not use WRITE_ONCE(). - Use uintptr_t instead of unsigned long in hlist_bl_add_before(). v2: https://www.redhat.com/archives/dm-devel/2019-March/msg00007.html Changes in v2: - Split third patch of v1 into three patches: 3/5, 4/5, 5/5. v1: https://www.redhat.com/archives/dm-devel/2018-December/msg00161.html Nikos Tsironis (6): list: Don't use WRITE_ONCE() in hlist_add_behind() list_bl: Add hlist_bl_add_before/behind helpers dm snapshot: Don't sleep holding the snapshot lock dm snapshot: Replace mutex with rw semaphore dm snapshot: Make exception tables scalable dm snapshot: Use fine-grained locking scheme drivers/md/dm-exception-store.h | 3 +- drivers/md/dm-snap.c | 359 +++++++++++++++++++++++++++------------- include/linux/list.h | 2 +- include/linux/list_bl.h | 26 +++ 4 files changed, 269 insertions(+), 121 deletions(-) Acked-by: Mikulas Patocka