From patchwork Sun Jun 17 02:00:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 10468283 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2C9F9600CC for ; Sun, 17 Jun 2018 02:03:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A62E28C79 for ; Sun, 17 Jun 2018 02:03:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0A45728C75; Sun, 17 Jun 2018 02:03:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 84DD028C71 for ; Sun, 17 Jun 2018 02:03:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0F0B6B0284; Sat, 16 Jun 2018 22:01:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 997A36B0286; Sat, 16 Jun 2018 22:01:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 837156B0287; Sat, 16 Jun 2018 22:01:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f69.google.com (mail-pl0-f69.google.com [209.85.160.69]) by kanga.kvack.org (Postfix) with ESMTP id 335846B0284 for ; Sat, 16 Jun 2018 22:01:16 -0400 (EDT) Received: by mail-pl0-f69.google.com with SMTP id t19-v6so7721494plo.9 for ; Sat, 16 Jun 2018 19:01:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=taP8BwheHg3CEkDXuvl7b4R9WQ7b3YzBrhJkOgu4qeA=; b=qkwuzoPSbPKm+Gu5/HaqkHThSQMtCn9KbPe9WgLwgX7ZXKg2SKVUALLxc0XVLDGpQT MfqTRDpN2FfPkhma93zn1K5O70g9tE3gtCT6JtBPUSw2ifvx6hsYCjxqHvrcUuK7r/PQ lwOkSHX5JcZyFrvqcqnLRpmlTSX+5T3NkodEW+s2gXDV6lU4u1RVu392BgM1SOBHOgin 3YDZGjLh7mZDav4ZmFJwG5R5dvGGjeumBcQeUS2Cz2im8ICYwdtPE7zIkw+CQjwmTJyI fN5haaksZljDt4Bf2pFwIVIwIKIrTcwfHAm7kAzJVmYEFy43u7kDREcQd6hRsJGwzkll m3+A== X-Gm-Message-State: APt69E3dal4V5Myltcrf7GFa7ExnlPcXVDQqEb/EqN1jPtk7vB9l+E9x xIQS4tVDLtutnfNlID5AFzcpiTiauXVPVM0v/84LJ+vTc8/b61KLEJ7HYMeNpT1UWhA7WW3eMaE JE/n32c2Aty02rSlrR+ZLQ2Rt0JFs58NmNYfooTziD7r2m5Rq7kGyi9FtFyDkSEy1zA== X-Received: by 2002:a62:fd0b:: with SMTP id p11-v6mr8132174pfh.52.1529200875882; Sat, 16 Jun 2018 19:01:15 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJZE0MYdLnPFgMySVZlkDfGXGVtoMCiqrbFn/jkqDPTFGgIQoOQ2gRIEtuZQTVeBixSoY2j X-Received: by 2002:a62:fd0b:: with SMTP id p11-v6mr8132134pfh.52.1529200874946; Sat, 16 Jun 2018 19:01:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529200874; cv=none; d=google.com; s=arc-20160816; b=sYXd1ZY1Rp3Kzpy2L5UXJ5KyUGAS+wn/anX7cIxJyu5ZW5D+htYtj9HY8DRbGWZsAB /scILkqm0tdXMcPAY3ozIfVr1YfhonTTfrCYGM76G6ejAnMvRIM9A9/vSYOnT4w/j2zv Mtbx4Fy6CsCVoIar9XdHBx+3vNOQh1MWVWNznWicVhcVESbVPIFY2DHzsB6dSbxous9Y 8FtDeVAlkGBMCg3YTH538WsBNJoiB7YWVw0GmHWybikFRBXQE6eYCo6haW+aONDpMXqq /RXQCZmg1LgDU/Eb5lzNHW3vZozG7DT+CZ5rqu7RqjSAr0/AV346+ylNIpTyaXIw75Gs PBuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=taP8BwheHg3CEkDXuvl7b4R9WQ7b3YzBrhJkOgu4qeA=; b=Dv8mJjclqoibDd0z6b/98paVkExlalzZvz2/320MHa9OUYlCH1Ng6u11zHxG4Rksd4 2awD9GoB/h5R+W4S+ufK7M2ookrp87+41iObDXqyZ8x+rKLUGGi9hwVefS36vM4/dlZc pugUPiJpPnpIlvzTMzY11KI+qdZ/2dK8W1HOiaKH8Zt4fDJsK++HpP8UOK+PeUcTmI0N Ch9wL3a7X+14s14PmybHJXqhIXwRf14hUhBxxsrVgyC64Xh8RV7yl/G6rd3u5lNO8WIj HQgTYN1v0jww/8EmPE72giyfjmeuu1YF2bcPuvOQqaD84FPViIrK5s2Cio2T8odoQZIB FO2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=mnOUCgq4; spf=pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=willy@infradead.org Received: from bombadil.infradead.org (bombadil.infradead.org. [2607:7c80:54:e::133]) by mx.google.com with ESMTPS id j20-v6si11853000pll.211.2018.06.16.19.01.14 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 16 Jun 2018 19:01:14 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) client-ip=2607:7c80:54:e::133; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=mnOUCgq4; spf=pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=willy@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=taP8BwheHg3CEkDXuvl7b4R9WQ7b3YzBrhJkOgu4qeA=; b=mnOUCgq42hLXgcqQvdSJolBa4 M2pHHerVUVQEAix/TXoPd/ZLNBXq46cwsAlwSXV34gaKqxdjMLDbIBuoLXtyO82BlJoMu8EfMh9ux pU3APwkzHc+vWtyGCLTLQ0cASLWRek2CsfN52jLPIlRjl9TV8tFIG/BNt6M9s9LZtM+AhKQ7EzJAp Y0JCImZZCPzVXB8OH2ekbiLa4xE5a2zPEzVoiptNgLCITKe5yxa70ZBRvxCZSbxxWiBqH4a2eOYWC +vtRoNDpb/8Lt7UHE2Cbqt6i3VnUNVTgVl8WLCtiLvyowoAKYnhgT5Dk3nnSM+cKTl0E7N2qZ6y0Q a/tTBK1sw==; Received: from willy by bombadil.infradead.org with local (Exim 4.90_1 #2 (Red Hat Linux)) id 1fUN0C-0001S2-0j; Sun, 17 Jun 2018 02:01:12 +0000 From: Matthew Wilcox To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Matthew Wilcox , Jan Kara , Jeff Layton , Lukas Czerner , Ross Zwisler , Christoph Hellwig , Goldwyn Rodrigues , Nicholas Piggin , Ryusuke Konishi , linux-nilfs@vger.kernel.org, Jaegeuk Kim , Chao Yu , linux-f2fs-devel@lists.sourceforge.net Subject: [PATCH v14 54/74] memfd: Convert memfd_wait_for_pins to XArray Date: Sat, 16 Jun 2018 19:00:32 -0700 Message-Id: <20180617020052.4759-55-willy@infradead.org> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180617020052.4759-1-willy@infradead.org> References: <20180617020052.4759-1-willy@infradead.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Simplify the locking by taking the spinlock while we walk the tree on the assumption that many acquires and releases of the lock will be worse than holding the lock while we process an entire batch of pages. Signed-off-by: Matthew Wilcox Reviewed-by: Mike Kravetz --- mm/memfd.c | 61 ++++++++++++++++++++++-------------------------------- 1 file changed, 25 insertions(+), 36 deletions(-) diff --git a/mm/memfd.c b/mm/memfd.c index 27069518e3c5..e7d6be725b7a 100644 --- a/mm/memfd.c +++ b/mm/memfd.c @@ -21,7 +21,7 @@ #include /* - * We need a tag: a new tag would expand every radix_tree_node by 8 bytes, + * We need a tag: a new tag would expand every xa_node by 8 bytes, * so reuse a tag which we firmly believe is never set or cleared on tmpfs * or hugetlbfs because they are memory only filesystems. */ @@ -72,9 +72,7 @@ static void memfd_tag_pins(struct address_space *mapping) */ static int memfd_wait_for_pins(struct address_space *mapping) { - struct radix_tree_iter iter; - void __rcu **slot; - pgoff_t start; + XA_STATE(xas, &mapping->i_pages, 0); struct page *page; int error, scan; @@ -82,7 +80,9 @@ static int memfd_wait_for_pins(struct address_space *mapping) error = 0; for (scan = 0; scan <= LAST_SCAN; scan++) { - if (!radix_tree_tagged(&mapping->i_pages, MEMFD_TAG_PINNED)) + unsigned int tagged = 0; + + if (!xas_tagged(&xas, MEMFD_TAG_PINNED)) break; if (!scan) @@ -90,45 +90,34 @@ static int memfd_wait_for_pins(struct address_space *mapping) else if (schedule_timeout_killable((HZ << scan) / 200)) scan = LAST_SCAN; - start = 0; - rcu_read_lock(); - radix_tree_for_each_tagged(slot, &mapping->i_pages, &iter, - start, MEMFD_TAG_PINNED) { - - page = radix_tree_deref_slot(slot); - if (radix_tree_exception(page)) { - if (radix_tree_deref_retry(page)) { - slot = radix_tree_iter_retry(&iter); - continue; - } - - page = NULL; - } - - if (page && - page_count(page) - page_mapcount(page) != 1) { - if (scan < LAST_SCAN) - goto continue_resched; - + xas_set(&xas, 0); + xas_lock_irq(&xas); + xas_for_each_tagged(&xas, page, ULONG_MAX, MEMFD_TAG_PINNED) { + bool clear = true; + if (xa_is_value(page)) + continue; + if (page_count(page) - page_mapcount(page) != 1) { /* * On the last scan, we clean up all those tags * we inserted; but make a note that we still * found pages pinned. */ - error = -EBUSY; + if (scan == LAST_SCAN) + error = -EBUSY; + else + clear = false; } + if (clear) + xas_clear_tag(&xas, MEMFD_TAG_PINNED); + if (++tagged % XA_CHECK_SCHED) + continue; - xa_lock_irq(&mapping->i_pages); - radix_tree_tag_clear(&mapping->i_pages, - iter.index, MEMFD_TAG_PINNED); - xa_unlock_irq(&mapping->i_pages); -continue_resched: - if (need_resched()) { - slot = radix_tree_iter_resume(slot, &iter); - cond_resched_rcu(); - } + xas_pause(&xas); + xas_unlock_irq(&xas); + cond_resched(); + xas_lock_irq(&xas); } - rcu_read_unlock(); + xas_unlock_irq(&xas); } return error;