From patchwork Sun Jun 17 02:00:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 10468317 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C928F600CC for ; Sun, 17 Jun 2018 02:04:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B93F628CC9 for ; Sun, 17 Jun 2018 02:04:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ABC2728CCE; Sun, 17 Jun 2018 02:04:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E59C328CCD for ; Sun, 17 Jun 2018 02:04:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0017B6B0291; Sat, 16 Jun 2018 22:01:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E9B176B0292; Sat, 16 Jun 2018 22:01:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0D716B0294; Sat, 16 Jun 2018 22:01:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f70.google.com (mail-pl0-f70.google.com [209.85.160.70]) by kanga.kvack.org (Postfix) with ESMTP id 6DDC36B0291 for ; Sat, 16 Jun 2018 22:01:35 -0400 (EDT) Received: by mail-pl0-f70.google.com with SMTP id w6-v6so7701322plp.14 for ; Sat, 16 Jun 2018 19:01:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=NR83KzpqxOCMo/mNGinQ7Yp0eJl39Zsg1RoPo491hDg=; b=gnj/muA/3kst9NCSxuIDP8GHN1+Xmh7lDwwddY2U6Eb/W9U+/jib1t7nE/E7QcvtCb DqMEGvmTnFtSM6MyD8DBQ4DpPdy4nqOKeJxt8xCkJ8nDkbHx1Dp3JfOOSgxgiiidaZon LGFBvh4pI8gc5B+bVtABPlX56JrgqxaYnXmIRjAXl1HdrR6h2TQM73Y0SpSHuigy9D7a AyBo+IeM2NrenZR5B5S15VTrGFQDAUF4dDOIGN74H6TuIPzp/mN04n0HM+dGDmKpGMih J283/Ughl822B7O+1pyi49Ng+r5SjuH5nO0zyL0FQ2pv7kNmeLahb33p9DAbPTYjNyRM YdnQ== X-Gm-Message-State: APt69E2L3lT6ALqLELz7rd2ssQ+f5xH7/K2tCffanGtxX4/4TA5X59Pj n2kN0QntdwbciJJMB3pewYQyR5zbxo+umJ1EbsF2XPzT5TGZDksTTLCvFv0GMatix0B2Sc8jTsQ /unmw5frverYPDe2+47MnfCCdyevGs8eycrpi73D52AFKlZJbBsSMiq2/w76IQyfmzg== X-Received: by 2002:a63:ba56:: with SMTP id l22-v6mr6619345pgu.161.1529200895106; Sat, 16 Jun 2018 19:01:35 -0700 (PDT) X-Google-Smtp-Source: ADUXVKL75gV+1WV5ZlxlBqC7AOxiQBJJ3P9lMPxdR9EkVCVN4Uq9wlgFd1Yy/AsRVfkh9NNEzr3v X-Received: by 2002:a63:ba56:: with SMTP id l22-v6mr6619299pgu.161.1529200893928; Sat, 16 Jun 2018 19:01:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529200893; cv=none; d=google.com; s=arc-20160816; b=Dxxd1T3lqYpuCxOttY1lHwJqaT3EUIbZoG3KQdoJaTb74Xy10R/mplRo3sy7X4RmPH b0/ZyvVVs0LHDL0Wut31BeyY6eBMXsX65lMcOid3F4iXsWarjzwwNMfkQiCA8wFr3fsL XMla4SCg2pP9PR0DUAQf6wyfTIj6yFQOe8OrZgkMBO7Uy2qqHb2qzhg/iZhYwEU2SFBA GTGC/IK3ffKKK9KxqRgJMLe5uJ5opHrWHNCU4Ct5R+P8fUUKwsQCgxEsaZcZUnXWR7SR 7yoPnehT7R3Byqejw/FRfHzs1wMiEDZVLaPkc7VE0UMy2sZD1/sfwhwIUBJba1VXbQaD S55g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=NR83KzpqxOCMo/mNGinQ7Yp0eJl39Zsg1RoPo491hDg=; b=fp3GQWWLrAREIRCGk2j31zz6Dn9e+dCVLV2lONPd79SNh4T52qNc9zMqU8x6vVsc5k qKQgCA1YCS7ASSlwMi13762Aqi8XcvUKaZuntMnoVvO71uFlkkrktJ+VJen0qh9shqn1 I+oCNy1OQZXQOxms+1pahXM+XufCOlbBrR2ERY+v0MJZQ6jtW984d1ccFW+3Bo81LoNj LMGE9JpxFGZg32XUGknN47+RB9cds4wDTRKwKU3+j6SdbVPdvAyFamHEDtNYyZrYX8E3 DkVZ0uGx2MaQHRn1al0hswNnVtUfBv/+qwE7/tbTaF7VULDXl/J8i6CJyfdVDPnT8ruW KxxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=itzEUVdS; spf=pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=willy@infradead.org Received: from bombadil.infradead.org (bombadil.infradead.org. [2607:7c80:54:e::133]) by mx.google.com with ESMTPS id q69-v6si11435347pfj.51.2018.06.16.19.01.33 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 16 Jun 2018 19:01:33 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) client-ip=2607:7c80:54:e::133; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=itzEUVdS; spf=pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=willy@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=NR83KzpqxOCMo/mNGinQ7Yp0eJl39Zsg1RoPo491hDg=; b=itzEUVdSx+m4jyLwMg1g/keER nDNphqo4osY+ueyPND3c9xL8mW6s3u7W3juedLrbUhj/J7hBVFNHM/kH0JiKwfXdPzeBRDMNBjgvC t/jr9Y3bMbEQQChEn4BOj+VHI2TR8TCPwutUu7C3wnyrMaj/4sNQmd2UtOX0bmDKPHO0SEsJVUw0j uNJ9r/UJme9kUnUOGikwUBOOP/ixJl3LSyUloqOdIPVEvivGZVm/WZzh9YhJGhQ4uEKH+jbhHukCY AocqCUSOZqVBP1nJ2+IASF3/lM2IXmvruDgXm5+uafip6QHZ04MMgOcC62LqhARSW+Gyemj3IdzFJ qcXHd9FhQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.90_1 #2 (Red Hat Linux)) id 1fUN0W-0001dO-9C; Sun, 17 Jun 2018 02:01:32 +0000 From: Matthew Wilcox To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Matthew Wilcox , Jan Kara , Jeff Layton , Lukas Czerner , Ross Zwisler , Christoph Hellwig , Goldwyn Rodrigues , Nicholas Piggin , Ryusuke Konishi , linux-nilfs@vger.kernel.org, Jaegeuk Kim , Chao Yu , linux-f2fs-devel@lists.sourceforge.net Subject: [PATCH v14 64/74] dax: Convert dax_insert_pfn_mkwrite to XArray Date: Sat, 16 Jun 2018 19:00:42 -0700 Message-Id: <20180617020052.4759-65-willy@infradead.org> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180617020052.4759-1-willy@infradead.org> References: <20180617020052.4759-1-willy@infradead.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add some XArray-based helper functions to replace the radix tree based metaphors currently in use. The biggest change is that converted code doesn't see its own lock bit; get_unlocked_entry() always returns an entry with the lock bit clear, and locking the entry now returns void. So we don't have to mess around loading the current entry and clearing the lock bit; we can just store the entry that we were using. Signed-off-by: Matthew Wilcox --- fs/dax.c | 145 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 113 insertions(+), 32 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index b7f54e386da8..71181f4bb1d3 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -38,6 +38,17 @@ #define CREATE_TRACE_POINTS #include +static inline unsigned int pe_order(enum page_entry_size pe_size) +{ + if (pe_size == PE_SIZE_PTE) + return PAGE_SHIFT - PAGE_SHIFT; + if (pe_size == PE_SIZE_PMD) + return PMD_SHIFT - PAGE_SHIFT; + if (pe_size == PE_SIZE_PUD) + return PUD_SHIFT - PAGE_SHIFT; + return ~0; +} + /* We choose 4096 entries - same as per-zone page wait tables */ #define DAX_WAIT_TABLE_BITS 12 #define DAX_WAIT_TABLE_ENTRIES (1 << DAX_WAIT_TABLE_BITS) @@ -46,6 +57,9 @@ #define PG_PMD_COLOUR ((PMD_SIZE >> PAGE_SHIFT) - 1) #define PG_PMD_NR (PMD_SIZE >> PAGE_SHIFT) +/* The order of a PMD entry */ +#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) + static wait_queue_head_t wait_table[DAX_WAIT_TABLE_ENTRIES]; static int __init init_dax_wait_table(void) @@ -85,10 +99,15 @@ static void *dax_make_locked(unsigned long pfn, unsigned long flags) DAX_LOCKED); } +static bool dax_is_locked(void *entry) +{ + return xa_to_value(entry) & DAX_LOCKED; +} + static unsigned int dax_entry_order(void *entry) { if (xa_to_value(entry) & DAX_PMD) - return PMD_SHIFT - PAGE_SHIFT; + return PMD_ORDER; return 0; } @@ -181,6 +200,77 @@ static void dax_wake_mapping_entry_waiter(struct xarray *xa, __wake_up(wq, TASK_NORMAL, wake_all ? 0 : 1, &key); } +static void dax_wake_entry(struct xa_state *xas, void *entry, bool wake_all) +{ + return dax_wake_mapping_entry_waiter(xas->xa, xas->xa_index, entry, + wake_all); +} + +/* + * Look up entry in page cache, wait for it to become unlocked if it + * is a DAX entry and return it. The caller must subsequently call + * put_unlocked_entry() if it did not lock the entry or put_locked_entry() + * if it did. + * + * Must be called with the i_pages lock held. + */ +static void *get_unlocked_entry(struct xa_state *xas) +{ + void *entry; + struct wait_exceptional_entry_queue ewait; + wait_queue_head_t *wq; + + init_wait(&ewait.wait); + ewait.wait.func = wake_exceptional_entry_func; + + for (;;) { + entry = xas_load(xas); + if (!entry || WARN_ON_ONCE(!xa_is_value(entry)) || + !dax_is_locked(entry)) + return entry; + + wq = dax_entry_waitqueue(xas->xa, xas->xa_index, entry, + &ewait.key); + prepare_to_wait_exclusive(wq, &ewait.wait, + TASK_UNINTERRUPTIBLE); + xas_unlock_irq(xas); + xas_reset(xas); + schedule(); + finish_wait(wq, &ewait.wait); + xas_lock_irq(xas); + } +} + +static void put_unlocked_entry(struct xa_state *xas, void *entry) +{ + /* If we were the only waiter woken, wake the next one */ + if (entry) + dax_wake_entry(xas, entry, false); +} + +/* + * We used the xa_state to get the entry, but then we locked the entry and + * dropped the xa_lock, so we know the xa_state is stale and must be reset + * before use. + */ +static void put_locked_entry(struct xa_state *xas, void *entry) +{ + void *old; + + xas_reset(xas); + xas_lock_irq(xas); + old = xas_store(xas, entry); + xas_unlock_irq(xas); + BUG_ON(!dax_is_locked(old)); + dax_wake_entry(xas, entry, false); +} + +static void dax_lock_entry(struct xa_state *xas, void *entry) +{ + unsigned long v = xa_to_value(entry); + xas_store(xas, xa_mk_value(v | DAX_LOCKED)); +} + /* * Check whether the given slot is locked. Must be called with the i_pages * lock held. @@ -1705,50 +1795,46 @@ EXPORT_SYMBOL_GPL(dax_iomap_fault); /* * dax_insert_pfn_mkwrite - insert PTE or PMD entry into page tables * @vmf: The description of the fault - * @pe_size: Size of entry to be inserted * @pfn: PFN to insert + * @order: Order of entry to insert. * * This function inserts a writeable PTE or PMD entry into the page tables * for an mmaped DAX file. It also marks the page cache entry as dirty. */ -static vm_fault_t dax_insert_pfn_mkwrite(struct vm_fault *vmf, - enum page_entry_size pe_size, - pfn_t pfn) +static vm_fault_t +dax_insert_pfn_mkwrite(struct vm_fault *vmf, pfn_t pfn, unsigned int order) { struct address_space *mapping = vmf->vma->vm_file->f_mapping; - void *entry, **slot; - pgoff_t index = vmf->pgoff; + XA_STATE_ORDER(xas, &mapping->i_pages, vmf->pgoff, order); + void *entry; vm_fault_t ret; - xa_lock_irq(&mapping->i_pages); - entry = get_unlocked_mapping_entry(mapping, index, &slot); + xas_lock_irq(&xas); + entry = get_unlocked_entry(&xas); /* Did we race with someone splitting entry or so? */ if (!entry || - (pe_size == PE_SIZE_PTE && !dax_is_pte_entry(entry)) || - (pe_size == PE_SIZE_PMD && !dax_is_pmd_entry(entry))) { - put_unlocked_mapping_entry(mapping, index, entry); - xa_unlock_irq(&mapping->i_pages); + (order == 0 && !dax_is_pte_entry(entry)) || + (order == PMD_ORDER && (xa_is_internal(entry) || + !dax_is_pmd_entry(entry)))) { + put_unlocked_entry(&xas, entry); + xas_unlock_irq(&xas); trace_dax_insert_pfn_mkwrite_no_entry(mapping->host, vmf, VM_FAULT_NOPAGE); return VM_FAULT_NOPAGE; } - radix_tree_tag_set(&mapping->i_pages, index, PAGECACHE_TAG_DIRTY); - entry = lock_slot(mapping, slot); - xa_unlock_irq(&mapping->i_pages); - switch (pe_size) { - case PE_SIZE_PTE: + xas_set_tag(&xas, PAGECACHE_TAG_DIRTY); + dax_lock_entry(&xas, entry); + xas_unlock_irq(&xas); + if (order == 0) ret = vmf_insert_mixed_mkwrite(vmf->vma, vmf->address, pfn); - break; #ifdef CONFIG_FS_DAX_PMD - case PE_SIZE_PMD: + else if (order == PMD_ORDER) ret = vmf_insert_pfn_pmd(vmf->vma, vmf->address, vmf->pmd, pfn, true); - break; #endif - default: + else ret = VM_FAULT_FALLBACK; - } - put_locked_mapping_entry(mapping, index); + put_locked_entry(&xas, entry); trace_dax_insert_pfn_mkwrite(mapping->host, vmf, ret); return ret; } @@ -1768,17 +1854,12 @@ vm_fault_t dax_finish_sync_fault(struct vm_fault *vmf, { int err; loff_t start = ((loff_t)vmf->pgoff) << PAGE_SHIFT; - size_t len = 0; + unsigned int order = pe_order(pe_size); + size_t len = PAGE_SIZE << order; - if (pe_size == PE_SIZE_PTE) - len = PAGE_SIZE; - else if (pe_size == PE_SIZE_PMD) - len = PMD_SIZE; - else - WARN_ON_ONCE(1); err = vfs_fsync_range(vmf->vma->vm_file, start, start + len - 1, 1); if (err) return VM_FAULT_SIGBUS; - return dax_insert_pfn_mkwrite(vmf, pe_size, pfn); + return dax_insert_pfn_mkwrite(vmf, pfn, order); } EXPORT_SYMBOL_GPL(dax_finish_sync_fault);