From patchwork Fri Oct 28 15:15:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13023825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 818F5ECAAA1 for ; Fri, 28 Oct 2022 15:15:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDF6E6B0085; Fri, 28 Oct 2022 11:15:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D41E28E0001; Fri, 28 Oct 2022 11:15:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBB356B0088; Fri, 28 Oct 2022 11:15:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A6A456B0085 for ; Fri, 28 Oct 2022 11:15:29 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 56E331408B6 for ; Fri, 28 Oct 2022 15:15:29 +0000 (UTC) X-FDA: 80070707178.03.F291F86 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id 10E45140013 for ; Fri, 28 Oct 2022 15:15:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=iHo0dXYsbgANrVfcUqE3Pj13Q6JiA9ERSsXw+Ir/P94=; b=ZIPz155/TgaAnxkWjnlin75+ms s6pJfHadSYXZsRaq7yCZ0nYG0ggNKifdPTHJ2NlZ6wZsi5VrMhxgPGnKpzPNt1/ysrIzlzExK4WqX wr9Al8lR99lPNfj9M+ZRalOzc9xo4nsQWx5xN4JO0yRTsUp93l8LDFyw/furcAUW1MjpLmRr6iFnL Oh4v9O9CZ6owIvp6DmMjHUsMjiCR5StDVSiQTPRCaMikCJoaPuKazc7cGECOPouxOH3/iJe3i6SIX P64c38VIgCEUUwm/AlHdM4CerYb4dF1LpELIjyEhEPO4kx2UOjRlSbIj6vs65jp2DLH/Vg1Q7DIuX Gnjp6n8Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1ooR4x-001LAK-PP; Fri, 28 Oct 2022 15:15:27 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , Uladzislau Rezki , David Howells , Dave Chinner , linux-fsdevel@vger.kernel.org, Thomas Gleixner , Ira Weiny , "Fabio M. De Francesco" , Luis Chamberlain Subject: [PATCH 1/1] mm: Add folio_map_local() Date: Fri, 28 Oct 2022 16:15:26 +0100 Message-Id: <20221028151526.319681-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221028151526.319681-1-willy@infradead.org> References: <20221028151526.319681-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666970128; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iHo0dXYsbgANrVfcUqE3Pj13Q6JiA9ERSsXw+Ir/P94=; b=z/4nH9LTnV7fJsuAyzu+b7KKuD2nhnTyG92yOORPx/P70bjC+OuBA9aiahlDtbC75zPwAF S5X5rAwONp6rxhdRdmKR5KKzc8jCB0Uei/YE5E5rnWws2YPgKsByODy+xeq8M/CITbCZWd UQgSg6ezqF6IPKlotMEniUDs4E1vzRI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="ZIPz155/"; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666970128; a=rsa-sha256; cv=none; b=t4kUZk4c+tmrj8q5gGu6NWnfK7F1qnDBEenGIlVGedjB0O4jgQla7U39k1Cpka41P6ibnL KpHFI1CWjxej8voQIdpLVe41582jcGlZKolEdZHey4cSky541DvPnXl2odhRCxpQmh1NgS twpUQWNltOwUA49bqyjRiD342CHUqt0= X-Stat-Signature: 8i1ozbu5g8boqa9omdjidr8ot6d5f14g X-Rspamd-Queue-Id: 10E45140013 X-Rspam-User: Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="ZIPz155/"; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam04 X-HE-Tag: 1666970126-866996 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some filesystems benefit from being able to map the entire folio. On 32-bit platforms with HIGHMEM, we fall back to using vmap, which will be slow. If it proves to be a performance problem, we can look at optimising it in a number of ways. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/highmem.h | 40 ++++++++++++++++++++++++++++++++- include/linux/vmalloc.h | 6 +++-- mm/vmalloc.c | 50 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 93 insertions(+), 3 deletions(-) diff --git a/include/linux/highmem.h b/include/linux/highmem.h index e9912da5441b..e8159243d88d 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -10,6 +10,7 @@ #include #include #include +#include #include "highmem-internal.h" @@ -132,6 +133,44 @@ static inline void *kmap_local_page(struct page *page); */ static inline void *kmap_local_folio(struct folio *folio, size_t offset); +/** + * folio_map_local - Map an entire folio. + * @folio: The folio to map. + * + * Unlike kmap_local_folio(), map an entire folio. This should be undone + * with folio_unmap_local(). The address returned should be treated as + * stack-based, and local to this CPU, like kmap_local_folio(). + * + * Context: May allocate memory using GFP_KERNEL if it takes the vmap path. + * Return: A kernel virtual address which can be used to access the folio, + * or NULL if the mapping fails. + */ +static inline __must_check void *folio_map_local(struct folio *folio) +{ + might_alloc(GFP_KERNEL); + + if (!IS_ENABLED(CONFIG_HIGHMEM)) + return folio_address(folio); + if (folio_test_large(folio)) + return vm_map_folio(folio); + return kmap_local_page(&folio->page); +} + +/** + * folio_unmap_local - Unmap an entire folio. + * @addr: Address returned from folio_map_local() + * + * Undo the result of a previous call to folio_map_local(). + */ +static inline void folio_unmap_local(const void *addr) +{ + if (!IS_ENABLED(CONFIG_HIGHMEM)) + return; + if (is_vmalloc_addr(addr)) + vunmap(addr); + kunmap_local(addr); +} + /** * kmap_atomic - Atomically map a page for temporary usage - Deprecated! * @page: Pointer to the page to be mapped @@ -426,5 +465,4 @@ static inline void folio_zero_range(struct folio *folio, { zero_user_segments(&folio->page, start, start + length, 0, 0); } - #endif /* _LINUX_HIGHMEM_H */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 096d48aa3437..4bb34c939c01 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -13,6 +13,7 @@ #include struct vm_area_struct; /* vma defining user mapping in mm_types.h */ +struct folio; /* also mm_types.h */ struct notifier_block; /* in notifier.h */ /* bits in flags of vmalloc's vm_struct below */ @@ -163,8 +164,9 @@ extern void *vcalloc(size_t n, size_t size) __alloc_size(1, 2); extern void vfree(const void *addr); extern void vfree_atomic(const void *addr); -extern void *vmap(struct page **pages, unsigned int count, - unsigned long flags, pgprot_t prot); +void *vmap(struct page **pages, unsigned int count, unsigned long flags, + pgprot_t prot); +void *vm_map_folio(struct folio *folio); void *vmap_pfn(unsigned long *pfns, unsigned int count, pgprot_t prot); extern void vunmap(const void *addr); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ccaa461998f3..265b860c9550 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2283,6 +2283,56 @@ void *vm_map_ram(struct page **pages, unsigned int count, int node) } EXPORT_SYMBOL(vm_map_ram); +#ifdef CONFIG_HIGHMEM +/** + * vm_map_folio() - Map an entire folio into virtually contiguous space. + * @folio: The folio to map. + * + * Maps all pages in @folio into contiguous kernel virtual space. This + * function is only available in HIGHMEM builds; for !HIGHMEM, use + * folio_address(). The pages are mapped with PAGE_KERNEL permissions. + * + * Return: The address of the area or %NULL on failure + */ +void *vm_map_folio(struct folio *folio) +{ + size_t size = folio_size(folio); + unsigned long addr; + void *mem; + + might_sleep(); + + if (likely(folio_nr_pages(folio) <= VMAP_MAX_ALLOC)) { + mem = vb_alloc(size, GFP_KERNEL); + if (IS_ERR(mem)) + return NULL; + addr = (unsigned long)mem; + } else { + struct vmap_area *va; + va = alloc_vmap_area(size, PAGE_SIZE, VMALLOC_START, + VMALLOC_END, NUMA_NO_NODE, GFP_KERNEL); + if (IS_ERR(va)) + return NULL; + + addr = va->va_start; + mem = (void *)addr; + } + + if (vmap_range_noflush(addr, addr + size, + folio_pfn(folio) << PAGE_SHIFT, + PAGE_KERNEL, folio_shift(folio))) { + vm_unmap_ram(mem, folio_nr_pages(folio)); + return NULL; + } + flush_cache_vmap(addr, addr + size); + + mem = kasan_unpoison_vmalloc(mem, size, KASAN_VMALLOC_PROT_NORMAL); + + return mem; +} +EXPORT_SYMBOL(vm_map_folio); +#endif + static struct vm_struct *vmlist __initdata; static inline unsigned int vm_area_page_order(struct vm_struct *vm)