From patchwork Wed Apr 10 10:19:52 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chanho Park X-Patchwork-Id: 2420291 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) by patchwork1.kernel.org (Postfix) with ESMTP id 587523FD8C for ; Wed, 10 Apr 2013 10:21:48 +0000 (UTC) Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1UPs9y-0001pz-LQ; Wed, 10 Apr 2013 10:21:46 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1UPs9w-0001F0-2P; Wed, 10 Apr 2013 10:21:44 +0000 Received: from mailout2.samsung.com ([203.254.224.25]) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1UPs9p-0001EG-4N for linux-arm-kernel@lists.infradead.org; Wed, 10 Apr 2013 10:21:40 +0000 Received: from epcpsbgr1.samsung.com (u141.gpu120.samsung.co.kr [203.254.230.141]) by mailout2.samsung.com (Oracle Communications Messaging Server 7u4-24.01 (7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTP id <0ML100FQ6BFT3MH0@mailout2.samsung.com> for linux-arm-kernel@lists.infradead.org; Wed, 10 Apr 2013 19:21:34 +0900 (KST) Received: from epcpsbgm2.samsung.com ( [172.20.52.116]) by epcpsbgr1.samsung.com (EPCPMTA) with SMTP id EB.54.20872.D2D35615; Wed, 10 Apr 2013 19:21:33 +0900 (KST) X-AuditID: cbfee68d-b7f786d000005188-eb-51653d2d81b1 Received: from epmmp1.local.host ( [203.254.227.16]) by epcpsbgm2.samsung.com (EPCPMTA) with SMTP id 43.1E.13494.D2D35615; Wed, 10 Apr 2013 19:21:33 +0900 (KST) Received: from localhost.localdomain ([10.90.51.45]) by mmp1.samsung.com (Oracle Communications Messaging Server 7u4-24.01 (7.0.4.24.0) 64bit (built Nov 17 2011)) with ESMTPA id <0ML100MMIBFUDO70@mmp1.samsung.com>; Wed, 10 Apr 2013 19:21:33 +0900 (KST) From: Chanho Park To: linux@arm.linux.org.uk, linux-arm-kernel@lists.infradead.org Subject: [PATCHv2] arm: mm: lockless get_user_pages_fast support Date: Wed, 10 Apr 2013 19:19:52 +0900 Message-id: <1365589192-5883-1-git-send-email-chanho61.park@samsung.com> X-Mailer: git-send-email 1.7.9.5 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrCLMWRmVeSWpSXmKPExsWyRsSkRFfXNjXQ4MBUU4v3y3oYLS7v17aY dH8Ci8XZpjfsFpseX2O1uLfmP6vF7cu8FrcbV7BZzF/0k9Vi0pttTBYvP55gceD2WDNvDaNH S3MPm8fOWXfZPTZ9msTusXlJvUffllWMAWxRXDYpqTmZZalF+nYJXBm3Jt5jLLhlWvHz3zfm BsapWl2MnBwSAiYSnT3tzBC2mMSFe+vZuhi5OIQEljJKTL11kgmm6PGKbjBbSGARo8TsNQYQ RS1MEl/OH2QBSbAJ6Epsef6KEcQWEXCQWHb1MAtIEbPAU0aJX83trCAJYaDEtNYzYJNYBFQl 9n1sBVvNK+AhcW/lFCCbA2ibgsScSTYQi1exS1xdEgdRLiDxbfIhFogSWYlNB6COlpQ4uOIG ywRGwQWMDKsYRVMLkguKk9KLDPWKE3OLS/PS9ZLzczcxAkP89L9nvTsYbx+wPsSYDDRuIrOU aHI+MEbySuINjc2MLExNTI2NzC3NSBNWEudVa7EOFBJITyxJzU5NLUgtii8qzUktPsTIxMEp 1cDodbD0VpLRH2uJ7aY3u1ScdiS+tl/KejC887t2wAzTGYvXRRoaL3/MvjsyTdHSteLJtde2 Qr96WVMP9Z34eq1JqH///x8Pdp09bmGsf/bmlSBLZle/I8xXd/pcfe7iPv1z/PXzZTU7twdH XelvaD1nyqVgMv/8Zdb0b5+OCIn9us6bfbrdrKVPiaU4I9FQi7moOBEAsDkuJIcCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrEIsWRmVeSWpSXmKPExsVy+t9jAV1d29RAg0krtS3eL+thtLi8X9ti 0v0JLBZnm96wW2x6fI3V4t6a/6wWty/zWtxuXMFmMX/RT1aLSW+2MVm8/HiCxYHbY828NYwe Lc09bB47Z91l99j0aRK7x+Yl9R59W1YxBrBFNTDaZKQmpqQWKaTmJeenZOal2yp5B8c7x5ua GRjqGlpamCsp5CXmptoqufgE6Lpl5gCdp6RQlphTChQKSCwuVtK3wzQhNMRN1wKmMULXNyQI rsfIAA0krGHMuDXxHmPBLdOKn/++MTcwTtXqYuTkkBAwkXi8opsJwhaTuHBvPRuILSSwiFFi 9hqDLkYuILuFSeLL+YMsIAk2AV2JLc9fMYLYIgIOEsuuHmYBKWIWeMoo8au5nRUkIQyUmNZ6 Bmwqi4CqxL6PrcwgNq+Ah8S9lVOAbA6gbQoScybZTGDkXsDIsIpRNLUguaA4KT3XSK84Mbe4 NC9dLzk/dxMjOIaeSe9gXNVgcYhRgINRiYd3gWFKoBBrYllxZe4hRgkOZiURXgut1EAh3pTE yqrUovz4otKc1OJDjMlAyycyS4km5wPjO68k3tDYxMzI0sjc0MLI2Jw0YSVx3oOt1oFCAumJ JanZqakFqUUwW5g4OKUaGFuZjr45yhrjej2Q22f58gOHlu/cdKDTLjtoAes77+0TbodprfGr +S0l/WNRQvXyp8s6NH5zK+2eL/mbrar4P//U/KX1/XvniOcvszw9W3VRgXTsNdXeDcsjCxP3 6r67aHP9X3bxfseVZmGNtxfOKJrx7lRzlvDSaj/rtSbXFBPXiIjcC1qiulyJpTgj0VCLuag4 EQCqODRJ5QIAAA== DLP-Filter: Pass X-MTR: 20000000000000000@CPGS X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20130410_062137_937719_C2B83808 X-CRM114-Status: GOOD ( 21.61 ) X-Spam-Score: -7.5 (-------) X-Spam-Report: SpamAssassin version 3.3.2 on merlin.infradead.org summary: Content analysis details: (-7.5 points) pts rule name description ---- ---------------------- -------------------------------------------------- -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high trust [203.254.224.25 listed in list.dnswl.org] 1.7 KHOP_BIG_TO_CC Sent to 10+ recipients instaed of Bcc or a list -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -2.4 RP_MATCHES_RCVD Envelope sender domain matches handover relay domain -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] Cc: Steve.Capper@arm.com, catalin.marinas@arm.com, will.deacon@arm.com, inki.dae@samsung.com, linux-mm@kvack.org, kyungmin.park@samsung.com, myungjoo.ham@samsung.com, notasas@gmail.com, Chanho Park X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This patch adds get_user_pages_fast(old name is "fast_gup") for ARM. The fast_gup can walk pagetable without taking mmap_sem or any locks. If there is not a pte with the correct permissions for the access, we fall back to slow path(get_user_pages) to get remaining pages. This patch is written on reference the x86's gup implementation. When Bill's hugetlb patchset[1] is applied, gup may need a implementation of gup_huge_pud to traverse hugepages. The patch also includes __get_user_pages_fast which is same with normal gup except its IRQ-safe. It will be needed in futex for THP. [1]: http://lists.infradead.org/pipermail/linux-arm-kernel/2012-October/126382.html Signed-off-by: Chanho Park Signed-off-by: Kyungmin Park --- Changes from v1: - Remove unnecessary smb_rmb barrier in gup_pte_range - Add __get_user_pages_fast implementation arch/arm/mm/Makefile | 2 +- arch/arm/mm/gup.c | 204 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 205 insertions(+), 1 deletion(-) create mode 100644 arch/arm/mm/gup.c diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile index 4e333fa..1c2896e 100644 --- a/arch/arm/mm/Makefile +++ b/arch/arm/mm/Makefile @@ -6,7 +6,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ iomap.o obj-$(CONFIG_MMU) += fault-armv.o flush.o idmap.o ioremap.o \ - mmap.o pgd.o mmu.o + mmap.o pgd.o mmu.o gup.o ifneq ($(CONFIG_MMU),y) obj-y += nommu.o diff --git a/arch/arm/mm/gup.c b/arch/arm/mm/gup.c new file mode 100644 index 0000000..0f9adce --- /dev/null +++ b/arch/arm/mm/gup.c @@ -0,0 +1,204 @@ +/* + * linux/arch/arm/mm/gup.c - Lockless get_user_pages_fast for arm + * + * Copyright (c) 2013 Samsung Electronics Co., Ltd. + * http://www.samsung.com + * Author : Chanho Park + * + * This code is written on reference from the x86 and PowerPC versions, by: + * + * Copyright (C) 2008 Nick Piggin + * Copyright (C) 2008 Novell Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +#include +#include +#include +#include +#include + +/* + * The performance critical leaf functions are made noinline otherwise gcc + * inlines everything into a single function which results in too much + * register pressure. + */ +static noinline int gup_pte_range(pmd_t *pmdp, unsigned long addr, + unsigned long end, int write, struct page **pages, int *nr) +{ + pte_t *ptep, pte; + + ptep = pte_offset_kernel(pmdp, addr); + do { + struct page *page; + + pte = *ptep; + + if (!pte_present_user(pte) || (write && !pte_write(pte))) + return 0; + + VM_BUG_ON(!pfn_valid(pte_pfn(pte))); + page = pte_page(pte); + + if (!page_cache_get_speculative(page)) + return 0; + + pages[*nr] = page; + (*nr)++; + + } while (ptep++, addr += PAGE_SIZE, addr != end); + + return 1; +} + +static int gup_pmd_range(pud_t *pudp, unsigned long addr, unsigned long end, + int write, struct page **pages, int *nr) +{ + unsigned long next; + pmd_t *pmdp; + + pmdp = pmd_offset(pudp, addr); + do { + next = pmd_addr_end(addr, end); + if (pmd_none(*pmdp) || pmd_bad(*pmdp)) + return 0; + else if (!gup_pte_range(pmdp, addr, next, write, pages, nr)) + return 0; + } while (pmdp++, addr = next, addr != end); + + return 1; +} + +static int gup_pud_range(pgd_t *pgdp, unsigned long addr, unsigned long end, + int write, struct page **pages, int *nr) +{ + unsigned long next; + pud_t *pudp; + + pudp = pud_offset(pgdp, addr); + do { + next = pud_addr_end(addr, end); + if (pud_none(*pudp)) + return 0; + else if (!gup_pmd_range(pudp, addr, next, write, pages, nr)) + return 0; + } while (pudp++, addr = next, addr != end); + + return 1; +} + +/* + * Like get_user_pages_fast() except its IRQ-safe in that it won't fall + * back to the regular GUP. + */ +int __get_user_pages_fast(unsigned long start, int nr_pages, int write, + struct page **pages) +{ + struct mm_struct *mm = current->mm; + unsigned long addr, len, end; + unsigned long next, flags; + pgd_t *pgdp; + int nr = 0; + + start &= PAGE_MASK; + addr = start; + len = (unsigned long) nr_pages << PAGE_SHIFT; + end = start + len; + + if (unlikely(!access_ok(write ? VERIFY_WRITE : VERIFY_READ, + start, len))) + return 0; + + /* + * This doesn't prevent pagetable teardown, but does prevent + * the pagetables from being freed on arm. + * + * So long as we atomically load page table pointers versus teardown, + * we can follow the address down to the the page and take a ref on it. + */ + local_irq_save(flags); + + pgdp = pgd_offset(mm, addr); + do { + next = pgd_addr_end(addr, end); + if (pgd_none(*pgdp)) + break; + else if (!gup_pud_range(pgdp, addr, next, write, pages, &nr)) + break; + } while (pgdp++, addr = next, addr != end); + + local_irq_restore(flags); + + return nr; +} + +int get_user_pages_fast(unsigned long start, int nr_pages, int write, + struct page **pages) +{ + struct mm_struct *mm = current->mm; + unsigned long addr, len, end; + unsigned long next; + pgd_t *pgdp; + int nr = 0; + + start &= PAGE_MASK; + addr = start; + len = (unsigned long) nr_pages << PAGE_SHIFT; + end = start + len; + + if (unlikely(!access_ok(write ? VERIFY_WRITE : VERIFY_READ, + start, len))) + goto slow_irqon; + + /* + * This doesn't prevent pagetable teardown, but does prevent + * the pagetables from being freed on arm. + * + * So long as we atomically load page table pointers versus teardown, + * we can follow the address down to the the page and take a ref on it. + */ + local_irq_disable(); + + pgdp = pgd_offset(mm, addr); + do { + next = pgd_addr_end(addr, end); + if (pgd_none(*pgdp)) + goto slow; + else if (!gup_pud_range(pgdp, addr, next, write, pages, &nr)) + goto slow; + } while (pgdp++, addr = next, addr != end); + + local_irq_enable(); + + VM_BUG_ON(nr != (end - start) >> PAGE_SHIFT); + return nr; + + { + int ret; + +slow: + local_irq_enable(); +slow_irqon: + /* Try to get the remaining pages with get_user_pages */ + start += nr << PAGE_SHIFT; + pages += nr; + + down_read(&mm->mmap_sem); + ret = get_user_pages(current, mm, start, + (end - start) >> PAGE_SHIFT, write, 0, pages, NULL); + up_read(&mm->mmap_sem); + + /* Have to be a bit careful with return values */ + if (nr > 0) { + if (ret < 0) + ret = nr; + else + ret += nr; + } + + return ret; + } +}