From patchwork Fri Dec 20 14:25:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A . Shutemov" X-Patchwork-Id: 11305595 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A5B416C1 for ; Fri, 20 Dec 2019 14:26:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 65B0224682 for ; Fri, 20 Dec 2019 14:26:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=shutemov-name.20150623.gappssmtp.com header.i=@shutemov-name.20150623.gappssmtp.com header.b="PckJPvOu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 65B0224682 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 31C3E8E0184; Fri, 20 Dec 2019 09:26:01 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 27EBC8E019D; Fri, 20 Dec 2019 09:26:01 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 147518E0184; Fri, 20 Dec 2019 09:26:01 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0135.hostedemail.com [216.40.44.135]) by kanga.kvack.org (Postfix) with ESMTP id EB35B8E019D for ; Fri, 20 Dec 2019 09:26:00 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 9CA5E180AD81F for ; Fri, 20 Dec 2019 14:26:00 +0000 (UTC) X-FDA: 76285744080.25.door39_8ba5c38433b2b X-Spam-Summary: 2,0,0,e8d99fa4c5108abc,d41d8cd98f00b204,kirill@shutemov.name,:akpm@linux-foundation.org:thomas.willhalm@intel.com:dan.j.williams@intel.com:otto.g.bruggeman@intel.com:aneesh.kumar@linux.vnet.ibm.com::x86@kernel.org:linux-kernel@vger.kernel.org:kirill.shutemov@linux.intel.com,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2559:2562:2693:2890:2892:2899:3138:3139:3140:3141:3142:3355:3865:3867:3868:3870:3871:3872:3874:4042:4118:4250:4321:4605:5007:6119:6120:6261:6653:7875:7901:7903:8603:9036:9592:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12679:12683:12895:12986:13141:13161:13184:13191:13192:13221:13229:13230:13894:14096:14181:14394:14721:21080:21444:21450:21451:21627:21795:21990:30001:30003:30051:30054:30064:30075,0,RBL:209.85.208.193:@shutemov.name:.lbl8.mailshell.net-62.8.0.100 66.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0 ,MSF:not X-HE-Tag: door39_8ba5c38433b2b X-Filterd-Recvd-Size: 7391 Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Fri, 20 Dec 2019 14:25:59 +0000 (UTC) Received: by mail-lj1-f193.google.com with SMTP id u1so10187929ljk.7 for ; Fri, 20 Dec 2019 06:25:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1aqJaMPhZ/LdjH/pzYf9S/QghCRA9LSMRsvTbXKKDeI=; b=PckJPvOugvXBXBrEBUqMIwlz8UBJbGbd2J43VDNcLAolBMRZ4byslRdYeWQaVz7Obx 4R0B6iJrB+Bcc73zIRff+qQ7uUsqdgp1GaoBiVcszCYtEptuvAqElel306DaUnKO+NXk 0JIuCY3jRAmkbGOLfedKYnbbKTOYXA6T+oKrdIWSXph/zZwskDO8ekzaHUbBnEIGlRmm 1/Sd5mCU5lobONzW7fTpTuvo9m9T2+FvaOSRiwjhdQ5+NW8A6CX7O6wsovBibyN31Tci 92EAdKncazj099YD/eoptvL8C0m50EHtVdf6a7jD/Bvq0VJbkt+T5sHWroDYqz1XRwsQ h2ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1aqJaMPhZ/LdjH/pzYf9S/QghCRA9LSMRsvTbXKKDeI=; b=EwH7v3TbqvcCvRLeZoFtm2G/xvokb5TNj5MTnBsIyin0rMo/v12sB2p8ZpYBFSgYSh St0wWPDs4V0m1fzA3cdgLN/xdKxmJF1GdP5ycH4ULlH4tmsNXgec+hP4445hgOnOqic5 47xSrpHRtAERew6kFNrI6uFUiLWkVNgLFqgr8ciLFHDGK9C9+l75tTdma20pZs8LfUJC HIzq2dd6lA7onZHiJvgdsG7h/WoztLYlnu0kqKK8Mzc3OTKJajy+WIQRabDRS7t22ES1 9+iKcQ+yNmwi4Dr7UnJorGdvMJNDQPsGpJf8uIaJIKVt6VsZMuenUB//NKy2Ifh2jbyX RuFg== X-Gm-Message-State: APjAAAUWJnuneHyRyUBE3X5hKvyaNt8GufT4EF8lCCZKcTbT73RtS7Fl 7FKLhTAJheQSBgJBBEG8ysnQIg== X-Google-Smtp-Source: APXvYqwgRZair3v7dcpcAV9A4xU5RH+vM2FKRLJfLSIhHs9JOyWmUyRsV2oGsT0Cbiqqk0ZHhQQkuQ== X-Received: by 2002:a2e:9806:: with SMTP id a6mr9736008ljj.178.1576851958332; Fri, 20 Dec 2019 06:25:58 -0800 (PST) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id h10sm4310863ljc.39.2019.12.20.06.25.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Dec 2019 06:25:56 -0800 (PST) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 373B01006EA; Fri, 20 Dec 2019 17:25:59 +0300 (+03) To: Andrew Morton Cc: "Willhalm, Thomas" , Dan Williams , "Bruggeman, Otto G" , "Aneesh Kumar K . V" , linux-mm@kvack.org, x86@kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 1/2] thp: Fix conflict of above-47bit hint address and PMD alignment Date: Fri, 20 Dec 2019 17:25:47 +0300 Message-Id: <20191220142548.7118-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20191220142548.7118-1-kirill.shutemov@linux.intel.com> References: <20191220142548.7118-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Filesystems use thp_get_unmapped_area() to provide THP-friendly mappings. For DAX in particular. Normally, the kernel doesn't create userspace mappings above 47-bit, even if the machine allows this (such as with 5-level paging on x86-64). Not all user space is ready to handle wide addresses. It's known that at least some JIT compilers use higher bits in pointers to encode their information. Userspace can ask for allocation from full address space by specifying hint address (with or without MAP_FIXED) above 47-bits. If the application doesn't need a particular address, but wants to allocate from whole address space it can specify -1 as a hint address. Unfortunately, this trick breaks thp_get_unmapped_area(): the function would not try to allocate PMD-aligned area if *any* hint address specified. Modify the routine to handle it correctly: - Try to allocate the space at the specified hint address with length padding required for PMD alignment. - If failed, retry without length padding (but with the same hint address); - If the returned address matches the hint address return it. - Otherwise, align the address as required for THP and return. The user specified hint address is passed down to get_unmapped_area() so above-47bit hint address will be taken into account without breaking alignment requirements. Signed-off-by: Kirill A. Shutemov Fixes: b569bab78d8d ("x86/mm: Prepare to expose larger address space to userspace") Reported-by: Thomas Willhalm Tested-by: Dan Williams --- mm/huge_memory.c | 38 ++++++++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 969653530c8f..dccadfbc9994 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -527,13 +527,13 @@ void prep_transhuge_page(struct page *page) set_compound_page_dtor(page, TRANSHUGE_PAGE_DTOR); } -static unsigned long __thp_get_unmapped_area(struct file *filp, unsigned long len, +static unsigned long __thp_get_unmapped_area(struct file *filp, + unsigned long addr, unsigned long len, loff_t off, unsigned long flags, unsigned long size) { - unsigned long addr; loff_t off_end = off + len; loff_t off_align = round_up(off, size); - unsigned long len_pad; + unsigned long len_pad, ret; if (off_end <= off_align || (off_end - off_align) < size) return 0; @@ -542,30 +542,40 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, unsigned long le if (len_pad < len || (off + len_pad) < off) return 0; - addr = current->mm->get_unmapped_area(filp, 0, len_pad, + ret = current->mm->get_unmapped_area(filp, addr, len_pad, off >> PAGE_SHIFT, flags); - if (IS_ERR_VALUE(addr)) + + /* + * The failure might be due to length padding. The caller will retry + * without the padding. + */ + if (IS_ERR_VALUE(ret)) return 0; - addr += (off - addr) & (size - 1); - return addr; + /* + * Do not try to align to THP boundary if allocation at the address + * hint succeeds. + */ + if (ret == addr) + return addr; + + ret += (off - ret) & (size - 1); + return ret; } unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags) { + unsigned long ret; loff_t off = (loff_t)pgoff << PAGE_SHIFT; - if (addr) - goto out; if (!IS_DAX(filp->f_mapping->host) || !IS_ENABLED(CONFIG_FS_DAX_PMD)) goto out; - addr = __thp_get_unmapped_area(filp, len, off, flags, PMD_SIZE); - if (addr) - return addr; - - out: + ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE); + if (ret) + return ret; +out: return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags); } EXPORT_SYMBOL_GPL(thp_get_unmapped_area); From patchwork Fri Dec 20 14:25:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A . Shutemov" X-Patchwork-Id: 11305593 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F3726C1 for ; Fri, 20 Dec 2019 14:26:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C0D6924682 for ; Fri, 20 Dec 2019 14:26:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=shutemov-name.20150623.gappssmtp.com header.i=@shutemov-name.20150623.gappssmtp.com header.b="zMuS2dgv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C0D6924682 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DA2368E019C; Fri, 20 Dec 2019 09:26:00 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D797E8E0184; Fri, 20 Dec 2019 09:26:00 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C69218E019C; Fri, 20 Dec 2019 09:26:00 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0216.hostedemail.com [216.40.44.216]) by kanga.kvack.org (Postfix) with ESMTP id B20188E0184 for ; Fri, 20 Dec 2019 09:26:00 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 6107F8249980 for ; Fri, 20 Dec 2019 14:26:00 +0000 (UTC) X-FDA: 76285744080.28.bat88_8b9f88b96a229 X-Spam-Summary: 2,0,0,8fa4f9e81aefd726,d41d8cd98f00b204,kirill@shutemov.name,:akpm@linux-foundation.org:thomas.willhalm@intel.com:dan.j.williams@intel.com:otto.g.bruggeman@intel.com:aneesh.kumar@linux.vnet.ibm.com::x86@kernel.org:linux-kernel@vger.kernel.org:kirill.shutemov@linux.intel.com,RULES_HIT:41:355:379:541:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2393:2559:2562:2693:2892:3138:3139:3140:3141:3142:3354:3865:3867:3868:3870:3871:3872:3874:4250:4321:5007:6119:6120:6261:6653:7901:7903:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12895:13191:13192:13229:13255:13894:14181:14394:14721:21080:21444:21451:21627:21740:21795:21809:21990:30003:30051:30054:30075,0,RBL:209.85.208.196:@shutemov.name:.lbl8.mailshell.net-62.8.0.100 66.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:39,LUA_SUMMARY:none X-HE-Tag: bat88_8b9f88b96a229 X-Filterd-Recvd-Size: 5804 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Fri, 20 Dec 2019 14:25:59 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id m26so7761063ljc.13 for ; Fri, 20 Dec 2019 06:25:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov-name.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9WqhstskUuONOhk/w+gyZwreQrEykNzgK1l1VnxD33s=; b=zMuS2dgvlkKxVxxI8JtGHSQUeKku32DXHsiX4HgwhvWFUTUQrOcWey2c7sxjWbQcb2 Lya2wZ2LgrMRnub1NYNq4lVqtPR7GXOZUNAxFJuWiQhzRgXz/bYw6QG+tOzMRQuxyD2d UidurIjM5hB5JSOahBqyYlwsq4VXOY9h2zJm6ACP7qLvGL34gFlLYX9z0N05hgsM9PDc NU+Fb1mzJA5EPJePj5fdiFzLdskrmz/sodj5LpO+WGGaoiQ/pdyZEfMar4KWDiR6VDbw DGsCfhoAYdBk1HjdXUjFq1lRHnGF0bXEHXGE5wSgPd0jizemwt6aBoJRo4a/fSzl1eOR xn3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9WqhstskUuONOhk/w+gyZwreQrEykNzgK1l1VnxD33s=; b=uZvyoOuRxIrdNGdkAc0ntgt91axwriUWAsFj0Yecpnc6gEoq2RE0+91Nx5OvcFsznt RJpa+n/qi2AqoxVAcB3Q+VO4j20uDSkfV3U5e3VyS03idzKiu+5HdnlN8QbM25RKJcOE /zVvPcVV9ZAkoD3eGBvwL+m5I49DG2EV1pVjkDE5x8l8SI8QjTFnZMZI9gF5/dET0ocv 7oXnRBetv1VzAIGEyASd1Cxsp5oH1CqQYpYriFdggBuCJTWa7jku6b4RYYQYO6hgkBQN 4MU+FFnbbQQcharmNgn5zs8CM/xPfRny5vI7Y49uCuN9pX/7NbBNSWES5rDo5/3iUU2/ u2nA== X-Gm-Message-State: APjAAAVYUAxUXUnx61uFR+fACsQs3zE48Fcn1qWiO8DklqlnDkWJQ3dE 6D8e8CVAEuUVpk22lUwjZ1D/tA== X-Google-Smtp-Source: APXvYqy1P1YNJauGj7NRz5haqIY7txUzCE4ENcU904LzYlGyPRf3lLLbOhOcL3xZoZexkkXT/5uXpQ== X-Received: by 2002:a2e:a0d5:: with SMTP id f21mr10236743ljm.106.1576851958001; Fri, 20 Dec 2019 06:25:58 -0800 (PST) Received: from box.localdomain ([86.57.175.117]) by smtp.gmail.com with ESMTPSA id w17sm4083642lfn.22.2019.12.20.06.25.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Dec 2019 06:25:56 -0800 (PST) From: "Kirill A. Shutemov" X-Google-Original-From: "Kirill A. Shutemov" Received: by box.localdomain (Postfix, from userid 1000) id 3E3631012A8; Fri, 20 Dec 2019 17:25:59 +0300 (+03) To: Andrew Morton Cc: "Willhalm, Thomas" , Dan Williams , "Bruggeman, Otto G" , "Aneesh Kumar K . V" , linux-mm@kvack.org, x86@kernel.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCH 2/2] thp, shmem: Fix conflict of above-47bit hint address and PMD alignment Date: Fri, 20 Dec 2019 17:25:48 +0300 Message-Id: <20191220142548.7118-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20191220142548.7118-1-kirill.shutemov@linux.intel.com> References: <20191220142548.7118-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Shmem/tmpfs tries to provide THP-friendly mappings if huge pages are enabled. But it doesn't work well with above-47bit hint address. Normally, the kernel doesn't create userspace mappings above 47-bit, even if the machine allows this (such as with 5-level paging on x86-64). Not all user space is ready to handle wide addresses. It's known that at least some JIT compilers use higher bits in pointers to encode their information. Userspace can ask for allocation from full address space by specifying hint address (with or without MAP_FIXED) above 47-bits. If the application doesn't need a particular address, but wants to allocate from whole address space it can specify -1 as a hint address. Unfortunately, this trick breaks THP alignment in shmem/tmp: shmem_get_unmapped_area() would not try to allocate PMD-aligned area if *any* hint address specified. This can be fixed by requesting the aligned area if the we failed to allocated at user-specified hint address. The request with inflated length will also take the user-specified hint address. This way we will not lose an allocation request from the full address space. Signed-off-by: Kirill A. Shutemov Fixes: b569bab78d8d ("x86/mm: Prepare to expose larger address space to userspace") Reported-by: kbuild test robot --- mm/shmem.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 165fa6332993..dc539482ce67 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2107,9 +2107,10 @@ unsigned long shmem_get_unmapped_area(struct file *file, /* * Our priority is to support MAP_SHARED mapped hugely; * and support MAP_PRIVATE mapped hugely too, until it is COWed. - * But if caller specified an address hint, respect that as before. + * But if caller specified an address hint and we allocated area there + * successfully, respect that as before. */ - if (uaddr) + if (uaddr == addr) return addr; if (shmem_huge != SHMEM_HUGE_FORCE) { @@ -2143,7 +2144,7 @@ unsigned long shmem_get_unmapped_area(struct file *file, if (inflated_len < len) return addr; - inflated_addr = get_area(NULL, 0, inflated_len, 0, flags); + inflated_addr = get_area(uaddr, 0, inflated_len, 0, flags); if (IS_ERR_VALUE(inflated_addr)) return addr; if (inflated_addr & ~PAGE_MASK)