From patchwork Tue Nov 3 00:28:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 11875629 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B3DC26A2 for ; Tue, 3 Nov 2020 00:29:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5B2A62225E for ; Tue, 3 Nov 2020 00:29:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="dtKkjXWV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5B2A62225E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A4C566B005D; Mon, 2 Nov 2020 19:29:05 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9B7A26B0070; Mon, 2 Nov 2020 19:29:05 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 792A06B0068; Mon, 2 Nov 2020 19:29:05 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id 3A1FD6B005D for ; Mon, 2 Nov 2020 19:29:05 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C92D23632 for ; Tue, 3 Nov 2020 00:29:04 +0000 (UTC) X-FDA: 77441222208.16.grape28_2007e60272b4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id A3825100E6917 for ; Tue, 3 Nov 2020 00:29:04 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,mike.kravetz@oracle.com,,RULES_HIT:30003:30054:30064:30070,0,RBL:141.146.126.79:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100;04y849sn5t66qx7y4znsnhwu8euugop3a9msqmrgw74jid9apqqdk6jzfx53ief.fknp8e4t1d8aj9c9dodtbhqjwfsaxjr18zw8qgu5rojpy7o4krt79hct5hpd53s.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:69,LUA_SUMMARY:none X-HE-Tag: grape28_2007e60272b4 X-Filterd-Recvd-Size: 9458 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Tue, 3 Nov 2020 00:29:03 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0A30Suko088119; Tue, 3 Nov 2020 00:28:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2020-01-29; bh=NmQ2PFivlGfrOZg00Ymm2k7iW7oVbhi3wCqoQABAwwQ=; b=dtKkjXWVML9UztitT+4RqPWJ2EXAlFbdCsZH6D9v0Wzit5qVEYx8akpEr0s416yB1jOo DhG6BzVkashN9SxtMnkqeAWF3mVw/bcI3LESx420vPVkfuzS30zMuS02d/MbUD7WIVG3 dAxKAxGAkySTQRiIigUqXQIPPJdRwppqS1lyfH2DxzuqEB9YRWLM8J2l4ntnSOD8C/UI r2l+PcX0LzGgcf36SnkamRph8FrI6NB1uQY64LhwphHQj2JrJOzEiXnQ45uA0JybB5fy Be3+d7ioFoSOK5N1YLjSS4dezVoTCzaLHmVcGl2t/Ax77ivIh69kYoQqmyzQYXtoc1yx 1w== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by aserp2130.oracle.com with ESMTP id 34hhb1xt6j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 03 Nov 2020 00:28:56 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0A30AkE3045781; Tue, 3 Nov 2020 00:28:50 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserp3030.oracle.com with ESMTP id 34jf47f62a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 03 Nov 2020 00:28:50 +0000 Received: from abhmp0015.oracle.com (abhmp0015.oracle.com [141.146.116.21]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 0A30SmUS028857; Tue, 3 Nov 2020 00:28:48 GMT Received: from monkey.oracle.com (/50.38.35.18) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 02 Nov 2020 16:28:48 -0800 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , Naoya Horiguchi , Michal Hocko , "Aneesh Kumar K . V" , Andrea Arcangeli , "Kirill A . Shutemov" , Davidlohr Bueso , Prakash Sangappa , Andrew Morton , Mike Kravetz , stable@vger.kernel.org Subject: [PATCH 1/4] Revert hugetlbfs: Use i_mmap_rwsem to address page fault/truncate race Date: Mon, 2 Nov 2020 16:28:38 -0800 Message-Id: <20201103002841.273161-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201103002841.273161-1-mike.kravetz@oracle.com> References: <20201026233150.371577-1-mike.kravetz@oracle.com> <20201103002841.273161-1-mike.kravetz@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9793 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 mlxscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011030000 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9793 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 suspectscore=0 clxscore=1015 mlxlogscore=999 impostorscore=0 malwarescore=0 lowpriorityscore=0 adultscore=0 spamscore=0 priorityscore=1501 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011030001 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit 87bf91d39bb5 ("hugetlbfs: Use i_mmap_rwsem to address page fault/truncate race") was made possible because a prior commit c0d0381ade79 ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization") took i_mmap_rwsem in read mode during huge page faults. Using i_mmap_rwsem for pmd sharing synchronization has proven problematic and will be removed in later patches. As a result, the assumptions upon which this patch was based will no longer be true. This reverts commit 87bf91d39bb52b688fb411d668fbe7df278b29ae Fixes 7bf91d39bb5 ("hugetlbfs: Use i_mmap_rwsem to address page fault/truncate race") Cc: Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 28 ++++++++-------------------- mm/hugetlb.c | 23 ++++++++++++----------- 2 files changed, 20 insertions(+), 31 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index b5c109703daa..c1057378dbf4 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -444,9 +444,10 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end) * In this case, we first scan the range and release found pages. * After releasing pages, hugetlb_unreserve_pages cleans up region/reserv * maps and global counts. Page faults can not race with truncation - * in this routine. hugetlb_no_page() holds i_mmap_rwsem and prevents - * page faults in the truncated range by checking i_size. i_size is - * modified while holding i_mmap_rwsem. + * in this routine. hugetlb_no_page() prevents page faults in the + * truncated range. It checks i_size before allocation, and again after + * with the page table lock for the page held. The same lock must be + * acquired to unmap a page. * hole punch is indicated if end is not LLONG_MAX * In the hole punch case we scan the range and release found pages. * Only when releasing a page is the associated region/reserv map @@ -486,15 +487,7 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, index = page->index; hash = hugetlb_fault_mutex_hash(mapping, index); - if (!truncate_op) { - /* - * Only need to hold the fault mutex in the - * hole punch case. This prevents races with - * page faults. Races are not possible in the - * case of truncation. - */ - mutex_lock(&hugetlb_fault_mutex_table[hash]); - } + mutex_lock(&hugetlb_fault_mutex_table[hash]); /* * If page is mapped, it was faulted in after being @@ -537,8 +530,7 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, } unlock_page(page); - if (!truncate_op) - mutex_unlock(&hugetlb_fault_mutex_table[hash]); + mutex_unlock(&hugetlb_fault_mutex_table[hash]); } huge_pagevec_release(&pvec); cond_resched(); @@ -576,8 +568,8 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset) BUG_ON(offset & ~huge_page_mask(h)); pgoff = offset >> PAGE_SHIFT; - i_mmap_lock_write(mapping); i_size_write(inode, offset); + i_mmap_lock_write(mapping); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0); i_mmap_unlock_write(mapping); @@ -699,11 +691,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, /* addr is the offset within the file (zero based) */ addr = index * hpage_size; - /* - * fault mutex taken here, protects against fault path - * and hole punch. inode_lock previously taken protects - * against truncation. - */ + /* mutex taken here, fault path and hole punch */ hash = hugetlb_fault_mutex_hash(mapping, index); mutex_lock(&hugetlb_fault_mutex_table[hash]); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fe76f8fd5a73..8a82b90ca3ee 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4335,17 +4335,16 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, } /* - * We can not race with truncation due to holding i_mmap_rwsem. - * i_size is modified when holding i_mmap_rwsem, so check here - * once for faults beyond end of file. + * Use page lock to guard against racing truncation + * before we get page_table_lock. */ - size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) - goto out; - retry: page = find_lock_page(mapping, idx); if (!page) { + size = i_size_read(mapping->host) >> huge_page_shift(h); + if (idx >= size) + goto out; + /* * Check for page in userfault range */ @@ -4451,6 +4450,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, } ptl = huge_pte_lock(h, mm, ptep); + size = i_size_read(mapping->host) >> huge_page_shift(h); + if (idx >= size) + goto backout; + ret = 0; if (!huge_pte_none(huge_ptep_get(ptep))) goto backout; @@ -4550,10 +4553,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, /* * Acquire i_mmap_rwsem before calling huge_pte_alloc and hold - * until finished with ptep. This serves two purposes: - * 1) It prevents huge_pmd_unshare from being called elsewhere - * and making the ptep no longer valid. - * 2) It synchronizes us with i_size modifications during truncation. + * until finished with ptep. This prevents huge_pmd_unshare from + * being called elsewhere and making the ptep no longer valid. * * ptep could have already be assigned via huge_pte_offset. That * is OK, as huge_pte_alloc will return the same value unless From patchwork Tue Nov 3 00:28:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 11875631 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8AFCD6A2 for ; Tue, 3 Nov 2020 00:29:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40A8B22268 for ; Tue, 3 Nov 2020 00:29:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="NIi8uErS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40A8B22268 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D0DE66B006E; Mon, 2 Nov 2020 19:29:05 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C9B5C6B005C; Mon, 2 Nov 2020 19:29:05 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A79FB6B006E; Mon, 2 Nov 2020 19:29:05 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 6564F6B005C for ; Mon, 2 Nov 2020 19:29:05 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E4E56181AC9CB for ; Tue, 3 Nov 2020 00:29:04 +0000 (UTC) X-FDA: 77441222208.19.week04_2a0e23c272b4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id C5D751AD1B4 for ; Tue, 3 Nov 2020 00:29:04 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,mike.kravetz@oracle.com,,RULES_HIT:30012:30054:30064:30070:30074:30075:30090,0,RBL:141.146.126.79:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100;04ygadjfoiqsyd96cqrtk6o6ot8nsyc43j8bwtfwxbb754ebse54kx3p1dh9t6z.8i8g9fz1teh3kbqz18p34ahqz3epu7cjfurbwjhdarut97hy8sfmopey6rnadkc.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:68,LUA_SUMMARY:none X-HE-Tag: week04_2a0e23c272b4 X-Filterd-Recvd-Size: 11085 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Tue, 3 Nov 2020 00:29:04 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0A30Susv088111; Tue, 3 Nov 2020 00:28:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2020-01-29; bh=h5+Lu0QflE5OZ+B9UFFP0ald5p6gRIVbsKDocV9JCg0=; b=NIi8uErSycKbmI66pWcHyz/fFTsk8wS3L5KQyx8c5yg++sknwOrbh6JtvOKBzNnpICvP xPn1oY7nw5jQOQXXrAVKGURg1WhR05YdcdFI6xH23zFtj0US43zlV+/z/f2gIfJHmbBY //ulyFsYNXRRzqWTKp8PMKsQWjTNuOVwLgqsEKwzNTRgs+Ui7Qny3uckxriFs/MsNwQO 5OrEhtgMqxgPKyAG1DAA0/+uZyF9sWnIOEGgRJLLDhMUuqtzKjDtz3+b9M5QBJrwByV3 YVOpjqr/2nNeInA3rQsAF4PucN0rX/pSlVjfnA9jBMqYtf2uwgzlWHUhn+kiuiOTqpdf Og== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 34hhb1xt6p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 03 Nov 2020 00:28:56 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0A30AQwE076604; Tue, 3 Nov 2020 00:28:53 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3020.oracle.com with ESMTP id 34hw0g068p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 03 Nov 2020 00:28:53 +0000 Received: from abhmp0015.oracle.com (abhmp0015.oracle.com [141.146.116.21]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 0A30Sppu000627; Tue, 3 Nov 2020 00:28:51 GMT Received: from monkey.oracle.com (/50.38.35.18) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 02 Nov 2020 16:28:51 -0800 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , Naoya Horiguchi , Michal Hocko , "Aneesh Kumar K . V" , Andrea Arcangeli , "Kirill A . Shutemov" , Davidlohr Bueso , Prakash Sangappa , Andrew Morton , Mike Kravetz , stable@vger.kernel.org Subject: [PATCH 2/4] hugetlbfs: add hinode_rwsem to hugetlb specific inode Date: Mon, 2 Nov 2020 16:28:39 -0800 Message-Id: <20201103002841.273161-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201103002841.273161-1-mike.kravetz@oracle.com> References: <20201026233150.371577-1-mike.kravetz@oracle.com> <20201103002841.273161-1-mike.kravetz@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9793 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 adultscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011030000 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9793 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 suspectscore=0 clxscore=1015 mlxlogscore=999 impostorscore=0 malwarescore=0 lowpriorityscore=0 adultscore=0 spamscore=0 priorityscore=1501 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011030001 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The hugetlb pmd sharing code needs additional synchronization. This is because sharing established via a call huge_pte_alloc, could be undone before control is returned to the caller. As a result, the returned value may be invalid. Ideally, i_mmap_rwsem would be used for this type of synchronization. However, previous attempts at using i_mmap_rwsem have failed. This is partly due to conflicts with the existing uses of i_mmap_rwsem that force a locking order not compatible with it's use for pmd sharing. Introduce a rwsem (hinode_rwsem) that resides in the hugetlb specific inode for the purpose of pmd sharing synchronization. This patch adds the semaphore to the inode and also provides routines for using the semaphore. To minimize performance impacts, the routines only acquire the semaphore if pmd sharing is possible. In addition, routines which can be used with lockdep to help ensure proper locking are also added. Use of the new semaphore and supporting routines will be provided in a later patch. Fixes: c0d0381ade79 ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization") Cc: Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 12 ++++ include/linux/hugetlb.h | 121 ++++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 13 ----- 3 files changed, 133 insertions(+), 13 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index c1057378dbf4..4f1404b9f354 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -85,6 +85,17 @@ static const struct fs_parameter_spec hugetlb_fs_parameters[] = { {} }; +#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE +static inline void init_hinode_rwsem(struct hugetlbfs_inode_info *info) +{ + init_rwsem(&info->hinode_rwsem); +} +#else +static inline void init_hinode_rwsem(struct hugetlbfs_inode_info *info) +{ +} +#endif + #ifdef CONFIG_NUMA static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma, struct inode *inode, pgoff_t index) @@ -831,6 +842,7 @@ static struct inode *hugetlbfs_get_inode(struct super_block *sb, inode->i_atime = inode->i_mtime = inode->i_ctime = current_time(inode); inode->i_mapping->private_data = resv_map; info->seals = F_SEAL_SEAL; + init_hinode_rwsem(info); switch (mode & S_IFMT) { default: init_special_inode(inode, mode, dev); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ebca2ef02212..c6a59c2dbc30 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -424,6 +424,9 @@ struct hugetlbfs_inode_info { struct shared_policy policy; struct inode vfs_inode; unsigned int seals; +#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE + struct rw_semaphore hinode_rwsem; +#endif }; static inline struct hugetlbfs_inode_info *HUGETLBFS_I(struct inode *inode) @@ -449,6 +452,101 @@ static inline struct hstate *hstate_inode(struct inode *i) { return HUGETLBFS_SB(i->i_sb)->hstate; } + +#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE +static inline bool vma_shareable(struct vm_area_struct *vma, unsigned long addr) +{ + unsigned long base = addr & PUD_MASK; + unsigned long end = base + PUD_SIZE; + + /* check on proper vm_flags and page table alignment */ + if (vma->vm_flags & VM_MAYSHARE && range_in_vma(vma, base, end)) + return true; + return false; +} + +/* + * hugetlb specific hinode_rwsem is used for pmd sharing synchronization. + * This routine will take the semaphore in read mode if necessary. If vma + * and addr are NULL, the routine will always acquire the semaphore. If + * values are supplied for vma and addr, they are used to determine if pmd + * sharing is actually possible, and only acquire the semaphore if possible. + * Returns true if lock was acquired, otherwise false. + */ +static inline bool hinode_lock_read(struct inode *inode, + struct vm_area_struct *vma, + unsigned long addr) +{ + if (vma && !addr) + addr = round_up(vma->vm_start, PUD_SIZE); + if (vma && !vma_shareable(vma, addr)) + return false; + + down_read(&HUGETLBFS_I(inode)->hinode_rwsem); + return true; +} + +static inline void hinode_unlock_read(struct inode *inode) +{ + up_read(&HUGETLBFS_I(inode)->hinode_rwsem); +} + +/* + * Take hinode_rwsem semaphore in write mode if necessary. See, + * hinode_lock_read for details. + * Returns true is lock was acquired, otherwise false. + */ +static inline bool hinode_lock_write(struct inode *inode, + struct vm_area_struct *vma, + unsigned long addr) +{ + if (vma && !addr) + addr = round_up(vma->vm_start, PUD_SIZE); + if (vma && !vma_shareable(vma, addr)) + return false; + + down_write(&HUGETLBFS_I(inode)->hinode_rwsem); + return true; +} + +static inline void hinode_unlock_write(struct inode *inode) +{ + up_write(&HUGETLBFS_I(inode)->hinode_rwsem); +} + +static inline void hinode_assert_locked(struct address_space *mapping) +{ + lockdep_assert_held(&HUGETLBFS_I(mapping->host)->hinode_rwsem); +} + +static inline void hinode_assert_write_locked(struct address_space *mapping) +{ + lockdep_assert_held_write(&HUGETLBFS_I(mapping->host)->hinode_rwsem); +} +#else +static inline bool hinode_lock_read(struct inode *inode, + struct vm_area_struct *vma, + unsigned long addr) +{ + return false; +} + +static inline void hinode_unlock_read(struct inode *inode) +{ +} + +static inline bool hinode_lock_write(struct inode *inode, + struct vm_area_struct *vma, + unsigned long addr) +{ + return false; +} + +static inline void hinode_unlock_write(struct inode *inode) +{ +} +#endif + #else /* !CONFIG_HUGETLBFS */ #define is_file_hugepages(file) false @@ -923,6 +1021,29 @@ static inline void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr pte_t *ptep, pte_t pte, unsigned long sz) { } + +static inline bool hinode_lock_read(struct inode *inode, + struct vm_area_struct *vma, + unsigned long addr) +{ + return false; +} + +static inline void hinode_unlock_read(struct inode *inode) +{ +} + +static inline bool hinode_lock_write(struct inode *inode, + struct vm_area_struct *vma, + unsigned long addr) +{ + return false; +} + +static inline void hinode_unlock_write(struct inode *inode) +{ +} + #endif /* CONFIG_HUGETLB_PAGE */ static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8a82b90ca3ee..da57018926e4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5296,19 +5296,6 @@ static unsigned long page_table_shareable(struct vm_area_struct *svma, return saddr; } -static bool vma_shareable(struct vm_area_struct *vma, unsigned long addr) -{ - unsigned long base = addr & PUD_MASK; - unsigned long end = base + PUD_SIZE; - - /* - * check on proper vm_flags and page table alignment - */ - if (vma->vm_flags & VM_MAYSHARE && range_in_vma(vma, base, end)) - return true; - return false; -} - /* * Determine if start,end range within vma could be mapped by shared pmd. * If yes, adjust start and end to cover range associated with possible From patchwork Tue Nov 3 00:28:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 11875633 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95209139F for ; Tue, 3 Nov 2020 00:29:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 131062225E for ; Tue, 3 Nov 2020 00:29:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="sGK+M3xf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 131062225E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6ACC26B0068; Mon, 2 Nov 2020 19:29:06 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 661926B0070; Mon, 2 Nov 2020 19:29:06 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43A1D6B0072; Mon, 2 Nov 2020 19:29:06 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0132.hostedemail.com [216.40.44.132]) by kanga.kvack.org (Postfix) with ESMTP id 010386B0068 for ; Mon, 2 Nov 2020 19:29:05 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A3178180AD80F for ; Tue, 3 Nov 2020 00:29:05 +0000 (UTC) X-FDA: 77441222250.28.cloth49_3215660272b4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 766FF6C2F for ; Tue, 3 Nov 2020 00:29:05 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,mike.kravetz@oracle.com,,RULES_HIT:30003:30012:30045:30051:30054:30064:30069:30070:30074:30090,0,RBL:156.151.31.85:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100;04yrotzxnin6ood1n65obiu1d6615ocz46addw8os3j4so1553owdmsoskp9nkb.cxsu5b7yp386ktp3wuqzuawz9jjopkczpiwyarbyq9w54xeo6bqhquza8837ad9.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:182,LUA_SUMMARY:none X-HE-Tag: cloth49_3215660272b4 X-Filterd-Recvd-Size: 31513 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Tue, 3 Nov 2020 00:29:04 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0A30Sv2B059904; Tue, 3 Nov 2020 00:28:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2020-01-29; bh=geuTP6PRzJMI7bS6hI4wGJMvdsQqPL1ta9Wf0sxrcz0=; b=sGK+M3xfzLYYH4Zfq3gxBAfgy0sOgvejhls3dz3E1ZiyYIgarKcRH2Su8CMnMZTjMCCm lVmc6KBAAg5zPcQWSxaBhaFKNyyYL5cYzoO8fwpxjh9L9D1uFoSRRVUOStTY3u9iUkBv daQUDFPKyNj0J9Y6Nm3LezVDLYlafPU3G3jJYt1tSunob/xX9EqAg3dghlNKECT+QBfn GXGwK53XjXMp4fuULa5GUE/joQLK37i6GayAqBzyLeaxMksSIIpmw+uC5d5afOdV1P2R lz9rSbkmuoCfNRVBJF6NulNB94EmjXgsOy+fS51t3QRSPDko94hIzWE3kC+8YA3fFD4F WA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2120.oracle.com with ESMTP id 34hhw2escy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 03 Nov 2020 00:28:56 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0A30APrj076536; Tue, 3 Nov 2020 00:28:55 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3020.oracle.com with ESMTP id 34hw0g069s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 03 Nov 2020 00:28:55 +0000 Received: from abhmp0015.oracle.com (abhmp0015.oracle.com [141.146.116.21]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 0A30SrTl000632; Tue, 3 Nov 2020 00:28:53 GMT Received: from monkey.oracle.com (/50.38.35.18) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 02 Nov 2020 16:28:52 -0800 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , Naoya Horiguchi , Michal Hocko , "Aneesh Kumar K . V" , Andrea Arcangeli , "Kirill A . Shutemov" , Davidlohr Bueso , Prakash Sangappa , Andrew Morton , Mike Kravetz , stable@vger.kernel.org Subject: [PATCH 3/4] hugetlbfs: use hinode_rwsem for pmd sharing synchronization Date: Mon, 2 Nov 2020 16:28:40 -0800 Message-Id: <20201103002841.273161-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201103002841.273161-1-mike.kravetz@oracle.com> References: <20201026233150.371577-1-mike.kravetz@oracle.com> <20201103002841.273161-1-mike.kravetz@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9793 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 adultscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011030000 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9793 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 mlxscore=0 suspectscore=0 clxscore=1015 priorityscore=1501 impostorscore=0 spamscore=0 lowpriorityscore=0 mlxlogscore=999 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011030001 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit c0d0381ade79 ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization") required changes to mm locking order that are hugetlb specific. Specifically, i_mmap_rwsem had to be taken before the page lock. This is not a big issue in hugetlb specific code, but becomes more problematic in the areas of page migration and memory failure where generic mm code had to deal with this change to lock ordering. An ugly routine 'hugetlb_page_mapping_lock_write' was added to help with these issues. Recently, Hugh Dickins diagnosed a migration BUG as caused by code introduced with hugetlb i_mmap_rwsem synchronization [1]. Subsequent discussion in that thread pointed out additional problems in the code. In the previous patch, a rw_semaphore (hinode_rwsem) was added to the hugetlbfs inode. Using hinode_rwsem instead of i_mmap_rwsem is actually a 'cleaner' approach to this problem as it can be inserted in the lock hierarchy where needed. And, there is no issue with other parts of the mm using this rw_semaphore. Change code to use hinode_rwsem instead of i_mmap_rwsem. [1] https://lore.kernel.org/linux-mm/alpine.LSU.2.11.2010071833100.2214@eggly.anvils/ Fixes: c0d0381ade79 ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization") Cc: Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 31 +++++-- include/linux/fs.h | 15 ---- include/linux/hugetlb.h | 8 -- mm/hugetlb.c | 188 +++++++++++----------------------------- mm/memory-failure.c | 34 +++----- mm/memory.c | 5 ++ mm/migrate.c | 34 ++++---- mm/rmap.c | 21 ++--- mm/userfaultfd.c | 17 ++-- 9 files changed, 124 insertions(+), 229 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 4f1404b9f354..bc9979382a1e 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -501,24 +501,35 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, mutex_lock(&hugetlb_fault_mutex_table[hash]); /* - * If page is mapped, it was faulted in after being - * unmapped in caller. Unmap (again) now after taking - * the fault mutex. The mutex will prevent faults - * until we finish removing the page. - * - * This race can only happen in the hole punch case. - * Getting here in a truncate operation is a bug. + * After taking fault mutex, check if page is mapped. + * If so, it was faulted in after being unmapped in + * caller. */ if (unlikely(page_mapped(page))) { + bool hinode_locked; + + /* + * Unmap (again) now after taking the fault + * mutex. The mutex will prevent faults until + * we finish removing the page. Be sure to + * take locks in the correct order. + * + * This race can only happen in the hole punch + * case. Getting here in a truncate operation + * is a bug. + */ BUG_ON(truncate_op); - mutex_unlock(&hugetlb_fault_mutex_table[hash]); + hinode_locked = + hinode_lock_write(inode, NULL, 0UL); i_mmap_lock_write(mapping); mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vmdelete_list(&mapping->i_mmap, index * pages_per_huge_page(h), (index + 1) * pages_per_huge_page(h)); i_mmap_unlock_write(mapping); + if (hinode_locked) + hinode_unlock_write(inode); } lock_page(page); @@ -575,15 +586,19 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset) pgoff_t pgoff; struct address_space *mapping = inode->i_mapping; struct hstate *h = hstate_inode(inode); + bool hinode_locked; BUG_ON(offset & ~huge_page_mask(h)); pgoff = offset >> PAGE_SHIFT; + hinode_locked = hinode_lock_write(inode, NULL, 0UL); i_size_write(inode, offset); i_mmap_lock_write(mapping); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0); i_mmap_unlock_write(mapping); + if (hinode_locked) + hinode_unlock_write(inode); remove_inode_hugepages(inode, offset, LLONG_MAX); return 0; } diff --git a/include/linux/fs.h b/include/linux/fs.h index 21cc971fd960..8123f281c275 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -493,11 +493,6 @@ static inline void i_mmap_lock_write(struct address_space *mapping) down_write(&mapping->i_mmap_rwsem); } -static inline int i_mmap_trylock_write(struct address_space *mapping) -{ - return down_write_trylock(&mapping->i_mmap_rwsem); -} - static inline void i_mmap_unlock_write(struct address_space *mapping) { up_write(&mapping->i_mmap_rwsem); @@ -513,16 +508,6 @@ static inline void i_mmap_unlock_read(struct address_space *mapping) up_read(&mapping->i_mmap_rwsem); } -static inline void i_mmap_assert_locked(struct address_space *mapping) -{ - lockdep_assert_held(&mapping->i_mmap_rwsem); -} - -static inline void i_mmap_assert_write_locked(struct address_space *mapping) -{ - lockdep_assert_held_write(&mapping->i_mmap_rwsem); -} - /* * Might pages of this file be mapped into userspace? */ diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index c6a59c2dbc30..a03475cccb77 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -154,8 +154,6 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx); pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud); -struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage); - extern int sysctl_hugetlb_shm_group; extern struct list_head huge_boot_pages; @@ -199,12 +197,6 @@ static inline unsigned long hugetlb_total_pages(void) return 0; } -static inline struct address_space *hugetlb_page_mapping_lock_write( - struct page *hpage) -{ - return NULL; -} - static inline int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long *addr, pte_t *ptep) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index da57018926e4..957abc2d02ff 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1568,106 +1568,6 @@ int PageHeadHuge(struct page *page_head) return page_head[1].compound_dtor == HUGETLB_PAGE_DTOR; } -/* - * Find address_space associated with hugetlbfs page. - * Upon entry page is locked and page 'was' mapped although mapped state - * could change. If necessary, use anon_vma to find vma and associated - * address space. The returned mapping may be stale, but it can not be - * invalid as page lock (which is held) is required to destroy mapping. - */ -static struct address_space *_get_hugetlb_page_mapping(struct page *hpage) -{ - struct anon_vma *anon_vma; - pgoff_t pgoff_start, pgoff_end; - struct anon_vma_chain *avc; - struct address_space *mapping = page_mapping(hpage); - - /* Simple file based mapping */ - if (mapping) - return mapping; - - /* - * Even anonymous hugetlbfs mappings are associated with an - * underlying hugetlbfs file (see hugetlb_file_setup in mmap - * code). Find a vma associated with the anonymous vma, and - * use the file pointer to get address_space. - */ - anon_vma = page_lock_anon_vma_read(hpage); - if (!anon_vma) - return mapping; /* NULL */ - - /* Use first found vma */ - pgoff_start = page_to_pgoff(hpage); - pgoff_end = pgoff_start + pages_per_huge_page(page_hstate(hpage)) - 1; - anon_vma_interval_tree_foreach(avc, &anon_vma->rb_root, - pgoff_start, pgoff_end) { - struct vm_area_struct *vma = avc->vma; - - mapping = vma->vm_file->f_mapping; - break; - } - - anon_vma_unlock_read(anon_vma); - return mapping; -} - -/* - * Find and lock address space (mapping) in write mode. - * - * Upon entry, the page is locked which allows us to find the mapping - * even in the case of an anon page. However, locking order dictates - * the i_mmap_rwsem be acquired BEFORE the page lock. This is hugetlbfs - * specific. So, we first try to lock the sema while still holding the - * page lock. If this works, great! If not, then we need to drop the - * page lock and then acquire i_mmap_rwsem and reacquire page lock. Of - * course, need to revalidate state along the way. - */ -struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage) -{ - struct address_space *mapping, *mapping2; - - mapping = _get_hugetlb_page_mapping(hpage); -retry: - if (!mapping) - return mapping; - - /* - * If no contention, take lock and return - */ - if (i_mmap_trylock_write(mapping)) - return mapping; - - /* - * Must drop page lock and wait on mapping sema. - * Note: Once page lock is dropped, mapping could become invalid. - * As a hack, increase map count until we lock page again. - */ - atomic_inc(&hpage->_mapcount); - unlock_page(hpage); - i_mmap_lock_write(mapping); - lock_page(hpage); - atomic_add_negative(-1, &hpage->_mapcount); - - /* verify page is still mapped */ - if (!page_mapped(hpage)) { - i_mmap_unlock_write(mapping); - return NULL; - } - - /* - * Get address space again and verify it is the same one - * we locked. If not, drop lock and retry. - */ - mapping2 = _get_hugetlb_page_mapping(hpage); - if (mapping2 != mapping) { - i_mmap_unlock_write(mapping); - mapping = mapping2; - goto retry; - } - - return mapping; -} - pgoff_t __basepage_index(struct page *page) { struct page *page_head = compound_head(page); @@ -3818,9 +3718,9 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, int cow; struct hstate *h = hstate_vma(vma); unsigned long sz = huge_page_size(h); - struct address_space *mapping = vma->vm_file->f_mapping; struct mmu_notifier_range range; int ret = 0; + bool hinode_locked; cow = (vma->vm_flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE; @@ -3829,16 +3729,17 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, vma->vm_start, vma->vm_end); mmu_notifier_invalidate_range_start(&range); - } else { - /* - * For shared mappings i_mmap_rwsem must be held to call - * huge_pte_alloc, otherwise the returned ptep could go - * away if part of a shared pmd and another thread calls - * huge_pmd_unshare. - */ - i_mmap_lock_read(mapping); } + /* + * For shared mappings hinode_rwsem must be held to call + * huge_pte_alloc, otherwise the returned ptep could go + * away if part of a shared pmd and another thread calls + * huge_pmd_unshare. + * + */ + hinode_locked = hinode_lock_read(vma->vm_file->f_inode, vma, 0UL); + for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; src_pte = huge_pte_offset(src, addr, sz); @@ -3914,8 +3815,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, if (cow) mmu_notifier_invalidate_range_end(&range); - else - i_mmap_unlock_read(mapping); + if (hinode_locked) + hinode_unlock_read(vma->vm_file->f_inode); return ret; } @@ -4311,7 +4212,8 @@ int huge_add_to_page_cache(struct page *page, struct address_space *mapping, static vm_fault_t hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *mapping, pgoff_t idx, - unsigned long address, pte_t *ptep, unsigned int flags) + unsigned long address, pte_t *ptep, unsigned int flags, + bool hinode_locked) { struct hstate *h = hstate_vma(vma); vm_fault_t ret = VM_FAULT_SIGBUS; @@ -4364,15 +4266,20 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, }; /* - * hugetlb_fault_mutex and i_mmap_rwsem must be - * dropped before handling userfault. Reacquire - * after handling fault to make calling code simpler. + * hugetlb_fault_mutex and inode mutex must be dropped + * before handling userfault. Reacquire after handling + * fault to make calling code simpler. */ hash = hugetlb_fault_mutex_hash(mapping, idx); mutex_unlock(&hugetlb_fault_mutex_table[hash]); - i_mmap_unlock_read(mapping); + if (hinode_locked) + hinode_unlock_read(mapping->host); + ret = handle_userfault(&vmf, VM_UFFD_MISSING); - i_mmap_lock_read(mapping); + + if (hinode_locked) + (void)hinode_lock_read(mapping->host, vma, + address); mutex_lock(&hugetlb_fault_mutex_table[hash]); goto out; } @@ -4534,6 +4441,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *mapping; int need_wait_lock = 0; unsigned long haddr = address & huge_page_mask(h); + bool hinode_locked; ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (ptep) { @@ -4552,7 +4460,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, } /* - * Acquire i_mmap_rwsem before calling huge_pte_alloc and hold + * Acquire hinode_rwsem before calling huge_pte_alloc and hold * until finished with ptep. This prevents huge_pmd_unshare from * being called elsewhere and making the ptep no longer valid. * @@ -4561,11 +4469,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * something has changed. */ mapping = vma->vm_file->f_mapping; - i_mmap_lock_read(mapping); + hinode_locked = hinode_lock_read(mapping->host, vma, address); ptep = huge_pte_alloc(mm, haddr, huge_page_size(h)); if (!ptep) { - i_mmap_unlock_read(mapping); - return VM_FAULT_OOM; + ret = VM_FAULT_OOM; + goto out_mutex; } /* @@ -4579,7 +4487,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, entry = huge_ptep_get(ptep); if (huge_pte_none(entry)) { - ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, flags); + ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, + flags, hinode_locked); goto out_mutex; } @@ -4661,7 +4570,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, } out_mutex: mutex_unlock(&hugetlb_fault_mutex_table[hash]); - i_mmap_unlock_read(mapping); + if (hinode_locked) + hinode_unlock_read(mapping->host); /* * Generally it's safe to hold refcount during waiting page lock. But * here we just wait to defer the next page fault to avoid busy loop and @@ -5002,6 +4912,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long pages = 0; bool shared_pmd = false; struct mmu_notifier_range range; + bool hinode_locked; /* * In the case of shared PMDs, the area to flush could be beyond @@ -5016,6 +4927,8 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, flush_cache_range(vma, range.start, range.end); mmu_notifier_invalidate_range_start(&range); + hinode_locked = hinode_lock_write(vma->vm_file->f_inode, vma, + range.start); i_mmap_lock_write(vma->vm_file->f_mapping); for (; address < end; address += huge_page_size(h)) { spinlock_t *ptl; @@ -5078,6 +4991,8 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, * See Documentation/vm/mmu_notifier.rst */ i_mmap_unlock_write(vma->vm_file->f_mapping); + if (hinode_locked) + hinode_unlock_write(vma->vm_file->f_inode); mmu_notifier_invalidate_range_end(&range); return pages << h->order; @@ -5327,16 +5242,11 @@ void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, * !shared pmd case because we can allocate the pmd later as well, it makes the * code much cleaner. * - * This routine must be called with i_mmap_rwsem held in at least read mode if - * sharing is possible. For hugetlbfs, this prevents removal of any page - * table entries associated with the address space. This is important as we - * are setting up sharing based on existing page table entries (mappings). - * - * NOTE: This routine is only called from huge_pte_alloc. Some callers of - * huge_pte_alloc know that sharing is not possible and do not take - * i_mmap_rwsem as a performance optimization. This is handled by the - * if !vma_shareable check at the beginning of the routine. i_mmap_rwsem is - * only required for subsequent processing. + * This must be called with hinode_rwsem held in read mode if sharing is + * possible. Otherwise, it could race with huge_pmd_unshare and the pte_t + * pointer could become invalid before being returned to the caller. Callers + * should use the helper routine hinode_lock_read() which will determine if + * sharing is possible and acquire the rwsem if necessary. */ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) { @@ -5352,8 +5262,9 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) if (!vma_shareable(vma, addr)) return (pte_t *)pmd_alloc(mm, pud, addr); + hinode_assert_locked(mapping); - i_mmap_assert_locked(mapping); + i_mmap_lock_read(mapping); vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) { if (svma == vma) continue; @@ -5383,6 +5294,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) spin_unlock(ptl); out: pte = (pte_t *)pmd_alloc(mm, pud, addr); + i_mmap_unlock_read(mapping); return pte; } @@ -5393,7 +5305,10 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud) * indicated by page_count > 1, unmap is achieved by clearing pud and * decrementing the ref count. If count == 1, the pte page is not shared. * - * Called with page table lock held and i_mmap_rwsem held in write mode. + * Called with page table lock held and hinode_rwsem held in write mode if + * sharing is possible. Callers should use the helper routine + * hinode_lock_write() which will determine if sharing is possible and acquire + * the rwsem if necessary. * * returns: 1 successfully unmapped a shared pte page * 0 the underlying pte page is not shared, or it is the last user @@ -5405,11 +5320,12 @@ int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, p4d_t *p4d = p4d_offset(pgd, *addr); pud_t *pud = pud_offset(p4d, *addr); - i_mmap_assert_write_locked(vma->vm_file->f_mapping); BUG_ON(page_count(virt_to_page(ptep)) == 0); if (page_count(virt_to_page(ptep)) == 1) return 0; + hinode_assert_write_locked(vma->vm_file->f_mapping); + pud_clear(pud); put_page(virt_to_page(ptep)); mm_dec_nr_pmds(mm); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index c0bb186bba62..593c109a3c80 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -992,7 +992,7 @@ static bool hwpoison_user_mappings(struct page *p, unsigned long pfn, enum ttu_flags ttu = TTU_IGNORE_MLOCK | TTU_IGNORE_ACCESS; struct address_space *mapping; LIST_HEAD(tokill); - bool unmap_success = true; + bool unmap_success; int kill = 1, forcekill; struct page *hpage = *hpagep; bool mlocked = PageMlocked(hpage); @@ -1054,31 +1054,19 @@ static bool hwpoison_user_mappings(struct page *p, unsigned long pfn, if (kill) collect_procs(hpage, &tokill, flags & MF_ACTION_REQUIRED); - if (!PageHuge(hpage)) { - unmap_success = try_to_unmap(hpage, ttu); - } else { + if (PageHuge(hpage) && !PageAnon(hpage) && mapping) { + bool hinode_locked; /* * For hugetlb pages, try_to_unmap could potentially call - * huge_pmd_unshare. Because of this, take semaphore in - * write mode here and set TTU_RMAP_LOCKED to indicate we - * have taken the lock at this higer level. - * - * Note that the call to hugetlb_page_mapping_lock_write - * is necessary even if mapping is already set. It handles - * ugliness of potentially having to drop page lock to obtain - * i_mmap_rwsem. + * huge_pmd_unshare. Because of this, take hinode_rwsem + * in write mode before calling. */ - mapping = hugetlb_page_mapping_lock_write(hpage); - - if (mapping) { - unmap_success = try_to_unmap(hpage, - ttu|TTU_RMAP_LOCKED); - i_mmap_unlock_write(mapping); - } else { - pr_info("Memory failure: %#lx: could not find mapping for mapped huge page\n", - pfn); - unmap_success = false; - } + hinode_locked = hinode_lock_write(mapping->host, NULL, 0UL); + unmap_success = try_to_unmap(hpage, ttu); + if (hinode_locked) + hinode_unlock_write(mapping->host); + } else { + unmap_success = try_to_unmap(hpage, ttu); } if (!unmap_success) pr_err("Memory failure: %#lx: failed to unmap page (mapcount=%d)\n", diff --git a/mm/memory.c b/mm/memory.c index c48f8df6e502..315d92bb68ff 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1465,9 +1465,14 @@ static void unmap_single_vma(struct mmu_gather *tlb, * safe to do nothing in this case. */ if (vma->vm_file) { + bool hinode_locked; + + hinode_locked = hinode_lock_write(vma->vm_file->f_inode, vma, 0UL); i_mmap_lock_write(vma->vm_file->f_mapping); __unmap_hugepage_range_final(tlb, vma, start, end, NULL); i_mmap_unlock_write(vma->vm_file->f_mapping); + if (hinode_locked) + hinode_unlock_write(vma->vm_file->f_inode); } } else unmap_page_range(tlb, vma, start, end, details); diff --git a/mm/migrate.c b/mm/migrate.c index 5ca5842df5db..a5685565cf1a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1280,7 +1280,6 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, int page_was_mapped = 0; struct page *new_hpage; struct anon_vma *anon_vma = NULL; - struct address_space *mapping = NULL; /* * Migratability of hugepages depends on architectures and their size. @@ -1328,36 +1327,33 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, goto put_anon; if (page_mapped(hpage)) { + struct address_space *mapping = NULL; + bool hinode_locked = false; + /* * try_to_unmap could potentially call huge_pmd_unshare. - * Because of this, take semaphore in write mode here and - * set TTU_RMAP_LOCKED to let lower levels know we have - * taken the lock. + * Take hinode_rwsem if sharing is possible. */ - mapping = hugetlb_page_mapping_lock_write(hpage); - if (unlikely(!mapping)) - goto unlock_put_anon; - + if (!PageAnon(hpage)) { + mapping = page_mapping(hpage); + if (mapping) + hinode_locked = hinode_lock_write(mapping->host, + NULL, 0UL); + } try_to_unmap(hpage, - TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS| - TTU_RMAP_LOCKED); + TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS); page_was_mapped = 1; - /* - * Leave mapping locked until after subsequent call to - * remove_migration_ptes() - */ + if (hinode_locked) + hinode_unlock_write(mapping->host); } if (!page_mapped(hpage)) rc = move_to_new_page(new_hpage, hpage, mode); - if (page_was_mapped) { + if (page_was_mapped) remove_migration_ptes(hpage, - rc == MIGRATEPAGE_SUCCESS ? new_hpage : hpage, true); - i_mmap_unlock_write(mapping); - } + rc == MIGRATEPAGE_SUCCESS ? new_hpage : hpage, false); -unlock_put_anon: unlock_page(new_hpage); put_anon: diff --git a/mm/rmap.c b/mm/rmap.c index 1b84945d655c..bb05ec810734 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -22,7 +22,8 @@ * * inode->i_mutex (while writing or truncating, not reading or faulting) * mm->mmap_lock - * page->flags PG_locked (lock_page) * (see huegtlbfs below) + * hugetlbfs inode->hinode_rwsem (hugetlbfs specific, see below) + * page->flags PG_locked (lock_page) * hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share) * mapping->i_mmap_rwsem * hugetlb_fault_mutex (hugetlbfs specific page fault mutex) @@ -45,10 +46,11 @@ * ->tasklist_lock * pte map lock * - * * hugetlbfs PageHuge() pages take locks in this order: - * mapping->i_mmap_rwsem - * hugetlb_fault_mutex (hugetlbfs specific page fault mutex) - * page->flags PG_locked (lock_page) + * hugetlbfs PageHuge() pages take locks in this order: + * hugetlbfs inode->hinode_rwsem + * mapping->i_mmap_rwsem + * hugetlb_fault_mutex ((hugetlbfs specific page fault mutex) + * page->flags PG_locked (NOT acquired with mapping->i_mmap_rwsem) */ #include @@ -1413,9 +1415,6 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, /* * If sharing is possible, start and end will be adjusted * accordingly. - * - * If called for a huge page, caller must hold i_mmap_rwsem - * in write mode as it is possible to call huge_pmd_unshare. */ adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end); @@ -1463,12 +1462,6 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, address = pvmw.address; if (PageHuge(page)) { - /* - * To call huge_pmd_unshare, i_mmap_rwsem must be - * held in write mode. Caller needs to explicitly - * do this outside rmap routines. - */ - VM_BUG_ON(!(flags & TTU_RMAP_LOCKED)); if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) { /* * huge_pmd_unshare unmapped an entire PMD diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 9a3d451402d7..b94101591027 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -220,6 +220,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, pgoff_t idx; u32 hash; struct address_space *mapping; + bool hinode_locked; /* * There is no default zero huge page for all huge page sizes as @@ -278,13 +279,14 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, BUG_ON(dst_addr >= dst_start + len); /* - * Serialize via i_mmap_rwsem and hugetlb_fault_mutex. - * i_mmap_rwsem ensures the dst_pte remains valid even + * Serialize via hinode_rwsem and hugetlb_fault_mutex. + * hinode_rwsem ensures the dst_pte remains valid even * in the case of shared pmds. fault mutex prevents * races with other faulting threads. */ mapping = dst_vma->vm_file->f_mapping; - i_mmap_lock_read(mapping); + hinode_locked = hinode_lock_read(mapping->host, dst_vma, + dst_addr); idx = linear_page_index(dst_vma, dst_addr); hash = hugetlb_fault_mutex_hash(mapping, idx); mutex_lock(&hugetlb_fault_mutex_table[hash]); @@ -293,7 +295,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, dst_pte = huge_pte_alloc(dst_mm, dst_addr, vma_hpagesize); if (!dst_pte) { mutex_unlock(&hugetlb_fault_mutex_table[hash]); - i_mmap_unlock_read(mapping); + if (hinode_locked) + hinode_unlock_read(mapping->host); goto out_unlock; } @@ -301,7 +304,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, dst_pteval = huge_ptep_get(dst_pte); if (!huge_pte_none(dst_pteval)) { mutex_unlock(&hugetlb_fault_mutex_table[hash]); - i_mmap_unlock_read(mapping); + if (hinode_locked) + hinode_unlock_read(mapping->host); goto out_unlock; } @@ -309,7 +313,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, dst_addr, src_addr, &page); mutex_unlock(&hugetlb_fault_mutex_table[hash]); - i_mmap_unlock_read(mapping); + if (hinode_locked) + hinode_unlock_read(mapping->host); vm_alloc_shared = vm_shared; cond_resched(); From patchwork Tue Nov 3 00:28:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 11875627 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12452697 for ; Tue, 3 Nov 2020 00:29:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 986E3223AC for ; Tue, 3 Nov 2020 00:29:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="qcGdrIHL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 986E3223AC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7A40C6B006C; Mon, 2 Nov 2020 19:29:05 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6F5376B006E; Mon, 2 Nov 2020 19:29:05 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 572BA6B006C; Mon, 2 Nov 2020 19:29:05 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0116.hostedemail.com [216.40.44.116]) by kanga.kvack.org (Postfix) with ESMTP id 2A2D76B005C for ; Mon, 2 Nov 2020 19:29:05 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B815A8249980 for ; Tue, 3 Nov 2020 00:29:04 +0000 (UTC) X-FDA: 77441222208.14.crook33_2b07f64272b4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 9B0ED18229818 for ; Tue, 3 Nov 2020 00:29:04 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,mike.kravetz@oracle.com,,RULES_HIT:30003:30012:30054:30064:30069,0,RBL:141.146.126.79:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100;04yfu4whu1okqe7ykpnmdq76y48esopuwt7abiorsuo3i47he75e4dx93pnmd4g.3ktbpf5ugxfgipc7jp79opn76p8ym65wujdnaefsbitpw1etyamwqy38dg981kp.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:68,LUA_SUMMARY:none X-HE-Tag: crook33_2b07f64272b4 X-Filterd-Recvd-Size: 8755 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Tue, 3 Nov 2020 00:29:03 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0A30SvUN088145; Tue, 3 Nov 2020 00:28:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2020-01-29; bh=FKqQw36FTOdMVzkcUWz8onJ82dve1+T6c2U3WfZd7e8=; b=qcGdrIHL8YLuByFfODBHdI16MREm5sXx9Dg+SR5HmpjQbrzmW9UEsbrrb4z5o5M//1Lg jaeTwM76R3HnrOCt2I2/kAsgE8snUV04FHrS+Ll18x3V8aUvMyiYpgunUX+Pkp6HFfJG sr2+zAe229gVk3ojuGGNl4CtHGWPbmLulUAjIzUA0YrplE70NvQsPaYv0zD4Z0vitHvG dssIXinP+XLfk0j/Z6QdsS9CGM2pqKoPrjJUn/3wvqJxvvPXqggT3Q/IQUUdTAIE5Rdx WXq9J+rIlUqjYMPqoTzIEp9XVUF6TxHSiQgc48oxET5r4RrAl0FdPqR6BlWIi+VftBfl gA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2130.oracle.com with ESMTP id 34hhb1xt6v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 03 Nov 2020 00:28:57 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 0A309foH086478; Tue, 3 Nov 2020 00:28:56 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userp3030.oracle.com with ESMTP id 34hvrutq2a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 03 Nov 2020 00:28:56 +0000 Received: from abhmp0015.oracle.com (abhmp0015.oracle.com [141.146.116.21]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id 0A30SsNn011245; Tue, 3 Nov 2020 00:28:55 GMT Received: from monkey.oracle.com (/50.38.35.18) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 02 Nov 2020 16:28:54 -0800 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , Naoya Horiguchi , Michal Hocko , "Aneesh Kumar K . V" , Andrea Arcangeli , "Kirill A . Shutemov" , Davidlohr Bueso , Prakash Sangappa , Andrew Morton , Mike Kravetz , stable@vger.kernel.org Subject: [PATCH 4/4] huegtlbfs: handle page fault/truncate races Date: Mon, 2 Nov 2020 16:28:41 -0800 Message-Id: <20201103002841.273161-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201103002841.273161-1-mike.kravetz@oracle.com> References: <20201026233150.371577-1-mike.kravetz@oracle.com> <20201103002841.273161-1-mike.kravetz@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9793 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 suspectscore=2 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011030000 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9793 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 suspectscore=2 clxscore=1015 mlxlogscore=999 impostorscore=0 malwarescore=0 lowpriorityscore=0 adultscore=0 spamscore=0 priorityscore=1501 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2011030001 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A huegtlb page fault can race with page truncation. Make the code identifying and handling these races more robust. Page fault handling needs to back out pages added to page cache beyond file size (i_size). When backing out the page, take care to restore reserve map entries and counts as necessary. File truncation (remove_inode_hugepages) needs to handle page mapping changes before locking the page. This could happen if page was added to page cache and later backed out in fault processing. Fixes 7bf91d39bb5 ("hugetlbfs: Use i_mmap_rwsem to address page fault/truncate race") Cc: Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 34 ++++++++++++++++++++-------------- mm/hugetlb.c | 40 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 58 insertions(+), 16 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index bc9979382a1e..6b975377558e 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -534,23 +534,29 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, lock_page(page); /* - * We must free the huge page and remove from page - * cache (remove_huge_page) BEFORE removing the - * region/reserve map (hugetlb_unreserve_pages). In - * rare out of memory conditions, removal of the - * region/reserve map could fail. Correspondingly, - * the subpool and global reserve usage count can need - * to be adjusted. + * After locking page, make sure mapping is the same. + * We could have raced with page fault populate and + * backout code. */ - VM_BUG_ON(PagePrivate(page)); - remove_huge_page(page); - freed++; - if (!truncate_op) { - if (unlikely(hugetlb_unreserve_pages(inode, + if (page_mapping(page) == mapping) { + /* + * We must free the huge page and remove from + * page cache (remove_huge_page) BEFORE + * removing the region/reserve map. In rare + * out of memory conditions, removal of the + * region/reserve map could fail and the + * subpool and global reserve usage count + * will need to be adjusted. + */ + VM_BUG_ON(PagePrivate(page)); + remove_huge_page(page); + freed++; + if (!truncate_op) { + if (unlikely(hugetlb_unreserve_pages(inode, index, index + 1, 1))) - hugetlb_fix_reserve_counts(inode); + hugetlb_fix_reserve_counts(inode); + } } - unlock_page(page); mutex_unlock(&hugetlb_fault_mutex_table[hash]); } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 957abc2d02ff..6b348d344f23 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4224,6 +4224,9 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, spinlock_t *ptl; unsigned long haddr = address & huge_page_mask(h); bool new_page = false; + bool page_cache = false; + bool reserve_alloc = false; + bool beyond_i_size = false; /* * Currently, we are forced to kill the process in the event the @@ -4311,6 +4314,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, clear_huge_page(page, address, pages_per_huge_page(h)); __SetPageUptodate(page); new_page = true; + if (PagePrivate(page)) + reserve_alloc = true; if (vma->vm_flags & VM_MAYSHARE) { int err = huge_add_to_page_cache(page, mapping, idx); @@ -4320,6 +4325,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, goto retry; goto out; } + page_cache = true; } else { lock_page(page); if (unlikely(anon_vma_prepare(vma))) { @@ -4358,8 +4364,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, ptl = huge_pte_lock(h, mm, ptep); size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) + if (idx >= size) { + beyond_i_size = true; goto backout; + } ret = 0; if (!huge_pte_none(huge_ptep_get(ptep))) @@ -4397,8 +4405,36 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, backout: spin_unlock(ptl); backout_unlocked: + if (new_page) { + if (page_cache && beyond_i_size) { + /* + * Back out pages added to page cache beyond i_size. + * Otherwise, they will 'sit' there until the file + * is removed. + */ + ClearPageDirty(page); + ClearPageUptodate(page); + delete_from_page_cache(page); + } + + if (reserve_alloc) { + /* + * If reserve was consumed, set PagePrivate so that + * it will be restored in free_huge_page(). + */ + SetPagePrivate(page); + } + + if (!beyond_i_size) { + /* + * Do not restore reserve map entries beyond i_size. + * there will be leaks when the file is removed. + */ + restore_reserve_on_error(h, vma, haddr, page); + } + + } unlock_page(page); - restore_reserve_on_error(h, vma, haddr, page); put_page(page); goto out; }