From patchwork Tue Dec 18 22:35:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 10736451 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C9A626C2 for ; Tue, 18 Dec 2018 22:36:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B7EB02AA1A for ; Tue, 18 Dec 2018 22:36:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AB00D2B05C; Tue, 18 Dec 2018 22:36:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC8932AA1A for ; Tue, 18 Dec 2018 22:36:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E54368E0001; Tue, 18 Dec 2018 17:36:18 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E01E98E000E; Tue, 18 Dec 2018 17:36:18 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB6E18E0001; Tue, 18 Dec 2018 17:36:18 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by kanga.kvack.org (Postfix) with ESMTP id 962FC8E0001 for ; Tue, 18 Dec 2018 17:36:18 -0500 (EST) Received: by mail-yb1-f200.google.com with SMTP id s18-v6so10700034ybm.16 for ; Tue, 18 Dec 2018 14:36:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=IuX36jtKJXAkPMUKIbK1MQXTpY0GwOAcxsWsjJtl40g=; b=qPH7RGfB+Vl6kOShWMNcmGfQKdMfvhuknmHuwz8KSeeBsBwhHd6F7IOrahBIsGU+uK 3Jr/wa24lW3jpMaoN09+AvMac6jcR0DXZtnRque/Au//Q1B/qyLo4HrQ/Fj1cz4Blt7s 2ec0yNiRvj8cAr7KbQy7GmCiE8Krw2vxB4NqzW4Qnom3anESZT/34/BTk0ERiDG9mwJB cJwLRloBXAR01A5qaWGUhi8Q8K52diPjr2zGM9TgPmrwTHwEY+Gfzc+ktTmIOaRYsreL zaob4I+0iwou9pGenDK+Q9qvkxwPLCtOxjC+lboThjXl2S7aunuNcUSn8xkBGZgBBONn kT+g== X-Gm-Message-State: AA+aEWYANJzH9E/a4EhgkjCMu8C165htZfypD5pMzS5ol+r+kcjtdbIE cePzCPipedUWT8s8EwgztC3cSCeNGPJJoMFaJiKbXWIXamRj3tbmcfW9a0JG1n27WhwuQyqM1vD fOgy4bS/WygreJfwrwzW7G1vkIr7LRpDPJcvrplgXsbN1iXkWx4q/ADq2VRK1jqr4mA== X-Received: by 2002:a25:9207:: with SMTP id b7mr19227288ybo.173.1545172578294; Tue, 18 Dec 2018 14:36:18 -0800 (PST) X-Google-Smtp-Source: AFSGD/Wwmx9BSfNWtSHfwvLxYRXdnnYsu7mSGpB9vPdD3eTPXBA790oVyIJM37D8NxHBimcBaPZq X-Received: by 2002:a25:9207:: with SMTP id b7mr19227248ybo.173.1545172577296; Tue, 18 Dec 2018 14:36:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545172577; cv=none; d=google.com; s=arc-20160816; b=Qy21PZlXh2HFRKLtDu3XzyPH4Euc+TC6iX8CrJeSfeakLAzF4OvnghUQ/aIpMj2St6 GBMIesFEd93SsE7t5d0X5NIzzwDSgzMtCuc+IgNV1vBjmIeNHuFC/V4rFz9amB72AAuU Mv0khXmZnJ5uMpwmz872Sir6aF9mzlVeVIVRnmH91d0m2b0GTgQB29fcu7qdUgK+hBzc 69KpLHTJtHel0q6Ph9ELelkbBl+F50MkRC3ROCsq+XEZcWO9ieF4hl1TfbnYRxd2hh0D I9USeXzIcgvRTqFu6ljJ1MIPLbWi51IemCk6b6/WcmsHP1pZ2buv/wnBC1qtuAEWH2Xz 2Kzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IuX36jtKJXAkPMUKIbK1MQXTpY0GwOAcxsWsjJtl40g=; b=ZezusCcefKf8wgk3LN8Ly1SDCJg5wY4LarIEZuw54ezH/6o+l4dv98hQ15NI5xwCAw 351lgOTZ53llD5hh4mmTrWGw8sP1lRQ7NpkVvlEJxijS7I+1XxQjDFbg73bvCEAYaeYW oV8OAPDXii+s/7qTlc06Q2JEhGsrw+BV2aXT2WnXUsZ00DCvSz0OuF9uu86rIodKNEIA ifVWSTIQeRY0apXW9cnKI4dTw6JKD4jfftCZoLwOrVe4H+LkROdZ5vjo/mXCQ8V8Cbd0 LxARoLpf2tpd4LhY+wKMTmv9ML2kMdukvmVNlci31Gv9PuzOSP1i/Oq87A13mvx8AKMM VYlA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=ptyJSaLc; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from userp2130.oracle.com (userp2130.oracle.com. [156.151.31.86]) by mx.google.com with ESMTPS id m81si10738396ybc.383.2018.12.18.14.36.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Dec 2018 14:36:17 -0800 (PST) Received-SPF: pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.86 as permitted sender) client-ip=156.151.31.86; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2018-07-02 header.b=ptyJSaLc; spf=pass (google.com: domain of mike.kravetz@oracle.com designates 156.151.31.86 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wBIMXqh6038564; Tue, 18 Dec 2018 22:36:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=IuX36jtKJXAkPMUKIbK1MQXTpY0GwOAcxsWsjJtl40g=; b=ptyJSaLcYAi6ziDxENUIv9Zemx15qU1A+ldPu3wVKAzfavX2qRNwJkGaKsW2+HY+tOEc YMP619ed6PvSBRfJWUr9OGqrBxbNxzrCEDMo+lW0/NTaH6JwkWIOikZVkisUM9xbEaRz ERiBTdhQ+4W0NYeZB087fEzYV3xEoKwI0SdXnQBmbGUgpHdmdWO0pIKy08UTEQhSkASD Dm6T3j+DTuJikl8gHsMBJDLCkZy75ni0xAzqbPT+65nUu91zAjg0XhHCeBA2MfPsnufT AomPrnApKF3IGpyk7yNXf0/P2Bf3IG7XzCT2q69jIbgtbkl9oiZClR/aMwOTGZ4ImT0g Qw== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2130.oracle.com with ESMTP id 2pcs1tp7c0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 18 Dec 2018 22:36:10 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wBIMa9bY000873 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 18 Dec 2018 22:36:10 GMT Received: from abhmp0019.oracle.com (abhmp0019.oracle.com [141.146.116.25]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id wBIMa9HL018736; Tue, 18 Dec 2018 22:36:09 GMT Received: from monkey.oracle.com (/50.38.38.67) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 18 Dec 2018 14:36:09 -0800 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Hugh Dickins , Naoya Horiguchi , "Aneesh Kumar K . V" , Andrea Arcangeli , "Kirill A . Shutemov" , Davidlohr Bueso , Prakash Sangappa , Andrew Morton , Mike Kravetz , stable@vger.kernel.org Subject: [PATCH v2 2/2] hugetlbfs: Use i_mmap_rwsem to fix page fault/truncate race Date: Tue, 18 Dec 2018 14:35:57 -0800 Message-Id: <20181218223557.5202-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181218223557.5202-1-mike.kravetz@oracle.com> References: <20181218223557.5202-1-mike.kravetz@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9111 signatures=668680 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812180184 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP hugetlbfs page faults can race with truncate and hole punch operations. Current code in the page fault path attempts to handle this by 'backing out' operations if we encounter the race. One obvious omission in the current code is removing a page newly added to the page cache. This is pretty straight forward to address, but there is a more subtle and difficult issue of backing out hugetlb reservations. To handle this correctly, the 'reservation state' before page allocation needs to be noted so that it can be properly backed out. There are four distinct possibilities for reservation state: shared/reserved, shared/no-resv, private/reserved and private/no-resv. Backing out a reservation may require memory allocation which could fail so that needs to be taken into account as well. Instead of writing the required complicated code for this rare occurrence, just eliminate the race. i_mmap_rwsem is now held in read mode for the duration of page fault processing. Hold i_mmap_rwsem longer in truncation and hold punch code to cover the call to remove_inode_hugepages. With this modification, code in remove_inode_hugepages checking for races becomes 'dead' as it can not longer happen. Remove the dead code and expand comments to explain reasoning. Similarly, checks for races with truncation in the page fault path can be simplified and removed. Cc: Fixes: ebed4bfc8da8 ("hugetlb: fix absurd HugePages_Rsvd") Signed-off-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 50 +++++++++++++++----------------------------- mm/hugetlb.c | 21 +++++++++---------- 2 files changed, 27 insertions(+), 44 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 32920a10100e..a9c00c6ef80d 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -383,17 +383,16 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end) * truncation is indicated by end of range being LLONG_MAX * In this case, we first scan the range and release found pages. * After releasing pages, hugetlb_unreserve_pages cleans up region/reserv - * maps and global counts. Page faults can not race with truncation - * in this routine. hugetlb_no_page() prevents page faults in the - * truncated range. It checks i_size before allocation, and again after - * with the page table lock for the page held. The same lock must be - * acquired to unmap a page. + * maps and global counts. * hole punch is indicated if end is not LLONG_MAX * In the hole punch case we scan the range and release found pages. * Only when releasing a page is the associated region/reserv map * deleted. The region/reserv map for ranges without associated - * pages are not modified. Page faults can race with hole punch. - * This is indicated if we find a mapped page. + * pages are not modified. + * + * Callers of this routine must hold the i_mmap_rwsem in write mode to prevent + * races with page faults. + * * Note: If the passed end of range value is beyond the end of file, but * not LLONG_MAX this routine still performs a hole punch operation. */ @@ -423,32 +422,14 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, for (i = 0; i < pagevec_count(&pvec); ++i) { struct page *page = pvec.pages[i]; - u32 hash; index = page->index; - hash = hugetlb_fault_mutex_hash(h, current->mm, - &pseudo_vma, - mapping, index, 0); - mutex_lock(&hugetlb_fault_mutex_table[hash]); - /* - * If page is mapped, it was faulted in after being - * unmapped in caller. Unmap (again) now after taking - * the fault mutex. The mutex will prevent faults - * until we finish removing the page. - * - * This race can only happen in the hole punch case. - * Getting here in a truncate operation is a bug. + * A mapped page is impossible as callers should unmap + * all references before calling. And, i_mmap_rwsem + * prevents the creation of additional mappings. */ - if (unlikely(page_mapped(page))) { - BUG_ON(truncate_op); - - i_mmap_lock_write(mapping); - hugetlb_vmdelete_list(&mapping->i_mmap, - index * pages_per_huge_page(h), - (index + 1) * pages_per_huge_page(h)); - i_mmap_unlock_write(mapping); - } + VM_BUG_ON(page_mapped(page)); lock_page(page); /* @@ -470,7 +451,6 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart, } unlock_page(page); - mutex_unlock(&hugetlb_fault_mutex_table[hash]); } huge_pagevec_release(&pvec); cond_resched(); @@ -505,8 +485,8 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset) i_mmap_lock_write(mapping); if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)) hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0); - i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, offset, LLONG_MAX); + i_mmap_unlock_write(mapping); return 0; } @@ -540,8 +520,8 @@ static long hugetlbfs_punch_hole(struct inode *inode, loff_t offset, loff_t len) hugetlb_vmdelete_list(&mapping->i_mmap, hole_start >> PAGE_SHIFT, hole_end >> PAGE_SHIFT); - i_mmap_unlock_write(mapping); remove_inode_hugepages(inode, hole_start, hole_end); + i_mmap_unlock_write(mapping); inode_unlock(inode); } @@ -624,7 +604,11 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, /* addr is the offset within the file (zero based) */ addr = index * hpage_size; - /* mutex taken here, fault path and hole punch */ + /* + * fault mutex taken here, protects against fault path + * and hole punch. inode_lock previously taken protects + * against truncation. + */ hash = hugetlb_fault_mutex_hash(h, mm, &pseudo_vma, mapping, index, addr); mutex_lock(&hugetlb_fault_mutex_table[hash]); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ab4c77b8c72c..25a0cd2f8b39 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3760,16 +3760,16 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, } /* - * Use page lock to guard against racing truncation - * before we get page_table_lock. + * We can not race with truncation due to holding i_mmap_rwsem. + * Check once here for faults beyond end of file. */ + size = i_size_read(mapping->host) >> huge_page_shift(h); + if (idx >= size) + goto out; + retry: page = find_lock_page(mapping, idx); if (!page) { - size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) - goto out; - /* * Check for page in userfault range */ @@ -3859,9 +3859,6 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, } ptl = huge_pte_lock(h, mm, ptep); - size = i_size_read(mapping->host) >> huge_page_shift(h); - if (idx >= size) - goto backout; ret = 0; if (!huge_pte_none(huge_ptep_get(ptep))) @@ -3964,8 +3961,10 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, /* * Acquire i_mmap_rwsem before calling huge_pte_alloc and hold - * until finished with ptep. This prevents huge_pmd_unshare from - * being called elsewhere and making the ptep no longer valid. + * until finished with ptep. This serves two purposes: + * 1) It prevents huge_pmd_unshare from being called elsewhere + * and making the ptep no longer valid. + * 2) It synchronizes us with file truncation. * * ptep could have already be assigned via huge_pte_offset. That * is OK, as huge_pte_alloc will return the same value unless