From patchwork Fri Nov 18 01:10:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 173F2C43217 for ; Fri, 18 Nov 2022 01:10:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA6558E0001; Thu, 17 Nov 2022 20:10:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B372A6B0073; Thu, 17 Nov 2022 20:10:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C5EA6B0078; Thu, 17 Nov 2022 20:10:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 672116B0071 for ; Thu, 17 Nov 2022 20:10:33 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 44E78140226 for ; Fri, 18 Nov 2022 01:10:33 +0000 (UTC) X-FDA: 80144782746.25.18746B1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id DCCFB40008 for ; Fri, 18 Nov 2022 01:10:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733832; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oUaUMokAqyi3NHrr8ZrIfiPSDBorB4zEKfPSOjIyLzs=; b=NCAnpiPt+tc1mIfjedss7udThNf5UmUQ5mc3nevJ92rWnn57VteJso03Kjkc7c8exkaROa MctIKYmmTGoj4rkbK9rQ1P0BMoDpFIn03m1gl/+ab9aM6FNEFkrvcD7Z+l9j7REK5c6coo V2oflIZ3tlA7ZcSz9gjDzKngbSWoPfo= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-648-5m0aTwnaM7CJ3k1LmRqjgA-1; Thu, 17 Nov 2022 20:10:31 -0500 X-MC-Unique: 5m0aTwnaM7CJ3k1LmRqjgA-1 Received: by mail-qk1-f198.google.com with SMTP id x2-20020a05620a448200b006fa7dad5c1cso4401217qkp.10 for ; Thu, 17 Nov 2022 17:10:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oUaUMokAqyi3NHrr8ZrIfiPSDBorB4zEKfPSOjIyLzs=; b=GcaXjTpfvB3n7GV3cFkP8RqSltkmHnic+wzw4P+nQG5vyPwiNGvskS1R0OzcGdkr0E YzimQ22Ify6rJT//wUhOtQ/OzDOtnW/Fdg2zsH5ynclN2Ah8VIy2Ahz2bmSEOFpitpNq NC5ghj8cE8MaiNFi/d8FtnU9rrR9qJ5urxRmXM8RBN3lNrG9cn2jHKE/Y1XY1VCQUccp xIduUB/rYIPlx1Z1ownuHpB8mxyM7DQiGbn7GrC9XBtMevHBA6OqPhXim74agOEMQbjH I/CXZI1Oh7eR76cw8uhsWZo4VdYY8E2TDCaRrhxRXO091nHKnbZImz6c0okOD6BI503J kwhg== X-Gm-Message-State: ANoB5pnLVKOiJXEWUxsHqCw1hKvAgISpTMlDCg2xRySJBvEB78ZMZyj8 fH//Htj2eSa9a6CqzhaOPyufVE1jAVmhB1UNfIBY+/6l5VXiU8BNbI2oIuv26antzGUdWqEuG9S fTCc6WDIVsGE6zLUhnrlRTLmod0fkHQviwKnMN2m2nOdpsVLQFuRrfwojEAq3 X-Received: by 2002:a05:620a:d41:b0:6fa:1af0:7c2a with SMTP id o1-20020a05620a0d4100b006fa1af07c2amr4043649qkl.777.1668733830622; Thu, 17 Nov 2022 17:10:30 -0800 (PST) X-Google-Smtp-Source: AA0mqf75PFHtqCG9q2KgS1yvrcaA+UfK7n2hAr9m/2KGhekXpy9/jpuVsMotL2vGdRkhVFniO41W+w== X-Received: by 2002:a05:620a:d41:b0:6fa:1af0:7c2a with SMTP id o1-20020a05620a0d4100b006fa1af07c2amr4043628qkl.777.1668733830341; Thu, 17 Nov 2022 17:10:30 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:29 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 01/12] mm/hugetlb: Let vma_offset_start() to return start Date: Thu, 17 Nov 2022 20:10:14 -0500 Message-Id: <20221118011025.2178986-2-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733832; a=rsa-sha256; cv=none; b=aMyC5DP1jSE0wknpCNUhdrvY21STe/3b44pr4tHzbgs2pUKaV2jYjP1Tcm7RcrDIJkn1lb 5P4Bnw3gG3XUW4nQBbA0k98EDi/sa6xQJkkZUTivUC4Uig5jcA0CH67yXyCpX85QtJEC4U LBNOM64r/YnVds3zv3r1BgOWgn8ujkw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NCAnpiPt; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733832; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oUaUMokAqyi3NHrr8ZrIfiPSDBorB4zEKfPSOjIyLzs=; b=aJt12UWd7AGuB3K6uDKG69jhdSwGTzQyVIvg4eIzeEPQjkcgOw4tDRvyNOj0cOH7qhDMWe SjEij/pVEPLJVXvN2h/D/i+3QHcNWTQJITtKkmtpZD2t5jGXZ4xO9hvgsZs8OmSKDP6XBZ 657l+SYVorxjA92/R+PTV59d0yQZU6E= X-Stat-Signature: tn1rrn1mzh4hqk6foxhzf86ojbnpmpcu X-Rspamd-Queue-Id: DCCFB40008 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NCAnpiPt; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1668733832-607617 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Even though vma_offset_start() is named like that, it's not returning "the start address of the range" but rather the offset we should use to offset the vma->vm_start address. Make it return the real value of the start vaddr, and it also helps for all the callers because whenever the retval is used, it'll be ultimately added into the vma->vm_start anyway, so it's better. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- fs/hugetlbfs/inode.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 790d2727141a..fdb16246f46e 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -412,10 +412,12 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, */ static unsigned long vma_offset_start(struct vm_area_struct *vma, pgoff_t start) { + unsigned long offset = 0; + if (vma->vm_pgoff < start) - return (start - vma->vm_pgoff) << PAGE_SHIFT; - else - return 0; + offset = (start - vma->vm_pgoff) << PAGE_SHIFT; + + return vma->vm_start + offset; } static unsigned long vma_offset_end(struct vm_area_struct *vma, pgoff_t end) @@ -457,7 +459,7 @@ static void hugetlb_unmap_file_folio(struct hstate *h, v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - if (!hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) + if (!hugetlb_vma_maps_page(vma, v_start, page)) continue; if (!hugetlb_vma_trylock_write(vma)) { @@ -473,8 +475,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h, break; } - unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, - NULL, ZAP_FLAG_DROP_MARKER); + unmap_hugepage_range(vma, v_start, v_end, NULL, + ZAP_FLAG_DROP_MARKER); hugetlb_vma_unlock_write(vma); } @@ -507,10 +509,9 @@ static void hugetlb_unmap_file_folio(struct hstate *h, */ v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - if (hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) - unmap_hugepage_range(vma, vma->vm_start + v_start, - v_end, NULL, - ZAP_FLAG_DROP_MARKER); + if (hugetlb_vma_maps_page(vma, v_start, page)) + unmap_hugepage_range(vma, v_start, v_end, NULL, + ZAP_FLAG_DROP_MARKER); kref_put(&vma_lock->refs, hugetlb_vma_lock_release); hugetlb_vma_unlock_write(vma); @@ -540,8 +541,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, - NULL, zap_flags); + unmap_hugepage_range(vma, v_start, v_end, NULL, zap_flags); /* * Note that vma lock only exists for shared/non-private From patchwork Fri Nov 18 01:10:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF758C4332F for ; Fri, 18 Nov 2022 01:10:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A2858E0002; Thu, 17 Nov 2022 20:10:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 652D46B0073; Thu, 17 Nov 2022 20:10:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42FEF8E0002; Thu, 17 Nov 2022 20:10:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 232D36B0071 for ; Thu, 17 Nov 2022 20:10:35 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E6C9C1A0D98 for ; Fri, 18 Nov 2022 01:10:34 +0000 (UTC) X-FDA: 80144782788.09.09827E0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 9988AC000A for ; Fri, 18 Nov 2022 01:10:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9OM954c43rVIe2ciAr/4Oap5Q/Q5UCXLA+pQIGS7ALA=; b=EIfjjx7NY+Fi1PVK+JkJnTH4LA5GnBP8uic1aMryn79BQidEjCZnhFLM13lb0xEimcvvoD MgjYKfs5A3MPtL/Skdwju5bWrNXX3eB28pgO2iBQdmF5OSYZtfE54ilZAGPsq4MEp0Fho+ r+/vS7nCRmd/qQfJN3eSnS2ZvuqxH9Y= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-207-63HhgjF_O0aLTzjxlYhqFw-1; Thu, 17 Nov 2022 20:10:33 -0500 X-MC-Unique: 63HhgjF_O0aLTzjxlYhqFw-1 Received: by mail-qk1-f199.google.com with SMTP id bj1-20020a05620a190100b006fa12a05188so4404640qkb.4 for ; Thu, 17 Nov 2022 17:10:32 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9OM954c43rVIe2ciAr/4Oap5Q/Q5UCXLA+pQIGS7ALA=; b=3+f2A2relQRGRzuJzvmn/DjW1A3XOitxDl3tsQJqdnSyqXetClFMK2K0/65VzrfhBV WsjWKwFSbxdmSHof5qQO1yykGj6JL+lLkEtfAuV5fGKd8SKlAu0DhENve/oyolNDd1Rq phRWwLWYrWYd3oUgq8kKI/VOAz0Q5q/vqpZCPAMxov0s/QrjeEZyNTxPXWuOLY1lFP8K Xse0QmJL1LU8bpeh/kWsoRvrePrL3grLuS2QCuQbYT/RiiM2wPQB3zlsR+b3plIw5UYn OjpuqzJqOxqzA6epsRVpzFyLi+peQoswzBUSGQvemK+4Pvv+OZ1eqhCqoJzZMQb9aHZp 8KGw== X-Gm-Message-State: ANoB5pnPaP25GfSYEN2NbzhU0CV/T8HwFWxNUFaBlYE93Q0a3Vfkg7KD XrlnjxUXIgt1Tx1jKBqBk47jyGMXCqTSUJUkDhyHNW+Yg+/bTs7abKL3emEs1pNU8quKbv7SALC 4nw2mX+pc7iJelI4oB6ZHekVv/XrgAFA2BME/z4uaAmyOPyem8a3C4efLxGcs X-Received: by 2002:a05:620a:260d:b0:6fb:a9af:2238 with SMTP id z13-20020a05620a260d00b006fba9af2238mr4174802qko.457.1668733831852; Thu, 17 Nov 2022 17:10:31 -0800 (PST) X-Google-Smtp-Source: AA0mqf5Aqw8l8m4fWogYc3FoWyNkgwxEBla8HpkXPW9zfRaGkpX2OEDEon4bGw7svTSEDlw3e0SH+w== X-Received: by 2002:a05:620a:260d:b0:6fb:a9af:2238 with SMTP id z13-20020a05620a260d00b006fba9af2238mr4174778qko.457.1668733831551; Thu, 17 Nov 2022 17:10:31 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:31 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 02/12] mm/hugetlb: Move swap entry handling into vma lock for fault Date: Thu, 17 Nov 2022 20:10:15 -0500 Message-Id: <20221118011025.2178986-3-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EIfjjx7N; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733834; a=rsa-sha256; cv=none; b=NY5IP0/MsPn9wevEavsZyjFzDk1eFI1i/vgp1D1o/HUVC9gL67xnwLGB7L690qUg61Q/K4 gpYEppiOzd4UEYu8G6+FFRjg9nYbkNQQW9iqMwPsDVsqiZfEKgojkKYNB6o30wALmx055z G7v8xuj5SqKSYSDHmYVRfJYjpbY5Tig= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9OM954c43rVIe2ciAr/4Oap5Q/Q5UCXLA+pQIGS7ALA=; b=PdO+81E/DStOBeNBXYnRvWvoeK+RF9ukZ9+F780I7MU2lFjBt86D9yubKnqdTp4gl8XRQT ZWz8bWsV2M3z2MBmzBTW6XzwHhSASxdPXe7rFcuv+qT+a/GM7vm5fl+aeB42Kf7kLpMuDx ZMzb8fb04DZMR9oWtu7l+Cqz9Qj4R90= Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EIfjjx7N; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam02 X-Rspam-User: X-Stat-Signature: wumojbaqdziokqrfxzjsm6jyb1xs6cpx X-Rspamd-Queue-Id: 9988AC000A X-HE-Tag: 1668733834-155478 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In hugetlb_fault(), there used to have a special path to handle swap entry at the entrance using huge_pte_offset(). That's unsafe because huge_pte_offset() for a pmd sharable range can access freed pgtables if without either the walker lock or vma lock. Here the simplest solution for making it safe is just to move the swap handling to be after the vma lock being held. We may need to take the fault mutex on either migration or hwpoison entries now (also the vma lock, but that's really needed), however neither of them is hot path so it should be fine. Signed-off-by: Peter Xu --- mm/hugetlb.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c3aab6d5b7aa..62ff3fc51d4e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5824,22 +5824,6 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, int need_wait_lock = 0; unsigned long haddr = address & huge_page_mask(h); - ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); - if (ptep) { - /* - * Since we hold no locks, ptep could be stale. That is - * OK as we are only making decisions based on content and - * not actually modifying content here. - */ - entry = huge_ptep_get(ptep); - if (unlikely(is_hugetlb_entry_migration(entry))) { - migration_entry_wait_huge(vma, ptep); - return 0; - } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) - return VM_FAULT_HWPOISON_LARGE | - VM_FAULT_SET_HINDEX(hstate_index(h)); - } - /* * Serialize hugepage allocation and instantiation, so that we don't * get spurious allocation failures if two CPUs race to instantiate @@ -5886,8 +5870,14 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * fault, and is_hugetlb_entry_(migration|hwpoisoned) check will * properly handle it. */ - if (!pte_present(entry)) + if (!pte_present(entry)) { + if (unlikely(is_hugetlb_entry_migration(entry))) + migration_entry_wait_huge(vma, ptep); + else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) + ret = VM_FAULT_HWPOISON_LARGE | + VM_FAULT_SET_HINDEX(hstate_index(h)); goto out_mutex; + } /* * If we are going to COW/unshare the mapping later, we examine the From patchwork Fri Nov 18 01:10:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3357EC433FE for ; Fri, 18 Nov 2022 01:10:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CA556B0071; Thu, 17 Nov 2022 20:10:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 679C56B0073; Thu, 17 Nov 2022 20:10:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F09C8E0006; Thu, 17 Nov 2022 20:10:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 39B578E0005 for ; Thu, 17 Nov 2022 20:10:38 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 03116804D8 for ; Fri, 18 Nov 2022 01:10:37 +0000 (UTC) X-FDA: 80144782956.25.6B84872 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id B547880006 for ; Fri, 18 Nov 2022 01:10:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U+Gtvi2x1OMFo8SBrSP3Pl5SiCw4KKSlLL8MzskCOZw=; b=baOfAKtrhbiKxjDPoeJnpmzl3rgMObZ3/3RbW8cuq5liAJNJOdPZ6yYUmX9NU5B9lOe40V n9WLw6+sCCoZdNX3QoISblZxD45imgFgVee7DlxO0OYPOcq8/m6n2iZs/ifGvio2f0Zsoi ZFHwSxxX3NJV/Ntg/8gui+FD+vxB/wM= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-137-KhJk9QM9NGqyGSTwDDB7vw-1; Thu, 17 Nov 2022 20:10:34 -0500 X-MC-Unique: KhJk9QM9NGqyGSTwDDB7vw-1 Received: by mail-qv1-f69.google.com with SMTP id on28-20020a056214449c00b004bbf12d7976so3235555qvb.18 for ; Thu, 17 Nov 2022 17:10:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U+Gtvi2x1OMFo8SBrSP3Pl5SiCw4KKSlLL8MzskCOZw=; b=ooLT2Z0bskZz2bpp6/fuuvn8rLLcBb02GvXPHV4kWEawEo191KjaUWZfEU3yjPTzQ5 9UMbRdBPM04+HLUTF8cHF+HPdPQWi1P4LjeqhHUJrBeSWgfpH/VldzfsNMWEdW66k1cm Awtluer5Z9z1tqBFPNCgK24/+KNy10zV/EzFmy9rDs4smmjWafbqmquS82OvWTxpPJJn DCN//p8nCGL6pGzsIdtJQrenoVH1ekOFYEbhuNXmr5RJtbvzpf9Ht49tMuFIqvWt8myJ jf7PnuqkQhJa3it3S8ea1ea64SjtxuXTGlKBCTx/fcedelPbz4e4jR1kZl5SGNuhqoLH QRJg== X-Gm-Message-State: ANoB5plV68lv2NFbk7ZyBRy7FCEFsz9ed/+IWRvsT8/1pHfZ0T8N7xTr x4Z1ZI8VXmdTeBWtBlxCqoWk6gfzdTs7q4jWDgMN6FF/oqgtDdwBjWoWiaWqxyXL6ELnvUkKA9M oYtBsU2cEx0HmvRh3K4wukSjJ6HjcKGn6fGvnTY/pH0ADGFgo7Vlm2N1rz6Y4 X-Received: by 2002:a05:6214:aab:b0:4bb:625c:e300 with SMTP id ew11-20020a0562140aab00b004bb625ce300mr4898438qvb.96.1668733833686; Thu, 17 Nov 2022 17:10:33 -0800 (PST) X-Google-Smtp-Source: AA0mqf6NCzhzda4Hbt+DXca7JaSEwUx4nLg4wsJGPVAjeHHodiz4M9R0WBhRFqNCAHbtCQKOIPJeFg== X-Received: by 2002:a05:6214:aab:b0:4bb:625c:e300 with SMTP id ew11-20020a0562140aab00b004bb625ce300mr4898416qvb.96.1668733833478; Thu, 17 Nov 2022 17:10:33 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:33 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 03/12] mm/hugetlb: Don't wait for migration entry during follow page Date: Thu, 17 Nov 2022 20:10:16 -0500 Message-Id: <20221118011025.2178986-4-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733836; a=rsa-sha256; cv=none; b=HIz8sN6F8G9z3dX2zktPxheXgOIHB4w3UHb5x3IGs3ZdHqs53/bHD1aGytAK3nSgfDMZFX slwLq5V0H5RvIS5YoSNfECkFtqjuVTdawac5lY2rtAqoLy4VVbaCDNOt2L8ZI6xvs8XCIW Kwbs8WqTVghnT2fj6w2ONSKYJJ8LpoQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=baOfAKtr; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733836; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U+Gtvi2x1OMFo8SBrSP3Pl5SiCw4KKSlLL8MzskCOZw=; b=dyZ1k3LX5i8o715DhWdWLIgSEsrnKrJougoLTkjpVG1SRLWn1L5nd53H5kHZgs1/DiKFE0 phIPPm/o3NZWQGDmGWqL4yluXDaQVmncgHCUd7jEOByBfKzsp20cH1Hv5K0SJyxHrWJORA OhzuABQhBon84aGc4iWWNR4HhJygPuM= Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=baOfAKtr; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com X-Rspam-User: X-Stat-Signature: q7k8c513xzmceheidt9yuohmucxdwmyc X-Rspamd-Queue-Id: B547880006 X-Rspamd-Server: rspam11 X-HE-Tag: 1668733836-725211 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: That's what the code does with !hugetlb pages, so we should logically do the same for hugetlb, so migration entry will also be treated as no page. This is probably also the last piece in follow_page code that may sleep, the last one should be removed in cf994dd8af27 ("mm/gup: remove FOLL_MIGRATION", 2022-11-16). Signed-off-by: Peter Xu --- mm/hugetlb.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 62ff3fc51d4e..61a1fa678172 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6222,7 +6222,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, if (WARN_ON_ONCE(flags & FOLL_PIN)) return NULL; -retry: pte = huge_pte_offset(mm, haddr, huge_page_size(h)); if (!pte) return NULL; @@ -6245,16 +6244,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, page = NULL; goto out; } - } else { - if (is_hugetlb_entry_migration(entry)) { - spin_unlock(ptl); - __migration_entry_wait_huge(pte, ptl); - goto retry; - } - /* - * hwpoisoned entry is treated as no_page_table in - * follow_page_mask(). - */ } out: spin_unlock(ptl); From patchwork Fri Nov 18 01:10:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DE47C43217 for ; Fri, 18 Nov 2022 01:10:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36C218E0003; Thu, 17 Nov 2022 20:10:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F70A6B0073; Thu, 17 Nov 2022 20:10:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16E7D8E0003; Thu, 17 Nov 2022 20:10:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0069E6B0071 for ; Thu, 17 Nov 2022 20:10:37 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D51091604E1 for ; Fri, 18 Nov 2022 01:10:37 +0000 (UTC) X-FDA: 80144782914.14.80733BA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 7E4718000B for ; Fri, 18 Nov 2022 01:10:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733837; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bpOw2twfKJ5Aty2Xl+QxZU1Og2jk2hUFVZDT66cF6Tk=; b=cJcCUpxeOeHM0GOyo3NtGXGU8EfEIhRfxSeRZJHL6Po//9LALlhZ2/jUJOx+e7HlHkq6Ug G3SguVdXch3LyEf4f19uzBRfh5m2S1FFvPR3uK+DSC/Wr+gbsDCEZTe6iCHzlgruG4t+Ca FeFHQFNbIQkV49eTQ4gTFOzcmHlxUGc= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-45-Z6ggdCvGMjSOTlEas9S1Bw-1; Thu, 17 Nov 2022 20:10:35 -0500 X-MC-Unique: Z6ggdCvGMjSOTlEas9S1Bw-1 Received: by mail-qk1-f199.google.com with SMTP id h13-20020a05620a244d00b006fb713618b8so4459958qkn.0 for ; Thu, 17 Nov 2022 17:10:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bpOw2twfKJ5Aty2Xl+QxZU1Og2jk2hUFVZDT66cF6Tk=; b=t9r4uGWiZw4QmyH+3ixb7/9sQQgFloAmJUIRizycVXl3I85zZClwgRIKBSoltfOUUl q6Ek06udx9vyvTewF6sXHMoh75YydlosiZQnutF3eiPlxuYiupCEjxDDkqvK8CovsqJp oSXfIRTgfSMUn7orPcDYIbex6xYTQn0wfNdcPI/c8ZPWTMBiybV69HzULHg+c7icegwR dOwevBwdJCLfzi9sty/H/cxTighU6+8zfXXazQos73FQ1PbQ9Hj7gwncBH4E+QFKQu25 bSU8ohRa5/upOaWB6qwvaG/ECQCj+o5IdFBTRQtLSKL1WWCOb799/79cphue9+TjASpg IAdw== X-Gm-Message-State: ANoB5pkRGDtzzQhlrt0gq71CnCuJuLDXmge651Q8J57y/DbraIwSb1/W P2qIeBYInXwFSllg6KntJ3l/HIIsT3QNo4ccnqWY4Q7RoUwrYwxJFrm64wPgDegP6HdlIOYV/L7 JS+NyaIYUPD0PhY3Lj2ZGOi6decWa9b7aDNAbKRhkZrT08EJsLnsZytcGuijs X-Received: by 2002:ac8:4e0c:0:b0:3a5:279d:dd34 with SMTP id c12-20020ac84e0c000000b003a5279ddd34mr4655000qtw.531.1668733835078; Thu, 17 Nov 2022 17:10:35 -0800 (PST) X-Google-Smtp-Source: AA0mqf7L/DTG7iZGWFpqDk5hsUhnfTTrsJsKX9hBQZCwDrkhBBpJLcBKb29jKauqxKgKWLHLgAsufg== X-Received: by 2002:ac8:4e0c:0:b0:3a5:279d:dd34 with SMTP id c12-20020ac84e0c000000b003a5279ddd34mr4654969qtw.531.1668733834702; Thu, 17 Nov 2022 17:10:34 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:34 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 04/12] mm/hugetlb: Add pgtable walker lock Date: Thu, 17 Nov 2022 20:10:17 -0500 Message-Id: <20221118011025.2178986-5-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733837; a=rsa-sha256; cv=none; b=hBoveURbz+4W2VuFKfIBpRf4B7PndpfvI8eS3E27Bz/vMR/yMd82P4QN4IrVOx6ZzOh6sC F4r1iJ7s5s9ZLp75RjjZ/JWdarEvCDJYCfqVgb5xsB7f5ZS5WOxqf3sZRiUIXCkWl4cjHu 1z7E+ZKTR9Zq85mGDp/C7v+1u9e46oU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cJcCUpxe; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733837; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bpOw2twfKJ5Aty2Xl+QxZU1Og2jk2hUFVZDT66cF6Tk=; b=cG1ASuD13IvGIQH6PvcewZkrEZFU/5IpjFew4Tg6AX5A7bgLeJrb5ytek8y8kVC+0eZoxj wwlCADR1Rk2qVx6tIbaB09IW9EkqcicqQwZMyDDQOGFo9558WlrudlLPjJQA3iq5Tm8PwV n5LCj/xb5JlxLoNKiFMzeMMMSELDRhM= X-Stat-Signature: dtkdgnhuygcay69jue3643n5gwwx5bxr X-Rspamd-Queue-Id: 7E4718000B Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cJcCUpxe; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1668733837-972528 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_pte_offset() is potentially a pgtable walker, looking up pte_t* for a hugetlb address. Normally, it's always safe to walk a generic pgtable as long as we're with the mmap lock held for either read or write, because that guarantees the pgtable pages will always be valid during the process. But it's not true for hugetlbfs, especially shared: hugetlbfs can have its pgtable freed by pmd unsharing, it means that even with mmap lock held for current mm, the PMD pgtable page can still go away from under us if pmd unsharing is possible during the walk. So we have three ways to make it safe: (1) If the mapping is private we're not prone to pmd sharing or unsharing, so it's okay. (2) If we're with the hugetlb vma lock held for either read/write, it's okay too because pmd unshare cannot happen at all. (3) Otherwise we may need the pgtable walker lock The pgtable walker is implemented in different ways depending on the config: - if !ARCH_WANT_HUGE_PMD_SHARE: it's no-op - else if MMU_GATHER_RCU_TABLE_FREE: it should be rcu_read_lock() - else: it should be local_irq_disable() A more efficient way to take the lock is only taking them with valid vmas that are possible to have pmd sharing (aka, want_pmd_share() returns true), but since the lock is mostly lightweighted (only riscv will use irq disable, x86 & arm will use rcu) just take them unconditionally for now. Document all these explicitly for huge_pte_offset() too, because it's not that obvious. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 99 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 551834cd5299..8f85ad0d5bdb 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -12,6 +12,8 @@ #include #include #include +#include +#include struct ctl_table; struct user_struct; @@ -24,6 +26,71 @@ typedef struct { unsigned long pd; } hugepd_t; #define __hugepd(x) ((hugepd_t) { (x) }) #endif +/** + * @hugetlb_walker_[un]lock(): protects software hugetlb pgtable walkers + * + * The hugetlb walker lock needs to be taken before huge_pte_offset() and + * needs to be released after finishing using the pte_t* returned. It's + * used to guarantee we won't access a freed pgtable page. Normally in + * this case we already have the mmap lock held for read. + * + * When holding the hugetlb walker lock, the thread cannot sleep, nor can + * it already in irq context (just to simplify unlock with no-save for + * !RCU_TABLE_TREE). Before the thread yields itself, it should release + * the pgtable lock. After the lock released, pte_t* can become invalid + * anytime so it cannot be accessed anymore. + * + * The only reason for the lock is to protect concurrent pmd unsharing + * followed by e.g. munmap() to drop the pgtable page. For no-pmd-sharing + * archs, it's no-op because it's always safe to access hugetlb pgtables + * just like generic pgtables. + */ +#if !defined(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) +static inline void hugetlb_walker_lock(void) +{ +} +static inline void hugetlb_walker_unlock(void) +{ +} +static inline bool __hugetlb_walker_locked(void) +{ + return true; +} +#elif defined(CONFIG_MMU_GATHER_RCU_TABLE_FREE) +static inline void hugetlb_walker_lock(void) +{ + rcu_read_lock(); +} +static inline void hugetlb_walker_unlock(void) +{ + rcu_read_unlock(); +} +static inline bool __hugetlb_walker_locked(void) +{ + return rcu_read_lock_held(); +} +#else +static inline void hugetlb_walker_lock(void) +{ + WARN_ON_ONCE(irqs_disabled()); + local_irq_disable(); +} +static inline void hugetlb_walker_unlock(void) +{ + local_irq_enable(); +} +static inline bool __hugetlb_walker_locked(void) +{ + return irqs_disabled(); +} +#endif + +#ifdef CONFIG_LOCKDEP +#define hugetlb_walker_locked() __hugetlb_walker_locked() +#else +#define hugetlb_walker_locked() true +#endif + #ifdef CONFIG_HUGETLB_PAGE #include @@ -192,6 +259,38 @@ extern struct list_head huge_boot_pages; pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz); +/* + * huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. + * Returns the pte_t* if found, or NULL if the address is not mapped. + * + * NOTE: since this function will walk all the pgtable pages (including not + * only high-level pgtable page, but also PUD entry that can be unshared + * concurrently for VM_SHARED), the caller of this function should be + * responsible of its thread safety. One can follow this rule: + * + * (1) For private mappings: pmd unsharing is not possible, so it'll + * always be safe if we're with the mmap sem for either read or write. + * This is normally always the case, IOW we don't need to do anything + * special. + * + * (2) For shared mappings: pmd unsharing is possible (so the PUD-ranged + * pgtable page can go away from under us! It can be done by a pmd + * unshare with a follow up munmap() on the other process), then we + * need either: + * + * (2.1) hugetlb vma lock read or write held, to make sure pmd unshare + * won't happen upon the range (it also makes sure the pte_t we + * read is the right and stable one), or, + * + * (2.2) pgtable walker lock, to make sure even pmd unsharing happened, + * the old shared PUD page won't get freed from under us, so even + * the pteval can be obsolete, at least it's still always safe to + * access the pgtable page (e.g., de-referencing pte_t* would not + * cause use-after-free). + * + * PS: from the regard of (2.2), it's the same logic of fast-gup being safe + * for generic mm when walking the pgtables even without mmap lock. + */ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz); unsigned long hugetlb_mask_last_page(struct hstate *h); From patchwork Fri Nov 18 01:10:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCF6EC4332F for ; Fri, 18 Nov 2022 01:10:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9116E8E0006; Thu, 17 Nov 2022 20:10:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C1C38E0005; Thu, 17 Nov 2022 20:10:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 714208E0006; Thu, 17 Nov 2022 20:10:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 632EC8E0005 for ; Thu, 17 Nov 2022 20:10:39 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 22F4CA0708 for ; Fri, 18 Nov 2022 01:10:39 +0000 (UTC) X-FDA: 80144782998.06.353B190 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id D4D84A000E for ; Fri, 18 Nov 2022 01:10:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733838; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U2GgktAyXhDRBB8fn5rbcYoK/H0Sll/0lbzArKYhQzY=; b=ApQeGQkmbSMVPsr+Xdpa/9MFmm/79+V0L+f94Hc/CYj0TXFGgSB9PhlKS2vxxgoWXiA+e6 Jr1SVbYaIudklgmSqTZ2tDIcT0693TUHc1kqi6HBQQWb52OvLMD3xHTMxhkuBjfK2zQbIR h9EsczfXbLd4Yai/AGva9x8AURGvGT4= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-146-WG-8xd0VMIWcnb4C2HMFLQ-1; Thu, 17 Nov 2022 20:10:37 -0500 X-MC-Unique: WG-8xd0VMIWcnb4C2HMFLQ-1 Received: by mail-qk1-f199.google.com with SMTP id q14-20020a05620a0d8e00b006ef0350dae1so4380221qkl.12 for ; Thu, 17 Nov 2022 17:10:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U2GgktAyXhDRBB8fn5rbcYoK/H0Sll/0lbzArKYhQzY=; b=RDZ0px2fkupxFDrVcpYxnrw5LApJl5bW9CGjwNd5w2e4Df4WInJqwqO3oYbWttEtL5 gAwbahxToxYiczuQDnPaFHhpa5HVdbFIONuHYIaQ8DD+oWodfN8DMxNlgXzq7k0bXkbU CIJpQa/dxDFmWgcLin8IwqzS5Wg3PW66ghpJYAqEKkI94TSz5t4qRERZlLi7+MHXxqPN GDujIwGmHUTLdxf0Gpjus6iydKETlX1hoDlujy0L+T9BzgLyX52+FhXkunPBfqtFGUKO nyk7izHI4bpg7vsXRzh71hxtNb2meUKjdFIwyEeKJMtXoGOR7CemS5t+a2twpSuc6EH7 S2Xg== X-Gm-Message-State: ANoB5pnXbJADiFPGRc2Hw8xgaTkHkZgF/WosiK2Bp/FRnHzF1nCw+WPD 1bUaQrxINBrtTnHRK3olL2qkvDH/35I8N2YBaTVHDk3qIXv1gFjs3fLAzcV/1mDe8A+6rYjwVC3 v/ZxvBAZ39u5NSCfSA81Qd0fmo4NOBK1MXpw3E+OeHtdpfo/nrl9AhF/+w+AJ X-Received: by 2002:ac8:48ce:0:b0:3a5:65ac:9c8a with SMTP id l14-20020ac848ce000000b003a565ac9c8amr4698806qtr.457.1668733836753; Thu, 17 Nov 2022 17:10:36 -0800 (PST) X-Google-Smtp-Source: AA0mqf5rnz6DDTlt9Ny28N2rVHhXYOuDlhdIfEgZs3xq/IpEsL+2rvvX9NavYJjdTwg6ClUXgreJUw== X-Received: by 2002:ac8:48ce:0:b0:3a5:65ac:9c8a with SMTP id l14-20020ac848ce000000b003a565ac9c8amr4698782qtr.457.1668733836471; Thu, 17 Nov 2022 17:10:36 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:35 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 05/12] mm/hugetlb: Make userfaultfd_huge_must_wait() safe to pmd unshare Date: Thu, 17 Nov 2022 20:10:18 -0500 Message-Id: <20221118011025.2178986-6-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733838; a=rsa-sha256; cv=none; b=TAzVCLdYVDLxP00zK5xV2RHzqqyuElOqSvgbD2zd4yJEA6A9KbwEQS4jxz19uyfRMh3V03 Ni/0CBbjQK1vnrq7GV4ud9gHC4/Ui+4osNfcamEVsmfz5Avh3O+rWINxFgkC+AD5TIaPWw A/eMln1xr32rNKtPDvD2NlxSbPO05I8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ApQeGQkm; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733838; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U2GgktAyXhDRBB8fn5rbcYoK/H0Sll/0lbzArKYhQzY=; b=e+q+VDp6lglVzRRLWt+CFs2P4Ri2ud6R83tN0PXXvjtArqPVBnSTzimbO7Vti/SMvqNxl7 W+DKiJ6qPFvRcS/vhu91IMcTBHqVLuicoHf+yk60HC4tFuQzSgGlQqJPHT3TKQZrDPCjqZ QYiSde1SGrMyaWbAFQqIcTNPTYwOYAo= X-Stat-Signature: cj7twnhfacg9za8ao9c7cckbr1cwy47j X-Rspamd-Queue-Id: D4D84A000E Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ApQeGQkm; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1668733838-804482 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can take the hugetlb walker lock, here taking vma lock directly. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 07c81ab3fd4d..a602f008dde5 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -376,7 +376,8 @@ static inline unsigned int userfaultfd_get_blocking_state(unsigned int flags) */ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) { - struct mm_struct *mm = vmf->vma->vm_mm; + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; struct userfaultfd_ctx *ctx; struct userfaultfd_wait_queue uwq; vm_fault_t ret = VM_FAULT_SIGBUS; @@ -403,7 +404,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) */ mmap_assert_locked(mm); - ctx = vmf->vma->vm_userfaultfd_ctx.ctx; + ctx = vma->vm_userfaultfd_ctx.ctx; if (!ctx) goto out; @@ -493,6 +494,13 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) blocking_state = userfaultfd_get_blocking_state(vmf->flags); + /* + * This stablizes pgtable for hugetlb on e.g. pmd unsharing. Need + * to be before setting current state. + */ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_lock_read(vma); + spin_lock_irq(&ctx->fault_pending_wqh.lock); /* * After the __add_wait_queue the uwq is visible to userland @@ -507,13 +515,15 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) set_current_state(blocking_state); spin_unlock_irq(&ctx->fault_pending_wqh.lock); - if (!is_vm_hugetlb_page(vmf->vma)) + if (!is_vm_hugetlb_page(vma)) must_wait = userfaultfd_must_wait(ctx, vmf->address, vmf->flags, reason); else - must_wait = userfaultfd_huge_must_wait(ctx, vmf->vma, + must_wait = userfaultfd_huge_must_wait(ctx, vma, vmf->address, vmf->flags, reason); + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_unlock_read(vma); mmap_read_unlock(mm); if (likely(must_wait && !READ_ONCE(ctx->released))) { From patchwork Fri Nov 18 01:10:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F651C433FE for ; Fri, 18 Nov 2022 01:10:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 149A18E0007; Thu, 17 Nov 2022 20:10:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FB1B8E0005; Thu, 17 Nov 2022 20:10:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF1EB8E0007; Thu, 17 Nov 2022 20:10:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C87D28E0005 for ; Thu, 17 Nov 2022 20:10:40 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A4071A0290 for ; Fri, 18 Nov 2022 01:10:40 +0000 (UTC) X-FDA: 80144783040.24.DDED779 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 4F8122000C for ; Fri, 18 Nov 2022 01:10:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ek0v3IZFacfv4QAamvdXXDg+rLS9iFnyxQu22HNSG2w=; b=cUNduIw9p51c1oOsfMI+bUWHghF9FF2t4a8jDrG+pEkKu6xK8I6ie26PDp0Dud4vEgODm+ r4YxtMFaUKQv1244x/zk0gNMdEQjG+hdabLjTZxjSsZ1/wp93fpxN1Spt6wcMFB2lIQoh3 d5FxLy9hJSkscqeA3Qtk5/2sAqXWFfo= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-267-cwQ99g8VN02nXnA_Ot8YCg-1; Thu, 17 Nov 2022 20:10:38 -0500 X-MC-Unique: cwQ99g8VN02nXnA_Ot8YCg-1 Received: by mail-qk1-f199.google.com with SMTP id w4-20020a05620a444400b006fa24b2f394so4362556qkp.15 for ; Thu, 17 Nov 2022 17:10:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ek0v3IZFacfv4QAamvdXXDg+rLS9iFnyxQu22HNSG2w=; b=CVIRLIxQKXvUUp3Kc6YU3etWzNp18sCNDhkrT6JsFey4pfpgPdcQLUASM8IJkTHyE+ twQVrQ+vEmySz9dR1MKSPHgJKpvxMjdHdhqwy2Ay12Y6fVMQT8vKuRcYumT4jrYjtVuu JooKXMOYOvYqte63fN5d66f0TdydjoF5bS0sIOYbPn2t8OsGrfFVPjL4gNUxW0sbEENd 44H3DgrHqlDlB4yazlR0yDRO5mPsd1W8gDMN8c2NDYHBGvKvCPtO53ggPdLtAOQv4MLp uG27NFrnASiWqhuOb217RIDy3HqsCPa1sKmTClmFOu6uSgCWjbQE3wtED3JizN9ALac7 +gCg== X-Gm-Message-State: ANoB5pkqWxRjrDjGRXOx8Wz1tukxGJWHMCKnpRPFhZtqFky3Cnd9q8k5 5oXOtt2LgBNsLw//3dTyvI3uJSCane4BAZ11gN7YJ4XzE23ALiDPTfAX5AUxQ4tDQwYJrCC66/z VC0vbkxR4cycXVlyfUhx1pprS+ti4L5+7QdU3I6e7jZT4aqWBwtgmZRcttySW X-Received: by 2002:ac8:7608:0:b0:3a5:6797:c57a with SMTP id t8-20020ac87608000000b003a56797c57amr4788577qtq.34.1668733838017; Thu, 17 Nov 2022 17:10:38 -0800 (PST) X-Google-Smtp-Source: AA0mqf4YUHL//KLcKTO3eXesn/AhPVyYy7y3vCnFwX7NxkFnL7BxzMVqhjH97L9Gr/XOknNuKwkO9w== X-Received: by 2002:ac8:7608:0:b0:3a5:6797:c57a with SMTP id t8-20020ac87608000000b003a56797c57amr4788551qtq.34.1668733837733; Thu, 17 Nov 2022 17:10:37 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:37 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 06/12] mm/hugetlb: Protect huge_pmd_share() with walker lock Date: Thu, 17 Nov 2022 20:10:19 -0500 Message-Id: <20221118011025.2178986-7-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cUNduIw9; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733840; a=rsa-sha256; cv=none; b=pjlx/U/hjAIhpPckIDAn8CJtqC6PjiJ/h3EOVT35UDHs4ontI5EQHiURxsvR29IkQpeMk3 lB0KMVcsqkVPBh8NtSGZjW14I+I483hdtUbL1lfc33Dw3YvuPZ2bM+pHu9VgJFP8O2MN0x wripCyIkeyEyD5wx+Gq07WcOef4LSlg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733840; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ek0v3IZFacfv4QAamvdXXDg+rLS9iFnyxQu22HNSG2w=; b=o1VuqzyKqu1zcRm4TnMDBaxcQzlYvsJlK2W/+mPUIpY8GrWfGNfxNXo/5fb+NmNAqjEQ8H +fPqi/oYiCH7JYrED1892Xze6zsZSku/u/L2EVjXBh5SeUoRuHfn+Hzjj9zK6u6Y2Rqice WNV49dus4KIOWxc85yOaPPPZC1Vek3w= Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cUNduIw9; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam02 X-Rspam-User: X-Stat-Signature: pe7zzzex1dhgetehpa64mw3jbduhksje X-Rspamd-Queue-Id: 4F8122000C X-HE-Tag: 1668733840-233843 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_pmd_share() is normally called with vma lock held already, it lets us feel like we don't really need the walker lock. But that's not true, because we only took the vma lock for "current" vma, but not all the vma pgtables we're going to walk upon. Taking each vma lock may lead to deadlock and hard to order. The safe approach is just to use the walker lock which guarantees the pgtable page being alive, then we should use get_page_unless_zero() rather than get_page() only, to make sure the pgtable page is not being freed at the meantime. Signed-off-by: Peter Xu --- mm/hugetlb.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 61a1fa678172..5ef883184885 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -7008,6 +7008,13 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, spinlock_t *ptl; i_mmap_lock_read(mapping); + + /* + * NOTE: even if we've got the vma read lock, here we still need to + * take the walker lock, because we're not walking the current vma, + * but some other mm's! + */ + hugetlb_walker_lock(); vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) { if (svma == vma) continue; @@ -7016,12 +7023,15 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, if (saddr) { spte = huge_pte_offset(svma->vm_mm, saddr, vma_mmu_pagesize(svma)); - if (spte) { - get_page(virt_to_page(spte)); + /* + * When page ref==0, it means it's probably being + * freed; continue with the next one. + */ + if (spte && get_page_unless_zero(virt_to_page(spte))) break; - } } } + hugetlb_walker_unlock(); if (!spte) goto out; From patchwork Fri Nov 18 01:10:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9F35C4332F for ; Fri, 18 Nov 2022 01:10:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C495D8E0008; Thu, 17 Nov 2022 20:10:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B34D98E0005; Thu, 17 Nov 2022 20:10:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 910618E0008; Thu, 17 Nov 2022 20:10:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 742F58E0005 for ; Thu, 17 Nov 2022 20:10:42 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 47E9181206 for ; Fri, 18 Nov 2022 01:10:42 +0000 (UTC) X-FDA: 80144783124.09.C946769 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id BEF06A0012 for ; Fri, 18 Nov 2022 01:10:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733841; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M2c5rXQnd1kf7yTmDBDfaoawKYfDXFaD9281dnoZiZE=; b=D/ZZJvo5cGIr+uUmeTgrRoZ0Ej9vbFPHSteEslytmFJOZyUHrbzacAJkje03m0a5a47Nhp +XDxndNjO/PCTUR6Yfso5ZHufW3VXigcEQJQ6kTAXx/qtKGUQt5FgHRP/HYlP3gcZQ5GOF GYAbE0FIVOfVk1JN6J1sSYH1bwjDIQs= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-581-0DT8o0baOPSZJesFdGPnOQ-1; Thu, 17 Nov 2022 20:10:40 -0500 X-MC-Unique: 0DT8o0baOPSZJesFdGPnOQ-1 Received: by mail-qv1-f72.google.com with SMTP id mo15-20020a056214330f00b004b96d712bccso3255308qvb.22 for ; Thu, 17 Nov 2022 17:10:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M2c5rXQnd1kf7yTmDBDfaoawKYfDXFaD9281dnoZiZE=; b=MYfcWhWZqwkdzwPSXJ9phkvcaEEXfH1yTcWIGEGA6lKkA8yorbDxdeCL7laqLwjldu gIWP5irmrDZMqrF8phLSdlJtl7UPLyqA+CjREbPkhQDYGQ2p3eoeQPD+TY231TmM7BSs Gu84qq+FDC9g7iThlmoCXFc1w3GfxoJdfylzmlD3DvcRnHjF2RfFuW7+YFQfHM+prJkg jk02bw06GjRq17OpVkx/FWg841Z/qtNskXT0WLFaYEsUA1Fs97/VAdK1c6VWDvRan15b Ev5TrmgXjmt6AAX5i9staphIPTyNdUYwUMdFrEIk/6h+SQMropKCXqI6T+s7qiQ3H1Q1 Bqiw== X-Gm-Message-State: ANoB5plrc6cWIpzz56+WqPwHAevHLtJ2dKIIHHUPoRanmPWx6CyAsk1D Db4AilVzaMzfDQR+Uf8eXIRhCccZ2WIQEZvJsK9kUNGN/ucaNLGHFdeDwbtsqw9dXQPQHtbKVdP 6T8hYeUMCK0Auc/P8+fyj9jKPPbxr3pn0A03hoYnaKtIc/zIityq3RK0CZuuz X-Received: by 2002:a05:620a:10a3:b0:6fa:156e:44c0 with SMTP id h3-20020a05620a10a300b006fa156e44c0mr4093375qkk.293.1668733839464; Thu, 17 Nov 2022 17:10:39 -0800 (PST) X-Google-Smtp-Source: AA0mqf5KpYFuSckUVSGa4lgr9LFNowF9nCCUTnXJySXZCtSwHy4gRXenmckPtcF9nzTIBjvTUthQDw== X-Received: by 2002:a05:620a:10a3:b0:6fa:156e:44c0 with SMTP id h3-20020a05620a10a300b006fa156e44c0mr4093352qkk.293.1668733839139; Thu, 17 Nov 2022 17:10:39 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:38 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 07/12] mm/hugetlb: Use hugetlb walker lock in hugetlb_follow_page_mask() Date: Thu, 17 Nov 2022 20:10:20 -0500 Message-Id: <20221118011025.2178986-8-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733841; a=rsa-sha256; cv=none; b=q+lv8Xlwqa2rfO2WkE6Egw5CxMvRbLNFccNJ7re1ibC1LjP7CRxUZ4fPSvm+W9mMxXTkel 3A55KeOS9qolai/+paoVeih26CpqZj0u88xZUZzExcMVDKp0diOYijHuieVfOdvalHM3yy rl9edW8ORt6CkAh2eVqKAcu0e8zDzKU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="D/ZZJvo5"; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733841; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M2c5rXQnd1kf7yTmDBDfaoawKYfDXFaD9281dnoZiZE=; b=AR0uuIn7/W/z5vjS44w9G8IcyBYI6yuf/PijSnE8r1g4pUfP9VNestj7BE4psGqSqVkXXx cmD6uApCJ1vDIMGmcc/86D0ZKjbjeoX+asJHGvktFyHK76UeykfgaIFb78kgQZL9mhO8zz kFF6U3cb/v1CxqcTbqFBqOdzreCpZOM= X-Stat-Signature: sa5j7r9popxbj45krbmosn8n3c35ghg8 X-Rspamd-Queue-Id: BEF06A0012 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="D/ZZJvo5"; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam04 X-Rspam-User: X-HE-Tag: 1668733841-122053 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hugetlb walker lock makes sure the pte_t* won't go away from under us. Signed-off-by: Peter Xu --- mm/hugetlb.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5ef883184885..fc49e3ca8acd 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6222,9 +6222,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, if (WARN_ON_ONCE(flags & FOLL_PIN)) return NULL; + hugetlb_walker_lock(); pte = huge_pte_offset(mm, haddr, huge_page_size(h)); if (!pte) - return NULL; + goto out_unlock; ptl = huge_pte_lock(h, mm, pte); entry = huge_ptep_get(pte); @@ -6247,6 +6248,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, } out: spin_unlock(ptl); +out_unlock: + hugetlb_walker_unlock(); return page; } From patchwork Fri Nov 18 01:10:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E01FDC43217 for ; Fri, 18 Nov 2022 01:10:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73F868E0009; Thu, 17 Nov 2022 20:10:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 651528E0005; Thu, 17 Nov 2022 20:10:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3923F8E0009; Thu, 17 Nov 2022 20:10:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 240D58E0005 for ; Thu, 17 Nov 2022 20:10:44 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id ED35B1202F1 for ; Fri, 18 Nov 2022 01:10:43 +0000 (UTC) X-FDA: 80144783166.26.26894B2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id A853220008 for ; Fri, 18 Nov 2022 01:10:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733843; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qjnChz0h9yEvsDVybPnS+PQNkUKEm7fVo3IYCZg/FQ4=; b=g1Vpboepen4xbDn22+iuycZ5V1QEIwfomXwZs+IbsweuK5PGyQGONzXprAIvVj4Pu+X2MQ gFhC8uW0zrDKBpoad5fQffJgNOzSDcRqzTYPYcF1Zlwi3rFsWoVeZ4F2fBUP+SG+karOOC dafZzC8vfFbm+VjhDOxtO5WR5Hp9iVA= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-570-YwwT6QYPP_uLWAwQH8uxwQ-1; Thu, 17 Nov 2022 20:10:41 -0500 X-MC-Unique: YwwT6QYPP_uLWAwQH8uxwQ-1 Received: by mail-qt1-f198.google.com with SMTP id y19-20020a05622a121300b003a526e0ff9bso3387348qtx.15 for ; Thu, 17 Nov 2022 17:10:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qjnChz0h9yEvsDVybPnS+PQNkUKEm7fVo3IYCZg/FQ4=; b=4aGeFl3Yx1Pgo1l85QQDxfzJ6kOoaKdZPiPY0z43f5u/DobCs68eRIeRHpD0U82ibV uUh3f8HPgkl4zxch81rFsyLP/hYRcHOegH1i4coGsu2n5X8+IxCrwMEMMQ7kylQuhgSP XFrJEgIbD+/TPLCH14dbIYFVCYZ49ZYQuDHWtY7W4A509ilybQn2YwR2VY9S333UB84j yYH0xCC4lP1pkUlSq8SI7f5+vCmJEqkmx7KIjbq2KKa39D1XTGYz8003UDyhGpt0Bx1o gzjp6jDTdnAZRpTrgui4iXiXuhG74vTNSL+R4ElNCrVBFdwf5F3qqxb+LcBQf7L4YX69 p46w== X-Gm-Message-State: ANoB5ploweYqH+dWH/OUgJFAYZwW+PJZNRdhEqruA6lZMalB1K8ALNJM cbCyt7WX9xdcj7NGl8mDQZjxjUMiNWXNX1AVTbVOvfiDFRxeJoUT8x2S4zfE27ctkukTe3fwZQR +NECb3fVfTz3oQVaR8rm6nue6yYVlMm1rcEf4uixFGboM8w2PuFltfzQa1RjA X-Received: by 2002:a05:620a:c04:b0:6cf:7988:75a9 with SMTP id l4-20020a05620a0c0400b006cf798875a9mr4033260qki.613.1668733841079; Thu, 17 Nov 2022 17:10:41 -0800 (PST) X-Google-Smtp-Source: AA0mqf54cqseGLR98vZZaTQtgNM/mQInkImtMr4ZFy3PyyVl91SKAmCECTcH2P+eiXPqv9u2h3rDQw== X-Received: by 2002:a05:620a:c04:b0:6cf:7988:75a9 with SMTP id l4-20020a05620a0c0400b006cf798875a9mr4033233qki.613.1668733840789; Thu, 17 Nov 2022 17:10:40 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:40 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 08/12] mm/hugetlb: Use hugetlb walker lock in follow_hugetlb_page() Date: Thu, 17 Nov 2022 20:10:21 -0500 Message-Id: <20221118011025.2178986-9-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733843; a=rsa-sha256; cv=none; b=19i+7As+TOHHZAKKrkasx8fdfXmKfSBgRFyC1vLyZFiuHwTCCaZ7r2tBCSxbgQDflii8Pp TVSJdo1J9ciUdzKwekcqvq/CFv2D9yWQaX1ct/9eoTGLLrD440TY3RNjlaDeYv8CbfEuOy WmwQP3ss5AP/WNMeMdPxrpisU0ZKjLA= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=g1Vpboep; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733843; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qjnChz0h9yEvsDVybPnS+PQNkUKEm7fVo3IYCZg/FQ4=; b=sW1OW+ISM2Nd8nYKWmCDzPXU9m+lzcPkr2aVEmpg791EYU/cGkLohp+j/XGQjZErK4GBSd X80iNnfGQJROvF8R86qnP8+K/nVVHIo/Bknwz8uL8fc4lJQjxu0eUVWIgxEaNd3Nj3aCpW XVT0K2N3c4sXZgKoyNJerk3hdJ9pIFo= X-Stat-Signature: ydai7i4ycnr8wcb9cgobqfns6aw7hj6i X-Rspamd-Queue-Id: A853220008 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=g1Vpboep; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1668733843-182570 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hugetlb walker lock makes sure the pte_t* won't go away from under us. Some trick is used to release the walker lock slightly earlier when we found present pte. Signed-off-by: Peter Xu --- mm/hugetlb.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fc49e3ca8acd..e81af6a46c59 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6280,6 +6280,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, break; } + hugetlb_walker_lock(); /* * Some archs (sparc64, sh*) have multiple pte_ts to * each hugepage. We have to make sure we get the @@ -6304,6 +6305,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, !hugetlbfs_pagecache_present(h, vma, vaddr)) { if (pte) spin_unlock(ptl); + hugetlb_walker_unlock(); remainder = 0; break; } @@ -6325,6 +6327,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, if (pte) spin_unlock(ptl); + hugetlb_walker_unlock(); + if (flags & FOLL_WRITE) fault_flags |= FAULT_FLAG_WRITE; else if (unshare) @@ -6367,6 +6371,15 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, continue; } + /* + * When reach here, it means the pteval is not absent, so + * anyone who wants to free and invalidate the pgtable page + * (aka, pte*) should need to first unmap the entries which + * relies on the pgtable lock. Since we're holding it, + * we're safe even without the walker lock anymore. + */ + hugetlb_walker_unlock(); + pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; page = pte_page(huge_ptep_get(pte)); From patchwork Fri Nov 18 01:10:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B473C433FE for ; Fri, 18 Nov 2022 01:10:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A9B058E000A; Thu, 17 Nov 2022 20:10:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A22178E0005; Thu, 17 Nov 2022 20:10:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 825198E000A; Thu, 17 Nov 2022 20:10:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6F6788E0005 for ; Thu, 17 Nov 2022 20:10:45 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4C491A084E for ; Fri, 18 Nov 2022 01:10:45 +0000 (UTC) X-FDA: 80144783250.13.85AC971 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id E84D0180003 for ; Fri, 18 Nov 2022 01:10:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733844; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ap7neQhploF5O2byy6eWIJyWsli20vMRANWIX9Kzu5I=; b=P1IX7psOa/NnYfYPqPJ1kd126SeNdbqEdLG3F67xKHFwx82gdpYlvjqYnUVIxhuiv7MJPj fa80cpwvPLKMbRwzghTDILBdNXnT0yTkhBMIBhHW9MBmyniD/aIcafM7KrdltGg0Cu3/14 IxWVERb6Lf3sVcFuqRx73T8dkoExFLw= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-594-sEibE29kM5es_afH7Tm_Og-1; Thu, 17 Nov 2022 20:10:43 -0500 X-MC-Unique: sEibE29kM5es_afH7Tm_Og-1 Received: by mail-qt1-f200.google.com with SMTP id f4-20020a05622a114400b003a57f828277so3424377qty.22 for ; Thu, 17 Nov 2022 17:10:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ap7neQhploF5O2byy6eWIJyWsli20vMRANWIX9Kzu5I=; b=U720Rk7tQXk2EZep4ylwTAZ0CZxk9BpNn4mkofKGh6UdT2RNRaHaQr1xwMFr+A9bRB rlBgArjCrfsWaUBjUokA6KTF4NXeNZFdlzSuOvozT11pc9DSbZPchxhqz3aJn1ZarIN2 qQmSOWKjbunPd/QIrLiq0QDivooTyHJSRZEy6KBtDzQss0xhji2VO4g/Qg0i2a+mFSus 71DM3c4D3QfAsWYuJ9HdP8O+gi93ANfN4I/lK2ld/XpPwWkGdf+msWA1iLn2Ela+ND5Z ozrZXTccvt/iyHSV4faEJvDaFcMTD0NnVe+1xzoj8GG7ui0JXxNFuBq2FlyfMMxDGKK+ 6HOw== X-Gm-Message-State: ANoB5pldM3DwrR+gdvhhsdypcoUtkEziG+nItNitIy9GkGttSfNfyUwq RAfeykrDCo+dghDxyR2pFwsrATkQRv6CXarqXIDITGoLE+XKAN2MmPG789RVPm29fM2/y1K5fPY g/orjbc/TaQ+5KgdrUdyUBTIDZA13hEFXGawVIhUHDfWRxqw/nVK7BEoWyLey X-Received: by 2002:a0c:c682:0:b0:4c6:80d9:844d with SMTP id d2-20020a0cc682000000b004c680d9844dmr4921265qvj.34.1668733842432; Thu, 17 Nov 2022 17:10:42 -0800 (PST) X-Google-Smtp-Source: AA0mqf5ri6W+Ys/uE6Yihgvm71dHXQ64Qx2xPRu/QHx/egaYRiWuxzZqmcFd9on3+WOH893ouf3hGw== X-Received: by 2002:a0c:c682:0:b0:4c6:80d9:844d with SMTP id d2-20020a0cc682000000b004c680d9844dmr4921236qvj.34.1668733842116; Thu, 17 Nov 2022 17:10:42 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:41 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 09/12] mm/hugetlb: Use hugetlb walker lock in hugetlb_vma_maps_page() Date: Thu, 17 Nov 2022 20:10:22 -0500 Message-Id: <20221118011025.2178986-10-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733845; a=rsa-sha256; cv=none; b=pIpm7NTjERExPH16U3/SaqPMROS/o8itCR8OXxrVLf6XOJxMMKJzzy+A+16QWkOPz9dM7Y siUa49WBHFjppD62O4cN7dK7V8/cPDP7f1IJ+C5/LuS/PU0hSWeoDW4Q8hVOwxipLCYnID QjotQplytLj0IwSo0ntYW/Mlzjp0bJY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P1IX7psO; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733845; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ap7neQhploF5O2byy6eWIJyWsli20vMRANWIX9Kzu5I=; b=TVkuGywXH1MVttN86XmsbONXzdaqIPIrqNCWKs8rQdBjajjU2SF7VTzxp4DydEb+qcok+F DQmd+NDQf1r90SV08g8aKD4v9WLKs2k4xPap0zytv72ch0h0dqehKOuNGRgHGT+3emXOEr KxEezkC8bLOKAi9aPKDveTKhhJ9zrCk= Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=P1IX7psO; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com X-Rspam-User: X-Stat-Signature: qnwb8omo3mgciu4n5oozryzh7m8mg5pd X-Rspamd-Queue-Id: E84D0180003 X-Rspamd-Server: rspam11 X-HE-Tag: 1668733844-118745 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hugetlb walker lock makes sure the pte_t* won't go away from under us. Signed-off-by: Peter Xu --- fs/hugetlbfs/inode.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index fdb16246f46e..265508981ba1 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -387,21 +387,24 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, unsigned long addr, struct page *page) { pte_t *ptep, pte; + bool result = false; + hugetlb_walker_lock(); ptep = huge_pte_offset(vma->vm_mm, addr, huge_page_size(hstate_vma(vma))); if (!ptep) - return false; + goto out; pte = huge_ptep_get(ptep); if (huge_pte_none(pte) || !pte_present(pte)) - return false; + goto out; if (pte_page(pte) == page) - return true; - - return false; + result = true; +out: + hugetlb_walker_unlock(); + return result; } /* From patchwork Fri Nov 18 01:10:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8D07C433FE for ; Fri, 18 Nov 2022 01:10:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 762DC8E000B; Thu, 17 Nov 2022 20:10:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7137B8E0005; Thu, 17 Nov 2022 20:10:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B43F8E000B; Thu, 17 Nov 2022 20:10:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4B0278E0005 for ; Thu, 17 Nov 2022 20:10:50 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1F5B2160669 for ; Fri, 18 Nov 2022 01:10:50 +0000 (UTC) X-FDA: 80144783460.13.E8763EA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 3FE311C0006 for ; Fri, 18 Nov 2022 01:10:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xhz/Ckbg8bcdF0GpIUPbn4oL7x6YrFJTnL8UpEBPxx4=; b=PkuZEJ3dctm5sHPy0w3qwC0AzBlz0/AQ9WX3wc93hLJEG5SCXOGffUq1Ha3UeEMmECWGuu 2VCWYFqs5VN+nqQzMmsKJrPXt7ZRQpgoTN/6hldrqld9TzgRbIlCoEbb4OCJXm15wQ3VIj xH3e9cLr1Tpc7aqCXWlSWONPCCUZG70= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-608--cmmcWxxNNC1vRCzE1gENA-1; Thu, 17 Nov 2022 20:10:46 -0500 X-MC-Unique: -cmmcWxxNNC1vRCzE1gENA-1 Received: by mail-qk1-f199.google.com with SMTP id q14-20020a05620a0d8e00b006ef0350dae1so4380610qkl.12 for ; Thu, 17 Nov 2022 17:10:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xhz/Ckbg8bcdF0GpIUPbn4oL7x6YrFJTnL8UpEBPxx4=; b=4EV9TEbfjrITR4oOZaEv4d4LBcvurSYEF0eQYuUptJnSCyRddbm91v49lUoIGA9PNj rl/HyzYhhnojyI3CzHIYG8m5EXeyKrAfQ7PP1NO5ttE3GWaUhJqW8nAkUWP2+U6PQQ3B ZEEXq5T/5WKDCVSi6B4XziYu0gzhuq6P/iGZF0IcIjhqrU3yGQ7NvIlX4cVWiOeIdIDf rpOl3hBvphuPH+gekrO4Hwg6GYrtY0bs837It2b95v/IEJGfdfnVJHOdgiKEdiAzI4Dm OVMp+uRGqItvov95dL6aPATCbQGD6/xRk9768idip/68lF3XQBuDU+NBsx95BUeAsTTj 1j6Q== X-Gm-Message-State: ANoB5plMvpX3pYH55L8SGZv8ktMqw0hLePiRNnMfUMyyx2MVFy49pHSX swUAAz+8Dzdm+VfG0UXTSTg9+KgKMMIkFzHoP0rgoaI0o17V+5xvxQDcS5hb1vmBHjIJxXGR7ta v75s+9JnYYOlstLNYBrXStYvUQPUrG0qayJdMAT7jtoA+Lpk2VUtjAy4O8v0e X-Received: by 2002:a05:620a:a10:b0:6fb:a7e6:96e with SMTP id i16-20020a05620a0a1000b006fba7e6096emr3968838qka.675.1668733844123; Thu, 17 Nov 2022 17:10:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf7dJcHqBiR7Zv4m+XJwPOWgqJH93kZp3F2AOeTKPKZTLrUWczK3rZnxkV5Lt7km6veLvhXYzg== X-Received: by 2002:a05:620a:a10:b0:6fb:a7e6:96e with SMTP id i16-20020a05620a0a1000b006fba7e6096emr3968816qka.675.1668733843810; Thu, 17 Nov 2022 17:10:43 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:43 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 10/12] mm/hugetlb: Use hugetlb walker lock in walk_hugetlb_range() Date: Thu, 17 Nov 2022 20:10:23 -0500 Message-Id: <20221118011025.2178986-11-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733849; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xhz/Ckbg8bcdF0GpIUPbn4oL7x6YrFJTnL8UpEBPxx4=; b=6GgEpyR7a+Bl6DTclrZMd4QE4MG42Wn2c5YSXsJpKmtD5QqNU7rLhLcl/laC2/Mi2BHpE/ KaGEqCwUPL4hHhlR8lJcuKy82ElpMrfasPNcMyhPYRF6wLpsOnb/HkcU68QsiNS7pWuycf tZPHtFAoEJYZDLWOpgqckMfr3GVkcPQ= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PkuZEJ3d; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733849; a=rsa-sha256; cv=none; b=eI1fCgvanOyPj7miBPDHTGYXL75CS5HJbVHxqKi7GNr9n0Mvm8mtX2MMYxmtZ8PiNMpOph MSps1ur0chxpWPVV3m3RNWLcnJNmPAcUV95h1COdDkKSUlCIwS7G3+zBao+mmQ7fJK4JfI W3ZQ73EyvbC3pqKv9h2BeXUfZi+dK84= X-Rspam-User: X-Stat-Signature: smdn9ipa5u84cx4q67fjt6w8ysfs6b3x X-Rspamd-Queue-Id: 3FE311C0006 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PkuZEJ3d; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam07 X-HE-Tag: 1668733848-896725 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hugetlb walker lock makes sure the pte_t* won't go away from under us. One thing to mention is there're two hugetlb_entry() users that can yield the thread within hugetlb_entry(), that'll need to add unlock/lock pair around the yield, meanwhile document hugetlb_entry() explaining the lock for sleepable hugetlb_entry()s. Signed-off-by: Peter Xu --- arch/s390/mm/gmap.c | 2 ++ fs/proc/task_mmu.c | 2 ++ include/linux/pagewalk.h | 9 ++++++++- mm/pagewalk.c | 2 ++ 4 files changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 02d15c8dc92e..fb2938e8d1c7 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2644,7 +2644,9 @@ static int __s390_enable_skey_hugetlb(pte_t *pte, unsigned long addr, end = start + HPAGE_SIZE - 1; __storage_key_init_range(start, end); set_bit(PG_arch_1, &page->flags); + hugetlb_walker_unlock(); cond_resched(); + hugetlb_walker_lock(); return 0; } diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 89338950afd3..ed750a52e60b 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1612,7 +1612,9 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, frame++; } + hugetlb_walker_unlock(); cond_resched(); + hugetlb_walker_lock(); return err; } diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index 959f52e5867d..7fa3724c6eb5 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -21,7 +21,14 @@ struct mm_walk; * depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD. * Any folded depths (where PTRS_PER_P?D is equal to 1) * are skipped. - * @hugetlb_entry: if set, called for each hugetlb entry + * @hugetlb_entry: if set, called for each hugetlb entry. Note that + * each pte_t* is protected by hugetlb_walker_lock(), + * and the lock does not allow sleep. If explicit + * sleep in the entry fn needed, the caller needs to + * release the lock (hugetlb_walker_unlock()), then + * relock it (hugetlb_walker_lock()) before return. + * After the unlock, the pte_t* may become invalid + * anytime so cannot be accessed anymore. * @test_walk: caller specific callback function to determine whether * we walk over the current vma or not. Returning 0 means * "do page table walk over the current vma", returning diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 7f1c9b274906..abf310011ab1 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -302,6 +302,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, const struct mm_walk_ops *ops = walk->ops; int err = 0; + hugetlb_walker_lock(); do { next = hugetlb_entry_end(h, addr, end); pte = huge_pte_offset(walk->mm, addr & hmask, sz); @@ -314,6 +315,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, if (err) break; } while (addr = next, addr != end); + hugetlb_walker_unlock(); return err; } From patchwork Fri Nov 18 01:11:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A170C43219 for ; Fri, 18 Nov 2022 01:11:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 391C66B0071; Thu, 17 Nov 2022 20:11:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 342CB6B0072; Thu, 17 Nov 2022 20:11:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20AF16B0073; Thu, 17 Nov 2022 20:11:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 10A2A6B0071 for ; Thu, 17 Nov 2022 20:11:38 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E164DA0290 for ; Fri, 18 Nov 2022 01:11:37 +0000 (UTC) X-FDA: 80144785434.10.67F28F3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 832EC1C0008 for ; Fri, 18 Nov 2022 01:11:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xcizgw09qxh5kxz+rR5aBntzCJEVxf2HEif8HyYLI/s=; b=PqdIUPDEcyT8Kjc18rxTtRuMgUDPHSkdOKrG66FljHRPfP4BkiywRvvXo05onknEZujc8T gf62WEk4iFu5fX732wrSgL/tVw/gcQTjjw+YTHhGMUAusXNYolqPNNQqAJS4YeBobXSRK8 K184uYRl6ers5OWq+xSFhU9NG0C3Q4c= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-356-KC71_G_OMuuvQPKs-HZ5dA-1; Thu, 17 Nov 2022 20:11:35 -0500 X-MC-Unique: KC71_G_OMuuvQPKs-HZ5dA-1 Received: by mail-qv1-f69.google.com with SMTP id nk7-20020a056214350700b004c68c912c93so2425624qvb.16 for ; Thu, 17 Nov 2022 17:11:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xcizgw09qxh5kxz+rR5aBntzCJEVxf2HEif8HyYLI/s=; b=EBvVLLP+WtATAdhCP09Pr3eE1vNbPyAFRo2XM0qZPi6lwE5w8VSlgTzIE0NE4v8bbv oxtZq6c/yvco7x6igZDNKND369M2SJncYIOrH3z2SaGEueG0wG/OZLyatZCzYHpx6iXV O65i5yG7cxgttfB9RK3tOUCCheLfcdEpQlipwcrouOmBt38TQtEhp9rXDiqNkCZN8v6f QN9agReBMDsVkewG6XL4DMi21rkKDojNZPCHVUNtTmnw27kp9h+XsRmTQxjKaLZtPUtT bap8DJVtL+gadMtRZhwMbJvYEZStEp+gq1rsha5ouOCJnR991lCmfmnWIWztp9+NAyPN sfNg== X-Gm-Message-State: ANoB5pl184V0pBC6JEbyvIzW2igGQIFWEvpYxinnPmv4PGPjtT8WjJGy wPhC9afCoi8PQuZya/oQAjxx/IbLy2hbIA4HG75AYIgiH+fcYZv4z+2HKPh4p0rxl0zBXMDdJ20 knCH4gWy5VSA= X-Received: by 2002:a0c:b3db:0:b0:4b3:e8bc:b06d with SMTP id b27-20020a0cb3db000000b004b3e8bcb06dmr4998972qvf.72.1668733895401; Thu, 17 Nov 2022 17:11:35 -0800 (PST) X-Google-Smtp-Source: AA0mqf5zY9gm/9Sq/rUG10fAy37Oz9ULSDk0Pq8azqj/JTmW8KGkcU5gXTaCvi5BCEs8EdrHiJsuRQ== X-Received: by 2002:a0c:b3db:0:b0:4b3:e8bc:b06d with SMTP id b27-20020a0cb3db000000b004b3e8bcb06dmr4998963qvf.72.1668733895220; Thu, 17 Nov 2022 17:11:35 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id i5-20020a05620a404500b006fb11eee465sm1507162qko.64.2022.11.17.17.11.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:11:34 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , Muchun Song , peterx@redhat.com, Nadav Amit , Miaohe Lin , James Houghton , David Hildenbrand , Andrew Morton , Rik van Riel , Andrea Arcangeli Subject: [PATCH RFC v2 11/12] mm/hugetlb: Use hugetlb walker lock in page_vma_mapped_walk() Date: Thu, 17 Nov 2022 20:11:33 -0500 Message-Id: <20221118011133.2179145-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733897; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xcizgw09qxh5kxz+rR5aBntzCJEVxf2HEif8HyYLI/s=; b=KYCmGJlocewuyr5LhWLbUnxT4VPA/rxnnMa49Nz6/kYyEp0Gb4Voe0nktefM5APedpJMl7 OxVgFCloyDlQDeEWKeudGxp7Po7hWJx2wxDAJVptJ6npVNoLR8cojaaDVPnoCD4a561oYW YByBPDpoN1Dyot3nHgPZm4q/WpiGbno= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PqdIUPDE; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733897; a=rsa-sha256; cv=none; b=YFBuR08GBcjvCU0Mwb27++OgOdSz3zUp310T1Bm84soZbOtw1Amo+FF/RuJIohFZSSi3VE mp1zOQ27P4nR01/fBTE100GNlyLoIFtfPIbByDy+y1E26UkSaP4N5F+tPK/SPVabbcaS4l nw2SxBoy14NE8NrFTl1LPnm0tEzpjjY= X-Rspam-User: X-Stat-Signature: 8ce3aq15h4gieo6g8i6a6hi8ec5npa1g X-Rspamd-Queue-Id: 832EC1C0008 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=PqdIUPDE; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam07 X-HE-Tag: 1668733897-308305 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hugetlb walker lock makes sure the pte_t* won't go away from under us. Signed-off-by: Peter Xu --- include/linux/rmap.h | 4 ++++ mm/page_vma_mapped.c | 5 ++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 011a7530dc76..94d25a67db2b 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -13,6 +13,7 @@ #include #include #include +#include /* * The anon_vma heads a list of private "related" vmas, to scan if @@ -408,6 +409,9 @@ static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw) pte_unmap(pvmw->pte); if (pvmw->ptl) spin_unlock(pvmw->ptl); + /* This needs to be after unlock of the spinlock */ + if (is_vm_hugetlb_page(pvmw->vma)) + hugetlb_walker_unlock(); } bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw); diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 93e13fc17d3c..5ac8a89130f6 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -169,10 +169,13 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->pte) return not_found(pvmw); + hugetlb_walker_lock(); /* when pud is not present, pte will be NULL */ pvmw->pte = huge_pte_offset(mm, pvmw->address, size); - if (!pvmw->pte) + if (!pvmw->pte) { + hugetlb_walker_unlock(); return false; + } pvmw->ptl = huge_pte_lock(hstate, mm, pvmw->pte); if (!check_pte(pvmw)) From patchwork Fri Nov 18 01:11:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99A92C433FE for ; Fri, 18 Nov 2022 01:11:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B91F8E0002; Thu, 17 Nov 2022 20:11:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 369028E0001; Thu, 17 Nov 2022 20:11:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1BC688E0002; Thu, 17 Nov 2022 20:11:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0C0078E0001 for ; Thu, 17 Nov 2022 20:11:43 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D11AE80657 for ; Fri, 18 Nov 2022 01:11:42 +0000 (UTC) X-FDA: 80144785644.20.222D6F0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 8A9C5140009 for ; Fri, 18 Nov 2022 01:11:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733902; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qhNG9bVlD1FFKnsASjagzPbRT3g9EE2vJHNL4cDHL60=; b=XXwlpIgiXKHQhGhXbUeRI8LOqZ7v0gDgYnlLNRmZKK35Jc38HBnaTxC4QZqT9ImKOtX+K5 Fgl4VkL0FsUFK7pllk7ADqDtN48clTb14FgO2mvg5jntbbVZvKWtGL0mKNkDXk82hhG72y FnMts/Rk9YCzorXzCGT+sxp5zxhnAII= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-539-HZ-Y2CwCNj6MMPdWPRtZhw-1; Thu, 17 Nov 2022 20:11:38 -0500 X-MC-Unique: HZ-Y2CwCNj6MMPdWPRtZhw-1 Received: by mail-qv1-f71.google.com with SMTP id on28-20020a056214449c00b004bbf12d7976so3237927qvb.18 for ; Thu, 17 Nov 2022 17:11:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qhNG9bVlD1FFKnsASjagzPbRT3g9EE2vJHNL4cDHL60=; b=F7zNHshIpfijJmiEH1gY8+QD+dnZET7ENugO2PKREovICEuuZb2NfSeqKRDnIPDM4K +GtN5I6BekHMUkYrVKFdKE7ruaI8jvgtCw7v0bOzX+VJz0RCkoxna0jYS+BZvEzidkIK nn2WYN7bqj2lBA0sUCaLQkf7i2dtf6UhXgKnjAVxKdRggiUfivVS63/4EYeQwOYmvbnK CAp7mXctFTyGTCxabcpTJhSNX39eXl7ocmOHP5Kyog2h2hhcAnZ7K8EC3cImnTzxBQJt rbHrRN20pRx7UT3dfTeOd4V5WCUT1BcMVUENvsq8+lB/PuzZztojGAtSVuEtSJ5Vw71d Lxvg== X-Gm-Message-State: ANoB5pllxdYlDN1QVXvi8i2B1pR1sOJ0xMd1hOVGVK9r05tnwzmu/EDq uAAD7dEGDF+uVmHCyA9bSBFpxIWcDXF/eWxZUmlfBsE/UwZ/nVWUFMPPu5t0slYbMRCVpoc9FTc L+aIk7fSNkRI= X-Received: by 2002:a05:620a:1345:b0:6f8:cdc2:b7a2 with SMTP id c5-20020a05620a134500b006f8cdc2b7a2mr4025130qkl.132.1668733897219; Thu, 17 Nov 2022 17:11:37 -0800 (PST) X-Google-Smtp-Source: AA0mqf4uP5nsobxUnrOU2Lysw6mCLTzR3h7mZxUqm52XhJt+bnVXiO5fslhFD3Bo224uvm+vlLV2vg== X-Received: by 2002:a05:620a:1345:b0:6f8:cdc2:b7a2 with SMTP id c5-20020a05620a134500b006f8cdc2b7a2mr4025109qkl.132.1668733896876; Thu, 17 Nov 2022 17:11:36 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id l15-20020a05620a28cf00b006bb2cd2f6d1sm1462185qkp.127.2022.11.17.17.11.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:11:36 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , Muchun Song , peterx@redhat.com, Nadav Amit , Miaohe Lin , James Houghton , David Hildenbrand , Andrew Morton , Rik van Riel , Andrea Arcangeli Subject: [PATCH RFC v2 12/12] mm/hugetlb: Introduce hugetlb_walk() Date: Thu, 17 Nov 2022 20:11:35 -0500 Message-Id: <20221118011135.2179157-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733902; a=rsa-sha256; cv=none; b=lKAVlfB64YJd94ezYKt1dsko2u04UCkXPP64/gWXwqpFJlWbxH+WoXyaNaV0p2yVamP7xJ fyHG3v6FM6Dm/RVJXWDQtedGiOJf9dqIxpf9BQ9N/zpRODAG5oHm07BONdGBRjvdhNjTEX 2FJUChK1kxRstxQeYQqeMKFQ6nr/wg8= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XXwlpIgi; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733902; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qhNG9bVlD1FFKnsASjagzPbRT3g9EE2vJHNL4cDHL60=; b=QCBL5DkRGYGXr/xEHUIwkkeRQ48uK6cO0zmlgIC8St3v6jSVBSyrOalinY125mulYKpCs5 B1PFHqfV3KpOWyUu2/091aoy9qrQ5MlI4euAW2b41LP1a0lZbUZBDmw5Pf6PvNccgq7mPb u+tOJKduf25TryBV5NFcllVmKW8Jv0c= X-Stat-Signature: j6zu8cfkmb4ch9c5p3f4pckypdrbcs1h X-Rspamd-Queue-Id: 8A9C5140009 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XXwlpIgi; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1668733902-37362 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_pte_offset() is the main walker function for hugetlb pgtables. The name is not really representing what it does, though. Instead of renaming it, introduce a wrapper function called hugetlb_walk() which will use huge_pte_offset() inside. Assert on the locks so when walking the pgtable. Note, the vma lock assertion will be a no-op for private mappings. Signed-off-by: Peter Xu --- fs/hugetlbfs/inode.c | 4 +--- fs/userfaultfd.c | 6 ++---- include/linux/hugetlb.h | 13 +++++++++++++ mm/hugetlb.c | 28 ++++++++++++++-------------- mm/page_vma_mapped.c | 2 +- mm/pagewalk.c | 4 +--- 6 files changed, 32 insertions(+), 25 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 265508981ba1..ed7934015290 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -390,9 +390,7 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, bool result = false; hugetlb_walker_lock(); - ptep = huge_pte_offset(vma->vm_mm, addr, - huge_page_size(hstate_vma(vma))); - + ptep = hugetlb_walk(vma, addr, huge_page_size(hstate_vma(vma))); if (!ptep) goto out; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index a602f008dde5..f31fe1a9f4c5 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -237,14 +237,12 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, unsigned long flags, unsigned long reason) { - struct mm_struct *mm = ctx->mm; pte_t *ptep, pte; bool ret = true; - mmap_assert_locked(mm); - - ptep = huge_pte_offset(mm, address, vma_mmu_pagesize(vma)); + mmap_assert_locked(ctx->mm); + ptep = hugetlb_walk(vma, address, vma_mmu_pagesize(vma)); if (!ptep) goto out; diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 8f85ad0d5bdb..e75b1ffb93a5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1296,4 +1296,17 @@ bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr); #define flush_hugetlb_tlb_range(vma, addr, end) flush_tlb_range(vma, addr, end) #endif +/* + * Safe version of huge_pte_offset() to check the locks. See comments + * above huge_pte_offset(). + */ +static inline pte_t * +hugetlb_walk(struct vm_area_struct *vma, unsigned long addr, unsigned long sz) +{ + if (!hugetlb_walker_locked()) + hugetlb_vma_assert_locked(vma); + + return huge_pte_offset(vma->vm_mm, addr, sz); +} + #endif /* _LINUX_HUGETLB_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e81af6a46c59..6c77ae7a3d94 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4814,7 +4814,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } else { /* * For shared mappings the vma lock must be held before - * calling huge_pte_offset in the src vma. Otherwise, the + * calling hugetlb_walk() in the src vma. Otherwise, the * returned ptep could go away if part of a shared pmd and * another thread calls huge_pmd_unshare. */ @@ -4824,7 +4824,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, last_addr_mask = hugetlb_mask_last_page(h); for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; - src_pte = huge_pte_offset(src, addr, sz); + src_pte = hugetlb_walk(src_vma, addr, sz); if (!src_pte) { addr |= last_addr_mask; continue; @@ -5028,7 +5028,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, hugetlb_vma_lock_write(vma); i_mmap_lock_write(mapping); for (; old_addr < old_end; old_addr += sz, new_addr += sz) { - src_pte = huge_pte_offset(mm, old_addr, sz); + src_pte = hugetlb_walk(vma, old_addr, sz); if (!src_pte) { old_addr |= last_addr_mask; new_addr |= last_addr_mask; @@ -5091,7 +5091,7 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct last_addr_mask = hugetlb_mask_last_page(h); address = start; for (; address < end; address += sz) { - ptep = huge_pte_offset(mm, address, sz); + ptep = hugetlb_walk(vma, address, sz); if (!ptep) { address |= last_addr_mask; continue; @@ -5404,7 +5404,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vma_lock_read(vma); spin_lock(ptl); - ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); + ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) goto retry_avoidcopy; @@ -5442,7 +5442,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * before the page tables are altered */ spin_lock(ptl); - ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); + ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); @@ -5839,7 +5839,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * until finished with ptep. This prevents huge_pmd_unshare from * being called elsewhere and making the ptep no longer valid. * - * ptep could have already be assigned via huge_pte_offset. That + * ptep could have already be assigned via hugetlb_walk(). That * is OK, as huge_pte_alloc will return the same value unless * something has changed. */ @@ -6223,7 +6223,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, return NULL; hugetlb_walker_lock(); - pte = huge_pte_offset(mm, haddr, huge_page_size(h)); + pte = hugetlb_walk(vma, haddr, huge_page_size(h)); if (!pte) goto out_unlock; @@ -6288,8 +6288,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, * * Note that page table lock is not held when pte is null. */ - pte = huge_pte_offset(mm, vaddr & huge_page_mask(h), - huge_page_size(h)); + pte = hugetlb_walk(vma, vaddr & huge_page_mask(h), + huge_page_size(h)); if (pte) ptl = huge_pte_lock(h, mm, pte); absent = !pte || huge_pte_none(huge_ptep_get(pte)); @@ -6481,7 +6481,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, last_addr_mask = hugetlb_mask_last_page(h); for (; address < end; address += psize) { spinlock_t *ptl; - ptep = huge_pte_offset(mm, address, psize); + ptep = hugetlb_walk(vma, address, psize); if (!ptep) { address |= last_addr_mask; continue; @@ -7037,8 +7037,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, saddr = page_table_shareable(svma, vma, addr, idx); if (saddr) { - spte = huge_pte_offset(svma->vm_mm, saddr, - vma_mmu_pagesize(svma)); + spte = hugetlb_walk(svma, saddr, + vma_mmu_pagesize(svma)); /* * When page ref==0, it means it's probably being * freed; continue with the next one. @@ -7400,7 +7400,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) hugetlb_vma_lock_write(vma); i_mmap_lock_write(vma->vm_file->f_mapping); for (address = start; address < end; address += PUD_SIZE) { - ptep = huge_pte_offset(mm, address, sz); + ptep = hugetlb_walk(vma, address, sz); if (!ptep) continue; ptl = huge_pte_lock(h, mm, ptep); diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 5ac8a89130f6..72d72cd73c8f 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -171,7 +171,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) hugetlb_walker_lock(); /* when pud is not present, pte will be NULL */ - pvmw->pte = huge_pte_offset(mm, pvmw->address, size); + pvmw->pte = hugetlb_walk(vma, pvmw->address, size); if (!pvmw->pte) { hugetlb_walker_unlock(); return false; diff --git a/mm/pagewalk.c b/mm/pagewalk.c index abf310011ab1..469e60b1b096 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -305,13 +305,11 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, hugetlb_walker_lock(); do { next = hugetlb_entry_end(h, addr, end); - pte = huge_pte_offset(walk->mm, addr & hmask, sz); - + pte = hugetlb_walk(vma, addr & hmask, sz); if (pte) err = ops->hugetlb_entry(pte, hmask, addr, next, walk); else if (ops->pte_hole) err = ops->pte_hole(addr, next, -1, walk); - if (err) break; } while (addr = next, addr != end);