From patchwork Tue May 2 16:34:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Stoakes X-Patchwork-Id: 13229154 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C74E6C77B73 for ; Tue, 2 May 2023 16:34:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FF61900007; Tue, 2 May 2023 12:34:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AF3B900002; Tue, 2 May 2023 12:34:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 450C2900007; Tue, 2 May 2023 12:34:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by kanga.kvack.org (Postfix) with ESMTP id EB512900002 for ; Tue, 2 May 2023 12:34:18 -0400 (EDT) Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-2f6401ce8f8so2515404f8f.3 for ; Tue, 02 May 2023 09:34:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683045258; x=1685637258; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JD+whRS851erhVVABMqdG0VTRc0I6Z1rUQ+rUiOFVdo=; b=LOnfftTmcJ81+iwcBByQXB7Ir64O4Y1k4pQkOcPV6TI0bAkgfp3UaIZJ+zuRf48mH+ sKQ+qJFhhAT+ExIv4raZCe4Y7g6YIyog7zt61v/DO+Fa7aU1K2QXQwuKAeXJn8SIoYiB W4rnS2m03quL9SZbgo+gI56ky1WuHrcAT7sa4wKL20+7gCA8GdMCVBH1cDnLcEb6fDXA c27XUVFOEl1cVsIdPhrXIdFzaWhtobeae2CFGmZSQpcLNg81bdYOgyqtNdKwNp+4Ve3p UgEWWsgUHsHxSNWn8K/VU+Ldw1J4kZz6hnohQGiLK02HgtBSbtURRyI2GNirMOgnJNRQ Xo9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683045258; x=1685637258; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JD+whRS851erhVVABMqdG0VTRc0I6Z1rUQ+rUiOFVdo=; b=WF/VReSHZBuT0xS6RxC1hfb7EJ3oAWbnWCOpoTBc38mzVfLNm3c3RlVnxKkHgg38h+ Qx3R8xQlHg9tBCRUHg+1ar9558MmyxRfC4oTkvIy5El98h/waC3MIhWw9idDH09Jzz9U gvB6AhLWE15Ke+j/JWUY4cba/jUR4z7lZWXydthUmIUNOkDeu7N8AX3Wch9Ceihy2hLo +iKKHPNAFFX4o6aIzgd06BqhkBZHWxAuEoqTgQFbmGcgT8IBYuBMWnzreIp4H4hN8rON 2rP1wolUn81AgvCfWmCxeuMnF07CdpgtekKFzRrD1NeFSVAYfNhppzA1Yo5bg2R8GwcX htFA== X-Gm-Message-State: AC+VfDwKoebelS4ivFXxjwdLJO73Y9h0UFWAzrGOjmop1NdAcN+7C9UH blQ2EAKmg2foDQmlRh2/1/B69oDtVszOhA== X-Google-Smtp-Source: ACHHUZ71Yj6T5jkZc/shg2Kn5A9sybaTNToTec3q07n6D5PpB37fQKAx38VozXIWishADRjyKrMnbg== X-Received: by 2002:adf:edcf:0:b0:306:2cd4:b01b with SMTP id v15-20020adfedcf000000b003062cd4b01bmr5305262wro.37.1683045257681; Tue, 02 May 2023 09:34:17 -0700 (PDT) Received: from lucifer.home (host86-156-84-164.range86-156.btcentralplus.com. [86.156.84.164]) by smtp.googlemail.com with ESMTPSA id b10-20020a5d550a000000b0030639a86f9dsm1789919wrv.51.2023.05.02.09.34.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 May 2023 09:34:17 -0700 (PDT) From: Lorenzo Stoakes To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton Cc: Jason Gunthorpe , Jens Axboe , Matthew Wilcox , Dennis Dalessandro , Leon Romanovsky , Christian Benvenuti , Nelson Escobar , Bernard Metzler , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Mark Rutland , Alexander Shishkin , Jiri Olsa , Namhyung Kim , Ian Rogers , Adrian Hunter , Bjorn Topel , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Brauner , Richard Cochran , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , linux-fsdevel@vger.kernel.org, linux-perf-users@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, Oleg Nesterov , Jason Gunthorpe , John Hubbard , Jan Kara , "Kirill A . Shutemov" , Pavel Begunkov , Mika Penttila , David Hildenbrand , Dave Chinner , Theodore Ts'o , Peter Xu , Matthew Rosato , "Paul E . McKenney" , Christian Borntraeger , Lorenzo Stoakes Subject: [PATCH v7 3/3] mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings Date: Tue, 2 May 2023 17:34:05 +0100 Message-Id: X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Writing to file-backed dirty-tracked mappings via GUP is inherently broken as we cannot rule out folios being cleaned and then a GUP user writing to them again and possibly marking them dirty unexpectedly. This is especially egregious for long-term mappings (as indicated by the use of the FOLL_LONGTERM flag), so we disallow this case in GUP-fast as we have already done in the slow path. We have access to less information in the fast path as we cannot examine the VMA containing the mapping, however we can determine whether the folio is anonymous and then whitelist known-good mappings - specifically hugetlb and shmem mappings. While we obtain a stable folio for this check, the mapping might not be, as a truncate could nullify it at any time. Since doing so requires mappings to be zapped, we can synchronise against a TLB shootdown operation. For some architectures TLB shootdown is synchronised by IPI, against which we are protected as the GUP-fast operation is performed with interrupts disabled. Equally, we are protected from architectures which specify CONFIG_MMU_GATHER_RCU_TABLE_FREE as the interrupts being disabled imply an RCU lock as well. We whitelist anonymous mappings (and those which otherwise do not have a valid mapping), shmem and hugetlb mappings, none of which require dirty tracking so are safe to long-term pin. It's important to note that there are no APIs allowing users to specify FOLL_FAST_ONLY for a PUP-fast let alone with FOLL_LONGTERM, so we can always rely on the fact that if we fail to pin on the fast path, the code will fall back to the slow path which can perform the more thorough check. Suggested-by: David Hildenbrand Suggested-by: Kirill A . Shutemov Suggested-by: Peter Zijlstra Signed-off-by: Lorenzo Stoakes --- mm/gup.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 2 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 6e209ca10967..93b4aa39e5a5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -95,6 +96,52 @@ static inline struct folio *try_get_folio(struct page *page, int refs) return folio; } +/* + * Used in the GUP-fast path to determine whether a FOLL_PIN | FOLL_LONGTERM | + * FOLL_WRITE pin is permitted for a specific folio. + * + * This assumes the folio is stable and pinned. + * + * Writing to pinned file-backed dirty tracked folios is inherently problematic + * (see comment describing the writeable_file_mapping_allowed() function). We + * therefore try to avoid the most egregious case of a long-term mapping doing + * so. + * + * This function cannot be as thorough as that one as the VMA is not available + * in the fast path, so instead we whitelist known good cases. + */ +static bool folio_longterm_write_pin_allowed(struct folio *folio) +{ + struct address_space *mapping; + + /* + * GUP-fast disables IRQs - this prevents IPIs from causing page tables + * to disappear from under us, as well as preventing RCU grace periods + * from making progress (i.e. implying rcu_read_lock()). + * + * This means we can rely on the folio remaining stable for all + * architectures, both those that set CONFIG_MMU_GATHER_RCU_TABLE_FREE + * and those that do not. + * + * We get the added benefit that given inodes, and thus address_space, + * objects are RCU freed, we can rely on the mapping remaining stable + * here with no risk of a truncation or similar race. + */ + lockdep_assert_irqs_disabled(); + + /* + * If no mapping can be found, this implies an anonymous or otherwise + * non-file backed folio so in this instance we permit the pin. + * + * shmem and hugetlb mappings do not require dirty-tracking so we + * explicitly whitelist these. + * + * Other non dirty-tracked folios will be picked up on the slow path. + */ + mapping = folio_mapping(folio); + return !mapping || shmem_mapping(mapping) || folio_test_hugetlb(folio); +} + /** * try_grab_folio() - Attempt to get or pin a folio. * @page: pointer to page to be grabbed @@ -123,6 +170,8 @@ static inline struct folio *try_get_folio(struct page *page, int refs) */ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) { + bool is_longterm = flags & FOLL_LONGTERM; + if (unlikely(!(flags & FOLL_PCI_P2PDMA) && is_pci_p2pdma_page(page))) return NULL; @@ -136,8 +185,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) * right zone, so fail and let the caller fall back to the slow * path. */ - if (unlikely((flags & FOLL_LONGTERM) && - !is_longterm_pinnable_page(page))) + if (unlikely(is_longterm && !is_longterm_pinnable_page(page))) return NULL; /* @@ -148,6 +196,16 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) if (!folio) return NULL; + /* + * Can this folio be safely pinned? We need to perform this + * check after the folio is stabilised. + */ + if ((flags & FOLL_WRITE) && is_longterm && + !folio_longterm_write_pin_allowed(folio)) { + folio_put_refs(folio, refs); + return NULL; + } + /* * When pinning a large folio, use an exact count to track it. *