From patchwork Thu Mar 7 01:02:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13584934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 749B0C5475B for ; Thu, 7 Mar 2024 01:03:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF6BA6B00C5; Wed, 6 Mar 2024 20:03:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA6D16B00C6; Wed, 6 Mar 2024 20:03:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C95946B00C7; Wed, 6 Mar 2024 20:03:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BA89B6B00C5 for ; Wed, 6 Mar 2024 20:03:00 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5D955121055 for ; Thu, 7 Mar 2024 01:03:00 +0000 (UTC) X-FDA: 81868443720.22.E1ED7C8 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 96C3E180009 for ; Thu, 7 Mar 2024 01:02:58 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Cctboyh9; spf=pass (imf16.hostedemail.com: domain of 3QRLpZQoKCEAlvjqwijvqpiqqing.eqonkpwz-oomxcem.qti@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3QRLpZQoKCEAlvjqwijvqpiqqing.eqonkpwz-oomxcem.qti@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709773378; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=EeijRANHOrxfrX3k4EnadZJQGC1/GCLM0wv9ckDJZjk=; b=exLs4LduwRffrFJEuRM2aNWX4AzIh8ZyE8APYVlhE+o1SJlyl1gvLs/UvruyeLRaBoA4Zl d25ottB1bCrKuRLKNUJx116824YmX9aB4u/rkwgVXRs6fYrlu1g4lkqK/Qeqdy3Ed3FV7n VSg47uxqtuJ30deohY6G8jYar7Pd42k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709773378; a=rsa-sha256; cv=none; b=BUSNx8t6MiH1Dx+iXMiv1EIccjCOwx9OIhc3ZEUmu6hQJMxsi3X5Bas2w7LtGHlfrr88b3 doRzkawkYT7/nzu7ZW3ogsz75o3VdBs5t/fGwLpL4Uc9ZHBoiBdaFFgswB57NCnYTLyfHH TTx1UJZLLgKWTla6/9KUtY7TwocmQVU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Cctboyh9; spf=pass (imf16.hostedemail.com: domain of 3QRLpZQoKCEAlvjqwijvqpiqqing.eqonkpwz-oomxcem.qti@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3QRLpZQoKCEAlvjqwijvqpiqqing.eqonkpwz-oomxcem.qti@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b267bf11so446465276.2 for ; Wed, 06 Mar 2024 17:02:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709773377; x=1710378177; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=EeijRANHOrxfrX3k4EnadZJQGC1/GCLM0wv9ckDJZjk=; b=Cctboyh9AfWEqMXiUl5spGQEPSdNaVOeh+do4CW3uBHKrXx0FEFMohwIh+sH5ByEUk vmZgaAxFSYhG7koBxnhQ0w8vKgXBKIW8gupRGeuPAqrCGX9XHvomwrr/Tr/c+6OQ8nBV EaMI+qrpi2/SqYRxv6qQrRrFED05cbkpfRvl4ms9+Q02LeZzzWUmQdrm03eHdXO4j0vQ NJv0AHUcr9IRD2+WS7ZcLhpyJST68o3FZsXZdTQyCHKxBZIhfv5cY94IAIo/kCP9ZK8F CCuClHL7QPAghqHYE0GhiTXDqUGityjiipnnyMHHnqxqO5CckeaupubCyrlvqnTN/xe6 98fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709773377; x=1710378177; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=EeijRANHOrxfrX3k4EnadZJQGC1/GCLM0wv9ckDJZjk=; b=QqxqJBUXhuSTgBEpkjOGADqXk1ICPYkhe16WO/CWDaWEkYU4PdnhcFPIJ4N+yUrsP4 Lw86BVp+2ivqK4AxQQ/MEHi//sxkyN5LDlwZE5cIjQxsHfftnH045amHAhU5LMy4cMqC rPL546+zOGuDGsH8q9Jhf5VfH1m/qkx2pntzjNYEX3a/55wOUrHOkmg7+Gogamhiautb 8KlBATsCruBI4u6CsTrPHGyCS1HM1K20P7B74yjJMthAGT2Vq36TWSOx9buovHf0Ll8a xL1Zm44i8gOp/5zkhVieSJY1giKhaz5W1eMu/TNkTkd3icGJLljIwc0YPS4BWkS38sso MqYQ== X-Forwarded-Encrypted: i=1; AJvYcCX3J9PjhkuuTVfBBSDAD4cuX1KuwIr1KggVSgDzBwkZJl3bSX4imM4R8AeZ3tNpO95DknWd1alktmh2+OSq1y//DoI= X-Gm-Message-State: AOJu0YzsP/O1AGmT5uM08GfaHj+K/L3mZzg9fAEmlSePA5cxJmm9Anid DdzQaCYG6alprWKvDAN/qkLkHsAPrAcCSipV3VwTAMurF+mC7x0GKBRmPeeodeyb0IVdPBcCxw5 OFj4OsqyJJgfW3LWICw== X-Google-Smtp-Source: AGHT+IFZ8DADJ7M68KA1PQljW8T0FB6a9j/RVD47/RG11udI58JqzGCUYlva3YcLanF4It0jZKeiEEgevI6/RKY6 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6902:70d:b0:dc6:c94e:fb85 with SMTP id k13-20020a056902070d00b00dc6c94efb85mr559337ybt.2.1709773377534; Wed, 06 Mar 2024 17:02:57 -0800 (PST) Date: Thu, 7 Mar 2024 01:02:50 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240307010250.3847179-1-jthoughton@google.com> Subject: [PATCH v2] mm: Add an explicit smp_wmb() to UFFDIO_CONTINUE From: James Houghton To: Peter Xu , Axel Rasmussen , Andrew Morton Cc: Muchun Song , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: 96C3E180009 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: xnsaaicwsixxay8kd4g1oz6z4ya4b67x X-HE-Tag: 1709773378-197172 X-HE-Meta: U2FsdGVkX1+l9NC0gGxdVdwrypt3ZdXuButa8rIa2w1uI8flZUOmT3a3Gm3Z1UoVRslzvYbkRBVipBjs4KcKMh4AcjDB4eUz3ULEIZxk9C/cfvUxW5Bh+qghSVddkriMYlEth30F9ozvl9nmzMgJ7aj+3B807doVRxjrEVW/2ZRxXRGB/lR7l9jQ+4Qwv4N/2Xd71fNQj+Qp7TvI0ukbedY6BOE86hQd5vGMEiVpDaZPv6A+lfYUc/0Xdkj5VdhEgWluG9cmB2WbcaKnMeMLVLCABTSW2fxfyMf+d+MJN2DdddeRMIFsdmSBAIXC0mOIT0aOfbdWWmFJzVhqiBabOqtv7wKvwYMMbibn3GeKzqyjWGSrZazegJTINt7z5+uQEkHorETW3GQAwRP8kYKnz5SvexpDB/P3LyZIWJgDMunXLmimsvIvLqaWCQoluxgHStz3tWrfpjTDRzGOzkBg8NfV1KC3MMo++7pqlIHhFQ2WsYLHrg/9daUl3jJF0UnWYu0K8t/BYlnnEO8M1AzKtBzSmc1ItnZ8ieiEhRxgCASH/ZQEVrxwhEOQ8E/UtZTovq3aNvTEiDoCo58aWjrMWG3hAJs3Jgp029AuKsNnz+uXY3MSADBWDUFch9l8XmrYQzis96RxONCPMJJ1faOWGiVBDBtY/XCvSHysuavOFWG3lycByGIPD3M0Vk5q16vhWQiDHnsFN97cEKr3TKKSVBPLfHJnWwaNeZiLGUVZ/0at635Tv2KByeqhagKnrdFeYm1BQMs53YkXQ13wGE3G6C/ElAHJLe8u+/Kl03HW4BoHhd8gTlDSfEjYrL59H8ScrY7w6NWyD17kapEvak2Xjm/YWzjKnwqIY7ZIcTfjeVnTZX3i+FZLtdw2bOWchj4O8h23h0HsCDAkGcB4DohAmSLCzuaQKDBMQBuVm0Sbx9bUWzVUE+JA7D3g/MDbIsDKmhWIzxyCL8tMMUMB5EX GPqVa8/W Y0maq4Jo6K3koLsiqzeG8B1Wj4KVD2tHUF3kFi36OcGY8gp0ctVFqElkTGybrtQdoQ35ockRSOuh4d12kE0RAgL22mp5eqhQVL0sX28npUINyxHDVNu2Rs5DXJ2DtcuAughx/CfTxeqtZwMAB4I2ZazfRtQdL9Q8yJ3G/8Fj9pp0E/PyDLnrCRgpujVdB9B2bO7246MROcXlKa9SsrnvqM5n2n+FW9D+cF5bLCRMLwya1MOBoKrjYkMaZIF5yZ5gvbPjILkRW5bO2HAkNV9OIIYmPxeD/vmWT0uUDmlEI7Fyezj/eBOrwzvJMVtZB9QAB72KmVGOYmMEeowQRKbyQHTwYISqXd6X8CWUcSETVuvAKi/eExYEjpgO1auy7JiELPKqnBLwSo8+p+oRZdx/JUTZfsJ5at+WZOP06wNViW8kc99bLN1JdkuVyjI+SX1BSCmxEw3V/F0S9J98geAUQvKSVAk/lMAlKxg6N9Hk79CzxZrhwiXkjVfiMKnuPHWK87QKDvkhh8ZH4AG3V3BExO8bifXKMrmVQ9e5ZREbF9wJhbWGU4n+q4AGlA7EzYx28y54z13hOzBE9kw/KUbGkNmJIHh80IhJbjdxsQKKaD0VFUQOe0+whg/wlyT2WfnB13x50vsSqp/lGEBd2B6GzFuW9XIKLTfIyu0CZ6zcEoBpc2LALDm6GEQoyKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Users of UFFDIO_CONTINUE may reasonably assume that a write memory barrier is included as part of UFFDIO_CONTINUE. That is, a user may believe that all writes it has done to a page that it is now UFFDIO_CONTINUE'ing are guaranteed to be visible to anyone subsequently reading the page through the newly mapped virtual memory region. Today, such a user happens to be correct. mmget_not_zero(), for example, is called as part of UFFDIO_CONTINUE (and comes before any PTE updates), and it implicitly gives us a write barrier. To be resilient against future changes, include an explicit smp_wmb(). While we're at it, optimize the smp_wmb() that is already incidentally present for the HugeTLB case. Merely making a syscall does not generally imply the memory ordering constraints that we need (including on x86). Signed-off-by: James Houghton Reviewed-by: Peter Xu --- mm/hugetlb.c | 17 +++++++++++++---- mm/userfaultfd.c | 9 +++++++++ 2 files changed, 22 insertions(+), 4 deletions(-) base-commit: f4239a5d7acc1b5ff9bac4d5471000b952279ef0 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bb17e5c22759..23ef240ba48a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6780,11 +6780,20 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } /* - * The memory barrier inside __folio_mark_uptodate makes sure that - * preceding stores to the page contents become visible before - * the set_pte_at() write. + * If we just allocated a new page, we need a memory barrier to ensure + * that preceding stores to the page become visible before the + * set_pte_at() write. The memory barrier inside __folio_mark_uptodate + * is what we need. + * + * In the case where we have not allocated a new page (is_continue), + * the page must already be uptodate. UFFDIO_CONTINUE already includes + * an earlier smp_wmb() to ensure that prior stores will be visible + * before the set_pte_at() write. */ - __folio_mark_uptodate(folio); + if (!is_continue) + __folio_mark_uptodate(folio); + else + WARN_ON_ONCE(!folio_test_uptodate(folio)); /* Add shared, newly allocated pages to the page cache. */ if (vm_shared && !is_continue) { diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 503ea77c81aa..712160cd41ec 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -845,6 +845,15 @@ ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx, ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long start, unsigned long len, uffd_flags_t flags) { + + /* + * A caller might reasonably assume that UFFDIO_CONTINUE contains an + * smp_wmb() to ensure that any writes to the about-to-be-mapped page by + * the thread doing the UFFDIO_CONTINUE are guaranteed to be visible to + * subsequent loads from the page through the newly mapped address range. + */ + smp_wmb(); + return mfill_atomic(ctx, start, 0, len, uffd_flags_set_mode(flags, MFILL_ATOMIC_CONTINUE)); }