From patchwork Mon Oct 9 06:42:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13412983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFBF2E95A95 for ; Mon, 9 Oct 2023 06:42:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09C216B019A; Mon, 9 Oct 2023 02:42:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3FA96B019C; Mon, 9 Oct 2023 02:42:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E56F96B019F; Mon, 9 Oct 2023 02:42:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D95856B019A for ; Mon, 9 Oct 2023 02:42:39 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A79A340146 for ; Mon, 9 Oct 2023 06:42:39 +0000 (UTC) X-FDA: 81324979638.17.E71CBC0 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id DE7A940003 for ; Mon, 9 Oct 2023 06:42:37 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xvaooVcG; spf=pass (imf27.hostedemail.com: domain of 33aAjZQYKCNwQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=33aAjZQYKCNwQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696833757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GMzA+bY5iQ55icqv7/c58xVjlf8IiONnVGODeN5A6AA=; b=XblMGsUB/KyzfG9XxFwRLA0+S/hjzNrcdUamNI7ZBs2qEuCeMKX4V4QecOqXRVV4EGd8KU 3qqON/Pe33DpU5Y7TSZeXAl6pOtDpVO0BsUjTEwT9Cfa85DNmh+FeF6oh2PE6GGmdPXvi1 QJyan5Nuzl1RirXYXMxrSXWeL2MWkyw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696833757; a=rsa-sha256; cv=none; b=0BeM3AlDMdG31oSdfRlSpohiAb9ksJPuM0XWv7XptmdfN9Ozjj4AgkrsIFgUO0bpt/oWZy IglgtTPBUWJ54ZTHtT35NmXBI/48qafZzPL5T7e0ho+MTcJRJKHE7xIhs9/UuYUHl4oEl3 h9tbKqBhPX2sUGDhKRbcunbns9YytUk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xvaooVcG; spf=pass (imf27.hostedemail.com: domain of 33aAjZQYKCNwQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=33aAjZQYKCNwQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-59f6902dc8bso66810397b3.0 for ; Sun, 08 Oct 2023 23:42:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696833757; x=1697438557; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GMzA+bY5iQ55icqv7/c58xVjlf8IiONnVGODeN5A6AA=; b=xvaooVcG8QWkK4a52nXE1sYz1SE1RyY+2BI4yh6k5RAvy4SyqqYaWAbHuSv/NQ4+AQ mezf5q/rUeyuQB38b2x/Fis648SuH3wfmW4xKVPy9mZfJ1Lo2j/0yumK5tPe39WxHMw9 EZFkJfk6sTl3+SEg2c6n9zzz4OhZpDhoeBfDBLHd3kMcBEIHg/CGOMUIEH463AK/z3a2 ikCpJGPex/0vvNt/qOut7FbIBVeFFnIVKwTBKStdKi6cGZKpm5RK4jvasK+B5N00xoeH 5F7zq4Pt5anxXrvy9IsBmoVkO8xfFNK/fLgBaxsOO1KHVfyfTa9aXrRFSnSLU9F1+E/e m0QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696833757; x=1697438557; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GMzA+bY5iQ55icqv7/c58xVjlf8IiONnVGODeN5A6AA=; b=c/HRXiygcCGz3dEVZ3As5GF1woG/7iKW0U7YkzuNLgbLRHW+W+xWAv4eD/ng36jpCx UDneomUTYIJyCgZ+l3WcmHR8WeiTKwc5UsHDWREuQ7bnFxY1ai8fTVyeG1yEjrMvyc12 rmQqmE5+ZHE+Y3ZEj4s82ew6Jet7hry3L5/B7+AeiQhh05h/PMBktSfLBAYjHcSWHCCL hlH4r0SlnMOFxEleSXVP+SOUe25niZs+t00BvWaCdwSyAQfdEhEqDFbm2GlBuZ6pnR/Q 6tvPPHj4A/N3PAL8FMNFv6oDE3jkpgXAaEA/G31vma4evpvDVdSF8OkBI2dD+GbZ67fQ LgEQ== X-Gm-Message-State: AOJu0Yw9uiXD7fIGaCQVNd6ZnvRY9ljJ8fb8+lMANEI1VOXP5DqElVKR NWRfyQrSvUHyKXHTgqH/muiI8wyKs8M= X-Google-Smtp-Source: AGHT+IFtPbkNFD9Rxrba570q5f/tqYHiGOqoU9c9uHOquBYgh1LiUto5yoJwsPheCUdVazB3Qxt4pauoLww= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:3e83:601e:dc42:a455]) (user=surenb job=sendgmr) by 2002:a81:a809:0:b0:59b:ebe0:9fce with SMTP id f9-20020a81a809000000b0059bebe09fcemr261242ywh.5.1696833757037; Sun, 08 Oct 2023 23:42:37 -0700 (PDT) Date: Sun, 8 Oct 2023 23:42:26 -0700 In-Reply-To: <20231009064230.2952396-1-surenb@google.com> Mime-Version: 1.0 References: <20231009064230.2952396-1-surenb@google.com> X-Mailer: git-send-email 2.42.0.609.gbb76f46606-goog Message-ID: <20231009064230.2952396-2-surenb@google.com> Subject: [PATCH v3 1/3] mm/rmap: support move to different root anon_vma in folio_move_anon_rmap() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: viro@zeniv.linux.org.uk, brauner@kernel.org, shuah@kernel.org, aarcange@redhat.com, lokeshgidra@google.com, peterx@redhat.com, david@redhat.com, hughd@google.com, mhocko@suse.com, axelrasmussen@google.com, rppt@kernel.org, willy@infradead.org, Liam.Howlett@oracle.com, jannh@google.com, zhangpeng362@huawei.com, bgeffon@google.com, kaleshsingh@google.com, ngeoffray@google.com, jdduke@google.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@android.com X-Stat-Signature: xbjzbi5tcscua9xj3is5a5kd4s5iykr5 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DE7A940003 X-Rspam-User: X-HE-Tag: 1696833757-857915 X-HE-Meta: U2FsdGVkX1/wK5xJET4G4JU1e4p3BaZJ1KbfQcqUdU1c8R3L2/maGC17Yy1o8RFxoSNq908WYfVVNpk6ckz9/WlMd2bpGYBKpazTgSxSPvY+uojpKApAyjwF27GOytaemG5mzWlw6XENCgZ+OaiLxS9ijl44K/ne/a4KnoNwR6X9KhoH9UKjlwPyKUZ9931+z+UOw6aXK/g+xwBZNrz+xRElhIsca2FYIsclv/yKJhu1eOJmfDVamqahoYJ1jfv3NCRB2rBUFDhig3P4zDAjOiLHO6m7n2itrgDYlEOmphu7XROocJl3oIXdVq5FO2hpcj/NvuQe4GAiRId+tzi/GImiw/vqNRQBA7ME1H7xAyD6GAuNVAVDGKo+3nb+jcTEJgo49rVRxa0SPkOiiSw/5Zj9YofaZ0sYPTwCP4uRbU6K4n6BCPQNkU4BYfN9FSDNgBlQEVo6+CZM8vHEliqTB13Eske4qAGZO9EMdFX2ggtD2iH9yejy+ZDekdfA2Uadk/ILkJCASEANNTQ5KvK4DvR7yVPiuoKUPQRV9VFw2vsNVtlFQiQ74T5tdrAdRgmEoc9m/jjNz9vPRef/4q1rL8AAHJCR51EKzl4cuYrGZw3Cl8sTaq+KdhRVitxbgMTazDJ17UjnPirOrB9YnrZRGoUiW/ynAGTE9f7P3x9GPLVFGFrwSZzEPpU0Hl7DEu/ahIcevY+uthK6p+urXW4gEA0nBYrnv4clRcy9ruDQOkOzV4AN1eV4Ks1UJ/UJW6+j//RJ8MUkwtVAHMQ33BLp18vZdFxPHBE2ANOZM8/ZDQO0Z9lr7lxM/4n3YumHFmvOLeRX0Y9o5sHg8zcUuUIv9/KqdQHCmb/7QcT6j6dwcoYbtybGzRAsbe9YrwuH0Rdbj0LVsxfO3D5XwptHTlcpuxRkkcNkDpM7mO0B0gDy806UqlHdoZHOiBVjSC2XqP3jHjfIVXa9ja+sTTv1kEu VRwweqtG XMDjiZzVN5LuXOlun8ghTfo324I5NlsHlqbmDiCcrWyKqy31NXLf2FacwrR7dGiuu/ocuLrr0Gv21kQMlgX/P5czctBYz4D2oawanpaB54T9aSw6LKquObRC8a9hr1iW1td5ytSrCxKm8+ilVRAE5s8eE5Un3MAL6LRPrKNGhfrfan70qkgMfH2kQkkPGEqLm67zEGVsFDMJ6Sw90kChNnNB5L3Vob57EB0UKuMv0nMjvEb6o5iplc5v8dTlCie1XDpu3lTyGASFHPk19ktX/CVJnJIKz5sxG8+03q+yL0VMbiJyXo7Quxl3dRvnqIRxRRNQAFlUMo2d07vT7Qyk2sQh50yC1HLjsJ2kEFpz7EdvB8Iz3GWUilEqHTMaRs/XUQwoOupRegJ2kUa/dyuYmvKHZKx8RPpfCbbQ0LrTkO0Zb24fKxuyH5+g/wm62p/mJiohmI0hwfiO4DfPcqaD5KMQn0UihK28TzgmOtURPmz+O4B0MP++nKzqnE+uwpJcfgfvslySYmFewD+njFuOm6dmMw0TyK3UCn3P5y8nL2ZFfPdGWKZAEm8qHyoOozAC3Aj1isltCFKiIL3c3jDabv0XDZvgaDwwyi5psU0qmu15pzZboJTrJOTS+6iCBwyN10MQgxSbQYkELCcQilZ9Q9IN5dTxGSteAU0/sQo6u2Ozssq358qKPMJ7bWQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrea Arcangeli For now, folio_move_anon_rmap() was only used to move a folio to a different anon_vma after fork(), whereby the root anon_vma stayed unchanged. For that, it was sufficient to hold the folio lock when calling folio_move_anon_rmap(). However, we want to make use of folio_move_anon_rmap() to move folios between VMAs that have a different root anon_vma. As folio_referenced() performs an RMAP walk without holding the folio lock but only holding the anon_vma in read mode, holding the folio lock is insufficient. When moving to an anon_vma with a different root anon_vma, we'll have to hold both, the folio lock and the anon_vma lock in write mode. Consequently, whenever we succeeded in folio_lock_anon_vma_read() to read-lock the anon_vma, we have to re-check if the mapping was changed in the meantime. If that was the case, we have to retry. Note that folio_move_anon_rmap() must only be called if the anon page is exclusive to a process, and must not be called on KSM folios. This is a preparation for UFFDIO_MOVE, which will hold the folio lock, the anon_vma lock in write mode, and the mmap_lock in read mode. Signed-off-by: Andrea Arcangeli Signed-off-by: Suren Baghdasaryan --- mm/rmap.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/mm/rmap.c b/mm/rmap.c index c1f11c9dbe61..f9ddc50269d2 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -542,7 +542,9 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, struct anon_vma *root_anon_vma; unsigned long anon_mapping; +retry: rcu_read_lock(); +retry_under_rcu: anon_mapping = (unsigned long)READ_ONCE(folio->mapping); if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON) goto out; @@ -552,6 +554,16 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON); root_anon_vma = READ_ONCE(anon_vma->root); if (down_read_trylock(&root_anon_vma->rwsem)) { + /* + * folio_move_anon_rmap() might have changed the anon_vma as we + * might not hold the folio lock here. + */ + if (unlikely((unsigned long)READ_ONCE(folio->mapping) != + anon_mapping)) { + up_read(&root_anon_vma->rwsem); + goto retry_under_rcu; + } + /* * If the folio is still mapped, then this anon_vma is still * its anon_vma, and holding the mutex ensures that it will @@ -586,6 +598,18 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, rcu_read_unlock(); anon_vma_lock_read(anon_vma); + /* + * folio_move_anon_rmap() might have changed the anon_vma as we might + * not hold the folio lock here. + */ + if (unlikely((unsigned long)READ_ONCE(folio->mapping) != + anon_mapping)) { + anon_vma_unlock_read(anon_vma); + put_anon_vma(anon_vma); + anon_vma = NULL; + goto retry; + } + if (atomic_dec_and_test(&anon_vma->refcount)) { /* * Oops, we held the last refcount, release the lock