From patchwork Fri Aug 2 15:55:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13751717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86644C52D6F for ; Fri, 2 Aug 2024 15:55:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD81D6B007B; Fri, 2 Aug 2024 11:55:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D89086B0083; Fri, 2 Aug 2024 11:55:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C04146B0085; Fri, 2 Aug 2024 11:55:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9A73A6B007B for ; Fri, 2 Aug 2024 11:55:43 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 52E1EA11CB for ; Fri, 2 Aug 2024 15:55:43 +0000 (UTC) X-FDA: 82407755766.04.7E1C6C2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 9759C180002 for ; Fri, 2 Aug 2024 15:55:41 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DKe9e2Cb; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722614094; a=rsa-sha256; cv=none; b=i87XepWbncqh5aBIMHHlPCeIZJOUs67pjGv99fG+6j5ZHVEf6Vy5jmJtLQGhVyQF5fuz25 QuLuBK0sb6g6XmDYykE9QyF/OFvcU1x0KF5gdnGKcx6vZnMYhw0jiex60gK5xEtQ+pGb6A gjCx7EIaqHDIRULN49QTwbri7PmR+kE= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DKe9e2Cb; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722614094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=L1rljUKvL0U4qJqT07GMyS3ShSSr+C5zU092arRl9Dk=; b=17tru1YMb7M9fSp6mZX4/s0ASTqL/9kMJwTVcLZk+lEo2WX6jbZobPX0vNMKvg+ax9HiGx XmmmpbhHJ49CAslPa2ibZ2TzY3ChJ956eu45YDyE4AT5kWviuOs4EdoDjV2ftIwZj7SzIX FNJ22JBIRqOlQ/Ub9xXmFmvJfoel0jI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722614141; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=L1rljUKvL0U4qJqT07GMyS3ShSSr+C5zU092arRl9Dk=; b=DKe9e2Cb7LLY7LyiMniK08C2SZBhBedRZ3sZNRk0pJE0uMg5ZrPKsRfDqYyd5hpzwCxhgH EngHsWZP3THk8U5J62j6nBTAM4+ed+cw6tSEbyAX8veqIZKeQty8keucyEGkNDwxbiJwf5 RxDBULw6oADGskY6sNEY8XxiCJ/3hC8= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-581-nld39d-uMKGpUalxEQkpag-1; Fri, 02 Aug 2024 11:55:35 -0400 X-MC-Unique: nld39d-uMKGpUalxEQkpag-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0C34D1944AB0; Fri, 2 Aug 2024 15:55:33 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.113]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 68195300018D; Fri, 2 Aug 2024 15:55:26 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-s390@vger.kernel.org, linux-fsdevel@vger.kernel.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Jonathan Corbet , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Sven Schnelle , Gerald Schaefer Subject: [PATCH v1 00/11] mm: replace follow_page() by folio_walk Date: Fri, 2 Aug 2024 17:55:13 +0200 Message-ID: <20240802155524.517137-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 9759C180002 X-Stat-Signature: pkzwi8ziwrspg6atak6u1fig9rh7fm6y X-Rspam-User: X-HE-Tag: 1722614141-381272 X-HE-Meta: U2FsdGVkX19AZaMDFeU8fd05IENTNSgGNPf7RaWJFwCrvIX46NXfsLe7ZxD+XwmOOifKC4WPBpdnoieHHKQac4kUAFLG6l2zb8IUy6ft77CFgEYSLN07/GUPdwHtG4bbdUhvZ2qrwyhSBBrkeOHw2lVgOI2KT39WCqNInRwkqfWIFRMvQbwbn2Nommw9O7Au7KoKTzX3SnGU2aYdS8GLZRMWLN8v99/cMlO5/scz3FyB4eymDw9ka2vF+EFOMk+V2GyjpjtbeM8KApxGv1cGa8WmZGbCrssEThJplOTDAH1HIJpwYxTSygLPuOlU0f9vBYMB0baSrMtDBtS0lzlnAudJovMi8nx+Spb49KVU6eooaMEZ8bVY2E1boQhMdINthB1AjeMqE1PMSMSvZD0/ugRBjCGYasFfa/WRXxe2fMmPqb1qdvh5vc33Kr60k8aX6EMb7GkrrUQiBYf1mKD9KRDOfxahIN2WZ68KwSe+9LhyS214KSJV1S7LRh2pOn4nyzkY/Lf/obffcMs4rFK3+UTq+S9Z9QaThuwr7pzclTb47kNEwnmGCESt553Lhhjcis+4diOU1T2uWcTjkPvDYYChHSuU5MQyA7+KqepEPiT//MuIgjqoZoWbK3SPZS1OMqBFU2+5Ut9Oi0wObs7sncbEcf9OiH7NHH+DwLx/CKPvWaN9xphjWpyh6vxIpieeyl0tTUWu2/2ZctAnz2vPV5xs2whPPUnaAZmuLwnP4LrJ3KctmzHdfreycNIr1rxDwAhMcX8pPZhUz9aghafKgUSs5jSHlpeG3s7nN1zpWtNH9MusFahVyYDnztU5kz6Ya9E5c1UjeGp4Pqp3Ix1HPD24IES0eUX4SSuzbEgO/rBl+qNeUDK3j/13KPbCrEpQEpws16jSboftXte8vbqXPWDOry0OayJKCYzMBZ0qldocZt9/rlJvCCMuQAXIscBoWFQVOQgtrqURVoRrU4f itAjri4n u0VMQMTpfUlFe1vm2akgxgg1/WAjKJf98vhMruMz3JHgDCMev8Pcihr1fqrkNcNoh8Hg59sJri9GwHLyNr7WzRUCmQxhVGgvEDRJJKlVH9fZGdLGKNTEYlipxVat/dUZvE0Hnnf2SXfgqf44a6sFugokxeJihm5o+Lc9OgWxXhhwVn7si9qDeLhVieAye7XHH/i+f9dQpyTEh21F910GZrpqNyQKh+R2mjTo4Uvtv3uSHpoHLNFZhDVUs+nUeD+2na+4YM2i81SqR0QIw5q5KMxEBxjOpMz7QcZfHyGUoGpAYtbiCMsBFTGpBpzn/jORl4HBtiGJptzc/rrK9AMr40ipVsVABIBPnx9JOQt2Nl70+pkL+tE2yNMpX3Wsgo56s0WHK4e1aJNizLIyLYMrxui+UjSHahdAm2vbICzqwwLLbFXI05FDdJ3gTFiU2+DWOJmlDFtQMqCZyPZb4IK/ikZ0UHwuLxglFkX8S38/I9TMzGh1GDhlKqr9HV7k2touthf+aK3lONSTayLU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Looking into a way of moving the last folio_likely_mapped_shared() call in add_folio_for_migration() under the PTL, I found myself removing follow_page(). This paves the way for cleaning up all the FOLL_, follow_* terminology to just be called "GUP" nowadays. The new page table walker will lookup a mapped folio and return to the caller with the PTL held, such that the folio cannot get unmapped concurrently. Callers can then conditionally decide whether they really want to take a short-term folio reference or whether the can simply unlock the PTL and be done with it. folio_walk is similar to page_vma_mapped_walk(), except that we don't know the folio we want to walk to and that we are only walking to exactly one PTE/PMD/PUD. folio_walk provides access to the pte/pmd/pud (and the referenced folio page because things like KSM need that), however, as part of this series no page table modifications are performed by users. We might be able to convert some other walk_page_range() users that really only walk to one address, such as DAMON with damon_mkold_ops/damon_young_ops. It might make sense to extend folio_walk in the future to optionally fault in a folio (if applicable), such that we can replace some get_user_pages() users that really only want to lookup a single page/folio under PTL without unconditionally grabbing a folio reference. I have plans to extend the approach to a range walker that will try batching various page table entries (not just folio pages) to be a better replace for walk_page_range() -- and users will be able to opt in which type of page table entries they want to process -- but that will require more work and more thoughts. KSM seems to work just fine (ksm_functional_tests selftests) and move_pages seems to work (migration selftest). I tested the leaf implementation excessively using various hugetlb sizes (64K, 2M, 32M, 1G) on arm64 using move_pages and did some more testing on x86-64. Cross compiled on a bunch of architectures. I am not able to test the s390x Secure Execution changes, unfortunately. Cc: Andrew Morton Cc: "Matthew Wilcox (Oracle)" Cc: Jonathan Corbet Cc: Christian Borntraeger Cc: Janosch Frank Cc: Claudio Imbrenda Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Sven Schnelle Cc: Gerald Schaefer David Hildenbrand (11): mm: provide vm_normal_(page|folio)_pmd() with CONFIG_PGTABLE_HAS_HUGE_LEAVES mm/pagewalk: introduce folio_walk_start() + folio_walk_end() mm/migrate: convert do_pages_stat_array() from follow_page() to folio_walk mm/migrate: convert add_page_for_migration() from follow_page() to folio_walk mm/ksm: convert get_mergeable_page() from follow_page() to folio_walk mm/ksm: convert scan_get_next_rmap_item() from follow_page() to folio_walk mm/huge_memory: convert split_huge_pages_pid() from follow_page() to folio_walk s390/uv: convert gmap_destroy_page() from follow_page() to folio_walk s390/mm/fault: convert do_secure_storage_access() from follow_page() to folio_walk mm: remove follow_page() mm/ksm: convert break_ksm() from walk_page_range_vma() to folio_walk Documentation/mm/transhuge.rst | 6 +- arch/s390/kernel/uv.c | 18 ++- arch/s390/mm/fault.c | 16 ++- include/linux/mm.h | 3 - include/linux/pagewalk.h | 58 ++++++++++ mm/filemap.c | 2 +- mm/gup.c | 24 +--- mm/huge_memory.c | 18 +-- mm/ksm.c | 127 +++++++++------------ mm/memory.c | 2 +- mm/migrate.c | 131 ++++++++++----------- mm/nommu.c | 6 - mm/pagewalk.c | 202 +++++++++++++++++++++++++++++++++ 13 files changed, 413 insertions(+), 200 deletions(-)