From patchwork Wed Sep 11 03:03:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13799614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5ED0CEE0201 for ; Wed, 11 Sep 2024 03:02:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B358B900007; Tue, 10 Sep 2024 23:02:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ABD8D8D0056; Tue, 10 Sep 2024 23:02:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95E48900007; Tue, 10 Sep 2024 23:02:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 739FC8D0056 for ; Tue, 10 Sep 2024 23:02:42 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 261CA1C3F59 for ; Wed, 11 Sep 2024 03:02:42 +0000 (UTC) X-FDA: 82550959764.30.C0D2441 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) by imf12.hostedemail.com (Postfix) with ESMTP id 9B4B940006 for ; Wed, 11 Sep 2024 03:02:38 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="YVNOJ/Io"; spf=pass (imf12.hostedemail.com: domain of matthew.brost@intel.com designates 198.175.65.16 as permitted sender) smtp.mailfrom=matthew.brost@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726023655; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g5WbKP4NA4g51cPKt8Q/SrXlLzaRan9l5rlPwkdocv0=; b=mKUWIwptg/Lx/lR1IWRkXXjGGZaKNdb5fGrz/Qh7Q59ffgJjl4H+JDvBcoVVUSaWsShrWP sBQSn96lyund46PjuVhantOj0EXD603TA35nJLmuu98MEqpI0YJDvlTDaDdclvwojaysRt 75Z5ox7oOD/clWzrl/Kc0CxpY1qH0IE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726023655; a=rsa-sha256; cv=none; b=EQ59hU9gr3Nht/QHg5G4oC+agQOOKU04sn52bkal9Q65I0GL2rBCKAo6l3g+0Wnjw9Ul+k erY7fQ4hCVAPT3jsgCXJEY0NdD+YIeIjb48SCx0x6rZiu316lHtJdnNeKL0vsMaJ7Gq4Mt u9hQVG26UPayMzxcTqtw4JL0KOBjUFU= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="YVNOJ/Io"; spf=pass (imf12.hostedemail.com: domain of matthew.brost@intel.com designates 198.175.65.16 as permitted sender) smtp.mailfrom=matthew.brost@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1726023759; x=1757559759; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=A7W8LEOsBbOr46JpvVY36Ibjl1xDtTX61OC/71XBNqY=; b=YVNOJ/IoQBlRXY3ptNVngs9u8XMqZLw0VJyaRsGwX7uHlRGRqesvhvSI VF/h6+oPblQF9zvE5C4SpzguQYwvrDfQ8wtGhpxj8zNjkPVLZN/G9Gdv/ IWzT+/cXW6aqDTSuAWJcLCN/wFSOPTNsKhP/tMxJlKfUNkvLHzIF/pTBR SONNMAsD28Ap2FhnPdzyO3GlErLlzDFl7n/rjE0aEuOiRYbDalDqPFuB5 4NFzjnkUOwYpdX26nzEWbimRm87Z6qrCqdPIj+SWVCRAvKLZcovRtUoY2 ++nYfCkaGY3lW4vb+N8e4WCI0fbV3HF1DeO2i+vuobmb2LJJGWelRiFH4 A==; X-CSE-ConnectionGUID: kaIQxiG9Ty2fxv4vRjQlUg== X-CSE-MsgGUID: MU30+fakToi0r1iMYq4yYA== X-IronPort-AV: E=McAfee;i="6700,10204,11191"; a="24907778" X-IronPort-AV: E=Sophos;i="6.10,218,1719903600"; d="scan'208";a="24907778" Received: from orviesa007.jf.intel.com ([10.64.159.147]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2024 20:02:37 -0700 X-CSE-ConnectionGUID: NM/m61pSQAqt4mnZYvCFUw== X-CSE-MsgGUID: FbyUWgeuSK+jdFDAFUU9Qw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,218,1719903600"; d="scan'208";a="67732255" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by orviesa007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2024 20:02:37 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, simona.vetter@ffwll.ch, Philip.Yang@amd.com, akpm@linux-foundation.org, felix.kuehling@amd.com, christian.koenig@amd.com Subject: [PATCH 1/1] mm/migrate: Trylock device page in do_swap_page Date: Tue, 10 Sep 2024 20:03:37 -0700 Message-Id: <20240911030337.870160-2-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240911030337.870160-1-matthew.brost@intel.com> References: <20240911030337.870160-1-matthew.brost@intel.com> MIME-Version: 1.0 X-Stat-Signature: e3m31sofuarwxdn6wphjsyqxwkkfyzsc X-Rspamd-Queue-Id: 9B4B940006 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1726023758-812054 X-HE-Meta: U2FsdGVkX19yiFb+146XL6ZVn0ZvDxoZTmv+g96/q5Khf8tStJnTyX/Clf+fpCTbJaoL/QZsTg+0ozHiMu2PjYnITmsozUVgylI6Y52qNN2cYGhx47ETRy4BBGYvX6y1DAMzAYN7biMsJiUl5sFdJfkgRa/FZtNcJzdUvk4fXttzN1jgNDC3MC92dVw1cnOxMfNTEVIFhyGyO0Aq5r8qZctZg+ipF3ahJroa5DnIYFDE/QsSlsEGli52xOBlZqP4UJkb7CJsnJbl1pjHgzMZcXVLOmpZ8FrdDwAMzxdaD0CQjPieOTyzdMuZIRY+2O3dtUKFGXRtf7kBn/mMOx54ahZM/zN5noSTBu86IiiRAAmAEcZINGEDhtzYvDnKiPWfY4NXjL7+KWl6hBKZaTUUeTEsWJuGlrW7fqdYx/XuEjYL6o/w91wlFmB9m/tlSrJ49zEglPRee8uo4ZJpouQW6B9Ogm2FX2VM+l1g9nUAZi7jxDpf/B5rjWDZw2ejumeLP/Xb1bWApowpyPp6BNpyjL1sR1nRcE1y2uZTezl+2LIWAqlRB1urr1PywUA+/prnmIJ+a6E6vkEB49StNGQQfHT/Is6DFkN14+eZt+DA9RyQrVi9qbrmXJGmT5fQgNWs6skYdziWB5AUI1ew8W0uO5xLieYSl7+oZIQoWmmZcM7jWWI0371JJ6OVGBfb72U+fVsEN57K3TtaNfqgA1SlSI+AN0LZwh2WwlPAzpprzG9pN93QQxAchEf7myxyB/qCj83lGjy3EqOAjecROQNrBpWVQIVv8ES8W9r7tC8qkbRt9gODVYjE5DKGvZfNJs9jyjnsmTDUVRKUgbkZUPOJbfC4DuO0UaUBt1iDetM3BQSfxNWNO/jrdBzoFW3mmQwvD6IJS/Qgar1sM2DlDLMF8ZaPMA8Qk1osXMuqFkEey7EqFOVU/6XPNmEracUQqMZCcSxLObhbpssyiHZwKCU yjmujcgk d23q09kqrW5GIzCI/KnfO7fBrBCYa7WRqVPgv88Ge9VoYV1v0hx8XDuhn2pHWNidHWxWoPU0BQMaSxMPJfkz0I8TH6zTcBWxcqmHVxo96X/Js5VrcQolR0l/JvSgyASyVZBdn1F9iGJ0NqVbkpV/ov6mmXgY1bfnufgl61UT7awxLck54mxP2rofsNwQICi5t+GSmt2Pl5HhKqjjLZQqbj0gOzOJBVT6fzj0GvFyKRRTv1BgqXWo1HahfnNWbHKeXBGGFYYZk0bo+ixgvmUQwe7niNVA2E84RHh/X7JYt7Ll1oT6au9CqqPVs5tCngb0my3+H X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Avoid multiple CPU page faults to the same device page racing by locking the page in do_swap_page before taking an additional reference to the page. This prevents scenarios where multiple CPU page faults each take an extra reference to a device page, which could abort migration in folio_migrate_mapping. With the device page locked in do_swap_page, the migrate_vma_* functions need to be updated to avoid locking the fault_page argument. Prior to this change, a livelock scenario could occur in Xe's (Intel GPU DRM driver) SVM implementation if enough threads faulted the same device page. Cc: Philip Yang Cc: Felix Kuehling Cc: Christian König Cc: Andrew Morton Suggessted-by: Simona Vetter Signed-off-by: Matthew Brost --- mm/memory.c | 13 +++++++--- mm/migrate_device.c | 60 +++++++++++++++++++++++++++++++-------------- 2 files changed, 50 insertions(+), 23 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 3c01d68065be..bbd97d16a96a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4046,10 +4046,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * Get a page reference while we know the page can't be * freed. */ - get_page(vmf->page); - pte_unmap_unlock(vmf->pte, vmf->ptl); - ret = vmf->page->pgmap->ops->migrate_to_ram(vmf); - put_page(vmf->page); + if (trylock_page(vmf->page)) { + get_page(vmf->page); + pte_unmap_unlock(vmf->pte, vmf->ptl); + ret = vmf->page->pgmap->ops->migrate_to_ram(vmf); + put_page(vmf->page); + unlock_page(vmf->page); + } else { + pte_unmap_unlock(vmf->pte, vmf->ptl); + } } else if (is_hwpoison_entry(entry)) { ret = VM_FAULT_HWPOISON; } else if (is_pte_marker_entry(entry)) { diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 6d66dc1c6ffa..049893a5a179 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -60,6 +60,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, struct mm_walk *walk) { struct migrate_vma *migrate = walk->private; + struct folio *fault_folio = migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; struct vm_area_struct *vma = walk->vma; struct mm_struct *mm = vma->vm_mm; unsigned long addr = start, unmapped = 0; @@ -88,11 +90,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, folio_get(folio); spin_unlock(ptl); - if (unlikely(!folio_trylock(folio))) + if (unlikely(fault_folio != folio && + !folio_trylock(folio))) return migrate_vma_collect_skip(start, end, walk); ret = split_folio(folio); - folio_unlock(folio); + if (fault_folio != folio) + folio_unlock(folio); folio_put(folio); if (ret) return migrate_vma_collect_skip(start, end, @@ -192,7 +196,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, * optimisation to avoid walking the rmap later with * try_to_migrate(). */ - if (folio_trylock(folio)) { + if (fault_folio == folio || folio_trylock(folio)) { bool anon_exclusive; pte_t swp_pte; @@ -204,7 +208,8 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (folio_try_share_anon_rmap_pte(folio, page)) { set_pte_at(mm, addr, ptep, pte); - folio_unlock(folio); + if (fault_folio != folio) + folio_unlock(folio); folio_put(folio); mpfn = 0; goto next; @@ -363,6 +368,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns, unsigned long npages, struct page *fault_page) { + struct folio *fault_folio = fault_page ? + page_folio(fault_page) : NULL; unsigned long i, restore = 0; bool allow_drain = true; unsigned long unmapped = 0; @@ -427,7 +434,8 @@ static unsigned long migrate_device_unmap(unsigned long *src_pfns, remove_migration_ptes(folio, folio, false); src_pfns[i] = 0; - folio_unlock(folio); + if (fault_folio != folio) + folio_unlock(folio); folio_put(folio); restore--; } @@ -536,6 +544,8 @@ int migrate_vma_setup(struct migrate_vma *args) return -EINVAL; if (args->fault_page && !is_device_private_page(args->fault_page)) return -EINVAL; + if (args->fault_page && !PageLocked(args->fault_page)) + return -EINVAL; memset(args->src, 0, sizeof(*args->src) * nr_pages); args->cpages = 0; @@ -799,19 +809,13 @@ void migrate_vma_pages(struct migrate_vma *migrate) } EXPORT_SYMBOL(migrate_vma_pages); -/* - * migrate_device_finalize() - complete page migration - * @src_pfns: src_pfns returned from migrate_device_range() - * @dst_pfns: array of pfns allocated by the driver to migrate memory to - * @npages: number of pages in the range - * - * Completes migration of the page by removing special migration entries. - * Drivers must ensure copying of page data is complete and visible to the CPU - * before calling this. - */ -void migrate_device_finalize(unsigned long *src_pfns, - unsigned long *dst_pfns, unsigned long npages) +static void __migrate_device_finalize(unsigned long *src_pfns, + unsigned long *dst_pfns, + unsigned long npages, + struct page *fault_page) { + struct folio *fault_folio = fault_page ? + page_folio(fault_page) : NULL; unsigned long i; for (i = 0; i < npages; i++) { @@ -838,7 +842,8 @@ void migrate_device_finalize(unsigned long *src_pfns, src = page_folio(page); dst = page_folio(newpage); remove_migration_ptes(src, dst, false); - folio_unlock(src); + if (fault_folio != src) + folio_unlock(src); if (is_zone_device_page(page)) put_page(page); @@ -854,6 +859,22 @@ void migrate_device_finalize(unsigned long *src_pfns, } } } + +/* + * migrate_device_finalize() - complete page migration + * @src_pfns: src_pfns returned from migrate_device_range() + * @dst_pfns: array of pfns allocated by the driver to migrate memory to + * @npages: number of pages in the range + * + * Completes migration of the page by removing special migration entries. + * Drivers must ensure copying of page data is complete and visible to the CPU + * before calling this. + */ +void migrate_device_finalize(unsigned long *src_pfns, + unsigned long *dst_pfns, unsigned long npages) +{ + return __migrate_device_finalize(src_pfns, dst_pfns, npages, NULL); +} EXPORT_SYMBOL(migrate_device_finalize); /** @@ -869,7 +890,8 @@ EXPORT_SYMBOL(migrate_device_finalize); */ void migrate_vma_finalize(struct migrate_vma *migrate) { - migrate_device_finalize(migrate->src, migrate->dst, migrate->npages); + __migrate_device_finalize(migrate->src, migrate->dst, migrate->npages, + migrate->fault_page); } EXPORT_SYMBOL(migrate_vma_finalize);