From patchwork Mon Feb 10 19:37:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13968597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52736C02198 for ; Mon, 10 Feb 2025 19:44:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B809E6B007B; Mon, 10 Feb 2025 14:44:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B096C6B0099; Mon, 10 Feb 2025 14:44:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95CAF6B009A; Mon, 10 Feb 2025 14:44:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 749976B007B for ; Mon, 10 Feb 2025 14:44:08 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3380CAF607 for ; Mon, 10 Feb 2025 19:44:08 +0000 (UTC) X-FDA: 83105060976.02.DE9156A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id E7906100004 for ; Mon, 10 Feb 2025 19:44:05 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CGyoavkB; spf=pass (imf14.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739216646; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oNpxFEEEyxoH6lwF3K+DQE9+aA/s7jzY02gjXCRiBPg=; b=RmABoBtm9Aa93NT+CjhS8I59K58G7rujbaKWYK/smqWdvvkP3vHZnLZtUEhnGxpefijLGC GQOUhAS5yS1wkKgkzMEgOhoDpWj7BU9uQz4sgN2SMmNQfVPO1LG3rg9pI5gIv5srowzgk8 Z9fxajCDog+ZyRnzuYNlysbvnpTRU1A= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CGyoavkB; spf=pass (imf14.hostedemail.com: domain of dhildenb@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhildenb@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739216646; a=rsa-sha256; cv=none; b=IwX+9ey5w5uNDkgdMNuK0zZaxnVGe/fzFrIR+Tow/xCrpeeQjxgP9Mk6d35Yi134Oeosf/ PHcLgDcKKYy+uDAhkes23/wA5JocA1GJrYCYEfbktKtKXELSEk5heIHmbSmsMsmLz3iXoI hgrqhnu1pqkUJopGoVOeJMDxV8hGSmU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739216645; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oNpxFEEEyxoH6lwF3K+DQE9+aA/s7jzY02gjXCRiBPg=; b=CGyoavkBVyC50MF8rOsLJc8xDqmIhI2/+zL/fr08U4M+Sjz3q6SsjaxLEerEz7DhCM6MhD w306AYtVj9oawzhhCfrjgGhhPoGnXn0kRLPvCohggr/Nz/ahttmsnHu/4OjEut5WicZCdR O+1CrdfGLKCgaNk6/mwWqtccMIqzRpM= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-588-wW5wHEQ8MzavGke3laYYLg-1; Mon, 10 Feb 2025 14:44:04 -0500 X-MC-Unique: wW5wHEQ8MzavGke3laYYLg-1 X-Mimecast-MFC-AGG-ID: wW5wHEQ8MzavGke3laYYLg Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-38dd692b6d9so895041f8f.3 for ; Mon, 10 Feb 2025 11:44:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739216342; x=1739821142; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oNpxFEEEyxoH6lwF3K+DQE9+aA/s7jzY02gjXCRiBPg=; b=oNNz2nCJL1Ax9lt83PFtdT5MSwl/SThjoKIa+tnLtgF97XV0rXaSv0Lg10KKaTRYmK wwI1Ew52pjyYwqvxjCRBfQ03rPcZtAENY3sB/uEpxH5yXa8nKvbQgH9KM0qOnk3L2ZNg uSjqMR2r++LogHMnm18A/HdWMso6XnMO45fk777uhmSSb+sk8JCrMaNBMtJrlYSvdvai aLAwckJTMGAamJw2zeQlDWUlVPbUzf36Lp+cbahtkc5bLYC5EUHllWWSuFW3fuZtIyVF 0jwb9FGu0HO/Cg+2i888OThoqL9F3mrIdTtqwj709qE8N1GY25aCuVPZC6d6AQvt3dZv 5zsQ== X-Forwarded-Encrypted: i=1; AJvYcCU6rGhpfXfHXww3JcPU37wByBg0lSlPiBGO3zYtAWYFMXE1eC8lBNfydFLuvkPk5EgPme+FCdkuvg==@kvack.org X-Gm-Message-State: AOJu0YzCIS0bA7h3mlVOG1GSYAtEuuDu9woGPLABxjqLsYnJ9OMzcnC9 +WalZTxBdh8HV2A05232d7ZyTZN/Od8476/wP3rxUajHIEA149rGEHFMNd9ntbothHCwD4d/UPg d5u27geHMhYeiT7ViYxAZkfAasq73BbqgJML3k+r9m+Lie+BC X-Gm-Gg: ASbGncvhkcdxTsp1YN6VFqQeP3ejlMRXJfg/aB2coEO5xcwRgd0NFe5sg/oKeZV8OwS Ln72RTR1mXv/a/jEoNQpEMYS24/DrckFuB/DEkhOmyWZhbo0aG5O/aSfe5lUEwYwHId7oNUfeod r0DDCDpVhaCP6INLfBGEj09M7Be24YbDg3nBjKtTEWmpWh+lZgnU8CySMjJlP+tl8zrqUtQSjFp 5ziP9tDwfrFQktKKmx0DQlLJy+8MKrxtJVGlpj8O+J8rCcJ8EPJcwIxMhVTEA3Ln0rpd3b6zM5i mLw6V/9GazGKCBNADp1XoBZcJC2coDxydmojV3kNpks7U+TRbAmPt0b2K60vemuObQ== X-Received: by 2002:a05:6000:1813:b0:38a:418e:21c7 with SMTP id ffacd0b85a97d-38dc935246fmr8277035f8f.53.1739216342067; Mon, 10 Feb 2025 11:39:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IHTiiTPkwAyfuLx0qL+LO6PapdaXuVNjUwBeGg/Z0ah/0RffyIckymQJ3LkKa1NUzxInLX6yQ== X-Received: by 2002:a05:6000:1813:b0:38a:418e:21c7 with SMTP id ffacd0b85a97d-38dc935246fmr8276996f8f.53.1739216341643; Mon, 10 Feb 2025 11:39:01 -0800 (PST) Received: from localhost (p200300cbc734b80012c465cd348aaee6.dip0.t-ipconnect.de. [2003:cb:c734:b800:12c4:65cd:348a:aee6]) by smtp.gmail.com with UTF8SMTPSA id ffacd0b85a97d-38dc4d00645sm11916376f8f.66.2025.02.10.11.38.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Feb 2025 11:39:00 -0800 (PST) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-doc@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, damon@lists.linux.dev, David Hildenbrand , Andrew Morton , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Jonathan Corbet , Alex Shi , Yanteng Si , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , Masami Hiramatsu , Oleg Nesterov , Peter Zijlstra , SeongJae Park , "Liam R. Howlett" , Lorenzo Stoakes , Vlastimil Babka , Jann Horn , Pasha Tatashin , Peter Xu , Alistair Popple , Jason Gunthorpe Subject: [PATCH v2 15/17] mm/damon: handle device-exclusive entries correctly in damon_folio_mkold_one() Date: Mon, 10 Feb 2025 20:37:57 +0100 Message-ID: <20250210193801.781278-16-david@redhat.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250210193801.781278-1-david@redhat.com> References: <20250210193801.781278-1-david@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 811Ez73u3823iX3-BWneyz3TLBxMTC-53NYH8Km-_bk_1739216643 X-Mimecast-Originator: redhat.com content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspam-User: X-Rspamd-Queue-Id: E7906100004 X-Stat-Signature: 5qs9rhb7fskj5y51mtjq3oduwpiupgsq X-Rspamd-Server: rspam03 X-HE-Tag: 1739216645-606623 X-HE-Meta: U2FsdGVkX19inY7SGYyFlE+7kpIhAz/qwg0dsLZktxvUMaQUOaEgoWRmDp0TzExcZdX3JGPuqTJeQ+T+CJKzugSVST8Hva48+7/eKmCQlE9SFH40EO72wJ+QobzkJA/VcV+utdHMZXGMo4bX9s5CA4ym4/dn+ZpW3898BqXCa8Qmd1f1eXRDb9NJeSGIIpEor9IDA7KYS1iODfGUwk5k4msT1aXsf+avxr1AmbBsUR8AdU5Wr993e1j77R4me12Ms3Ki4QqFT1IqhPw8R0VcFMHQ0gAnQTpt0GWh/7pXtPvzUqaEMJ72UNttFYp+iEYxUF7lFyP44ckpiVNSoK84XAdSbtQbO0ri2953hjycaUqnIpZDN715+ocjQzoC0jesdA7VWQqm7j1+JUYNShr1ZmQDIYta2pnQ3dhvp+P7H3WaamobIW1Izs4iGkWc33Y9u3bv4nAb647/n8U1td2r+mWBSpGMqQUefLO/yTwckiu4dlgQE6l3VRUSL49/6BzgT4haEYZTgATIbQwSvn9aqpkSU1GZRXiSU8GUakb3i2gaugxrFltIADU16R4rMlqJEEk+I0qgdAaFTbjf0PvjPfS6Di9yHzNaBQ1XSujiHFX6S+e7gEC8rRo5qkAFHCqlR9mzaPPcUEiWQU2Z7lY7/mQ6IbQTAmgKSWEwK9e0Y70FX/6rsGAMfbxgzvCN2NvVPJV8leo9Ybd9RzCTMR6TWCqp+v4iu/pwK0DtdteqMaCXt1qUNnkfy25tpRV5AcYH8UidaLu4MafjT0dbvJl/WlZBkUUamWYrh8+QOL7oukb3k8pd8A+S1S9znwI/PkAdwUYCw4g0MlP3xOGCuUe2tqny4PNMwTN+FZF3ZTHb/4s6v73PR4zAZZWz1KzBycqoSFEqbGjJeeptck0vUs2ZXZ5xG+BK/MRHeeFTCycz7InJpbtWl5BQRbjCRXsVmM0NYxge9GLX2N0tJhtknLd UdL/dLqx JcuU9ILe/uM7n5C3vd8jJGanl6W8CGR5DHD1aotPyu5hjvgDmJBS6v0M2j5XSj91efTLEOf28N50kT40a6VYZaO1xaaor+D4Vr/TKFwS5Zwsdyma0GJWlUO/OvhgPV+kbDUYHnBluhYksFk6znO6tMh24Do4Qz8MINg0W34USbycpbp/jBzSs447yONbFYpAQ55TGiaBmIOws82LfVjrWVNsNNKdhQpSPtcmIJRpFvUsL83BM81YAnd5pHyYW5zx+zmaGZahL2yenlrHk2G+lWm8NMNF8X40WHA7QpzGr7HREXKp3p1LhLW1N4KGlMJgfpWHFA8pFsA+CCW3Vmh5i+n1X/o3mFXieQTJzmTOkgoCArN3ozMtdjoSJH8MEtrm6BGPeJIL55FZhAdDetM6S9UnEexWhZBpOUoXKGw+2nUPEJp+wxFEPorfqaTgvYxQORriYiTbTJntJiDHi9hHyIiHqacYUZAFh3lqxKRFETgna9nV0SAomInxvjXSNDKkrQhLY3+JQuFXilNrCAumXClQ7dp3P93Zg8HZkXxb+F+Fu25gG1Hey41NzGokK2NWKetV2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.001960, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ever since commit b756a3b5e7ea ("mm: device exclusive memory access") we can return with a device-exclusive entry from page_vma_mapped_walk(). damon_folio_mkold_one() is not prepared for that and calls damon_ptep_mkold() with PFN swap PTEs. Teach damon_ptep_mkold() to deal with these PFN swap PTEs. Note that device-private entries are so far not applicable on that path, as damon_get_folio() filters out non-lru folios. Should we just skip PFN swap PTEs completely? Possible, but it seems straight forward to just handle it correctly. Note that we could currently only run into this case with device-exclusive entries on THPs. We still adjust the mapcount on conversion to device-exclusive; this makes the rmap walk abort early for small folios, because we'll always have !folio_mapped() with a single device-exclusive entry. We'll adjust the mapcount logic once all page_vma_mapped_walk() users can properly handle device-exclusive entries. Signed-off-by: David Hildenbrand Reviewed-by: SeongJae Park --- mm/damon/ops-common.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index d25d99cb5f2bb..86a50e8fbc806 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -9,6 +9,8 @@ #include #include #include +#include +#include #include "ops-common.h" @@ -39,12 +41,29 @@ struct folio *damon_get_folio(unsigned long pfn) void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr) { - struct folio *folio = damon_get_folio(pte_pfn(ptep_get(pte))); + pte_t pteval = ptep_get(pte); + struct folio *folio; + bool young = false; + unsigned long pfn; + + if (likely(pte_present(pteval))) + pfn = pte_pfn(pteval); + else + pfn = swp_offset_pfn(pte_to_swp_entry(pteval)); + folio = damon_get_folio(pfn); if (!folio) return; - if (ptep_clear_young_notify(vma, addr, pte)) + /* + * PFN swap PTEs, such as device-exclusive ones, that actually map pages + * are "old" from a CPU perspective. The MMU notifier takes care of any + * device aspects. + */ + if (likely(pte_present(pteval))) + young |= ptep_test_and_clear_young(vma, addr, pte); + young |= mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); + if (young) folio_set_young(folio); folio_set_idle(folio);