From patchwork Tue Jun 4 12:25:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13685127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79DDDC25B78 for ; Tue, 4 Jun 2024 12:26:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDB496B00B6; Tue, 4 Jun 2024 08:26:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E635D6B00B7; Tue, 4 Jun 2024 08:26:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D04E46B00B8; Tue, 4 Jun 2024 08:26:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B04436B00B6 for ; Tue, 4 Jun 2024 08:26:13 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 58615160980 for ; Tue, 4 Jun 2024 12:26:13 +0000 (UTC) X-FDA: 82193128626.18.9E5AFF5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 8778E1C001C for ; Tue, 4 Jun 2024 12:26:11 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hFWJ6+r5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717503971; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dhRcJw45QpvZbCGtXIGpN8SLS1SogYKGihoPbbdpvvk=; b=fClFvxP3uA+6PTWpcA1LgCAOvyUNFrdVVkGScBVky4BnZnoQFNu5r/Xjvive4C3TOpAU/M oeZ/92OIWyIwZ4oMxiapV9kAr3b3lepJZzItlOmWUTs0t+qt9AiBmRbecM6dzMTPZAXipC DAlPCkk/jz7kvbp749/G4u8xKWx1JyQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717503971; a=rsa-sha256; cv=none; b=Gr8DSUiY3NHjnGptfb24amHcqKabH1hk8tWGrR9ee1ePl4IXY2J2FwqqiWzeK8789P4GuJ ZyABjF/49xuxzEPOUBLQX//kP6BP898GQSzfR45MAP/mRfoOz7pqQWHyUEJyi7e2vfKah6 UXP/hBpkbZy9gvcrtYZE/vN8DUyhz6k= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=hFWJ6+r5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1717503970; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dhRcJw45QpvZbCGtXIGpN8SLS1SogYKGihoPbbdpvvk=; b=hFWJ6+r5L9NUMQIygz5Qg0zBSNvUAdW5O5DXmaGebpOUgacLAF8hKBExbEBllZsMNQOykb SS7lWdlBgbop1HaKHNog1DI4/fNThAX9CrQs/jPSC3xh031KKZTR7aOyJA0oR6/mqHOzNj og22FEqzLiwEoywak+J7qUCbo6FNqSM= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-623-thwmSiOnMeqZMJPmEBQ3dQ-1; Tue, 04 Jun 2024 08:26:07 -0400 X-MC-Unique: thwmSiOnMeqZMJPmEBQ3dQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C2F4F18BF6D6; Tue, 4 Jun 2024 12:26:04 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.193.118]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EADCC1955D89; Tue, 4 Jun 2024 12:25:57 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, David Hildenbrand , Andrew Morton , Guillaume Morin , Russell King , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , "Liang, Kan" Subject: [PATCH RFC 1/3] kernel/events/uprobes: pass VMA instead of MM to install_breakpoint() Date: Tue, 4 Jun 2024 14:25:46 +0200 Message-ID: <20240604122548.359952-2-david@redhat.com> In-Reply-To: <20240604122548.359952-1-david@redhat.com> References: <20240604122548.359952-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Stat-Signature: 8hbmri5fhnsissdtejwhcqa6m9cgdnmq X-Rspamd-Queue-Id: 8778E1C001C X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1717503971-382920 X-HE-Meta: U2FsdGVkX19h38g+l5aln1Oq/LlzJHTD3XE7FAQO+VvzrSeg0mAm53jpsIRJHfLN5Vmd9MWmfrGZFp+g8WOIQJzOkzz4fpGjB2pPCE07zc0sXnh7+9vl+FGGCnK+GM/YU1BRA0kIHww09iKypFgpxe6gP3vicjlQUH83xjXnEkOXkURztcSwz2MplgLkrJksqpm5W/APpVN7TYEhSFgl+uX8lgSSNzMOkqbx5W+p1IX30UB3owB+beCMXI+Z+OxwzFs1BNXMvgJ95uJE6cYeD799tddxquF3Ttw9Upk/Pwv1FCOK8ILpeSTDVPAijhvr1nEZws/MDEkiJ1ZoWlVeiSBVSA50FwrAKQLPzLZGAnNPjTx1oIslKJHCgUnoT3v7JcuEjilDfFRFJE6tOo4QmDdjzHDGyaa24ME3fke2BzH5lkUwUyakUkPXT8qIYHFaKyEdWxYCiMc0RLi/pg0iygkwKS07di0hqohfIcodkBZhsX7bFEjSvMXwALOHYnCvvAGbbInB6nU+D3MLOUcrGmeLWfoDCF6jyjoc/RPPZw9bVxsgKZx0rVB0KdX487RCs2pwPizq1AztzYNsqxfDAN/JoMLRsubexcShsScKPrdBYLAFOf/kl+Ricw+qdfhmJ5PND1YWuZkcckBet7T7jftKPD+8DTznTpzk83UnX4/kIEkKyrlV1Vn2jJ0fhhrxVFksDQ1g6Nir/vQnWPuXBJVvwJfHJoYry7SOzNtqOJpxLDLq0OaH83nr8L3lXYIJXqhir0U4tLujCNt7LjKtRH0BOfMYw5fr6raVnt7MmU9nyaO8XEaZLypWjMZKxzip7YuYgO+ih35YYxkzgRp2GfW6djfa0F0Bhu/7DvAe5S7CFIbleqw/eJRcFEJfKbYQ81fqE+QzGLd1rE28Ec+vGdefJ9H5eI/+zAbjqAhw4nm+N+Wm4iKd0r6EiFceO942895hBSmN+dI5tFKnqrr wtmxiUby WqFjq5aZqBpBPeDGesgd5iFT9LO1vrh0GvP+zBBxSETzViaorNDNxBh66UTrXCpmW0OpNN995TOgzJ1toBJMo9g81QXNOZvvGw6LLl5xQ/XoxFWSmzxum+V493xakTW/C7txfJW423gM9+pVgaxEYSqIRvtKC96EByWTh5RdUhENFwIhRUA4WXKZoIIqu9/72bxPaMhyvVfyL77wJL6eEFviJTMpHoXpMZNwGu+nTGc/+nEuD4fFWDFZr+PKzL3/N2GFAB5XSabH5Ja/Np0z+Hye0omVzMLAbYGSA5Wy25ckzI1mlgnknfulfFw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: ... and remove the "MM" argument from remove_breakpoint(), because it can easily be derived from the VMA. Signed-off-by: David Hildenbrand --- kernel/events/uprobes.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 2c83ba776fc7b..c6d6b7a8e4410 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -886,10 +886,10 @@ static bool filter_chain(struct uprobe *uprobe, return ret; } -static int -install_breakpoint(struct uprobe *uprobe, struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long vaddr) +static int install_breakpoint(struct uprobe *uprobe, struct vm_area_struct *vma, + unsigned long vaddr) { + struct mm_struct *mm = vma->vm_mm; bool first_uprobe; int ret; @@ -914,9 +914,11 @@ install_breakpoint(struct uprobe *uprobe, struct mm_struct *mm, return ret; } -static int -remove_breakpoint(struct uprobe *uprobe, struct mm_struct *mm, unsigned long vaddr) +static int remove_breakpoint(struct uprobe *uprobe, struct vm_area_struct *vma, + unsigned long vaddr) { + struct mm_struct *mm = vma->vm_mm; + set_bit(MMF_RECALC_UPROBES, &mm->flags); return set_orig_insn(&uprobe->arch, mm, vaddr); } @@ -1061,11 +1063,11 @@ register_for_each_vma(struct uprobe *uprobe, struct uprobe_consumer *new) /* consult only the "caller", new consumer. */ if (consumer_filter(new, UPROBE_FILTER_REGISTER, mm)) - err = install_breakpoint(uprobe, mm, vma, info->vaddr); + err = install_breakpoint(uprobe, vma, info->vaddr); } else if (test_bit(MMF_HAS_UPROBES, &mm->flags)) { if (!filter_chain(uprobe, UPROBE_FILTER_UNREGISTER, mm)) - err |= remove_breakpoint(uprobe, mm, info->vaddr); + err |= remove_breakpoint(uprobe, vma, info->vaddr); } unlock: @@ -1250,7 +1252,7 @@ static int unapply_uprobe(struct uprobe *uprobe, struct mm_struct *mm) continue; vaddr = offset_to_vaddr(vma, uprobe->offset); - err |= remove_breakpoint(uprobe, mm, vaddr); + err |= remove_breakpoint(uprobe, vma, vaddr); } mmap_read_unlock(mm); @@ -1386,7 +1388,7 @@ int uprobe_mmap(struct vm_area_struct *vma) if (!fatal_signal_pending(current) && filter_chain(uprobe, UPROBE_FILTER_MMAP, vma->vm_mm)) { unsigned long vaddr = offset_to_vaddr(vma, uprobe->offset); - install_breakpoint(uprobe, vma->vm_mm, vma, vaddr); + install_breakpoint(uprobe, vma, vaddr); } put_uprobe(uprobe); } From patchwork Tue Jun 4 12:25:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13685128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EF5CC25B7E for ; Tue, 4 Jun 2024 12:26:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F96A6B0085; Tue, 4 Jun 2024 08:26:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A88C6B00B8; Tue, 4 Jun 2024 08:26:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 84ABD6B009D; Tue, 4 Jun 2024 08:26:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 61E8A6B00B8 for ; Tue, 4 Jun 2024 08:26:22 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 29EC4A0564 for ; Tue, 4 Jun 2024 12:26:22 +0000 (UTC) X-FDA: 82193129004.19.9B2C13A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 73EDC1C0013 for ; Tue, 4 Jun 2024 12:26:20 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RYMtTkYy; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717503980; a=rsa-sha256; cv=none; b=q1nY6SJseAxMN1toZ10DTshTp2K9CVf/8HA1qRnQycPoaOPDVCDSvjRVuSmKwoMEVk9zKR pUdyjV9/ND6R4x3Q0//zGgoYK2BJZId4Y6Eu74mjj8JAbMhj59HWGG9/xwpPctP/U5CV3G hpuI4PIKxd6IoNhUwhgG1rftuBLplBs= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RYMtTkYy; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717503980; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aXfKzgnS1a7dNjnPxPlU/qmVEHFMnEf3bxdZwelxxd0=; b=sZvAEmtr6QfdOBCEhr5rq6epq9Mt+8Tq9aMX/eNTFu3dn8oszmOkGxSZqcGmTX2beHG/6F 9mK9wtkXXPBlvluQmpCc5fTwPpNiiBn5rqLPzQerAZRXEdWK9S4ZZlnjw+RepxR1VOAq+o eiXyq7OmqMYAT1Sb33FeF7xjbYe8bVk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1717503979; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aXfKzgnS1a7dNjnPxPlU/qmVEHFMnEf3bxdZwelxxd0=; b=RYMtTkYy2mtuMBcGWwcNUnJhkUQ6mEbdpm2Og76mw+bfZyeFl5+EMxEYiO3ktUF0W2g8rg xWvrwxR/aa5JY5wIU632IjI5SyXsctoybOlyqB8BQ6bfmy38c4JSggn+jDWaGGNzqTKwwh cawqlvm6WvU0APY/KBXLPjyNQ73KIE0= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-121-ODGmKQOXNA-4YSKHFzPAow-1; Tue, 04 Jun 2024 08:26:14 -0400 X-MC-Unique: ODGmKQOXNA-4YSKHFzPAow-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E85DF18BF6E4; Tue, 4 Jun 2024 12:26:10 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.193.118]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 1B8851955D8B; Tue, 4 Jun 2024 12:26:04 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, David Hildenbrand , Andrew Morton , Guillaume Morin , Russell King , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , "Liang, Kan" Subject: [PATCH RFC 2/3] kernel/events/uprobes: pass VMA to set_swbp(), set_orig_insn() and uprobe_write_opcode() Date: Tue, 4 Jun 2024 14:25:47 +0200 Message-ID: <20240604122548.359952-3-david@redhat.com> In-Reply-To: <20240604122548.359952-1-david@redhat.com> References: <20240604122548.359952-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Rspamd-Queue-Id: 73EDC1C0013 X-Rspam-User: X-Rspamd-Server: rspam12 X-Stat-Signature: bn8y7xyfjftup9xiu5y1ih1yxgboojbu X-HE-Tag: 1717503980-656458 X-HE-Meta: U2FsdGVkX1/zqYxbmpQCeGaBM5oVvm2jjBF7WKPntCxEerntwfI/92ZTM6rR+i9jLXrgpBY6fFZp4s07W7V4AVDHELPs44qYk6A/Z1gQy/cR20kkt6m0Y9/zkJVc1njx3e81/WlhOHbx268wcGOa1nJo+9H5B8UiE0TcW3FgA4OWqz9fOXet5aiWvk0+l7yPum386/TZPslA/pm5yvJHmU4+RGLI5TW1sg3fLei5Kti9j61QidkEs6rh2u0suIFkM9W/XClotLL2StIIL+STI+/g3EnxKC7MFub/1TRTd7Rk96Tkk9zZ535+kh2i5yOqxuWvMoagWLdeaZmyac9n2nI45uoiUWelqCSgkOblIOwYRhIc2RaOAqGXBKah0yDReuzcLDvJJCQQG3tfNSdVpfeNIWFDGsOnDBHtcN5SDH8qB83F/oLjv2FRfO2dj3cb+7lmJaPHoAu8YbxGAHAgRzm6FnQwfIHuCtE+bBGD4irUmNAczCP0LXExBKq4nXuwTVFV3grd0s43fREzgRdNiX4K8VZUqGQ8FsNvG9C3B8g8AB4K1nB9e+Cy8YyeSdhQSRxqCFVnK8b1bLKyzxrm2FoTSXGJDtle2DinQfl1Xhl1kpUvYHmi91WFQ4bngH/YKrKhpuU2GUp3PWGI3lGuJYwDGf04+89nBF24HnWizg/SZMhfQNOHEr6VHkCK1RMiSV9Ta/HLQCk2AQShziBLZY57JP86EehamqEVbMtpxyT/D1UJgYVd3rGijbFMyZSKADLccc4lxCXv8Wc7oxCV7SPctfYJ5Y2Osbc82qBpGATiBo+C3Uy4DFGgorOLGots7ppG0TZDiL6MjJfRc1m9W0qyPSOUiPRCvHHTHnzSEe4/WAgwUflYOletqpRvahGnPagtD48OScDg3g1G7NoGYM0RQ6rwNmblmkYHOViJj1/ZQdmELqGMJFx29+mxiEhSExjPALnD+sCHhneY2OM K6Se53Dg 5HyOPV65fybrRXBRjOXMzhyNuxDwxVsavvvk40ZBJhq0eX7qxTvwSF2XIm4xd34up1zgMTIV2wC6l17BxMG1wgcb/BqCtO4iVSgfxv2GFmEFKSkstii7rumJyn0FBwJRy1GGjYcGlxlMtb/h4T3oTb+cXimtg05ZlASIRG+JjtvQxpE2o2BUVT4eTVU95lPrHo5TyEkpGVq5rw55qvf56qNuX85YlDnuiqfFkTWxUq6Qs0+GJhBrFUt5S4tBYdKzntK8nCC++iwRIkgQu1KK1cYyUu/YuBvK6QBV+qlkzyWdbJpuwPV3cbxQyHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We already have the VMA, no need to look it up using get_user_page_vma_remote(). We can now switch to get_user_pages_remote(). Signed-off-by: David Hildenbrand --- arch/arm/probes/uprobes/core.c | 4 ++-- include/linux/uprobes.h | 6 +++--- kernel/events/uprobes.c | 33 +++++++++++++++++---------------- 3 files changed, 22 insertions(+), 21 deletions(-) diff --git a/arch/arm/probes/uprobes/core.c b/arch/arm/probes/uprobes/core.c index f5f790c6e5f89..885e0c5e8c20d 100644 --- a/arch/arm/probes/uprobes/core.c +++ b/arch/arm/probes/uprobes/core.c @@ -26,10 +26,10 @@ bool is_swbp_insn(uprobe_opcode_t *insn) (UPROBE_SWBP_ARM_INSN & 0x0fffffff); } -int set_swbp(struct arch_uprobe *auprobe, struct mm_struct *mm, +int set_swbp(struct arch_uprobe *auprobe, struct vm_area_struct *vma, unsigned long vaddr) { - return uprobe_write_opcode(auprobe, mm, vaddr, + return uprobe_write_opcode(auprobe, vma, vaddr, __opcode_to_mem_arm(auprobe->bpinsn)); } diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index f46e0ca0169c7..77580dba6f8de 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -103,13 +103,13 @@ struct uprobes_state { }; extern void __init uprobes_init(void); -extern int set_swbp(struct arch_uprobe *aup, struct mm_struct *mm, unsigned long vaddr); -extern int set_orig_insn(struct arch_uprobe *aup, struct mm_struct *mm, unsigned long vaddr); +extern int set_swbp(struct arch_uprobe *aup, struct vm_area_struct *vma, unsigned long vaddr); +extern int set_orig_insn(struct arch_uprobe *aup, struct vm_area_struct *vma, unsigned long vaddr); extern bool is_swbp_insn(uprobe_opcode_t *insn); extern bool is_trap_insn(uprobe_opcode_t *insn); extern unsigned long uprobe_get_swbp_addr(struct pt_regs *regs); extern unsigned long uprobe_get_trap_addr(struct pt_regs *regs); -extern int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr, uprobe_opcode_t); +extern int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, unsigned long vaddr, uprobe_opcode_t); extern int uprobe_register(struct inode *inode, loff_t offset, struct uprobe_consumer *uc); extern int uprobe_register_refctr(struct inode *inode, loff_t offset, loff_t ref_ctr_offset, struct uprobe_consumer *uc); extern int uprobe_apply(struct inode *inode, loff_t offset, struct uprobe_consumer *uc, bool); diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index c6d6b7a8e4410..f2b1fef7f1d62 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -449,19 +449,19 @@ static int update_ref_ctr(struct uprobe *uprobe, struct mm_struct *mm, * * uprobe_write_opcode - write the opcode at a given virtual address. * @auprobe: arch specific probepoint information. - * @mm: the probed process address space. + * @vma: the probed virtual memory area. * @vaddr: the virtual address to store the opcode. * @opcode: opcode to be written at @vaddr. * * Called with mm->mmap_lock held for write. * Return 0 (success) or a negative errno. */ -int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, - unsigned long vaddr, uprobe_opcode_t opcode) +int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, + unsigned long vaddr, uprobe_opcode_t opcode) { + struct mm_struct *mm = vma->vm_mm; struct uprobe *uprobe; struct page *old_page, *new_page; - struct vm_area_struct *vma; int ret, is_register, ref_ctr_updated = 0; bool orig_page_huge = false; unsigned int gup_flags = FOLL_FORCE; @@ -473,9 +473,9 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, if (is_register) gup_flags |= FOLL_SPLIT_PMD; /* Read the page with vaddr into memory */ - old_page = get_user_page_vma_remote(mm, vaddr, gup_flags, &vma); - if (IS_ERR(old_page)) - return PTR_ERR(old_page); + ret = get_user_pages_remote(mm, vaddr, 1, gup_flags, &old_page, NULL); + if (ret != 1) + return ret; ret = verify_opcode(old_page, vaddr, &opcode); if (ret <= 0) @@ -560,30 +560,31 @@ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, /** * set_swbp - store breakpoint at a given address. * @auprobe: arch specific probepoint information. - * @mm: the probed process address space. + * @vma: the probed virtual memory area. * @vaddr: the virtual address to insert the opcode. * * For mm @mm, store the breakpoint instruction at @vaddr. * Return 0 (success) or a negative errno. */ -int __weak set_swbp(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr) +int __weak set_swbp(struct arch_uprobe *auprobe, struct vm_area_struct *vma, + unsigned long vaddr) { - return uprobe_write_opcode(auprobe, mm, vaddr, UPROBE_SWBP_INSN); + return uprobe_write_opcode(auprobe, vma, vaddr, UPROBE_SWBP_INSN); } /** * set_orig_insn - Restore the original instruction. - * @mm: the probed process address space. + * @vma: the probed virtual memory area. * @auprobe: arch specific probepoint information. * @vaddr: the virtual address to insert the opcode. * * For mm @mm, restore the original opcode (opcode) at @vaddr. * Return 0 (success) or a negative errno. */ -int __weak -set_orig_insn(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr) +int __weak set_orig_insn(struct arch_uprobe *auprobe, + struct vm_area_struct *vma, unsigned long vaddr) { - return uprobe_write_opcode(auprobe, mm, vaddr, + return uprobe_write_opcode(auprobe, vma, vaddr, *(uprobe_opcode_t *)&auprobe->insn); } @@ -905,7 +906,7 @@ static int install_breakpoint(struct uprobe *uprobe, struct vm_area_struct *vma, if (first_uprobe) set_bit(MMF_HAS_UPROBES, &mm->flags); - ret = set_swbp(&uprobe->arch, mm, vaddr); + ret = set_swbp(&uprobe->arch, vma, vaddr); if (!ret) clear_bit(MMF_RECALC_UPROBES, &mm->flags); else if (first_uprobe) @@ -920,7 +921,7 @@ static int remove_breakpoint(struct uprobe *uprobe, struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; set_bit(MMF_RECALC_UPROBES, &mm->flags); - return set_orig_insn(&uprobe->arch, mm, vaddr); + return set_orig_insn(&uprobe->arch, vma, vaddr); } static inline bool uprobe_is_active(struct uprobe *uprobe) From patchwork Tue Jun 4 12:25:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13685129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85EFDC27C50 for ; Tue, 4 Jun 2024 12:26:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0AAC06B00BA; Tue, 4 Jun 2024 08:26:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 05BFA6B00BB; Tue, 4 Jun 2024 08:26:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E16706B00BC; Tue, 4 Jun 2024 08:26:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BC8706B00BA for ; Tue, 4 Jun 2024 08:26:27 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 70DB0407B3 for ; Tue, 4 Jun 2024 12:26:27 +0000 (UTC) X-FDA: 82193129214.23.A415494 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 91646C000C for ; Tue, 4 Jun 2024 12:26:25 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aoQplvrC; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717503985; a=rsa-sha256; cv=none; b=T2fGxnOfn5k1eY2XlwNYBLM1vv5pCPgblTaqQktHlOS8+rJq0wHAA48nS5nNBwVsMmmZjJ He37XycZuMuf9H3QaerFDPZm0v9ZQEyjLy7RncEJ8oSB0aMxDMQ/ToJZnGXTelIHODGdjl xBYF6lFuX6M+7Oiqav1vFtPjX8z1C9g= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aoQplvrC; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717503985; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CnFa3KkR11uKBOePM8iHmpy6uffAuJd7k5r6Ny89vc8=; b=KZqV1tUErbMxKLlGkNb47T2iaUtpnhZEBhaqHoB8ZX9byElpikpG6oixPWN5i6zPqMp9Qu tK2alBsxQIZCES/y15Cl+RCnycjBqUAKNWh2VwWlng6M/Hkd+PI0Nqym5yG0G57pwVK3ft ZM8B476OhkuRoI3BwVUbI78vG4m3dJU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1717503984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CnFa3KkR11uKBOePM8iHmpy6uffAuJd7k5r6Ny89vc8=; b=aoQplvrCW6nO77gWeNGN76sVdoTuNo7wIlgnJQfGliqvpl5UxhiB8kwmHkSOuiZXxvL8eL CfpW/AP2N+j7Ir05zw6/JuB860iMYcdmjAz46CerHkfhr4XRXI86iwMPhrSL/iEWM1bO7R 8R1LHsz/Tz/kXX+63Y7z0XGTF4/Fq6g= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-136-TgKOtLjzPtirdHQq3B4p2Q-1; Tue, 04 Jun 2024 08:26:20 -0400 X-MC-Unique: TgKOtLjzPtirdHQq3B4p2Q-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id BE98D1895DE9; Tue, 4 Jun 2024 12:26:17 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.193.118]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 6DBE11955D89; Tue, 4 Jun 2024 12:26:11 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, David Hildenbrand , Andrew Morton , Guillaume Morin , Russell King , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , "Liang, Kan" Subject: [PATCH RFC 3/3] kernel/events/uprobes: uprobe_write_opcode() rewrite Date: Tue, 4 Jun 2024 14:25:48 +0200 Message-ID: <20240604122548.359952-4-david@redhat.com> In-Reply-To: <20240604122548.359952-1-david@redhat.com> References: <20240604122548.359952-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 91646C000C X-Stat-Signature: duwheikytqzaoiz8ibr8nidqx11toawi X-HE-Tag: 1717503985-662156 X-HE-Meta: U2FsdGVkX1877Jl1mLGM3eAiUW8qZsLYce7MpdnLjRZOkQJCkB4yMjZ96AfSj3NCTZU1h2+uBaZRSOHyYLOcbRWRuVqafnZFpycEV4YYtfysmistfH1KMMU8Oq9hPE6Zte4Oa5CSSp2MXdu0K7fZRWQkw6iurqqn0Ws5Lwc/kGKx6EWfLzRX35lPUPZX5VuWUDp6ZtQU3kmbBnOalktejJpDqsBhCBQD5ZpexSplgCc2yhZoU4hVIUe/MccR1dfWSsOm9K/JIZALCLmcvJV/djRoH6qAOnvYiwHOkKEumbokL/0BQZAaA6zdiOPYyDlmuMcGvd9IdrEc75JotDjes532ypOAwF3+uRG0bXaxUB4uBb4yKeqwOTuVdmmIRYH1uuNegEm34HI0spxTpFeqWFqge1GUuCRDeQEnpu6zcKUX8xEAVHgEgJHqVGnvWn9HFNMpNpwVcVrAH+bN39u1pjf1H3FYdtawBX53cOY6BUoni4tLdvVwhFABo3oQIjSmpBm560XF8JIiOlqvNmjrWl9jmQGLFRzN+46K/pC4ZoxgUm6Wvb+GLH4avdYg5eGM7/S8t+Yp7PtyIU9kzSV32e7Af/Pxl6L0/KFdl8Kmo/AJfkvfShcVJbvyDjr3E1CogayC6LcOGeSMOXxjwevdrtnSrxHfs6Gw5WqiFUWIwK9+K8AWVQ6B1ClJnwg+nSbOejfCmBDZ4weljuQFuTtGdcISBHNg6Hmf6oVnAAcXnbg1CrOmcGSWudxPR0FbJrrDNk4YFYY/CIkbg2eCSLCBGV2Ek0SmR7DwcE0BPw4c+erJ+9zREhByhH5yje1fPj9DGO58g917Q5a79zZ/ZiThJRz9MLMpTaNNqe/z2B74O8K2VmWTcLbWdGKySGRsoROq0QxjTTcxIb4ELXkV8SGypgwfzGteqaMEz/YpCKeLqOjzQPiqlLNscopG1a3PyEjIzPDAIFB/ItYJpaH6CZj iF57sQJG cPiAW8B5PoX19UcBUYUGLBuu4JioJdcRxXpgPbG8kuOuRDjrgNW+ciFY740aZmA2TLvTPyo6IzdKZaBl4Hd83F7Mlx+ApzQv4HcuQ7pzEUldUKot7R3O2+a503ESvYVUFWLJ5DN+NJhav7JTsH2zXk1GVgNarjRxQt7dpWT6nVsAT/yvqVraXMRfYHG0FcWUkWqX+m8EpuZkGE4YtiLex/MAsmxSHpMDO003xOriWFbb4Kfe0Schqfn0hnEXR0rdrIWofR5YOR7f+kGAWxW4pyXg6W4dNDNnk+d02TIQAB7iKl4yw+m0AofKr9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: uprobe_write_opcode() does some pretty low-level things that really, it shouldn't be doing: for example, manually breaking COW by allocating anonymous folios and replacing mapped pages. Further, it does seem to do some shaky things: for example, writing to possible COW-shared anonymous pages or zapping anonymous pages that might be pinned. We're also not taking care of uffd, uffd-wp, softdirty ... although rather corner cases here. Let's just get it right like ordinary ptrace writes would. Let's rewrite the code, leaving COW-breaking to core-MM, triggered by FOLL_FORCE|FOLL_WRITE (note that the code was already using FOLL_FORCE). We'll use GUP to lookup/faultin the page and break COW if required. Then, we'll walk the page tables using walk_page_range_vma(), to perform our page modification atomically by temporarily unmap+flushing the PTE that maps an anonymous page. Unfortunately, we still have to implement the zapping logic manually, because we only want to zap in specific circumstances (e.g., page content identical). Note that we can handle large folios just fine. walk_page_range_vma() will call split_huge_pmd() for us. We can drop the WARN and stop using FOLL_SPLIT_PMD. If split_huge_pmd() decides to zap a PMD-mapped file THP, there is nothing to do when unregistering. Document that. Signed-off-by: David Hildenbrand --- kernel/events/uprobes.c | 342 +++++++++++++++++++++++----------------- 1 file changed, 194 insertions(+), 148 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index f2b1fef7f1d62..a22f2e9fd6456 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -26,6 +26,7 @@ #include #include #include +#include #include @@ -137,80 +138,6 @@ static loff_t vaddr_to_offset(struct vm_area_struct *vma, unsigned long vaddr) return ((loff_t)vma->vm_pgoff << PAGE_SHIFT) + (vaddr - vma->vm_start); } -/** - * __replace_page - replace page in vma by new page. - * based on replace_page in mm/ksm.c - * - * @vma: vma that holds the pte pointing to page - * @addr: address the old @page is mapped at - * @old_page: the page we are replacing by new_page - * @new_page: the modified page we replace page by - * - * If @new_page is NULL, only unmap @old_page. - * - * Returns 0 on success, negative error code otherwise. - */ -static int __replace_page(struct vm_area_struct *vma, unsigned long addr, - struct page *old_page, struct page *new_page) -{ - struct folio *old_folio = page_folio(old_page); - struct folio *new_folio; - struct mm_struct *mm = vma->vm_mm; - DEFINE_FOLIO_VMA_WALK(pvmw, old_folio, vma, addr, 0); - int err; - struct mmu_notifier_range range; - - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, addr, - addr + PAGE_SIZE); - - if (new_page) { - new_folio = page_folio(new_page); - err = mem_cgroup_charge(new_folio, vma->vm_mm, GFP_KERNEL); - if (err) - return err; - } - - /* For folio_free_swap() below */ - folio_lock(old_folio); - - mmu_notifier_invalidate_range_start(&range); - err = -EAGAIN; - if (!page_vma_mapped_walk(&pvmw)) - goto unlock; - VM_BUG_ON_PAGE(addr != pvmw.address, old_page); - - if (new_page) { - folio_get(new_folio); - folio_add_new_anon_rmap(new_folio, vma, addr); - folio_add_lru_vma(new_folio, vma); - } else - /* no new page, just dec_mm_counter for old_page */ - dec_mm_counter(mm, MM_ANONPAGES); - - if (!folio_test_anon(old_folio)) { - dec_mm_counter(mm, mm_counter_file(old_folio)); - inc_mm_counter(mm, MM_ANONPAGES); - } - - flush_cache_page(vma, addr, pte_pfn(ptep_get(pvmw.pte))); - ptep_clear_flush(vma, addr, pvmw.pte); - if (new_page) - set_pte_at(mm, addr, pvmw.pte, - mk_pte(new_page, vma->vm_page_prot)); - - folio_remove_rmap_pte(old_folio, old_page, vma); - if (!folio_mapped(old_folio)) - folio_free_swap(old_folio); - page_vma_mapped_walk_done(&pvmw); - folio_put(old_folio); - - err = 0; - unlock: - mmu_notifier_invalidate_range_end(&range); - folio_unlock(old_folio); - return err; -} - /** * is_swbp_insn - check if instruction is breakpoint instruction. * @insn: instruction to be checked. @@ -438,6 +365,120 @@ static int update_ref_ctr(struct uprobe *uprobe, struct mm_struct *mm, return ret; } +static bool orig_page_is_identical(struct vm_area_struct *vma, + unsigned long vaddr, struct page *page, bool *large) +{ + const pgoff_t index = vaddr_to_offset(vma, vaddr) >> PAGE_SHIFT; + struct page *orig_page = find_get_page(vma->vm_file->f_inode->i_mapping, + index); + struct folio *orig_folio; + bool identical; + + if (!orig_page) + return false; + orig_folio = page_folio(orig_page); + + *large = folio_test_large(orig_folio); + identical = folio_test_uptodate(orig_folio) && + pages_identical(page, orig_page); + folio_put(orig_folio); + return identical; +} + +#define UWO_NO_PGTABLE 0 +#define UWO_RETRY 1 +#define UWO_RETRY_WRITE_FAULT 2 +#define UWO_DONE 3 +#define UWO_DONE_COLLAPSE 4 + +struct uwo_data { + unsigned long opcode_vaddr; + uprobe_opcode_t opcode; +}; + +static int __write_opcode_pte(pte_t *ptep, unsigned long vaddr, + unsigned long next, struct mm_walk *walk) +{ + struct uwo_data *data = walk->private; + const bool is_register = !!is_swbp_insn(&data->opcode); + pte_t pte = ptep_get(ptep); + struct folio *folio = NULL; + struct page *page; + bool large; + + if (!pte_present(pte)) + return UWO_RETRY; + page = vm_normal_page(walk->vma, vaddr, pte); + if (page) + folio = page_folio(page); + + /* + * If we don't find an anonymous folio when unregistering, we're done. + */ + if (!folio || !folio_test_anon(folio)) + return is_register ? UWO_RETRY_WRITE_FAULT : UWO_DONE; + + /* + * See can_follow_write_pte(): we'd actually prefer a writable PTE here, + * but when unregistering we might no longer have VM_WRITE ... + */ + if (!pte_write(pte)) { + if (!PageAnonExclusive(page)) + return UWO_RETRY_WRITE_FAULT; + if (unlikely(userfaultfd_pte_wp(walk->vma, pte))) + return UWO_RETRY_WRITE_FAULT; + /* SOFTDIRTY is handled via pte_mkdirty() below. */ + } + + /* Unmap + flush the TLB, such that we can write atomically .*/ + flush_cache_page(walk->vma, vaddr, pte_pfn(pte)); + pte = ptep_clear_flush(walk->vma, vaddr, ptep); + copy_to_page(page, data->opcode_vaddr, &data->opcode, + UPROBE_SWBP_INSN_SIZE); + + /* + * When unregistering, we may only zap a PTE if uffd is disabled and + * there are no unexpected folio references ... + */ + if (is_register || userfaultfd_missing(walk->vma) || + (folio_ref_count(folio) != folio_mapcount(folio) + + folio_test_swapcache(folio) * folio_nr_pages(folio))) + goto remap; + + /* + * ... and the mapped page is identical to the original page that + * would get faulted in on next access. + */ + if (!orig_page_is_identical(walk->vma, vaddr, page, &large)) + goto remap; + + /* Zap it and try to reclaim swap space. */ + dec_mm_counter(walk->mm, MM_ANONPAGES); + folio_remove_rmap_pte(folio, page, walk->vma); + if (!folio_mapped(folio) && folio_test_swapcache(folio) && + folio_trylock(folio)) { + folio_free_swap(folio); + folio_unlock(folio); + } + folio_put(folio); + + return large ? UWO_DONE_COLLAPSE : UWO_DONE; +remap: + /* + * Make sure that our copy_to_page() changes become visible before the + * set_pte_at() write. + */ + smp_wmb(); + /* We modified the page. Make sure to mark the PTE dirty. */ + set_pte_at(walk->mm, vaddr, ptep, pte_mkdirty(pte)); + return UWO_DONE; +} + +static const struct mm_walk_ops write_opcode_ops = { + .pte_entry = __write_opcode_pte, + .walk_lock = PGWALK_WRLOCK, +}; + /* * NOTE: * Expect the breakpoint instruction to be the smallest size instruction for @@ -450,111 +491,116 @@ static int update_ref_ctr(struct uprobe *uprobe, struct mm_struct *mm, * uprobe_write_opcode - write the opcode at a given virtual address. * @auprobe: arch specific probepoint information. * @vma: the probed virtual memory area. - * @vaddr: the virtual address to store the opcode. - * @opcode: opcode to be written at @vaddr. + * @opcode_vaddr: the virtual address to store the opcode. + * @opcode: opcode to be written at @opcode_vaddr. * * Called with mm->mmap_lock held for write. * Return 0 (success) or a negative errno. */ int uprobe_write_opcode(struct arch_uprobe *auprobe, struct vm_area_struct *vma, - unsigned long vaddr, uprobe_opcode_t opcode) -{ - struct mm_struct *mm = vma->vm_mm; - struct uprobe *uprobe; - struct page *old_page, *new_page; - int ret, is_register, ref_ctr_updated = 0; - bool orig_page_huge = false; + const unsigned long opcode_vaddr, uprobe_opcode_t opcode) +{ + struct uprobe *uprobe = container_of(auprobe, struct uprobe, arch); + const unsigned long vaddr = opcode_vaddr & PAGE_MASK; + const bool is_register = !!is_swbp_insn(&opcode); + struct uwo_data data = { + .opcode = opcode, + .opcode_vaddr = opcode_vaddr, + }; unsigned int gup_flags = FOLL_FORCE; + struct mm_struct *mm = vma->vm_mm; + struct mmu_notifier_range range; + int ret, ref_ctr_updated = 0; + struct page *page; - is_register = is_swbp_insn(&opcode); - uprobe = container_of(auprobe, struct uprobe, arch); + if (WARN_ON_ONCE(!is_cow_mapping(vma->vm_flags))) + return -EINVAL; -retry: + /* + * When registering, we have to break COW to get an exclusive anonymous + * page that we can safely modify. Use FOLL_WRITE to trigger a write + * fault if required. When unregistering, we might be lucky and the + * anon page is already gone. So defer write faults until really + * required. + */ if (is_register) - gup_flags |= FOLL_SPLIT_PMD; - /* Read the page with vaddr into memory */ - ret = get_user_pages_remote(mm, vaddr, 1, gup_flags, &old_page, NULL); + gup_flags |= FOLL_WRITE; + +retry: + ret = get_user_pages_remote(mm, vaddr, 1, gup_flags, &page, NULL); if (ret != 1) - return ret; + goto out; - ret = verify_opcode(old_page, vaddr, &opcode); + ret = verify_opcode(page, opcode_vaddr, &opcode); + put_page(page); if (ret <= 0) - goto put_old; - - if (WARN(!is_register && PageCompound(old_page), - "uprobe unregister should never work on compound page\n")) { - ret = -EINVAL; - goto put_old; - } + goto out; /* We are going to replace instruction, update ref_ctr. */ if (!ref_ctr_updated && uprobe->ref_ctr_offset) { ret = update_ref_ctr(uprobe, mm, is_register ? 1 : -1); if (ret) - goto put_old; - + goto out; ref_ctr_updated = 1; } - ret = 0; - if (!is_register && !PageAnon(old_page)) - goto put_old; - - ret = anon_vma_prepare(vma); - if (ret) - goto put_old; - - ret = -ENOMEM; - new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vaddr); - if (!new_page) - goto put_old; - - __SetPageUptodate(new_page); - copy_highpage(new_page, old_page); - copy_to_page(new_page, vaddr, &opcode, UPROBE_SWBP_INSN_SIZE); - if (!is_register) { - struct page *orig_page; - pgoff_t index; - - VM_BUG_ON_PAGE(!PageAnon(old_page), old_page); - - index = vaddr_to_offset(vma, vaddr & PAGE_MASK) >> PAGE_SHIFT; - orig_page = find_get_page(vma->vm_file->f_inode->i_mapping, - index); - - if (orig_page) { - if (PageUptodate(orig_page) && - pages_identical(new_page, orig_page)) { - /* let go new_page */ - put_page(new_page); - new_page = NULL; - - if (PageCompound(orig_page)) - orig_page_huge = true; - } - put_page(orig_page); - } + /* + * In the common case, we'll be able to zap the page when + * unregistering. So trigger MMU notifiers now, as we won't + * be able to do it under PTL. + */ + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, + vaddr, vaddr + PAGE_SIZE); + mmu_notifier_invalidate_range_start(&range); } - ret = __replace_page(vma, vaddr & PAGE_MASK, old_page, new_page); - if (new_page) - put_page(new_page); -put_old: - put_page(old_page); + /* + * Perform another page table walk and perform the actual page + * modification. + */ + ret = walk_page_range_vma(vma, vaddr, vaddr + PAGE_SIZE, + &write_opcode_ops, &data); - if (unlikely(ret == -EAGAIN)) - goto retry; + if (!is_register) + mmu_notifier_invalidate_range_end(&range); + switch (ret) { + case UWO_NO_PGTABLE: /* walk_page_range_vma() returned 0. */ + /* + * walk_page_range_vma() did not find anything. If there + * would have been a PMD-mapped file THP, it could have + * simply been zapped; in that (unlikely) case, there is + * nothing to when unregistering. + * + * If we had a PMD-mapped anonymous folio, + * walk_page_range_vma() would have PTE-mapped it for us + * and walked it. + */ + if (is_register) + goto retry; + break; + case UWO_RETRY_WRITE_FAULT: + gup_flags |= FOLL_WRITE; + fallthrough; + case UWO_RETRY: + goto retry; + case UWO_DONE_COLLAPSE: + case UWO_DONE: + break; + default: + VM_WARN_ON_ONCE(1); + } +out: /* Revert back reference counter if instruction update failed. */ - if (ret && is_register && ref_ctr_updated) + if (ret < 0 && is_register && ref_ctr_updated) update_ref_ctr(uprobe, mm, -1); /* try collapse pmd for compound page */ - if (!ret && orig_page_huge) + if (ret == UWO_DONE_COLLAPSE) collapse_pte_mapped_thp(mm, vaddr, false); - return ret; + return ret < 0 ? ret : 0; } /**