From patchwork Fri Jul 12 17:00:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DD71C2BD09 for ; Fri, 12 Jul 2024 17:02:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93B006B00B1; Fri, 12 Jul 2024 13:01:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 875AF6B00B2; Fri, 12 Jul 2024 13:01:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 679056B00B3; Fri, 12 Jul 2024 13:01:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 43DDC6B00B1 for ; Fri, 12 Jul 2024 13:01:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D7B571C1F0D for ; Fri, 12 Jul 2024 17:01:48 +0000 (UTC) X-FDA: 82331717496.06.014C6F6 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf22.hostedemail.com (Postfix) with ESMTP id 8F5E8C001D for ; Fri, 12 Jul 2024 17:01:46 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d+7XX2wT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3eGGRZggKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3eGGRZggKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803668; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9uYrhf+8D7jqi5tPdexW+5CSq/ZRPpRaUw7HFMG3mLc=; b=gyxISH8DVobNB9+ENK1SBhEegoe0dByJZqwHOvbI1oeMOr7stqHAz9dLaArkxoOiNSWfCz YiCDsrJ3igXcJg/2ssq+/YNo/c2bSEKSQy/06mYNt75QU6NjBhI9Xiinizbp6J+6T9QV76 HQ6jML2f0oPxJRkAiXv9zMIL1CXrq2I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803668; a=rsa-sha256; cv=none; b=q0Rf9dvLH1I2zItSB2q+sXA8IgBRmhmQG831K5LBPTdMmF1jIW5OyoPfBOOp85tdqhS2N/ VPaymUdJK2SYedI+soSCt4bSkARnga8duGtDfGW6V9xJBZfAQx/QgzoASMqtIU2r5Np48x Md/HBpKToxf0OL9HfYMOIKmmJNKbSYY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d+7XX2wT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3eGGRZggKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3eGGRZggKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-426620721c2so15371235e9.2 for ; Fri, 12 Jul 2024 10:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803705; x=1721408505; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9uYrhf+8D7jqi5tPdexW+5CSq/ZRPpRaUw7HFMG3mLc=; b=d+7XX2wTN/SEWLNVHOCQWKyAhCzdMkRz3wHrg4c2SRfX45cgbkw+dEltlM8E0lYAiR 4oPsJurnVta8NijDYSAMvHN/v9GU4bqDDEzW9sNzdc9HH/tY8SjQUZeff6vdJ3u8a/KL pbFHzari8T5AK0jS8xC34Oqb+zsGKll+0geHYkmZFng/DlxpA8sLknyvVcLo1yyKdhwC MFFe4RCl20JM6VHyEySe83mzQf8vqsfwtDM+hna45IxOuirF4w754gEzSeie3N323dJU uJhetRSUCHTn8bq6hBRUKf2roim3mzmQ/tBR7ICOGko4OZOLNZdcpHMBdhPVtgV/0IMD mhDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803705; x=1721408505; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9uYrhf+8D7jqi5tPdexW+5CSq/ZRPpRaUw7HFMG3mLc=; b=Tz3/mhUV/xbRFZq8lDq6zmE5Ycv469PSFg2BadsTpLFtX+kKxFesFuzHkso5VwCdAw PfVXxZh5TnOsVO381iHjUfV1OnE7PupqodZIacw/EXIzsONdZypiM7qzMowhUhnyd1Pm 0kTfJfCNCsakJMYxRE/eI1d4HjyGxjhvFP8YxSqh4Aa3FtK1dpoxQqKVgCkVEnwzuQta hsnSQJ0ychjG0/ANjfMJCv1yQ+Al1Vj/8N+/HIjTWxA7iEHJq1hA6gaSajvHLyAAQYs/ IIqTpvTvckwtdc/nSgkVDxtSo0oG89uRjM6cd4Ye1LzEySNXww/xywyAhEgnZEpLWlNk 2SGA== X-Forwarded-Encrypted: i=1; AJvYcCUE/hjVczjKLjv5FmOcLChOeoBJCvU4ILmvVIYaKayT2C8xjEWwiOSTwvDkSlpy1WkIYyj6aWnbz987YHAed7+b7Ug= X-Gm-Message-State: AOJu0YzNsGakICwR0BZZo4vjV1dX52Tn+QTK9IFmQUj6G8Se6YDSjK49 BifOlo0XXyt2p0Kkn1mra9v3y9vAsphu7kVu7tY42XOwVm9Xk8qBt05+VWD6MYcKlPYcWU2ZBdV sHFHGDjLqgw== X-Google-Smtp-Source: AGHT+IEAch1YA6Hi8yZ9PqgivQRgArWS9QwTp9vfGmMIf5En6W6fVDOrfLBr4fzHwEnBua+4wRIDTzrg/vklEA== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:19cc:b0:426:6c7a:3a77 with SMTP id 5b1f17b1804b1-426705ced5bmr1906465e9.1.1720803704718; Fri, 12 Jul 2024 10:01:44 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:36 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-18-144b319a40d8@google.com> Subject: [PATCH 18/26] mm: asi: Map vmalloc/vmap data as nonsesnitive From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8F5E8C001D X-Stat-Signature: ohr4wpps5qfgg8kg7u9oz87kn8pqjqyb X-Rspam-User: X-HE-Tag: 1720803706-469418 X-HE-Meta: U2FsdGVkX1+/pvCehfLH5ulu0bPQ4puWKhOY+fw+5lpUKYWM+TlxklbHbUWHM/Jg21msDFASt0Kn4nE4omzCSJjePJkdaE5LFF8M70wHyfWYSr5WBMZEiKwj5KRewyyeKTwgfjqcFQ9w2qDYb2P2y0q5iXc2T5QZd2USIkOwb8S+z/ZqU1kNNvyAAGAG5GcnR6LrSRhA7c1sYb4w5zhQtJynooYxzg5RV5r2LA0QMD/kZ6PRde6WbNyLw5Y5jovuFp3PfxOjLWVJdfyh/D5JFUjhlIrrfawQMjrnn6aPAoSCsQBgJSUUdzLYCDsly1BFpf7RXRxqGOrGYbGJxVN/Xc2gjJ1LoWh+yn6aVhPojybVgAAl8O1f/F2aEMrchrWtSuAJx49Yphr5fBhPOSuCD9og0Fn8bVun+qODFwSf4Q8YBaFFo2cNwkzl3+XLJGhkvDNSRHctPVBpjbzpvReiPgmvjSDNHWhI3WaPqkmeCW3L1/c5Ivbd098z5MlDfmh6OdBViUOHnbDOSrZRQZAO4pOvk8ABeJe0idjN9qR+u1TW0r2QHHKfGaoDEZdOPqVB++QL0OdTGZxC+3gzzGwCHnuRnPdAyTS2fNEUmCyg+aoO0K7bUfHaXWcbNlHp517S0VAlC9nlMOZEr4Y5tOOrGsbmdFqCr6oO4PQO0YgV+7YS0T4gwQnRDZpN6EgOLxPBhVaLJal0RaWK0L83u4s27b6ePsOm7ucYP1LtxU7msLNv+8j2JlHfGdcIQk+nfXP88R1fI4nrmA31Yn/jvQRbiqgzgK9Quzu3lqIc8ycogIgzKwYgSTa1EUhtT3QeCf+WIAKxzN4fIwgklbPzMDzG8+r2CExLHmRSzIQxxXT/99puVbLsByDgOx6uGEhPnNPBXd1kACz+TGSHmlS2DcwZadM4Uu094w4xv5hr9baaSS/IlJUpVV36fGEVV+prR0aEGt+hzyPWO0D82P94gFP hZmvY9KT GkzrQPJDVimFDXZpQ6+RBGdFUAm3oNKxTL6Kg588oxRhptuTYSbJmJnWtxRdPJo6gWCztgMZ/hck/LxPZUhuZ1jJXK5OwRjb3tepKDOcOKnGaFKkSmvfRMF4EsBzPnHmauBuEQggg3nNgZINhdpijIQxbhTgnMrQtY0FYRZqin3GN1Rk+78cyvRUDwlKwd7Nt3/Tqh5QdvU13AlwBFe3t2WXv5pUEm8xXzNOOm29kXze+5qnSIgPXY9C2Gqu0RnPMYQwHNY9rDGIKOb5InmhyudmGFwSHafMu1m29Awe2t7aYj9trI6f1pr/39vBP08fgRUNLCgwE/oxLvYLF54wmhfZrtVZ0P5gQ2EzdZRTkFjlHJni1q7aiBu6UmSX7l4xFwCPCrUH1s2OkiEbXdvP0G/V0OSMniRULwtuJV/73RoKLqp8UfdcQ4f+Fiiz6GuD9DV9w+K9oqREI5ZEjHG3s+zEVt8Ml6/Fi8GD2Ldc/5W+m+O5j65LLFMjo4UrIah5GN11GSqRjGjvLLO1V/3vpM7iFkN+yj53/aDEq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We add new VM flags for sensitive and global-nonsensitive, parallel to the corresponding GFP flags. __get_vm_area_node and friends will default to creating global-nonsensitive VM areas, and vmap then calls asi_map as necessary. __vmalloc_node_range has additional logic to check and set defaults for the sensitivity of the underlying page allocation. It does this via an initial __set_asi_flags call - note that it then calls __get_vm_area_node which also calls __set_asi_flags. This second call is a NOP. By default, we mark the underlying page allocation as sensitive, even if the VM area is global-nonsensitive. This is just an optimization to avoid unnecessary asi_map etc, since presumably most code has no reason to access vmalloc'd data through the direct map. There are some details of the GFP-flag/VM-flag interaction that are not really obvious, for example: what should happen when callers of __vmalloc explicitly set GFP sensitivity flags? (That function has no VM flags argument). For the moment let's just not block on that and focus on adding the infastructure, though. At the moment, the high-level vmalloc APIs doesn't actually provide a way to conffigure sensitivity, this commit just adds the infrastructure. We'll have to decide how to expose this to allocation sites as we implement more denylist logic. vmap does already allow configuring vm flags. Signed-off-by: Brendan Jackman --- mm/vmalloc.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 7a8daf5afb7c..d14e2f692e42 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3189,6 +3189,7 @@ struct vm_struct *remove_vm_area(const void *addr) { struct vmap_area *va; struct vm_struct *vm; + unsigned long vm_addr; might_sleep(); @@ -3200,6 +3201,7 @@ struct vm_struct *remove_vm_area(const void *addr) if (!va || !va->vm) return NULL; vm = va->vm; + vm_addr = (unsigned long) READ_ONCE(vm->addr); debug_check_no_locks_freed(vm->addr, get_vm_area_size(vm)); debug_check_no_obj_freed(vm->addr, get_vm_area_size(vm)); @@ -3331,6 +3333,7 @@ void vfree(const void *addr) addr); return; } + asi_unmap(ASI_GLOBAL_NONSENSITIVE, vm->addr, get_vm_area_size(vm)); if (unlikely(vm->flags & VM_FLUSH_RESET_PERMS)) vm_reset_perms(vm); @@ -3370,12 +3373,14 @@ void vunmap(const void *addr) if (!addr) return; + vm = remove_vm_area(addr); if (unlikely(!vm)) { WARN(1, KERN_ERR "Trying to vunmap() nonexistent vm area (%p)\n", addr); return; } + asi_unmap(ASI_GLOBAL_NONSENSITIVE, vm->addr, get_vm_area_size(vm)); kfree(vm); } EXPORT_SYMBOL(vunmap); @@ -3424,16 +3429,21 @@ void *vmap(struct page **pages, unsigned int count, addr = (unsigned long)area->addr; if (vmap_pages_range(addr, addr + size, pgprot_nx(prot), - pages, PAGE_SHIFT) < 0) { - vunmap(area->addr); - return NULL; - } + pages, PAGE_SHIFT) < 0) + goto err; + + if (asi_map(ASI_GLOBAL_NONSENSITIVE, area->addr, + get_vm_area_size(area))) + goto err; /* The necessary asi_unmap() is in vunmap. */ if (flags & VM_MAP_PUT_PAGES) { area->pages = pages; area->nr_pages = count; } return area->addr; +err: + vunmap(area->addr); + return NULL; } EXPORT_SYMBOL(vmap); @@ -3701,6 +3711,10 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, goto fail; } + if (asi_map(ASI_GLOBAL_NONSENSITIVE, area->addr, + get_vm_area_size(area))) + goto fail; /* The necessary asi_unmap() is in vfree. */ + return area->addr; fail: @@ -3780,6 +3794,13 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, size = ALIGN(real_size, 1UL << shift); } + /* + * Assume nobody is interested in accessing these pages via the direct + * map, so there's no point in having them in ASI's global-nonsensitive + * physmap, which would just cost us a TLB flush later on. + */ + gfp_mask |= __GFP_SENSITIVE; + again: area = __get_vm_area_node(real_size, align, shift, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node,