From patchwork Thu Jun 9 18:18:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 12875897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4D4BC43334 for ; Thu, 9 Jun 2022 18:18:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69F1D8D0037; Thu, 9 Jun 2022 14:18:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64ED38D0034; Thu, 9 Jun 2022 14:18:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 517278D0037; Thu, 9 Jun 2022 14:18:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 44B718D0034 for ; Thu, 9 Jun 2022 14:18:52 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 1958D6152C for ; Thu, 9 Jun 2022 18:18:52 +0000 (UTC) X-FDA: 79559508504.19.5B25A7C Received: from out2.migadu.com (out2.migadu.com [188.165.223.204]) by imf02.hostedemail.com (Postfix) with ESMTP id 7A39A8003E for ; Thu, 9 Jun 2022 18:18:51 +0000 (UTC) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1654798730; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=EsGrAYAp6IJFxcDVn2/7Z4jT50O8LabBUCsC9BUqAmM=; b=XNdLbz37TBFf1v0b6csPsvWj5sDWbvqbE7S1KwnxS4SfiEscQLGgb7f5GXJzS9M1Gfrm0u tNERjdG7bv4iyPAyQkwUJmp5G6Q6965x/8gnLOV1CuT2rdkawheBHtlDCc/DurG82Y82fe J9SQ+dLevT+gkykVqB7AQ/67U/n49os= From: andrey.konovalov@linux.dev To: Andrew Morton Cc: Andrey Konovalov , Marco Elver , Alexander Potapenko , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 1/3] mm: rename kernel_init_free_pages to kernel_init_pages Date: Thu, 9 Jun 2022 20:18:45 +0200 Message-Id: <1ecaffc0a9c1404d4d7cf52efe0b2dc8a0c681d8.1654798516.git.andreyknvl@google.com> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1654798731; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=EsGrAYAp6IJFxcDVn2/7Z4jT50O8LabBUCsC9BUqAmM=; b=GuPDoxWVQ0342ZZIWxo8g61jTMimGNt94wnxhQMzUC5tGgakawHQVutA32St5jNbJU+X3E iLLhphsRAolCmqHZ4Y0rBhYseKBBXZ9mRkZx4F6vfaOGQUzDuF6X+A2VGhLZsvRfASw4Ll PwxFBLFRr4XmmKWWTKBP3vWdKENukKo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1654798731; a=rsa-sha256; cv=none; b=Agle+QoxB7X52PJeStiZ6JlIGFwFCz12M/WI+CexfR+ZmYPy26oYHutsNgaYRxySNoA7pg wVxG624k61TNU+DL8xKshXj2LMnL6Fna4FMSNOsx1do66w6U5JhN0ZoRU6x0+/7yxHuWCc h+u0C1J0BBrPZzrE/O43yeWPNeOrKlA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=XNdLbz37; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf02.hostedemail.com: domain of andrey.konovalov@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev X-Rspamd-Server: rspam11 X-Rspam-User: Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=XNdLbz37; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf02.hostedemail.com: domain of andrey.konovalov@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev X-Stat-Signature: nxysbx3wdzomi6pcny8uq8gx54jhj3t5 X-Rspamd-Queue-Id: 7A39A8003E X-HE-Tag: 1654798731-995339 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Rename kernel_init_free_pages() to kernel_init_pages(). This function is not only used for free pages but also for pages that were just allocated. Signed-off-by: Andrey Konovalov Reviewed-by: Muchun Song --- mm/page_alloc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e008a3df0485..66ef8c310dce 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1296,7 +1296,7 @@ static inline bool should_skip_kasan_poison(struct page *page, fpi_t fpi_flags) PageSkipKASanPoison(page); } -static void kernel_init_free_pages(struct page *page, int numpages) +static void kernel_init_pages(struct page *page, int numpages) { int i; @@ -1396,7 +1396,7 @@ static __always_inline bool free_pages_prepare(struct page *page, init = false; } if (init) - kernel_init_free_pages(page, 1 << order); + kernel_init_pages(page, 1 << order); /* * arch_free_page() can make the page's contents inaccessible. s390 @@ -2441,7 +2441,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, } /* If memory is still not initialized, do it now. */ if (init) - kernel_init_free_pages(page, 1 << order); + kernel_init_pages(page, 1 << order); /* Propagate __GFP_SKIP_KASAN_POISON to page flags. */ if (kasan_hw_tags_enabled() && (gfp_flags & __GFP_SKIP_KASAN_POISON)) SetPageSkipKASanPoison(page); From patchwork Thu Jun 9 18:18:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 12875898 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E177C433EF for ; Thu, 9 Jun 2022 18:18:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0117E8D0038; Thu, 9 Jun 2022 14:18:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F01658D0034; Thu, 9 Jun 2022 14:18:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC95C8D0038; Thu, 9 Jun 2022 14:18:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CBB858D0034 for ; Thu, 9 Jun 2022 14:18:52 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9D46A35BE6 for ; Thu, 9 Jun 2022 18:18:52 +0000 (UTC) X-FDA: 79559508504.22.71F9FD8 Received: from out2.migadu.com (out2.migadu.com [188.165.223.204]) by imf21.hostedemail.com (Postfix) with ESMTP id 2DFFB1C007D for ; Thu, 9 Jun 2022 18:18:52 +0000 (UTC) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1654798731; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fO4jbpVApVQcfTPwx9gBa9O1ee1UTzsQG3M5HGLfNnA=; b=Xa3doeBqLfHwMDbSeTb4ZiFXvBmIjIofPaa33n/qvrG/uWiPfuEdLm4uM0UENmXHeStFzI 05B2JyJIU0BKhPX6l1OEPU9ljfiSO4uJc8a8Ta+g8lK5davYDlg+I/F1PNsr9pOnGgg1I3 Or3TzC6G5wLjJyGaTo+AhgrnmvED5fw= From: andrey.konovalov@linux.dev To: Andrew Morton Cc: Andrey Konovalov , Marco Elver , Alexander Potapenko , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 2/3] mm: introduce clear_highpage_kasan_tagged Date: Thu, 9 Jun 2022 20:18:46 +0200 Message-Id: <4471979b46b2c487787ddcd08b9dc5fedd1b6ffd.1654798516.git.andreyknvl@google.com> In-Reply-To: <1ecaffc0a9c1404d4d7cf52efe0b2dc8a0c681d8.1654798516.git.andreyknvl@google.com> References: <1ecaffc0a9c1404d4d7cf52efe0b2dc8a0c681d8.1654798516.git.andreyknvl@google.com> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1654798732; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fO4jbpVApVQcfTPwx9gBa9O1ee1UTzsQG3M5HGLfNnA=; b=hYeLoh/Ry8sIk32kQ9qUKLLT7B9TcA05omYdedKAESlOijfaJbS62EmFRq8gWiWdGsU662 kqhM5kmU/FcIGwawYp3ql+eSQexm1B+8kikg4/FdAVzE7gkegb6KKAOEkk2tOAMKmg4KaE p1S6QT55trK7S28TwRdgRKf1d7CmtxY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1654798732; a=rsa-sha256; cv=none; b=BsXJILZMJg+e9mBq3P2IL7I9TI1btB97j6I5bfMXQ8tAXmVmKLdBRGQphepW43mWpSskOl kuuMobvUhACz5oMhCNNpxAkYGIj9YUO74MANtZuvTt3SidelW5ivcGjBYef9H1CMIH0AmW Igv77H+rBDkm8/VLP0ij8d72bWeUsdM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Xa3doeBq; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of andrey.konovalov@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev X-Rspamd-Server: rspam11 X-Rspam-User: Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=Xa3doeBq; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf21.hostedemail.com: domain of andrey.konovalov@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev X-Stat-Signature: tayaggdyp3a7aughzid3qmif6u7epjkj X-Rspamd-Queue-Id: 2DFFB1C007D X-HE-Tag: 1654798732-240451 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov Add a clear_highpage_kasan_tagged() helper that does clear_highpage() on a page potentially tagged by KASAN. This helper is used by the following patch. Signed-off-by: Andrey Konovalov --- Changes v1->v2: - Renamed clear_highpage_tagged() to clear_highpage_kasan_tagged(). - Removed extra empty line in clear_highpage_kasan_tagged(). --- include/linux/highmem.h | 10 ++++++++++ mm/page_alloc.c | 8 ++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/include/linux/highmem.h b/include/linux/highmem.h index 3af34de54330..70b496bbd2d9 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -243,6 +243,16 @@ static inline void clear_highpage(struct page *page) kunmap_local(kaddr); } +static inline void clear_highpage_kasan_tagged(struct page *page) +{ + u8 tag; + + tag = page_kasan_tag(page); + page_kasan_tag_reset(page); + clear_highpage(page); + page_kasan_tag_set(page, tag); +} + #ifndef __HAVE_ARCH_TAG_CLEAR_HIGHPAGE static inline void tag_clear_highpage(struct page *page) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 66ef8c310dce..76a02255f57c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1302,12 +1302,8 @@ static void kernel_init_pages(struct page *page, int numpages) /* s390's use of memset() could override KASAN redzones. */ kasan_disable_current(); - for (i = 0; i < numpages; i++) { - u8 tag = page_kasan_tag(page + i); - page_kasan_tag_reset(page + i); - clear_highpage(page + i); - page_kasan_tag_set(page + i, tag); - } + for (i = 0; i < numpages; i++) + clear_highpage_kasan_tagged(page + i); kasan_enable_current(); } From patchwork Thu Jun 9 18:18:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 12875899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B361C43334 for ; Thu, 9 Jun 2022 18:18:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1980B8D003B; Thu, 9 Jun 2022 14:18:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 147068D0034; Thu, 9 Jun 2022 14:18:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D54B88D003B; Thu, 9 Jun 2022 14:18:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C2B908D0034 for ; Thu, 9 Jun 2022 14:18:53 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 9484580FE6 for ; Thu, 9 Jun 2022 18:18:53 +0000 (UTC) X-FDA: 79559508546.10.B2EEF79 Received: from out2.migadu.com (out2.migadu.com [188.165.223.204]) by imf25.hostedemail.com (Postfix) with ESMTP id C09F5A0083 for ; Thu, 9 Jun 2022 18:18:52 +0000 (UTC) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1654798731; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eJy0WE1RZ6sxBZ/XENlV0bykPV6S6KO6JYi7Cn8ktdA=; b=ubs0Oifr9NeLgGYjoYCMK34TNA3W4sNPWTEZZCiza1bfWHwBX8EyPUpXrZHU7Cv6hCE5ak y1DkJYdc4Y3Oq6IOfRNfb63wfsGYD2DmY1lkte1HR6+HEYN5qgzkAsz0EloDVX9StUcauf ZW1okNYxSRCkWhmaPjux1cYCxrR7VrM= From: andrey.konovalov@linux.dev To: Andrew Morton Cc: Andrey Konovalov , Marco Elver , Alexander Potapenko , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 3/3] kasan: fix zeroing vmalloc memory with HW_TAGS Date: Thu, 9 Jun 2022 20:18:47 +0200 Message-Id: <4bc503537efdc539ffc3f461c1b70162eea31cf6.1654798516.git.andreyknvl@google.com> In-Reply-To: <1ecaffc0a9c1404d4d7cf52efe0b2dc8a0c681d8.1654798516.git.andreyknvl@google.com> References: <1ecaffc0a9c1404d4d7cf52efe0b2dc8a0c681d8.1654798516.git.andreyknvl@google.com> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1654798733; a=rsa-sha256; cv=none; b=pt56cMzaUEQp+3mw3U2abA/89hwCiUJLLQkK66f68qjYWNfyxbdwOFpQ55y9BkrwXXR13F A9m1FjcaWEBxMgSmMen/4ic2e795D12bAvNRuL/lUqtos6/je+ndcP9RBX68U+/9FITxJm veswhDOsz5ePDS2aF2mvxCXY1zuKt5c= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ubs0Oifr; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of andrey.konovalov@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1654798733; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eJy0WE1RZ6sxBZ/XENlV0bykPV6S6KO6JYi7Cn8ktdA=; b=rPaRo6qdD+preLZpHkFLbbv4neJYv27cTfNGgYi/6SKD2S2bDTV6VqefEfG6g1C2jMbBD8 0kQXs9IMPirgwAiMjmsw9x4RBeFYE+xxd60gy0RKseEfzjE03BEOTGL+DrlAYpoYzkOyfG zCWdOmy7AK8OEONsP+AoQDtpTc6DfAE= X-Rspamd-Queue-Id: C09F5A0083 X-Rspam-User: Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=ubs0Oifr; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf25.hostedemail.com: domain of andrey.konovalov@linux.dev designates 188.165.223.204 as permitted sender) smtp.mailfrom=andrey.konovalov@linux.dev X-Stat-Signature: jfuysmf167j8a9mwawukq7btp9gqbduj X-Rspamd-Server: rspam02 X-HE-Tag: 1654798732-611559 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Andrey Konovalov HW_TAGS KASAN skips zeroing page_alloc allocations backing vmalloc mappings via __GFP_SKIP_ZERO. Instead, these pages are zeroed via kasan_unpoison_vmalloc() by passing the KASAN_VMALLOC_INIT flag. The problem is that __kasan_unpoison_vmalloc() does not zero pages when either kasan_vmalloc_enabled() or is_vmalloc_or_module_addr() fail. Thus: 1. Change __vmalloc_node_range() to only set KASAN_VMALLOC_INIT when __GFP_SKIP_ZERO is set. 2. Change __kasan_unpoison_vmalloc() to always zero pages when the KASAN_VMALLOC_INIT flag is set. 3. Add WARN_ON() asserts to check that KASAN_VMALLOC_INIT cannot be set in other early return paths of __kasan_unpoison_vmalloc(). Also clean up the comment in __kasan_unpoison_vmalloc. Fixes: 23689e91fb22 ("kasan, vmalloc: add vmalloc tagging for HW_TAGS") Signed-off-by: Andrey Konovalov --- Changes v1->v2: - Add init_vmalloc_pages() helper. --- mm/kasan/hw_tags.c | 32 +++++++++++++++++++++++--------- mm/vmalloc.c | 10 +++++----- 2 files changed, 28 insertions(+), 14 deletions(-) diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c index 9e1b6544bfa8..9ad8eff71b28 100644 --- a/mm/kasan/hw_tags.c +++ b/mm/kasan/hw_tags.c @@ -257,27 +257,37 @@ static void unpoison_vmalloc_pages(const void *addr, u8 tag) } } +static void init_vmalloc_pages(const void *start, unsigned long size) +{ + const void *addr; + + for (addr = start; addr < start + size; addr += PAGE_SIZE) { + struct page *page = virt_to_page(addr); + + clear_highpage_kasan_tagged(page); + } +} + void *__kasan_unpoison_vmalloc(const void *start, unsigned long size, kasan_vmalloc_flags_t flags) { u8 tag; unsigned long redzone_start, redzone_size; - if (!kasan_vmalloc_enabled()) - return (void *)start; - - if (!is_vmalloc_or_module_addr(start)) + if (!kasan_vmalloc_enabled() || !is_vmalloc_or_module_addr(start)) { + if (flags & KASAN_VMALLOC_INIT) + init_vmalloc_pages(start, size); return (void *)start; + } /* - * Skip unpoisoning and assigning a pointer tag for non-VM_ALLOC - * mappings as: + * Don't tag non-VM_ALLOC mappings, as: * * 1. Unlike the software KASAN modes, hardware tag-based KASAN only * supports tagging physical memory. Therefore, it can only tag a * single mapping of normal physical pages. * 2. Hardware tag-based KASAN can only tag memory mapped with special - * mapping protection bits, see arch_vmalloc_pgprot_modify(). + * mapping protection bits, see arch_vmap_pgprot_tagged(). * As non-VM_ALLOC mappings can be mapped outside of vmalloc code, * providing these bits would require tracking all non-VM_ALLOC * mappers. @@ -289,15 +299,19 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size, * * For non-VM_ALLOC allocations, page_alloc memory is tagged as usual. */ - if (!(flags & KASAN_VMALLOC_VM_ALLOC)) + if (!(flags & KASAN_VMALLOC_VM_ALLOC)) { + WARN_ON(flags & KASAN_VMALLOC_INIT); return (void *)start; + } /* * Don't tag executable memory. * The kernel doesn't tolerate having the PC register tagged. */ - if (!(flags & KASAN_VMALLOC_PROT_NORMAL)) + if (!(flags & KASAN_VMALLOC_PROT_NORMAL)) { + WARN_ON(flags & KASAN_VMALLOC_INIT); return (void *)start; + } tag = kasan_random_tag(); start = set_tag(start, tag); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 07db42455dd4..0adf4aa1514d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3168,15 +3168,15 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, /* * Mark the pages as accessible, now that they are mapped. - * The init condition should match the one in post_alloc_hook() - * (except for the should_skip_init() check) to make sure that memory - * is initialized under the same conditions regardless of the enabled - * KASAN mode. + * The condition for setting KASAN_VMALLOC_INIT should complement the + * one in post_alloc_hook() with regards to the __GFP_SKIP_ZERO check + * to make sure that memory is initialized under the same conditions. * Tag-based KASAN modes only assign tags to normal non-executable * allocations, see __kasan_unpoison_vmalloc(). */ kasan_flags |= KASAN_VMALLOC_VM_ALLOC; - if (!want_init_on_free() && want_init_on_alloc(gfp_mask)) + if (!want_init_on_free() && want_init_on_alloc(gfp_mask) && + (gfp_mask & __GFP_SKIP_ZERO)) kasan_flags |= KASAN_VMALLOC_INIT; /* KASAN_VMALLOC_PROT_NORMAL already set if required. */ area->addr = kasan_unpoison_vmalloc(area->addr, real_size, kasan_flags);