From patchwork Thu Mar 16 12:18:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhenhua Huang X-Patchwork-Id: 13177501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D755C6FD19 for ; Thu, 16 Mar 2023 12:19:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5316D900003; Thu, 16 Mar 2023 08:19:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5124A900002; Thu, 16 Mar 2023 08:19:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D096900003; Thu, 16 Mar 2023 08:19:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2D5FD900002 for ; Thu, 16 Mar 2023 08:19:04 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 025CE80452 for ; Thu, 16 Mar 2023 12:19:03 +0000 (UTC) X-FDA: 80574665808.30.72CA13B Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by imf20.hostedemail.com (Postfix) with ESMTP id 2BB5B1C0006 for ; Thu, 16 Mar 2023 12:18:59 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=ZSYKbSe4; dmarc=pass (policy=none) header.from=quicinc.com; spf=pass (imf20.hostedemail.com: domain of quic_zhenhuah@quicinc.com designates 205.220.168.131 as permitted sender) smtp.mailfrom=quic_zhenhuah@quicinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678969140; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=pt6m/4EMXuhZyv7EYfDFeBNPiQPywUM74OkTDbTcPuM=; b=eTwYHcQUByZ5gR3zW/5Gr+R/18267rqS9A7S6XhlyjtY3WAkH2fxmD4yYI7VdHZXXZinDH pxbJkVfrue0TZr1IghAbQzivFjrbQquHEg6bYpS4chxkN0x3JMdwUscrytYtQ9XDNBJkFB P/7hu28o53uZ1ZXHliMCr/0FqzgNq7M= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=ZSYKbSe4; dmarc=pass (policy=none) header.from=quicinc.com; spf=pass (imf20.hostedemail.com: domain of quic_zhenhuah@quicinc.com designates 205.220.168.131 as permitted sender) smtp.mailfrom=quic_zhenhuah@quicinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678969140; a=rsa-sha256; cv=none; b=DanfLcHqVqMMYX6t163xib/NT8Sau2AyXh6KKrOVz0Jh89OKLSz9zdbufsoOy5F5WTDDdZ xvj9i7m7qWAOulFK/XuloEFwKV/ZZ7Sm5Noyeq37YrjXY1yK8wEykClNxGyPBUr553AD/P RbXbG7ur8ubxRG4h7FofN0Qpc7cMskE= Received: from pps.filterd (m0279865.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32G4kFOd011172; Thu, 16 Mar 2023 12:18:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : mime-version : content-type; s=qcppdkim1; bh=pt6m/4EMXuhZyv7EYfDFeBNPiQPywUM74OkTDbTcPuM=; b=ZSYKbSe44+uaUUOWIUbOsM+1ar2cpvmfbRuz86B1vXk8uW02DrFOt38hwq/8Vw997vQL 0pc97QDeM1m/VsGDsYmUbEEsJzvqBsIksK0ydIJ58nR2KSQLY58/JsIvjBfpYz67G3fT SvqyCwlzOLT7vecHDM9MNZAYhZed6NKYu/xcE8wqVuzOwIePpKFa+Tsw94VOqC1FG5yA 0RAVvic89wwuXbrukXMdLKCVSrtn5oJW3mL/Lbha57VJvpTzB5F0F8Cllmo1bvF4u691 HnMp3Z8V6A/fwSew0lFPPskzWjmHOasTCRDCnD591vRdVHnpxUGUbqnzwJ7AmGaIfxTl kQ== Received: from nalasppmta04.qualcomm.com (Global_NAT1.qualcomm.com [129.46.96.20]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3pbpxshs6r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 16 Mar 2023 12:18:43 +0000 Received: from nalasex01a.na.qualcomm.com (nalasex01a.na.qualcomm.com [10.47.209.196]) by NALASPPMTA04.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 32GCIgeF012416 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 16 Mar 2023 12:18:42 GMT Received: from zhenhuah-gv.qualcomm.com (10.80.80.8) by nalasex01a.na.qualcomm.com (10.47.209.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 16 Mar 2023 05:18:37 -0700 From: Zhenhua Huang To: , , , , , , , , , , CC: Zhenhua Huang , , , , , , Subject: [PATCH v10] mm,kfence: decouple kfence from page granularity mapping judgement Date: Thu, 16 Mar 2023 20:18:30 +0800 Message-ID: <1678969110-11941-1-git-send-email-quic_zhenhuah@quicinc.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01a.na.qualcomm.com (10.52.223.231) To nalasex01a.na.qualcomm.com (10.47.209.196) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: DCngU5f5BFwvwhE-Qhk82Xat-Ktq3CyO X-Proofpoint-ORIG-GUID: DCngU5f5BFwvwhE-Qhk82Xat-Ktq3CyO X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-16_08,2023-03-16_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 phishscore=0 lowpriorityscore=0 adultscore=0 priorityscore=1501 malwarescore=0 impostorscore=0 mlxscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303160102 X-Rspamd-Queue-Id: 2BB5B1C0006 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: au66wpey3m7oxmhc64poe4okhyb4ibxt X-HE-Tag: 1678969139-615885 X-HE-Meta: U2FsdGVkX1/k8xkEhWwT+gnwBJcIHRVSBF2rm5UN2+DSmVNBX+h2QOU1F/hFFFEzC9H0u7vSDFtDdkH5OUJ9T6uqUX4gAqo6mecputh431rweNfS2+yiqlSV0hVHzoYlU0UjV8wel5ODyXKqzwwhAcbd3qsmb1GHMmSZq0/crhWRSwXefOuZSFQuVlTDYxLM0Ct1N+6UyDLdMCZHPBdSHj1cAknYYdYnrVmTy6Iw3YiQYAPftyJiSECZbJsbSUOg6AHEfR6r/SiMWaLrz2eHCO2oFCYccCUd88PLAzvu4BW6orCr6/6hPQaToZ45YR4MXfSOVVOmX2v7gF+pqdT4SgxkR9qCWYMPDo2v7Q3wHmk1pSXJ/V1pfsJ/vny4rP6QeigLS9nYII3uJ8tGxiguAzV2YCGQV1BFyMTXXRYTasuumW4pY0HuREGirtY8f7FKeEWxDMWbQRV9TUP0ep9g4g4AmBV7KphpvXhBrtBxF/mHEoM4VeHeP9emmDPUysHKw6BSitdLTD9pUwbJkPpkgxNv5DuRXliaNrcybH1D7dM8izMPSyqwRvEqKXzXaMOE/ur3C3z3nOzJJiBjqr0DnDSLjJzyDQu7buuuXQLGhrRkKuGhDRXTRmBEQEgTTB4dhDtWDXO+fYjHk3f36rLN+1qmIEAS3uLpgCx+M629xbXLDVbkuF8rPNDEbEijWZtKxP/+ETCpfI7LQb3nZG9A0pNxCWJfn8PqOA8sk7wCCALwp+99szhaLowMOaIv12gyiwqfwtwKimZ186oL91fxWFS/t81q/FHHZJDK1+ghmnGxONNFxVSxxlKAxuelSrnYaSxXNuNT3m05hiI1IQpCAfTmRtl26tkJgwjqfUUlwEUL3JWRTiytmoVq1i3jlyi9tYwhbYixkvEI6sPrsGVKXyNWBsnsjkOdHF1V/fQIJ5MCHLG7b5nLn/r4WgTTkhCDwVMU23sQz+JpQd224Tu cQy4Wv+w OWkvtFEvc8rox7ooemXUKdLekTawboMAqtmY5JHJk6SVHGQ+zfvexfFtVFbKxnWdEyMbXmVH9judiROkXnmBgC0QGLqBVHRRTwP3BenxmDlQ0QVRJ89SdEagHdSpkEswsycWoD+semvIpVdfyJ5w3oiEi8o/TjrTZp7BNUvfqTrEOo05vBtp91Z7fFwduBaZJm5767hcsSIlMcas7iTzI3LogvPTJtdoLaWuFVA/gno8Zj/O/NLMOBPCa1kV+rJ31KFhZH2YT4xZFMq60CNSiR8R9ziLe4fC7xPp3BFjpnRApOcgWAtsh2zWLxPYo96cWpfMmAjwdfcLWirY3LF2CRc3mxgx3FkXcogUO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Kfence only needs its pool to be mapped as page granularity, if it is inited early. Previous judgement was a bit over protected. From [1], Mark suggested to "just map the KFENCE region a page granularity". So I decouple it from judgement and do page granularity mapping for kfence pool only. Need to be noticed that late init of kfence pool still requires page granularity mapping. Page granularity mapping in theory cost more(2M per 1GB) memory on arm64 platform. Like what I've tested on QEMU(emulated 1GB RAM) with gki_defconfig, also turning off rodata protection: Before: [root@liebao ]# cat /proc/meminfo MemTotal: 999484 kB After: [root@liebao ]# cat /proc/meminfo MemTotal: 1001480 kB To implement this, also relocate the kfence pool allocation before the linear mapping setting up, arm64_kfence_alloc_pool is to allocate phys addr, __kfence_pool is to be set after linear mapping set up. LINK: [1] https://lore.kernel.org/linux-arm-kernel/Y+IsdrvDNILA59UN@FVFF77S0Q05N/ Suggested-by: Mark Rutland Signed-off-by: Zhenhua Huang Reviewed-by: Kefeng Wang --- arch/arm64/include/asm/kfence.h | 10 +++++++ arch/arm64/mm/mmu.c | 61 +++++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/pageattr.c | 7 +++-- mm/kfence/core.c | 4 +++ 4 files changed, 80 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kfence.h b/arch/arm64/include/asm/kfence.h index aa855c6..a81937f 100644 --- a/arch/arm64/include/asm/kfence.h +++ b/arch/arm64/include/asm/kfence.h @@ -19,4 +19,14 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) return true; } +#ifdef CONFIG_KFENCE +extern bool kfence_early_init; +static inline bool arm64_kfence_can_set_direct_map(void) +{ + return !kfence_early_init; +} +#else /* CONFIG_KFENCE */ +static inline bool arm64_kfence_can_set_direct_map(void) { return false; } +#endif /* CONFIG_KFENCE */ + #endif /* __ASM_KFENCE_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ae25524d..aaf1801 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -38,6 +39,7 @@ #include #include #include +#include #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) @@ -521,12 +523,67 @@ static int __init enable_crash_mem_map(char *arg) } early_param("crashkernel", enable_crash_mem_map); +#ifdef CONFIG_KFENCE + +bool kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL; + +/* early_param() will be parsed before map_mem() below. */ +static int __init parse_kfence_early_init(char *arg) +{ + int val; + + if (get_option(&arg, &val)) + kfence_early_init = !!val; + return 0; +} +early_param("kfence.sample_interval", parse_kfence_early_init); + +static phys_addr_t arm64_kfence_alloc_pool(void) +{ + phys_addr_t kfence_pool; + + if (!kfence_early_init) + return 0; + + kfence_pool = memblock_phys_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); + if (!kfence_pool) { + pr_err("failed to allocate kfence pool\n"); + kfence_early_init = false; + return 0; + } + + /* Temporarily mark as NOMAP. */ + memblock_mark_nomap(kfence_pool, KFENCE_POOL_SIZE); + + return kfence_pool; +} + +static void arm64_kfence_map_pool(phys_addr_t kfence_pool, pgd_t *pgdp) +{ + if (!kfence_pool) + return; + + /* KFENCE pool needs page-level mapping. */ + __map_memblock(pgdp, kfence_pool, kfence_pool + KFENCE_POOL_SIZE, + pgprot_tagged(PAGE_KERNEL), + NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS); + memblock_clear_nomap(kfence_pool, KFENCE_POOL_SIZE); + __kfence_pool = phys_to_virt(kfence_pool); +} +#else /* CONFIG_KFENCE */ + +static inline phys_addr_t arm64_kfence_alloc_pool(void) { return 0; } +static inline void arm64_kfence_map_pool(phys_addr_t kfence_pool, pgd_t *pgdp) { } + +#endif /* CONFIG_KFENCE */ + static void __init map_mem(pgd_t *pgdp) { static const u64 direct_map_end = _PAGE_END(VA_BITS_MIN); phys_addr_t kernel_start = __pa_symbol(_stext); phys_addr_t kernel_end = __pa_symbol(__init_begin); phys_addr_t start, end; + phys_addr_t early_kfence_pool; int flags = NO_EXEC_MAPPINGS; u64 i; @@ -539,6 +596,8 @@ static void __init map_mem(pgd_t *pgdp) */ BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end)); + early_kfence_pool = arm64_kfence_alloc_pool(); + if (can_set_direct_map()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; @@ -604,6 +663,8 @@ static void __init map_mem(pgd_t *pgdp) } } #endif + + arm64_kfence_map_pool(early_kfence_pool, pgdp); } void mark_rodata_ro(void) diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index debdecf..dd1291a 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -11,6 +11,7 @@ #include #include #include +#include struct page_change_data { pgprot_t set_mask; @@ -22,12 +23,14 @@ bool rodata_full __ro_after_init = IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT_ENABLED bool can_set_direct_map(void) { /* - * rodata_full, DEBUG_PAGEALLOC and KFENCE require linear map to be + * rodata_full and DEBUG_PAGEALLOC require linear map to be * mapped at page granularity, so that it is possible to * protect/unprotect single pages. + * + * KFENCE pool requires page-granular mapping if initialized late. */ return (rodata_enabled && rodata_full) || debug_pagealloc_enabled() || - IS_ENABLED(CONFIG_KFENCE); + arm64_kfence_can_set_direct_map(); } static int change_page_range(pte_t *ptep, unsigned long addr, void *data) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 1417888..bf2f194c 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -824,6 +824,10 @@ void __init kfence_alloc_pool(void) if (!kfence_sample_interval) return; + /* if the pool has already been initialized by arch, skip the below. */ + if (__kfence_pool) + return; + __kfence_pool = memblock_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); if (!__kfence_pool)