From patchwork Thu Mar 16 08:50:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhenhua Huang X-Patchwork-Id: 13177214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1352BC6FD1F for ; Thu, 16 Mar 2023 08:50:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 35FAC900003; Thu, 16 Mar 2023 04:50:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3103E900002; Thu, 16 Mar 2023 04:50:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D81D900003; Thu, 16 Mar 2023 04:50:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0FDCE900002 for ; Thu, 16 Mar 2023 04:50:57 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B396D407FE for ; Thu, 16 Mar 2023 08:50:56 +0000 (UTC) X-FDA: 80574141312.01.479EBDB Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by imf12.hostedemail.com (Postfix) with ESMTP id A23ED4000F for ; Thu, 16 Mar 2023 08:50:54 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=NSFDRT5F; spf=pass (imf12.hostedemail.com: domain of quic_zhenhuah@quicinc.com designates 205.220.180.131 as permitted sender) smtp.mailfrom=quic_zhenhuah@quicinc.com; dmarc=pass (policy=none) header.from=quicinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678956654; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=e82hxHn52yqSiK6HiEB47+42nkEcrdxP6ydPxrg8nW8=; b=ktaknfONOldnfvwyiq+e8/jEfJ6pXYJKirzgYSBlj4d1C7ljwx2EHg8a1MtWIrr0mh2LlQ zPJvH71fZ89Nu59TbbGPOLMwxHI/AR1odF3we5fvE5gJcUHuqb3NNTsSk12pxhR92akjVu /SzhnUy+/DW8jDA8WGd+b3jtXccLEYk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=NSFDRT5F; spf=pass (imf12.hostedemail.com: domain of quic_zhenhuah@quicinc.com designates 205.220.180.131 as permitted sender) smtp.mailfrom=quic_zhenhuah@quicinc.com; dmarc=pass (policy=none) header.from=quicinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678956654; a=rsa-sha256; cv=none; b=0og1DzK34Qe1KIzVgiCbjmXDrhTcsmGeKv/0F85KX+wFXEb+GQejwb8CLHjTmZBTNUltkT p7AvS3MPJWtYdYfDaeMGl5ES6/yrdzdUYGVEWOnYfDyY79VUmXGW2KEdRHBqf7UGZnBEvq yM1lItZaGxtWu7zH9sS6JZc1HwESb84= Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32G6I5GD029854; Thu, 16 Mar 2023 08:50:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : mime-version : content-type; s=qcppdkim1; bh=e82hxHn52yqSiK6HiEB47+42nkEcrdxP6ydPxrg8nW8=; b=NSFDRT5Fd4Qvpg3nFILSr00slArVf+NOOGA85kObBI5dTt4XOwkBXhqs3f+pl1NNCTCG BzvudPl4qtKaaXKj+0lztIgQp8Se3VQPCMFsK6c03p8Kq4EuvlfDGA4BSlnfkq2KVgRl 5EowwRO5uklN1nw83sZsy9UrRnDG85wi5UVwT8rKXYPqz3mNGYGErNOGuwAn80BBlYZw z2gQOQzxySqSai1LBgR/gIKn+km2v7Md1f4Zo6KAxvtbDWTgu9cLuIG4e2dDlUQBmNGh 3ZsVqgWSSMDeZTZysFUc0fWAoq0CdzSbzFkP9JVKmF5y+oMcLZDfeVJwjCxOMbyrsNed ug== Received: from nalasppmta03.qualcomm.com (Global_NAT1.qualcomm.com [129.46.96.20]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3pbpy3h8a3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 16 Mar 2023 08:50:41 +0000 Received: from nalasex01a.na.qualcomm.com (nalasex01a.na.qualcomm.com [10.47.209.196]) by NALASPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 32G8oe0m000605 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 16 Mar 2023 08:50:40 GMT Received: from zhenhuah-gv.qualcomm.com (10.80.80.8) by nalasex01a.na.qualcomm.com (10.47.209.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 16 Mar 2023 01:50:35 -0700 From: Zhenhua Huang To: , , , , , , , , , , CC: Zhenhua Huang , , , , , , Subject: [PATCH v9] mm,kfence: decouple kfence from page granularity mapping judgement Date: Thu, 16 Mar 2023 16:50:20 +0800 Message-ID: <1678956620-26103-1-git-send-email-quic_zhenhuah@quicinc.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01b.na.qualcomm.com (10.46.141.250) To nalasex01a.na.qualcomm.com (10.47.209.196) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: 0HFkjlROkqOfuMDTe0_Xwv9AnTTKv4bV X-Proofpoint-ORIG-GUID: 0HFkjlROkqOfuMDTe0_Xwv9AnTTKv4bV X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-16_06,2023-03-15_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxscore=0 bulkscore=0 mlxlogscore=999 impostorscore=0 lowpriorityscore=0 suspectscore=0 phishscore=0 malwarescore=0 adultscore=0 clxscore=1015 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303160074 X-Stat-Signature: eoubffktu4tpcupghgbdozq8n3ejeucr X-Rspam-User: X-Rspamd-Queue-Id: A23ED4000F X-Rspamd-Server: rspam06 X-HE-Tag: 1678956654-81935 X-HE-Meta: U2FsdGVkX19nHa2Mro33FpQAppibUWlti30rCEbl1m+NQ+QcqrYlPPUiXICUcKsaYhEtV8bcRVlf94+KVXe25UPnNDKY4UzhEzKA/HhXxxT0dxOshrNVAtzGqyebK/7iRYmpGIKmzrxQF7JkTEmO1g8x5tzcMfz24u5TRk7DY8ajBG+YpCpGJdUG9DJWwLS2EAaI+E/q4mgrDGWCH6tyiQx3hzavRnlKnho/czLd/AWurRhMl/ZCHnnOG7Z9nMzd2YQL6x5K/L2/l6RD5022wGbw0XFYFgmxJ5gk6MNFuxnUCW/uga8PbzF4qcvlTLxZ1F3xf+e8aDD5gMedLQjy3G/RJgzRa0rrDLlA5e7QaI21u+913yi8soCgaI7/rFAoB8h9QNQMk01BZu+tF6u4vf/59U8CQ2ILPyac5Y2elRBUXzF7woC3A98kXgzQ3+Z7Sk0lNKfkMiVSslqIBGTU4A6qwM0YXkvK6WbuFUFtdL9ATE5EtXgOB7g02Yqmi/Muz+LPEo+OEDQ6n1CmqYX9sSX3bPfC+XgD3wK7xXJZZreK3IiAR8m8oZS9OxMGVMC/y/7G1F3PrZk4+r9potH3jmJ0BWC8ha/JII0DU/Jz2IiAf01XRuqhoJAaRBgWulTw8SehydL1Fiq/6hawBsiiDZigvOfKQNwM724XcaeRB175+7NXcH8zeYoLANcuCbz3EbpPZGh9OxKO8PVwyxg9Y9RFBvlxVxpzQ3X4ucx7LijbXJVZi3HeMHvEsk/Gi86jpnXfnBoAGKwvT5CgnDvINSGzP2HUJvQCVdaDTL4hdKYm7qte2vwxpPq1evc1nrIPRNnoDHC1GpexGDel6vqVCOZvPvK7Ww1S57ZfS1WIn0Jh46+vafDjYSP/Qcml7LIhrYNGnRZRyDJEzY3OyX/CJFCfOcZLBqB+uj5u4jhxVEuuxumCIhrCgj9eg1dpSXUgtj4uW0ydyqHzvG2cuwO d46T9sTc zRy0EGgs1gzOB6JtaPGjSGZ5fit+aAlv9z5QyFNeHt5sMgjpu5feL9cA3nWeMHbc04zXE5qQrRBLzB+tKytggilo6NzIVxCl1XuqkP/ih8vV4Y1Z5MgMwt/okRNamgKhzyKNEs5TUe071tGBnZo+2plhEEhq/P0mnGLJeCA/BmYuLqNON11uHuUkrZPBwgO7NEZLwydSForXog6WusOlnbR+eegZhTNlBp5jJp6gbiDXr5R4zM8ombVw20Klwdt4pAhdmptqep61RnlD6tglET91ya/jSU+28zOIkTjmHYn81J8+Pz9R5aLP7wG7mclviwwVpos2yZ+pByVxsUL9GbA6GzCIQqo7lGUMA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Kfence only needs its pool to be mapped as page granularity, if it is inited early. Previous judgement was a bit over protected. From [1], Mark suggested to "just map the KFENCE region a page granularity". So I decouple it from judgement and do page granularity mapping for kfence pool only. Need to be noticed that late init of kfence pool still requires page granularity mapping. Page granularity mapping in theory cost more(2M per 1GB) memory on arm64 platform. Like what I've tested on QEMU(emulated 1GB RAM) with gki_defconfig, also turning off rodata protection: Before: [root@liebao ]# cat /proc/meminfo MemTotal: 999484 kB After: [root@liebao ]# cat /proc/meminfo MemTotal: 1001480 kB To implement this, also relocate the kfence pool allocation before the linear mapping setting up, arm64_kfence_alloc_pool is to allocate phys addr, __kfence_pool is to be set after linear mapping set up. LINK: [1] https://lore.kernel.org/linux-arm-kernel/Y+IsdrvDNILA59UN@FVFF77S0Q05N/ Suggested-by: Mark Rutland Signed-off-by: Zhenhua Huang --- arch/arm64/include/asm/kfence.h | 16 +++++++++++ arch/arm64/mm/mmu.c | 59 +++++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/pageattr.c | 9 +++++-- include/linux/kfence.h | 1 + mm/kfence/core.c | 4 +++ 5 files changed, 87 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kfence.h b/arch/arm64/include/asm/kfence.h index aa855c6..8143c91 100644 --- a/arch/arm64/include/asm/kfence.h +++ b/arch/arm64/include/asm/kfence.h @@ -10,6 +10,22 @@ #include +extern phys_addr_t early_kfence_pool; + +#ifdef CONFIG_KFENCE + +extern char *__kfence_pool; +static inline void kfence_set_pool(phys_addr_t addr) +{ + __kfence_pool = phys_to_virt(addr); +} + +#else + +static inline void kfence_set_pool(phys_addr_t addr) { } + +#endif + static inline bool arch_kfence_init_pool(void) { return true; } static inline bool kfence_protect_page(unsigned long addr, bool protect) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 6f9d889..61944c70 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -38,6 +39,7 @@ #include #include #include +#include #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) @@ -525,6 +527,48 @@ static int __init enable_crash_mem_map(char *arg) } early_param("crashkernel", enable_crash_mem_map); +#ifdef CONFIG_KFENCE + +static bool kfence_early_init __initdata = !!CONFIG_KFENCE_SAMPLE_INTERVAL; +/* + * early_param can be parsed before linear mapping + * set up + */ +static int __init parse_kfence_early_init(char *p) +{ + int val; + + if (get_option(&p, &val)) + kfence_early_init = !!val; + return 0; +} +early_param("kfence.sample_interval", parse_kfence_early_init); + +static phys_addr_t arm64_kfence_alloc_pool(void) +{ + phys_addr_t kfence_pool; + + if (!kfence_early_init) + return 0; + + kfence_pool = memblock_phys_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); + if (!kfence_pool) + pr_err("failed to allocate kfence pool\n"); + + return kfence_pool; +} + +#else + +static phys_addr_t arm64_kfence_alloc_pool(void) +{ + return 0; +} + +#endif + +phys_addr_t early_kfence_pool; + static void __init map_mem(pgd_t *pgdp) { static const u64 direct_map_end = _PAGE_END(VA_BITS_MIN); @@ -543,6 +587,10 @@ static void __init map_mem(pgd_t *pgdp) */ BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end)); + early_kfence_pool = arm64_kfence_alloc_pool(); + if (early_kfence_pool) + memblock_mark_nomap(early_kfence_pool, KFENCE_POOL_SIZE); + if (can_set_direct_map()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; @@ -608,6 +656,17 @@ static void __init map_mem(pgd_t *pgdp) } } #endif + + /* Kfence pool needs page-level mapping */ + if (early_kfence_pool) { + __map_memblock(pgdp, early_kfence_pool, + early_kfence_pool + KFENCE_POOL_SIZE, + pgprot_tagged(PAGE_KERNEL), + NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS); + memblock_clear_nomap(early_kfence_pool, KFENCE_POOL_SIZE); + /* kfence_pool really mapped now */ + kfence_set_pool(early_kfence_pool); + } } void mark_rodata_ro(void) diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 79dd201..7ce5295 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -7,10 +7,12 @@ #include #include #include +#include #include #include #include +#include struct page_change_data { pgprot_t set_mask; @@ -22,12 +24,15 @@ bool rodata_full __ro_after_init = IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT_ENABLED bool can_set_direct_map(void) { /* - * rodata_full, DEBUG_PAGEALLOC and KFENCE require linear map to be + * rodata_full and DEBUG_PAGEALLOC require linear map to be * mapped at page granularity, so that it is possible to * protect/unprotect single pages. + * + * Kfence pool requires page granularity mapping also if we init it + * late. */ return (rodata_enabled && rodata_full) || debug_pagealloc_enabled() || - IS_ENABLED(CONFIG_KFENCE); + (IS_ENABLED(CONFIG_KFENCE) && !early_kfence_pool); } static int change_page_range(pte_t *ptep, unsigned long addr, void *data) diff --git a/include/linux/kfence.h b/include/linux/kfence.h index 726857a..91cbcc9 100644 --- a/include/linux/kfence.h +++ b/include/linux/kfence.h @@ -222,6 +222,7 @@ bool __kfence_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *sla #else /* CONFIG_KFENCE */ +#define KFENCE_POOL_SIZE 0 static inline bool is_kfence_address(const void *addr) { return false; } static inline void kfence_alloc_pool(void) { } static inline void kfence_init(void) { } diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 5349c37..e05ccf1 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -814,6 +814,10 @@ void __init kfence_alloc_pool(void) if (!kfence_sample_interval) return; + /* if the pool has already been initialized by arch, skip the below */ + if (__kfence_pool) + return; + __kfence_pool = memblock_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); if (!__kfence_pool)