From patchwork Wed Jul 31 02:54:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Tikhomirov X-Patchwork-Id: 13748012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 200C2C3DA49 for ; Wed, 31 Jul 2024 02:57:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 917A46B0089; Tue, 30 Jul 2024 22:57:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C7106B008A; Tue, 30 Jul 2024 22:57:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 768586B008C; Tue, 30 Jul 2024 22:57:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 530A46B0089 for ; Tue, 30 Jul 2024 22:57:20 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 941A7A01B5 for ; Wed, 31 Jul 2024 02:57:19 +0000 (UTC) X-FDA: 82398536598.23.4D3AB49 Received: from EUR03-VI1-obe.outbound.protection.outlook.com (mail-vi1eur03on2099.outbound.protection.outlook.com [40.107.103.99]) by imf17.hostedemail.com (Postfix) with ESMTP id 9F15740021 for ; Wed, 31 Jul 2024 02:57:16 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=virtuozzo.com header.s=selector2 header.b=jDWtmkpC; spf=pass (imf17.hostedemail.com: domain of ptikhomirov@virtuozzo.com designates 40.107.103.99 as permitted sender) smtp.mailfrom=ptikhomirov@virtuozzo.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=virtuozzo.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722394632; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZKeqphtEOBju11zfTP+TwSqnt0fLzZERCrRNXEXf0YY=; b=H/h1DtusC4d6iHJ4aAfXN05uOVuu4NBjQvP9sQijHvBZ0K8F59ZR9qF9KVetlvDsSQOlp8 1cvk9jvJ5Bia3gAyvibLosmp36pu1Eh/fWfq6lIy7JmnRPnmK/1MwSaES/iu5eeocr8G9O DQCzgCdzerQ8+Vu/Eod/cvgwuOfrLBw= ARC-Authentication-Results: i=2; imf17.hostedemail.com; dkim=pass header.d=virtuozzo.com header.s=selector2 header.b=jDWtmkpC; spf=pass (imf17.hostedemail.com: domain of ptikhomirov@virtuozzo.com designates 40.107.103.99 as permitted sender) smtp.mailfrom=ptikhomirov@virtuozzo.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=virtuozzo.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1722394632; a=rsa-sha256; cv=pass; b=OgSriVN2+fKid+s7Kx4qAvD1pmpGNfTYPcFTx8XdEXDDiufnqcHmTAKPfniZamTJgnP+j8 ej4ojptmlac7Bg0jgDtbAPwN+nwxCzomTXobbvri0T3JLDmuu6+K5Ha2nfhB13sAEO/Ilt wHxH9AlynUxt0lbl5O6UlciLZqr0NO8= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FGLQOsCUiz6k0MAw5yNAQ/VVJYfaBFH1jbZrAJvvioLc/+av6i3ANQslPscemKOcO3xUbJ6tEaEB+ArR1PDGSdvSul6nwa8w9QjhTVP+B0TAiBVpT7xoVbub5KROyjMpXFZwwg+kTdi9mlBMfF6RxsDyRKuid8s3+lfAv0Il7xxXo6elaKM9+XyDqJTF4aylRqsOQWeKP3T9U1qxCEKMnL6VWx8YU2yLrRFG2S7la/vwlEjbgRNFGylcb52GAjWH+xJTKUu80g2SHWZ6sPK+RDThai3NM8TFYKfh4D1MYVZvElcqYxygXjknRiiQgGwDvz30wIQxfTecs3AJrsZROg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZKeqphtEOBju11zfTP+TwSqnt0fLzZERCrRNXEXf0YY=; b=h67uS2lnqHUkjliWPOc/A+F5qy+RBAIiKrLWZkAJ2zll43FriVd+W7LhdzLB/pW8zMtxQOhCcVOXgT/8DtZRTLELium66SQtl+H327XulvseSdOV6oroK6nek03QS166VxG9UkmbB635a0THFGCAux2BCdRgGDC7brg++s1XUK4BO31TD5QnZACAC3RkOwj+edoQW9txVnoZsMwhAusCpRQdZvUOIz8NvLUsRn+0v0UP960c5HzUPD8TkgJ0x7W7KfsODDt8bOrRLmVRUvyyoa3WGZUZNSLPjWrGxmHSQ+qaFPvBdEVCF7sEzi2/misttqBLCBqTgUOebozVP1ja4Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZKeqphtEOBju11zfTP+TwSqnt0fLzZERCrRNXEXf0YY=; b=jDWtmkpCTQ9NnySdmhOL6e1iL23jZZ0Xemw+mnuxws393RFQdduc9gUyYQ5EMPozBoba07zxHLj+HAM/hKiL3Gw0gAIu+Tuf39/c6DALoXyVwKh20ygyFrA2PO3654uby0/82omfEl7wTyN0uz862NiszkqRbdZ/LcAeUd7CG/BfnH64pC/POZWPUCo8AcCL/JvgjqjjyHgJaLXYbfA2JXhOugjFNw6fAbzuiLr7keUOOKa4R2inmP/0ayt2tE4ncmpqELJb/8XMcj/f75lkv7A/eGJcSg5Z4nMpp4J4k8KSKtgVKpKHLJqAOCseua+fTGEJ5gVX01W00ljPXKjgWQ== Received: from DU0PR08MB9003.eurprd08.prod.outlook.com (2603:10a6:10:471::13) by AM8PR08MB6483.eurprd08.prod.outlook.com (2603:10a6:20b:315::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7828.21; Wed, 31 Jul 2024 02:55:45 +0000 Received: from DU0PR08MB9003.eurprd08.prod.outlook.com ([fe80::7261:fca8:8c2e:29ce]) by DU0PR08MB9003.eurprd08.prod.outlook.com ([fe80::7261:fca8:8c2e:29ce%4]) with mapi id 15.20.7828.016; Wed, 31 Jul 2024 02:55:45 +0000 From: Pavel Tikhomirov To: Catalin Marinas , Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Wei Yongjun , Chen Jun , Aleksandr Mikhalitsyn , Pavel Tikhomirov , kernel@openvz.org Subject: [PATCH v2 1/2] kmemleak: enable tracking for percpu pointers Date: Wed, 31 Jul 2024 10:54:09 +0800 Message-ID: <20240731025526.157529-2-ptikhomirov@virtuozzo.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240731025526.157529-1-ptikhomirov@virtuozzo.com> References: <20240731025526.157529-1-ptikhomirov@virtuozzo.com> X-ClientProxiedBy: SI2PR02CA0033.apcprd02.prod.outlook.com (2603:1096:4:195::20) To DU0PR08MB9003.eurprd08.prod.outlook.com (2603:10a6:10:471::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DU0PR08MB9003:EE_|AM8PR08MB6483:EE_ X-MS-Office365-Filtering-Correlation-Id: 720c689f-365f-4b35-6e8a-08dcb10c4629 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|52116014; X-Microsoft-Antispam-Message-Info: 1DvqCoG5JwbSXEFN7ffASGQD9x2xjVwVNsqFTWkEwal6rUBEHxZ/UUXLiTigPOZFQzmCK6ohTBFmQPkCy5WFQ+QrGXRy121YqnozZGzFolhutX9ViqVzarFFapESHSdmbftamhHuikMwK8VMXSZtk5zMrljjaG5o6lgDNZnRNDIqeyjeh3q8oqa0DOy4G0CZT+Xgw6P6dXy5Aon0GZReQWnJ8dTZYywcQeqbCPjKki19KJ6B1GNO+kQlKvPUrIV468AqZ9Vxb4Niq4JSCImIvAkpnpIGiqRyFvWZPeJR1l2DlsRASJOtYpOI2dTJXEx3uuWCnMpBKtYr14eQ1FVh/PuQ0RGclEIV5omyEWc5TyhKX8T48wGheBN0vql2xyUvs/4t7gw3kzVA1iSUiPZEutryJ0PB/Dbe3BGNeJml5+povBvmK8lreAHJhTyibaChPQxsOGQ5jastWzcX3AlqTtHoY6vZ0ut3+a+0BgKTsp+GXqzsxO01WG9wsS+AVaANOKuJLOvt4uYTzQxQ6wSFTwwdnMYsSsmKIbE2t/6y418wkZZ1th2jj1fo/Fcc4oRcExynNrmb4HkQA1pZD66jj/eExys1rupR2y2S8gbFX2xi231X1lPaKi/syiVgnZWEAIS3E5/ylINa6N+nCtMhd0uZYvbdtkUDKyYHTLOoAr/ckuKCueMQhKPpKB49L79ztRe0QkV2hcqo5FgQ7gVz2Vk5k2TEk7VSGlijNQLtbbQI60JIZwYNHZznYUG0Klp5uh5y0LcWGkJ4ipynX/xOsCtxdVtDpDaLo/NS6fpnXaWOBBlYxU0DKk01k78beeXYwDvch5Sy63ZeOvqpTAmv5psJn1DNOjS8CmjagP7EXfnRgCip5YJaseaGryXZ59fok1K18jh5KkLIlDHzf3xlFJKHRRzM5WNLocMmC4x08catPEL13uyPlpZ5Z/0z1eqfm0TCI2uAlA8jA7ARqIz4EYsnekOIEQzST8kYw5FGhZQwkxk+8fXdsSGKO/SCd6VBSm5WwomoEa2ZXUM7P4LClEbSQChDZvIUvijueOdGBDr9+ZwBS/BIYMG8Me7imZOtj13RV06pOuJtYkc/za6Dr+RZtKGIf3LMeFEmQoiNTng716Pd/Cv5T0wPH7A3qTViMHJj2UOLsPgiTF+OXeJt4KE1qy9TyXZhENXGrMlR/stxzNPVi71DkWngElOxUkG3qg7TtM//IotbCBVY+d+pm2oRIRoy7qiDZQEwv4rNPEVQqtaePTsPyXyIa5XCU3QXf7gzvYf3DWWMWRW8u/yGaFzEWWo0zhn0jXdeAKI1RSEEInqqJeoDVqXR4vF9CjsjnDceTd30fWM0LZn31wCSjg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DU0PR08MB9003.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(52116014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 5gZj7LdCCZcA7Z2CBQO73APmijBflRIb7ae25vCUCxAjNmBiMPZVZkPM2O+M9ZEnXPYimSoHeN5NIqc0+0D+/WKQ0iNJOMkrS42Gs2ZzaxU7GuNAec9ihUbYtioHyaUMYy00vcw+N9YvcFhG7k5Lu2Xzx08aTDitI4wmIlcVqeDmTlaAcg+MJtfqrawa47G7NiQuLfRCEOlwPMrwz8YqWECSTc1suowI4p1J6JYT/F2xSINvTIvunPXPaGD2lhwhvCDt3JbUxJRE5Y1t0yfzJ6M7LaEb4x301ZcAGrvm6puWIwYnyZhOIP3tYmp6QKo73PLMogQ9t7HSv4qek6AZ/TJ+bT+wxxi9lhnpowAV+Dv+6HTdNMCjEENeD78gG7JKdXiYqBgKLLv4lwdnvr/L+iAQI0k0ti3h1Np6qHyD5DnGwmgoX4/93U1WuIh0g2orO2WYUPlbToX1WAaBrS07sUj3KBffZbC2Sk+IhS93seerlUUhitQr91QAJzbGgPx7uCVbZ4shRe5QNtyFpStFigAAe2nIO7nCgMcfDv3QKZpHlk805OtpfiHedC/XF7FwtglZ/tIjIyxgs6Qnj5ky4zgVacwVBcW/FyebuCOeeRmap1MisrUHyu+j3JrlqydiGE1x+lJqvM0WJwhqA7oMA2+cRbnm72XSdjeMExWnqbJLM6jPdZ8CqMRMdNBQApMVEtZaHoJqvl9JN7eI7d2Z32vf0pinzJ6gjX2WwjVyw5Bn04Hvtr676LMxn/rsHiXYaw3Nzjw+e16Hb3cjmVMXI5RlmVBGaQvnPW5MJBmM942PVs/ZHZNFqyKFhNubLISS+X+Q7aDVFCgy0In3kSxe96WLqHStYyxTnHF6tjURpMHUGDNS8b1IwKOD3TS15TBFsr63t5esAeUUYwVveZ9kjIndxmSjjxFhBL4wgKIGiE12Vtg7LDlA/bIjsYL2n4NNOzbFY5HK7rnpKq/x1AmQ2a0VGteVbEueRltlTRRxGzwDN0oTFOM2TsdEyIysCoyAalTVqmywq+56dPPjRQsELytHAyg0cqsG6MDqllR4MZPIipg/NZpde510hOUrunWC/7znBVf0xpQvXQuLky7/fI0fqQ2NwTZVYvybcy7tvbNJJm/FCMazto7rRIjwFi6LCJWEO5u0QUK6wdk3QgIvG4QWQTHWEnU2dyAq2vXTM4FkVM/VEKe7cxw17hWbmFORNoG7gtZsJtbLEU4dNOdTDqHIz0uz/E0F6ZXYdIS8ONCGdVcmeb/1x2SztxLd2XwH3wQG4f5/T44gYpk8cyAewNoOtSJvv7v3xs7WPqnANI16aHIwV38l7iPArNJ8uVcIL0K4jt9cUUBdBTdsYFsauObMvpUMog6ixX4Lwqah7C0x9JRT1SQO5cMPfLxlR4K7cUMcY8OUFZZLoDtvELkMe3C5PJv6nEo4xqF3XR4mbO/O8rabEayLyIfOb4ZgMm0twJNAwCT+QXiTWFLeI44P54T8OSHVVpv2n8NEf3P5WXAHeQPMDOeAxrG/twQn9dx0PIi4G9Sv9/bQPdBwZg+v1Ix0/F/r/huJyM8bcjR/6U1uj4gwGZnZ3plMmIsueT7AqhtH29ErNbMyP5eUE2PlIep4jxafK8VhwhBsK0RR2k8iPnyFmxY4MS7h1EtSQ6Ai2qUeI/4fsEmZTHYDp6mm2Q== X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 720c689f-365f-4b35-6e8a-08dcb10c4629 X-MS-Exchange-CrossTenant-AuthSource: DU0PR08MB9003.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jul 2024 02:55:45.5376 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RwuAj9dAKqcsW+MIdkw7KnQ+ofNUWQytTpFCUBYvOFT/W2YB+rjd1V43Wv8bcNcJtr76Ido0eIySh6lLm4u8n6xY8zO1XZ7Pdp5clSjo7H4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB6483 X-Rspam-User: X-Stat-Signature: i8fst47nm38qiyaismso631fzzzq4u4j X-Rspamd-Queue-Id: 9F15740021 X-Rspamd-Server: rspam11 X-HE-Tag: 1722394636-442059 X-HE-Meta: U2FsdGVkX1+KI59sjwXPI6r1cxFtkUT6x9+0UrByXcyy82M/EaEdRWHz1BrOh1X3WZDqC0SpW8JS/XXVt6KGpkbrTH93oQEMPTrZEuLv+Rx8COZ3LJfnH5uxkLNKwkw/pM4vU+faLi0QRHTj6WxgVgHvZE4VWH30HhA1BgTG0KgU7JpQliz5BOPwiNPBY2m9C2ATIuFC2XKO8vH6Ec2WcQQ0YwbdHT4X5m6dN9vJcrN7eCD1KwAhW4V7TyuTc6csvuP3Y9qS6nBIlHGfkgiwjT33JPNT70KcbJ9n25NN1l0fmimR7d/0xp2HhOfKo0hsS0Lm1irr0zMks0FrQ8YTDcqAe5QVbaI+g1mQsaqi2NxtaJZMPwCuEETxMqQLyyuPcqopYqJySa5cX8h/hWILzit08+geIKV97dSqvMJgvaUCBkpxxBhjuEmLb0q6l7ZedRX0UM572iW4Ea426hPVtB45jmP05Dmvf4e8d8l7KWIlSSrCAV0/8K+E3N1SncRivFaZNmvZGGnxNPTSK+43NSuOgkPUl81luLb+bzmFDpf6Xh544SQQ86FFK9Q7lPUfNZ2/5XXGUdqhmUGoE0C3YozA2Obs23UJPC8QFeZEAyDPwFBjUpr1/dI/SO/40c0kCWQ0ER/WKfktUoPacxCk4U4NFEbbIXr+oxH3iZiKaGyi7a2DRdtC35hIRNmDXdxwPOhpD76yK0tkm9eKNmuts8LbHLHCCNq5t0bDmS0mbR475EJSf+R7jfQQpWsZRixoe+DVYZU5iZJbvcG1aLwWvYrkXwkFd5lK1YcfdOtFP0Fv1a00hP1POUDEIA8nNKnxtJK34bcq+8W5aNl5VbsRlWRuqn8qxKU+zGAx8MBpbS11c8SXq3HG0B/KNoHZ3FJ7ABfofbq+6ZByjelSdr9yGgSEXBUkMgNugUU0kS9LPq9PO0PHIgxYLy5YbAFd8+uSAgUuCq1g05WxgaRps1r foUVF0My JjVV+21WdDJ/rZ6hc9WiizONfRCRbKkKnIKuUuHR77fT5DOQGlJY5YgNH5GLkvJNra/geayk4MO3MvCVMa9ZZJO06eUGGRkRsQOL0ZJ6IXymQrCyIOGvSbYPxQkKc4EshI4dPv+sIVes4ExShcisq/tExUOiCuGFYu+YRX9xdKNfvn4UjlUP0z7WgcUg6LDB9YvibwGArle1UdSYLm/k7fs9eVxHsoJfqzetZuQHr1RrYtBoKu+ivBPpj7WuFswqXlK8BxTDdI66GJwac62fuPcooMPifz1D1Wma5eXKMl7dmdghXC3Mw4lZ6bV/kZyNiDxSGFZxjFeYNVsn3hFR0dv6TTKcBmfRbxe0BScH7IRKPZ+UY52pdq70viBxVJY336+4VLb78SgA3VfdkTnqcsMkYjQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This basically does: - Add min_percpu_addr and max_percpu_addr to filter out unrelated data similar to min_addr and max_addr; - Set min_count for percpu pointers to 1 to start tracking them; - Calculate checksum of percpu area as xor of crc32 for each cpu; - Split pointer lookup and update refs code into separate helper and use it twice: once as if the pointer is a virtual pointer and once as if it's percpu. CC: Wei Yongjun CC: Chen Jun Signed-off-by: Pavel Tikhomirov Reviewed-by: Catalin Marinas --- note: excess_ref can only be a virtual pointer at least for now v2: rename confusing scan_pointer to pointer_update_refs and move it just after update_refs --- mm/kmemleak.c | 153 +++++++++++++++++++++++++++++++------------------- 1 file changed, 94 insertions(+), 59 deletions(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 764b08100570b..6b498c6d9c34a 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -224,6 +224,10 @@ static int kmemleak_error; static unsigned long min_addr = ULONG_MAX; static unsigned long max_addr; +/* minimum and maximum address that may be valid per-CPU pointers */ +static unsigned long min_percpu_addr = ULONG_MAX; +static unsigned long max_percpu_addr; + static struct task_struct *scan_thread; /* used to avoid reporting of recently allocated objects */ static unsigned long jiffies_min_age; @@ -294,13 +298,20 @@ static void hex_dump_object(struct seq_file *seq, const u8 *ptr = (const u8 *)object->pointer; size_t len; - if (WARN_ON_ONCE(object->flags & (OBJECT_PHYS | OBJECT_PERCPU))) + if (WARN_ON_ONCE(object->flags & OBJECT_PHYS)) return; + if (object->flags & OBJECT_PERCPU) + ptr = (const u8 *)this_cpu_ptr((void __percpu *)object->pointer); + /* limit the number of lines to HEX_MAX_LINES */ len = min_t(size_t, object->size, HEX_MAX_LINES * HEX_ROW_SIZE); - warn_or_seq_printf(seq, " hex dump (first %zu bytes):\n", len); + if (object->flags & OBJECT_PERCPU) + warn_or_seq_printf(seq, " hex dump (first %zu bytes on cpu %d):\n", + len, raw_smp_processor_id()); + else + warn_or_seq_printf(seq, " hex dump (first %zu bytes):\n", len); kasan_disable_current(); warn_or_seq_hex_dump(seq, DUMP_PREFIX_NONE, HEX_ROW_SIZE, HEX_GROUP_SIZE, kasan_reset_tag((void *)ptr), len, HEX_ASCII); @@ -695,10 +706,14 @@ static int __link_object(struct kmemleak_object *object, unsigned long ptr, untagged_ptr = (unsigned long)kasan_reset_tag((void *)ptr); /* - * Only update min_addr and max_addr with object - * storing virtual address. + * Only update min_addr and max_addr with object storing virtual + * address. And update min_percpu_addr max_percpu_addr for per-CPU + * objects. */ - if (!(objflags & (OBJECT_PHYS | OBJECT_PERCPU))) { + if (objflags & OBJECT_PERCPU) { + min_percpu_addr = min(min_percpu_addr, untagged_ptr); + max_percpu_addr = max(max_percpu_addr, untagged_ptr + size); + } else if (!(objflags & OBJECT_PHYS)) { min_addr = min(min_addr, untagged_ptr); max_addr = max(max_addr, untagged_ptr + size); } @@ -1055,12 +1070,8 @@ void __ref kmemleak_alloc_percpu(const void __percpu *ptr, size_t size, { pr_debug("%s(0x%px, %zu)\n", __func__, ptr, size); - /* - * Percpu allocations are only scanned and not reported as leaks - * (min_count is set to 0). - */ if (kmemleak_enabled && ptr && !IS_ERR(ptr)) - create_object_percpu((unsigned long)ptr, size, 0, gfp); + create_object_percpu((unsigned long)ptr, size, 1, gfp); } EXPORT_SYMBOL_GPL(kmemleak_alloc_percpu); @@ -1304,12 +1315,23 @@ static bool update_checksum(struct kmemleak_object *object) { u32 old_csum = object->checksum; - if (WARN_ON_ONCE(object->flags & (OBJECT_PHYS | OBJECT_PERCPU))) + if (WARN_ON_ONCE(object->flags & OBJECT_PHYS)) return false; kasan_disable_current(); kcsan_disable_current(); - object->checksum = crc32(0, kasan_reset_tag((void *)object->pointer), object->size); + if (object->flags & OBJECT_PERCPU) { + unsigned int cpu; + + object->checksum = 0; + for_each_possible_cpu(cpu) { + void *ptr = per_cpu_ptr((void __percpu *)object->pointer, cpu); + + object->checksum ^= crc32(0, kasan_reset_tag((void *)ptr), object->size); + } + } else { + object->checksum = crc32(0, kasan_reset_tag((void *)object->pointer), object->size); + } kasan_enable_current(); kcsan_enable_current(); @@ -1340,6 +1362,64 @@ static void update_refs(struct kmemleak_object *object) } } +static void pointer_update_refs(struct kmemleak_object *scanned, + unsigned long pointer, unsigned int objflags) +{ + struct kmemleak_object *object; + unsigned long untagged_ptr; + unsigned long excess_ref; + + untagged_ptr = (unsigned long)kasan_reset_tag((void *)pointer); + if (objflags & OBJECT_PERCPU) { + if (untagged_ptr < min_percpu_addr || untagged_ptr >= max_percpu_addr) + return; + } else { + if (untagged_ptr < min_addr || untagged_ptr >= max_addr) + return; + } + + /* + * No need for get_object() here since we hold kmemleak_lock. + * object->use_count cannot be dropped to 0 while the object + * is still present in object_tree_root and object_list + * (with updates protected by kmemleak_lock). + */ + object = __lookup_object(pointer, 1, objflags); + if (!object) + return; + if (object == scanned) + /* self referenced, ignore */ + return; + + /* + * Avoid the lockdep recursive warning on object->lock being + * previously acquired in scan_object(). These locks are + * enclosed by scan_mutex. + */ + raw_spin_lock_nested(&object->lock, SINGLE_DEPTH_NESTING); + /* only pass surplus references (object already gray) */ + if (color_gray(object)) { + excess_ref = object->excess_ref; + /* no need for update_refs() if object already gray */ + } else { + excess_ref = 0; + update_refs(object); + } + raw_spin_unlock(&object->lock); + + if (excess_ref) { + object = lookup_object(excess_ref, 0); + if (!object) + return; + if (object == scanned) + /* circular reference, ignore */ + return; + raw_spin_lock_nested(&object->lock, SINGLE_DEPTH_NESTING); + update_refs(object); + raw_spin_unlock(&object->lock); + } +} + /* * Memory scanning is a long process and it needs to be interruptible. This * function checks whether such interrupt condition occurred. @@ -1372,13 +1452,10 @@ static void scan_block(void *_start, void *_end, unsigned long *start = PTR_ALIGN(_start, BYTES_PER_POINTER); unsigned long *end = _end - (BYTES_PER_POINTER - 1); unsigned long flags; - unsigned long untagged_ptr; raw_spin_lock_irqsave(&kmemleak_lock, flags); for (ptr = start; ptr < end; ptr++) { - struct kmemleak_object *object; unsigned long pointer; - unsigned long excess_ref; if (scan_should_stop()) break; @@ -1387,50 +1464,8 @@ static void scan_block(void *_start, void *_end, pointer = *(unsigned long *)kasan_reset_tag((void *)ptr); kasan_enable_current(); - untagged_ptr = (unsigned long)kasan_reset_tag((void *)pointer); - if (untagged_ptr < min_addr || untagged_ptr >= max_addr) - continue; - - /* - * No need for get_object() here since we hold kmemleak_lock. - * object->use_count cannot be dropped to 0 while the object - * is still present in object_tree_root and object_list - * (with updates protected by kmemleak_lock). - */ - object = lookup_object(pointer, 1); - if (!object) - continue; - if (object == scanned) - /* self referenced, ignore */ - continue; - - /* - * Avoid the lockdep recursive warning on object->lock being - * previously acquired in scan_object(). These locks are - * enclosed by scan_mutex. - */ - raw_spin_lock_nested(&object->lock, SINGLE_DEPTH_NESTING); - /* only pass surplus references (object already gray) */ - if (color_gray(object)) { - excess_ref = object->excess_ref; - /* no need for update_refs() if object already gray */ - } else { - excess_ref = 0; - update_refs(object); - } - raw_spin_unlock(&object->lock); - - if (excess_ref) { - object = lookup_object(excess_ref, 0); - if (!object) - continue; - if (object == scanned) - /* circular reference, ignore */ - continue; - raw_spin_lock_nested(&object->lock, SINGLE_DEPTH_NESTING); - update_refs(object); - raw_spin_unlock(&object->lock); - } + pointer_update_refs(scanned, pointer, 0); + pointer_update_refs(scanned, pointer, OBJECT_PERCPU); } raw_spin_unlock_irqrestore(&kmemleak_lock, flags); }