From patchwork Tue Jun 14 22:03:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12881649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48371C433EF for ; Tue, 14 Jun 2022 22:04:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 991B06B0071; Tue, 14 Jun 2022 18:04:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 940F36B0072; Tue, 14 Jun 2022 18:04:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 807CC6B0073; Tue, 14 Jun 2022 18:04:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 70F346B0071 for ; Tue, 14 Jun 2022 18:04:17 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 341FF21061 for ; Tue, 14 Jun 2022 22:04:17 +0000 (UTC) X-FDA: 79578220554.22.A5E7BB5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 8045F4009C for ; Tue, 14 Jun 2022 22:04:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655244256; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A77y1Lb+c0CpuBJ2A6h47CxYcHUb1O0vwhwXfFK1d3U=; b=VS8HY7l7xqIiy1mznxImPdQyNf3tUshXV2+ct7pL5GYnfQF6Z08zwnQ/SlAEqNSbHsP1/X ynDvjCfl63fG15mE4RV0Qn5X+eEvda9iRbHGvw/mHGi7CGRwA7wZ9+ZvO0NFTeOiM73q63 nSii5m0CaTAWDECgobEh+QP3cY36Mf8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-649-7YOsaToPPyuNnrqUVU661A-1; Tue, 14 Jun 2022 18:04:14 -0400 X-MC-Unique: 7YOsaToPPyuNnrqUVU661A-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5A1933C021A9; Tue, 14 Jun 2022 22:04:14 +0000 (UTC) Received: from llong.com (unknown [10.22.33.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id 185B5492CA2; Tue, 14 Jun 2022 22:04:14 +0000 (UTC) From: Waiman Long To: Catalin Marinas , Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Waiman Long Subject: [PATCH v2 1/3] mm/kmemleak: Use _irq lock/unlock variants in kmemleak_scan/_clear() Date: Tue, 14 Jun 2022 18:03:57 -0400 Message-Id: <20220614220359.59282-2-longman@redhat.com> In-Reply-To: <20220614220359.59282-1-longman@redhat.com> References: <20220614220359.59282-1-longman@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655244256; a=rsa-sha256; cv=none; b=NlUZesgrnirg4rllCKd0/jUVuPTjqKUva2Q5NKtFoEFeN5B5BYITCY7TQnPxDNjyg1tknp Rszm7M+MVrsq/lv7UWUUmdaB8vLx4vSwaK8E1OUP3Uf+IAXrOQUXuyXNW57Evb3sls5YHz 6D37Xy4eIsp/KoGhAn3smRpRH9wBHOQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VS8HY7l7; spf=none (imf27.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=longman@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655244256; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A77y1Lb+c0CpuBJ2A6h47CxYcHUb1O0vwhwXfFK1d3U=; b=FSButSMq8TOd/ciRU4oJNoDZFPp4hp8QGFT7QCVrSr90EIidXA4DToBQgcjL6ZYcSB0ESr V8JNYobjzcqlpwMnfX4DyJKwByo2gC/VSqSMyeAJjWr7f8xDcNlrrDN62T0xmjmizxHDy0 5MLiQ8POXppMMyGtrlKqXfCxHGK/v88= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8045F4009C X-Stat-Signature: 9pt5ynhbpqjexqqj7yo8ync1ex59jf1a Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VS8HY7l7; spf=none (imf27.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=longman@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1655244256-123643 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The kmemleak_scan() function is called only from the kmemleak scan thread or from write to the kmemleak debugfs file. Both are in task context and so we can directly use the simpler _irq() lock/unlock calls instead of the more complex _irqsave/_irqrestore variants. Similarly, kmemleak_clear() is called only from write to the kmemleak debugfs file. The same change can be applied. Signed-off-by: Waiman Long Reviewed-by: Muchun Song Reviewed-by: Catalin Marinas --- mm/kmemleak.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index a182f5ddaf68..dad9219c972c 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1413,7 +1413,6 @@ static void scan_gray_list(void) */ static void kmemleak_scan(void) { - unsigned long flags; struct kmemleak_object *object; struct zone *zone; int __maybe_unused i; @@ -1424,7 +1423,7 @@ static void kmemleak_scan(void) /* prepare the kmemleak_object's */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { - raw_spin_lock_irqsave(&object->lock, flags); + raw_spin_lock_irq(&object->lock); #ifdef DEBUG /* * With a few exceptions there should be a maximum of @@ -1441,7 +1440,7 @@ static void kmemleak_scan(void) if (color_gray(object) && get_object(object)) list_add_tail(&object->gray_list, &gray_list); - raw_spin_unlock_irqrestore(&object->lock, flags); + raw_spin_unlock_irq(&object->lock); } rcu_read_unlock(); @@ -1509,14 +1508,14 @@ static void kmemleak_scan(void) */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { - raw_spin_lock_irqsave(&object->lock, flags); + raw_spin_lock_irq(&object->lock); if (color_white(object) && (object->flags & OBJECT_ALLOCATED) && update_checksum(object) && get_object(object)) { /* color it gray temporarily */ object->count = object->min_count; list_add_tail(&object->gray_list, &gray_list); } - raw_spin_unlock_irqrestore(&object->lock, flags); + raw_spin_unlock_irq(&object->lock); } rcu_read_unlock(); @@ -1536,7 +1535,7 @@ static void kmemleak_scan(void) */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { - raw_spin_lock_irqsave(&object->lock, flags); + raw_spin_lock_irq(&object->lock); if (unreferenced_object(object) && !(object->flags & OBJECT_REPORTED)) { object->flags |= OBJECT_REPORTED; @@ -1546,7 +1545,7 @@ static void kmemleak_scan(void) new_leaks++; } - raw_spin_unlock_irqrestore(&object->lock, flags); + raw_spin_unlock_irq(&object->lock); } rcu_read_unlock(); @@ -1748,15 +1747,14 @@ static int dump_str_object_info(const char *str) static void kmemleak_clear(void) { struct kmemleak_object *object; - unsigned long flags; rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { - raw_spin_lock_irqsave(&object->lock, flags); + raw_spin_lock_irq(&object->lock); if ((object->flags & OBJECT_REPORTED) && unreferenced_object(object)) __paint_it(object, KMEMLEAK_GREY); - raw_spin_unlock_irqrestore(&object->lock, flags); + raw_spin_unlock_irq(&object->lock); } rcu_read_unlock(); From patchwork Tue Jun 14 22:03:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12881652 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CDEAC43334 for ; Tue, 14 Jun 2022 22:04:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D17266B0074; Tue, 14 Jun 2022 18:04:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C9C5A6B0075; Tue, 14 Jun 2022 18:04:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC7836B0078; Tue, 14 Jun 2022 18:04:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8B55D6B0074 for ; Tue, 14 Jun 2022 18:04:21 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 6737680F84 for ; Tue, 14 Jun 2022 22:04:21 +0000 (UTC) X-FDA: 79578220722.19.DD12D85 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id EFFD1C008C for ; Tue, 14 Jun 2022 22:04:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655244260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Dq7PRgfYueFTliKJ1ZMorr4opGVcM+i2DptJCykcaYo=; b=JKHLiqO4Wxr6CYLhwKIXzCWCweqft29Kn9XEpIpBbJay9T/4iDAYO+fcufRGrPR6X1FBrW Lc5+q8iYWdyhv03ACfloWxidpFiAktLam9J5upVV0SUWZbgq3hVgj6ypjBL8QLqXq9zPZT 1WuJiEOZ/hEu5srEgQkrkl3aJKtpoMI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-357-nbuAHervO9iXkg1fBAknOw-1; Tue, 14 Jun 2022 18:04:15 -0400 X-MC-Unique: nbuAHervO9iXkg1fBAknOw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A8AB685A581; Tue, 14 Jun 2022 22:04:14 +0000 (UTC) Received: from llong.com (unknown [10.22.33.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id 66D30492CA2; Tue, 14 Jun 2022 22:04:14 +0000 (UTC) From: Waiman Long To: Catalin Marinas , Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Waiman Long Subject: [PATCH v2 2/3] mm/kmemleak: Skip unlikely objects in kmemleak_scan() without taking lock Date: Tue, 14 Jun 2022 18:03:58 -0400 Message-Id: <20220614220359.59282-3-longman@redhat.com> In-Reply-To: <20220614220359.59282-1-longman@redhat.com> References: <20220614220359.59282-1-longman@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655244261; a=rsa-sha256; cv=none; b=wFST2fgizlbswErMz2ixXuAl/GZMDAukx+PE3saBfZiG3ktazvrvBf/ykURKqWNAGJvKet aD9Y5E++bt1Yt5YI6heIx79rlNPrfSj9GkBsMmrqBikLvXJI8R6dgBVz1AbEcXRugs9IpV O1omJrpkXiWdSgt0AMRmMl2eHBfHTlo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JKHLiqO4; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf10.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=longman@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655244261; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dq7PRgfYueFTliKJ1ZMorr4opGVcM+i2DptJCykcaYo=; b=H9i8nrKzR1jqFbhXxv1yvvqC8FckFBtyxurR+keE/r2AKRWhWQ1KoIR/kC0hQGMLWwCPLH GwOFJ5vltTa14H0o5YS5mFEbc0C7OUVP0EFykjpdbJtkRuHkcQlWRU0uie7eIdGxoEOCB9 LXpaSxESM/Nvj7f9qgtI53RhHyKVG50= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: EFFD1C008C X-Stat-Signature: jsx9am3gxo1u1x4zbzf3rh7zm9quba1d Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JKHLiqO4; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf10.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=longman@redhat.com X-Rspam-User: X-HE-Tag: 1655244260-186745 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are 3 RCU-based object iteration loops in kmemleak_scan(). Because of the need to take RCU read lock, we can't insert cond_resched() into the loop like other parts of the function. As there can be millions of objects to be scanned, it takes a while to iterate all of them. The kmemleak functionality is usually enabled in a debug kernel which is much slower than a non-debug kernel. With sufficient number of kmemleak objects, the time to iterate them all may exceed 22s causing soft lockup. watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kmemleak:625] In this particular bug report, the soft lockup happen in the 2nd iteration loop. In the 2nd and 3rd loops, most of the objects are checked and then skipped under the object lock. Only a selected fews are modified. Those objects certainly need lock protection. However, the lock/unlock operation is slow especially with interrupt disabling and enabling included. We can actually do some basic check like color_white() without taking the lock and skip the object accordingly. Of course, this kind of check is racy and may miss objects that are being modified concurrently. The cost of missed objects, however, is just that they will be discovered in the next scan instead. The advantage of doing so is that iteration can be done much faster especially with LOCKDEP enabled in a debug kernel. With a debug kernel running on a 2-socket 96-thread x86-64 system (HZ=1000), the 2nd and 3rd iteration loops speedup with this patch on the first kmemleak_scan() call after bootup is shown in the table below. Before patch After patch Loop # # of objects Elapsed time # of objects Elapsed time ------ ------------ ------------ ------------ ------------ 2 2,599,850 2.392s 2,596,364 0.266s 3 2,600,176 2.171s 2,597,061 0.260s This patch reduces loop iteration times by about 88%. This will greatly reduce the chance of a soft lockup happening in the 2nd or 3rd iteration loops. Even though the first loop runs a little bit faster, it can still be problematic if many kmemleak objects are there. As the object count has to be modified in every object, we cannot avoid taking the object lock. So other way to prevent soft lockup will be needed. Signed-off-by: Waiman Long Reviewed-by: Catalin Marinas --- mm/kmemleak.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index dad9219c972c..7dd64139a7c7 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1508,6 +1508,13 @@ static void kmemleak_scan(void) */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { + /* + * This is racy but we can save the overhead of lock/unlock + * calls. The missed objects, if any, should be caught in + * the next scan. + */ + if (!color_white(object)) + continue; raw_spin_lock_irq(&object->lock); if (color_white(object) && (object->flags & OBJECT_ALLOCATED) && update_checksum(object) && get_object(object)) { @@ -1535,6 +1542,13 @@ static void kmemleak_scan(void) */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { + /* + * This is racy but we can save the overhead of lock/unlock + * calls. The missed objects, if any, should be caught in + * the next scan. + */ + if (!color_white(object)) + continue; raw_spin_lock_irq(&object->lock); if (unreferenced_object(object) && !(object->flags & OBJECT_REPORTED)) { From patchwork Tue Jun 14 22:03:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12881650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75201C43334 for ; Tue, 14 Jun 2022 22:04:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 002446B0073; Tue, 14 Jun 2022 18:04:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ECE1A6B0075; Tue, 14 Jun 2022 18:04:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5FC46B0074; Tue, 14 Jun 2022 18:04:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C79B86B0072 for ; Tue, 14 Jun 2022 18:04:20 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9D9F8611B2 for ; Tue, 14 Jun 2022 22:04:20 +0000 (UTC) X-FDA: 79578220680.11.F3BE2D5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 244064009E for ; Tue, 14 Jun 2022 22:04:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655244259; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x10p2RYJ/7U/x7uwqRvY5a/J+6w+5v/qLg41hZfOPsk=; b=i6KmiBEnWPEzhA9GihlXOX/EswvVDgAI+oJa5bDrY3nY1IF0uFduwEsYzBzRmyOk3dPOEi NONZ0ulj9lka3gPq4cmPitiYmMiBnpPTAVDgJatioFoSBW0l7LLNcwFMdvktPh47c1UdeF h7ihaX7SMvGAMSoVGk1v/xZIH6lZYbs= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-591-dTAbzv03O6COUgz8sHXMEw-1; Tue, 14 Jun 2022 18:04:15 -0400 X-MC-Unique: dTAbzv03O6COUgz8sHXMEw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 046693C021AA; Tue, 14 Jun 2022 22:04:15 +0000 (UTC) Received: from llong.com (unknown [10.22.33.116]) by smtp.corp.redhat.com (Postfix) with ESMTP id B55A9492CA2; Tue, 14 Jun 2022 22:04:14 +0000 (UTC) From: Waiman Long To: Catalin Marinas , Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Waiman Long Subject: [PATCH v2 3/3] mm/kmemleak: Prevent soft lockup in first object iteration loop of kmemleak_scan() Date: Tue, 14 Jun 2022 18:03:59 -0400 Message-Id: <20220614220359.59282-4-longman@redhat.com> In-Reply-To: <20220614220359.59282-1-longman@redhat.com> References: <20220614220359.59282-1-longman@redhat.com> MIME-Version: 1.0 Content-type: text/plain X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655244260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x10p2RYJ/7U/x7uwqRvY5a/J+6w+5v/qLg41hZfOPsk=; b=wlKzUPDR5OkPRkLehzIejgGvvDyljGYa6jNN/+9DAje+aaiMH4ZSodsitCzNK314Q/VrAi EFVnr2S9Pw2VfHxKX8NMvAzs38mj+7VrfKtW1bLYEloJJXOzrryjyrloXWZeBdJnkCoB7W UymPSYYz8Wcj/TeoXyE280P9w+GkoKk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=i6KmiBEn; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf27.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=longman@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655244260; a=rsa-sha256; cv=none; b=r1HNuZZNoaQOlsNy/Kq1kxk3FpHkxeVG9tus1UGurDezn3NfJfN74vBcqDRrdTG3uenNmG o0kRgDq6O11wJE5Oz3nKI3+QxEGy9EJQ4K/yMnd60YbJz3Am/JMM2ADXdGNy9Y44EXaX5V Y9G/HAt6J2vUnMlwivTvCD6lSreZ0QU= X-Rspamd-Queue-Id: 244064009E X-Rspam-User: X-Stat-Signature: mgg9ug7by1ra4xbyqg4ay9kdk6mw91zw Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=i6KmiBEn; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf27.hostedemail.com: domain of longman@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=longman@redhat.com X-Rspamd-Server: rspam04 X-HE-Tag: 1655244259-208848 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The first RCU-based object iteration loop has to modify the object count. So we cannot skip taking the object lock. One way to avoid soft lockup is to insert occasional cond_resched() call into the loop. This cannot be done while holding the RCU read lock which is to protect objects from being freed. However, taking a reference to the object will prevent it from being freed. We can then do a cond_resched() call after every 64k objects safely. Signed-off-by: Waiman Long Reviewed-by: Catalin Marinas --- mm/kmemleak.c | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/mm/kmemleak.c b/mm/kmemleak.c index 7dd64139a7c7..abba063ae5ee 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1417,12 +1417,16 @@ static void kmemleak_scan(void) struct zone *zone; int __maybe_unused i; int new_leaks = 0; + int loop1_cnt = 0; jiffies_last_scan = jiffies; /* prepare the kmemleak_object's */ rcu_read_lock(); list_for_each_entry_rcu(object, &object_list, object_list) { + bool obj_pinned = false; + + loop1_cnt++; raw_spin_lock_irq(&object->lock); #ifdef DEBUG /* @@ -1437,10 +1441,32 @@ static void kmemleak_scan(void) #endif /* reset the reference count (whiten the object) */ object->count = 0; - if (color_gray(object) && get_object(object)) + if (color_gray(object) && get_object(object)) { list_add_tail(&object->gray_list, &gray_list); + obj_pinned = true; + } raw_spin_unlock_irq(&object->lock); + + /* + * Do a cond_resched() to avoid soft lockup every 64k objects. + * Make sure a reference has been taken so that the object + * won't go away without RCU read lock. + */ + if (!(loop1_cnt & 0xffff)) { + if (!obj_pinned && !get_object(object)) { + /* Try the next object instead */ + loop1_cnt--; + continue; + } + + rcu_read_unlock(); + cond_resched(); + rcu_read_lock(); + + if (!obj_pinned) + put_object(object); + } } rcu_read_unlock();