From patchwork Mon Mar 20 18:03:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21B15C6FD1D for ; Mon, 20 Mar 2023 18:12:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 884796B0074; Mon, 20 Mar 2023 14:12:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 834DD6B0078; Mon, 20 Mar 2023 14:12:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7233A6B007B; Mon, 20 Mar 2023 14:12:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5F0E06B0074 for ; Mon, 20 Mar 2023 14:12:08 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 383AC80F00 for ; Mon, 20 Mar 2023 18:12:08 +0000 (UTC) X-FDA: 80590070736.17.96A8B99 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 856A480017 for ; Mon, 20 Mar 2023 18:12:06 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=R8bHZZmX; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335926; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references:dkim-signature; bh=2/4zOYZZtH2WSGJk64M37M/sTAQLT68CCaLnUdPqKgU=; b=qzGZDdausqctxskjgBa6i332nUYS/+kZScdUtEmeJciSpFC/F3vZq6JTZo/teM27PsNd+w 5IWLlBbDQknbmDVMMzaJszYCovDx3Nrkp/cv50R2FddEg2gMhVMVcViT3ua48blbFodSxb 86WeV7mz0Zcl9tIEd5qTY1yA7FDS7sM= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=R8bHZZmX; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335926; a=rsa-sha256; cv=none; b=zZox+e3y/hHU4O3RNr4Vxsw0y9d/WPBSlZZ4HtouCF1zekIrpyfhCU1LYj/tJriBvzGXBB RGZ9uM6BQDgKTYpENcS3TRolGjVnJ9MvvQ9n40LVBx5V82HdNxnG+A4kj3sp/tSsdjuJ3k GWgI8z16RQtrdB0Khtz0tESQRcCxCqM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335925; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc; bh=2/4zOYZZtH2WSGJk64M37M/sTAQLT68CCaLnUdPqKgU=; b=R8bHZZmXypvxhfj+rUb96JwoX8FapPt5rvHMArmvCaOf0VftNdze7f0AWW9OU2jmY18uzL /6F4vNee5+k+B1/cjNb9I4sKCRzTgFxnHRGCArgyruNbHag+rcMhjws3D4lktLJftM4QGz JddGvSxD28/bCXZyD6KGZQiEWG2wPDw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-384-zhto1ZHtMbmUSZaKIzWElw-1; Mon, 20 Mar 2023 14:12:04 -0400 X-MC-Unique: zhto1ZHtMbmUSZaKIzWElw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9500A38173D9; Mon, 20 Mar 2023 18:12:03 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 20A08C15BA0; Mon, 20 Mar 2023 18:12:03 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id CF185403A3E4B; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180332.102837832@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:32 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko Subject: [PATCH v7 00/13] fold per-CPU vmstats remotely X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 856A480017 X-Stat-Signature: g1d95ep734mw747h3ybaeqmq3niqwsyk X-Rspam-User: X-HE-Tag: 1679335926-57546 X-HE-Meta: U2FsdGVkX18L/QLnUyAuIVrjI/9bHOGur1Ow76nXHoLG5csb2AkdlApaURtAdBX/juWh8b42eK+RGIcbVQTLKZs+r9YEpV6cJmPl9MEj3PAKG9xUz0t6ViDX37m2cYpFZhSS4Tp1QYluZQBjjcJXtRgGyxtj+tqEoVkJ9SMYDMKKhFuFuZ8Yp7cJz7+bAy4bRxKpHYLR1pURw4pCu3Ozi4R6g+6s3NSyvPt4OpWaNbbj7vsmEYnh7obNAe5o6QBK6CYo3HSyFpRMCm9GTvkBo/QNoNKolJL2wJMbZOFimhL3msy5XIoWDdepYQqqHpUQsmWuodZ/RELL5glJOvdQASlc5QtjJxiMr5dx5iyoKIuFHMbXAiSl9JdyUTooM+5u4JSLS7MdaLiZ7IgsN4R5FCz4skBuLP4Lms+cEIj3NrzbuLtb0GltUkttHhhuvygOnQLQGpGdyJcjYw7ATJAZouloj2su8uPMGbfyw+jmf9ifopdWjUWc7wGrt/9SEkM+NtpcNLdEqwpQHHEE8VIzTcx40/oPKjwtzJ35XhZdNdSLNYqynV58P4EabhI91rUajLaxVNP6cAXpkWOXvmjLMsEQO6PxP9B3ie0a+E0jGrLTyJVBJlfSydSbAH01q9XDiIT1ZoX0BtC3A6hpQ39TrlbRO3ISV/p7wSgFvTZijxd11WtjGQRIplMihBAIWhRKeIJap8U4+A7nVs3GMlOtW4wyVKqWD6CxWCX4VJx6GPgCUYEO9Anpf6BgjYm6mMLtVU1VASX+6tjdtdi4s+tvnRQtIVzwRGJBRmbKS/YexJqfNVNhSlHfI7Pv9DTLUz+jUx95RhgK4BuqhCNWh+TwYXclugQPFJpZmaEzYAtvN2CN1KWrMJc80V/2MaZqUOOuQgw1HX4xu5q4UIPT0YXOkiG4/Ngjfk32eFGR46HXkmz3dMtbBRFEKqbdxfCdZKQRwdctagfhn8l9wfqCJiB CnhP4qUj vJ1jjc6KWzqHnftvSm+dbTsayDufovh7e3+fzV03z2w6nq0sakkg21bTM92ISnlg4xw/oei8QU4ZK+CsnuJcOdcQKxGPK7rqs7tKCjRdVt4N/g73G8+u4NJdonRVQENdmkWMM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch series addresses the following two problems: 1. A customer provided evidence indicating that a process was stalled in direct reclaim: - The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES]. - On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty. - This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely. I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet. - In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat. Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark. 2. With a task that busy loops on a given CPU, the kworker interruption to execute vmstat_update is undesired and may exceed latency thresholds for certain applications. By having vmstat_shepherd flush the per-CPU counters to the global counters from remote CPUs. This is done using cmpxchg to manipulate the counters, both CPU locally (via the account functions), and remotely (via cpu_vm_stats_fold). Thanks to Aaron Tomlin for diagnosing issue 1 and writing the initial patch series. Performance details for the kworker interruption: oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... The example above shows an additional 7us for the oslat -> kworker -> oslat switches. In the case of a virtualized CPU, and the vmstat_update interruption in the host (of a qemu-kvm vcpu), the latency penalty observed in the guest is higher than 50us, violating the acceptable latency threshold for certain applications. v7: - Fix allow_direct_reclaim issue by using zone_page_state_snapshot (Michal Hocko) v6: - Add more information on throttle_direct_reclaim problem to commit logs (Michal Hocko) v5: - Drop "mm/vmstat: remove remote node draining" (Vlastimil Babka) - Implement remote node draining for cpu_vm_stats_fold (Vlastimil Babka) v4: - Switch per-CPU vmstat counters to s32, required by RISC-V, ARC architectures v3: - Removed unused drain_zone_pages and changes variable (David Hildenbrand) - Use xchg instead of cmpxchg in refresh_cpu_vm_stats (Peter Xu) - Add drain_all_pages to vmstat_refresh to make stats more accurate (Peter Xu) - Improve changelog of "mm/vmstat: switch counter modification to cmpxchg" (Peter Xu / David) - Improve changelog of "mm/vmstat: remove remote node draining" (David Hildenbrand) v2: - actually use LOCK CMPXCHG on counter mod/inc/dec functions (Christoph Lameter) - use try_cmpxchg for cmpxchg loops (Uros Bizjak / Matthew Wilcox) arch/arm64/include/asm/percpu.h | 16 ++ arch/loongarch/include/asm/percpu.h | 23 +++- arch/s390/include/asm/percpu.h | 5 arch/x86/include/asm/percpu.h | 39 +++--- include/asm-generic/percpu.h | 17 ++ include/linux/mmzone.h | 8 - include/linux/percpu-defs.h | 2 include/linux/vmstat.h | 2 kernel/fork.c | 2 kernel/scs.c | 2 mm/page_alloc.c | 5 mm/vmscan.c | 2 mm/vmstat.c | 440 +++++++++++++++++++++++++++++++++++++++++++++++------------------------------ 13 files changed, 361 insertions(+), 202 deletions(-)