From patchwork Mon Mar 20 18:03:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7A3DC7618A for ; Mon, 20 Mar 2023 18:12:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8DFA6B007B; Mon, 20 Mar 2023 14:12:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D16F46B007D; Mon, 20 Mar 2023 14:12:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDF056B007E; Mon, 20 Mar 2023 14:12:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9A9566B007D for ; Mon, 20 Mar 2023 14:12:11 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6329880D4E for ; Mon, 20 Mar 2023 18:12:11 +0000 (UTC) X-FDA: 80590070862.06.83B7194 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id A4BF340018 for ; Mon, 20 Mar 2023 18:12:09 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e9kpL083; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335929; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=nxDgSxDt58miOCbO29kj6Q7ncA44wbvUWM1rbcl0z7M=; b=T4jfqloN7HbRhwjSokABT3pS+zrRFSLjQ2G+vFmqCFbHzwujkvMTrqtgg/aTIs2O2+6jxR AJ2H6wjSv5abYUaq26qkNbL/ZII7BjJUcE0/p4uR2vZg9AhKnrErGFGP9UBSnHUftlPOKb G6aIDJMJ6xw2N6rZLkwOQ5O7en7KxeE= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=e9kpL083; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335929; a=rsa-sha256; cv=none; b=cummY9swhU7JKOTQ6kpgykE2Es9wfxy2QhFxQ2qBOA0BP71Nv2uocMJsZQ7U7qd7tp/W9x 4T1x90zWzEQ9QEFs+yqLTZdpuEtZ3LNeBlM6PAR8V4k4gaZPSANknTn358DPgaH16o5gjY Q06+2XbKWOGJxLnTAFpIWgI0RMa0pek= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=nxDgSxDt58miOCbO29kj6Q7ncA44wbvUWM1rbcl0z7M=; b=e9kpL083+4kEcsoNgC3xyJEQmqWxgdF8LfuOQmFt3nA7sIBrGZmknu1bsXJgAUaW/mTrlP Qs5KVYuLY+TGtbKMyxNtK9xfJ80xn49rtRfaWQUvyIPpFxng920eQU1q/1otmr/fO3UHUh wBrnrMO/cNd4LtIvPO41qAn7nsWJiJg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-138--rX6rrAwMOqsYpUa1YzcdA-1; Mon, 20 Mar 2023 14:12:05 -0400 X-MC-Unique: -rX6rrAwMOqsYpUa1YzcdA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2B9671C08786; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EFD072027062; Mon, 20 Mar 2023 18:12:04 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id D1AD6403BC89A; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180745.556821285@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:33 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 01/13] vmstat: allow_direct_reclaim should use zone_page_state_snapshot References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: A4BF340018 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: de6i6ak8t348ymndqixh86nsf4nddiwi X-HE-Tag: 1679335929-311444 X-HE-Meta: U2FsdGVkX18x2zra8rACEaBUw3dcCThiZlg1wCUA4cGdffAANciElGkyPYDCJGGEjVoIkgo/rRthItC+t48LyY3xFjp3wD0raIHxkVOPLTc0V6GbiK55gg3rt685BbvcwENNypQNJCLCcxZKIRUopfuxBCS3sAQKBcpZ04qVjKew6j8Qt2nA/PCWmr3+FLfcANA6UPH6XsecHUWX/UdDT6d/5SPArbcCocwCTUHEYvohYNCYJIrJ713DxNMzr0+L23Fm/YgNW9D7FPq8z2PBUhrqVo18jm2suT+PfqrtL9JZ81pyFqwGHI3Sgi5vN7rAWFJOEuqqqJGqX7+52m192d1iJJlK2DbIeeDibIOsEq/vk4BWWmcewIm7EjoariLJpFBR6s7erEYgzkSWVhdfV6i0+PtAlSpPR2Rwh6HdBqeFhjYW73sUbbSzrOsKIVLGf5VSK5WOzvPtatCrJbs/QIm9DF+sJ7NeXhCbkbMG+7gC/pdd5geB8pMyZD9HbKjH2UZuU/Adk+mRYh3tmWjuTG+UeaoaRZizha93/hxtaX50QJwrabCX9QWgD7No3BjVpxfGB+JOZ+gvqqeL8MzhnDvzypuum2xwgRwY9usqMxGvhoWJ0r/6Q9sUw9Y13PmWeiM4C2KEiulcznNZXF4fwGCYVYOVlyzoRR39lG0HQBGl80HPU1zZyNv5D341O3WjdHHnLcNrjVx9svJO3fPfxSu9aR24TEwwarq3Va3P/TeawKnx4bxhxAK3fLK67E5QCAk7MRwaDPeYV6KN2HGLROn3/jbfKKSK+jyRATEPxnlbAYE1UiZE0TmjC48UPaN104NaN95s49FLnrU+j3COgR0HdJZexOiJ72jenzsxZ04Uev8yKx4PTe0GNXfA8jMIAedy1T++XSORw/WbGu/EoVxXvt/UqxusXWDRP+lj664iM2WFrCIIdSgAGL4/Rzd3EKypd80xj9FC29WgLQh AiALHU9H WVLbOligZQCP7lpk5fHvLcSp2yRH+7WuxMQ2GjT8rxpfYSQ99Ih1P9kza2UtIKbt375eJfOKjhj8i46LDD9+FZd0eiHjVV3p3V1KS2FtB5AcqC3kbx8mQkKMchjU0ffQN3rtJQwT2JdrdULGUazibBSMcBt90JQsM+cpSfxGgsfomfl8aJA2OrwkY30a1LtUncUJMavMGG7fpKEAS1odE2a+hBnlSz+ZGFyiwlZ1zPvKRFtmLkvP2wBvRUkb/nTRWrjL265gPmdVX0e34yG5AIJqpuu717gLyRunBvlTa3W4LBLYKrAcs1iw1nw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A customer provided evidence indicating that a process was stalled in direct reclaim: - The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES]. - On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty. - This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely. I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet. - In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat. Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark. Change allow_direct_reclaim to use zone_page_state_snapshot, which allows a more precise version of the vmstat counters to be used. allow_direct_reclaim will only be called from try_to_free_pages, which is not a hot path. Suggested-by: Michal Hocko Signed-off-by: Marcelo Tosatti Acked-by: Michal Hocko Index: linux-vmstat-remote/mm/vmscan.c =================================================================== --- linux-vmstat-remote.orig/mm/vmscan.c +++ linux-vmstat-remote/mm/vmscan.c @@ -6861,7 +6861,7 @@ static bool allow_direct_reclaim(pg_data continue; pfmemalloc_reserve += min_wmark_pages(zone); - free_pages += zone_page_state(zone, NR_FREE_PAGES); + free_pages += zone_page_state_snapshot(zone, NR_FREE_PAGES); } /* If there are no reserves (unexpected config) then do not throttle */ From patchwork Mon Mar 20 18:03:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B499FC76195 for ; Mon, 20 Mar 2023 18:12:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E1326B0082; Mon, 20 Mar 2023 14:12:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FBE46B0089; Mon, 20 Mar 2023 14:12:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D17B86B0082; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 98B956B0087 for ; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 770EC140A30 for ; Mon, 20 Mar 2023 18:12:14 +0000 (UTC) X-FDA: 80590070988.01.397E587 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id C7742C0020 for ; Mon, 20 Mar 2023 18:12:12 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jBp7le+b; spf=pass (imf10.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335932; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=418Es5nqdsliVmHbTuXzLvQQdVHfWzYdcac5+iWJU4U=; b=Qi70WLkA/TF8G71BrIfv/MJCRWDM0PjRu61F7yiUtk35o80P3uhzTpNd6EnethXC4fhgwW udJsMkw/P/SE8kCSamVfl2utE8U2VAOSltYlW8MC1jK3Ny+L2OP94Cm4MpEVUkHYR8TIig 0JRXedutiVGBE3ik/5JRUrVG2UwoKFE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jBp7le+b; spf=pass (imf10.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335932; a=rsa-sha256; cv=none; b=fIw6LsLaz6ngLgFWnUw0P1g189v+0Y+TWBXemFiw5koZxVbMiKn3bcuO/2jBuV6NmlgJqF N3AE7GHo4S/q+cNWiuaw6egvppceGti4y/AUkHUjLy3PEiZCAb/18fU716lQo7bq03KkDB XVn3nFFePz+wplyw5WogfErK8ACfKqk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=418Es5nqdsliVmHbTuXzLvQQdVHfWzYdcac5+iWJU4U=; b=jBp7le+bXCvw3fhezDgxr5yRD0FKsPp0MI4tVe2EdXSMQRjIN90u/SOTQsxX8UiYQKX+TC yhZ1JOn7cMwS29e/FU+O8v8pzUmXfLVnYtvf1O2N5e5K2aetCdnFCvr/RcryM55qHlk39R bCh6sGmr/R16TAUjhR60Y9rnAZw+HaM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-228-d_AdcjSBOMiexEZSO9wA3Q-1; Mon, 20 Mar 2023 14:12:08 -0400 X-MC-Unique: d_AdcjSBOMiexEZSO9wA3Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2F2E98030CC; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 03D3E40C6E68; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id D51A9403BC8AE; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180745.582248645@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:34 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 02/13] this_cpu_cmpxchg: ARM64: switch this_cpu_cmpxchg to locked, add _local function References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C7742C0020 X-Stat-Signature: yfdgyih7a1sjmho9nbmfby4mzhmfiman X-Rspam-User: X-HE-Tag: 1679335932-898176 X-HE-Meta: U2FsdGVkX1/FruLQQ4UidMLQSMW0X812uWaxPkgnikQceO9Icx0dBEGnoCvSrMAwWzeSidUNkK+gBDzhikNwH/y/CURFPhl/McilMksauQiUQhQDAx8p8CLIDp5oDtqIPa/IBTrKcxiewg8i1n7gxWRafsvKnJdf8WgO+3IrJTM09KvnQKDSR927EzPztE2/bCwMys9Hr5eZXU4C8MNJHNR8i+TFuwz5/yt9uGxbJLa9ewvHlCLkL1IPfef9z4veYoc7CiT9lOHLkSTE4wmERyV5sgiGT2MnehB/u1w3UM6IJIXRNQryJM6bQ/48IH4sk2O6txpZtrySfRHmnlV8Wjg+dqaxx7CAkf8MaQiMcoDgBmN7rOIZR76W4iR2EXKJD8Ucy94fvkWbFr8+Lo8KSNor1KZBMbO6LpwOaMnr4zqPdH+gaScfUTj6f6ecva3sHPDBvyMBsqHmMHQfiL4tt/FSOuFul1ooytex5CyFg6zgg6OF55+3CSr0odJe2uQvEcPIZ5bloZU3NrzHPS03JmZcodQh2bSckG/fhdJil3ub2WTXfpQhCQIaBz3VWk2HoUf8eBHvKE1ihTk94r45GzAqSCDK9chHTexh5fnfziCJk3M0XDVMOPw0FMOgbIZCrG3Sm+ciJ68u8FSyJZ8pNrryZqH19u7ypj9x4XqPDd7C4dwho47f3EWgkz8D/D2DU10EkuAZLu98KoO4iqOgpbKBwb5JtZY1bOVnWy8/EHIVqbWE5wauNbDyMTrK2bdDg2xcbHrovRMQRh8U0qbj+V/cpusJi7rFN8TckyM475QRBHX0lao8Ow1CNNtxbibX0gHSfg8RjsPs40pihYEyai0/4iOBB+OZaROXYTkTmCDs6MEiB4Ck5qTCspXeBieCP1PtSjM8L9nu8RFTByAb5DWUoa1iOb3iFscvsYBocWZ0hG78r5H0rUaemoq4YB2ZY/A3GHKC3fQ1pCD+MJf wNRb7Acz AXmxxlJrLxCAOR1g2N6HYa3c390/Wwu9feDeEAzuU7C0poBRkkfTbWVJrrP8THSiCdCMzSSlDume98t5ylkBNkAdRTbVEbaPfDk/XIPMj6c3QZFwzFWJySeVa833rujY4G5NqJHdwV8c6+drkuZJae7Cqmu+748cpmhoIaVMpkc52f6MEDBqORJYCO9xrnIPJCLk2EfUWD9gK97R+2ZY+vy4p+egNeSaW8VQI62JgqogIePZo3Vh5JKyiCA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change ARM's this_cpu_cmpxchg_ helpers to be atomic, and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/arm64/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/arm64/include/asm/percpu.h +++ linux-vmstat-remote/arch/arm64/include/asm/percpu.h @@ -232,13 +232,23 @@ PERCPU_RET_OP(add, add, ldadd) _pcp_protect_return(xchg_relaxed, pcp, val) #define this_cpu_cmpxchg_1(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_2(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_4(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg, pcp, o, n) + +#define this_cpu_cmpxchg_local_1(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_2(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_4(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); From patchwork Mon Mar 20 18:03:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDDACC6FD1D for ; Mon, 20 Mar 2023 18:12:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 973BC6B0087; Mon, 20 Mar 2023 14:12:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8353F6B0088; Mon, 20 Mar 2023 14:12:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D5716B0089; Mon, 20 Mar 2023 14:12:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 591746B0087 for ; Mon, 20 Mar 2023 14:12:15 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2A49FAB565 for ; Mon, 20 Mar 2023 18:12:15 +0000 (UTC) X-FDA: 80590071030.23.2613130 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 5333080014 for ; Mon, 20 Mar 2023 18:12:13 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MKE9CSxn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335933; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=6QfpHdKEKCW5B9XIpfLylgQS/cYcg8nRN+BFpTs113M=; b=QLduv0eIyeYmmf+3qQSbNiedw7ShZVtC6yU8csfENKY8GmUyvdYUZfNVesdh7+eWK9DD/N Z9VjqMNpciaSwLQTqiQohtkYx0DysShQIU+xYAEFRSl71NPiBTJbK9sZeP9kE2mM/nBiFT GqlqfifOC7X9NuvHrZI1SU0XCjRMBRI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MKE9CSxn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335933; a=rsa-sha256; cv=none; b=G1UTAiWDbtnVZE99ZihEKoZXz61cRH1lUtY+23Ae2TQfc3fy2YoW8tG32BJMCUrnSHjgpl gPBA11RtQ3cwSWM6n98TIlh5VFNprA6SHYDoj/c6KNTI/tnAl8j5ftbTckA91N6TQ+zvh0 f+rqMWpyHGPYnYP16ucQ7Mwr5ZMx/So= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=6QfpHdKEKCW5B9XIpfLylgQS/cYcg8nRN+BFpTs113M=; b=MKE9CSxn08tMXYEpI8gE00qOiqkf6nceimBMUF0NMR2dSXdQSgv/O1+UYbR4omnM145e3P i/wC4QRYJ9I6drO5/e7ibYA5wnekIbgPV8T9QnrkiiwfcC2o0FXMYVbPXwTNvE2wHY3k2a bFOfx21tH/h2H0vEKuKBE2N+LohBtfM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-228-1GlfG6L4Pq2mh-MeYubarw-1; Mon, 20 Mar 2023 14:12:08 -0400 X-MC-Unique: 1GlfG6L4Pq2mh-MeYubarw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 30316858F09; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0114E40C20FA; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id D7661403BC8B0; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180745.607294360@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:35 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 03/13] this_cpu_cmpxchg: loongarch: switch this_cpu_cmpxchg to locked, add _local function References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Queue-Id: 5333080014 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: n3793mkybu3yr6tns3yifbu3zfrsqk4z X-HE-Tag: 1679335933-836918 X-HE-Meta: U2FsdGVkX1/iCJaLy8vS8z4b30hf6Y2LBh8BxzYlGucXxAqzEkHNVa3NPLe6tvf0lWn0QFG3RJk4G96aAzmBqlR6phFlAeRG+MdhTGXZx0AF/pzo3E6gzkZ99JbAvHMJ429Ct9Ezc2TYxFOvN42Y/LtyzIuLlxFKi1luM81cpAnk4PIYSUxjJR4PLB3K28wvqKiku0WeRVTLBN9mW284elOQ2p89E5ewpELfFeh0AQ4BQzcTLsXU8OEMw89abKn8837l8EOpuVQX+eam3h59ikt+d7vB39X8xWmDlQ0yEBz1g7D0vpgk2XK0BRLrvhLBa/y4nOHVY137nb13vcj3WK02qnU7I4IWNytaJSfftspF3+4MGQvia6yk5JSJF0iwBdo7y991/jH3+mC9GKuofdzRxu+ut0Ok40nuj/xu/TbBwElohssa7arc0B90HfYP73ruM6Uf7FxRqj5MuFevd2NmUU8DM2xrsNl70IietsMooaYNdG6ISINzLaPoOBIbb6vozhbuUKFbVY89WgEi1YJP0f1mJDBiivnWJmM1UalMN/OF07RILcD0MoKwj364+sL0zxTVP/dNwPNCAfLBAT+2+wURpFqo9yJ41iQDL/K23+qAPekKI1uTF8zXyGVR02GGcbCPYN2PoLWb+XB3eIbHP2TQ1fAxZb2fKL2em3tZRnb4xJuXE931szfG+KjpF6MO82965K+rTGqz+AtRICmvllUwzVIB9cjuLx6CaSqUaPlYl+/zIH/sSzuIiF98P0ak5rFVRJ/a8WeKuDilfzflpVmcQ9T7JiZXKAhjSv+GlFUS0CdIqyoYMhCMDynDO1lZZH/ifIu9jHWfDzS0cGlwaPZ4+MHVdM6jf9g7QCbIkZ+DCykWitBU+370VI8MlSzwhob8m5KZncRub8HbN+cskF6XVdWnxxY7/IXw0gmjrZUQPK8EXe6JvE+BVbqaB4vzOrnSERL0AIyr/HP eUfIJU+H UZKqhty5ciRQUuZ2VNl8xI+MSr2BKDl1m1YitNWem5Ypd6XPWnZPViF1y1FYBu1tMCTBpJ0oJNlt8N/zjS0gksFKWAEhHbgfsCJCUj616xC2R6cvMCoBdWt6ZeGsTIyei53POpnuwBYgJaf2hkeUEiybH/+YiDuvAaNv4akvlbe1eTqvU9k8QSz+62HgzTHs7MVX7+mrTRfnKR3b2qBjnaP1SM6rQ5QE+qfOohUvRrcV8v0DnEWgeYSw9Mg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add this_cpu_cmpxchg_local helpers to Loongarch. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/loongarch/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/loongarch/include/asm/percpu.h +++ linux-vmstat-remote/arch/loongarch/include/asm/percpu.h @@ -150,6 +150,16 @@ static inline unsigned long __percpu_xch } /* this_cpu_cmpxchg */ +#define _protect_cmpxchg(pcp, o, n) \ +({ \ + typeof(*raw_cpu_ptr(&(pcp))) __ret; \ + preempt_disable_notrace(); \ + __ret = cmpxchg(raw_cpu_ptr(&(pcp)), o, n); \ + preempt_enable_notrace(); \ + __ret; \ +}) + +/* this_cpu_cmpxchg_local */ #define _protect_cmpxchg_local(pcp, o, n) \ ({ \ typeof(*raw_cpu_ptr(&(pcp))) __ret; \ @@ -222,10 +232,15 @@ do { \ #define this_cpu_xchg_4(pcp, val) _percpu_xchg(pcp, val) #define this_cpu_xchg_8(pcp, val) _percpu_xchg(pcp, val) -#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) + +#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg(ptr, o, n) #include From patchwork Mon Mar 20 18:03:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BC5DC7618A for ; Mon, 20 Mar 2023 18:12:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EEFA16B0083; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E50916B0087; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C55586B0088; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8F5196B0083 for ; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3622FC0D96 for ; Mon, 20 Mar 2023 18:12:14 +0000 (UTC) X-FDA: 80590070988.07.C7317AD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 738AF140024 for ; Mon, 20 Mar 2023 18:12:12 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bVllhTjK; spf=pass (imf09.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335932; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=ZYdI/NkwLfDGmXJs7KnOK9vem40tm0vaN9MiHTPCcnM=; b=QvkE4cFZMaIDwXIRthIm7Vq8m3WYI6vq/HGS3wcv+CWWnVHxS0n0tM0AuYDuR0HgWurF2u dg+hPZB8JZjEi5PQeOGgYtEyhMUCkFUCMYGQbiwajsomadoQl6mSc1jYUCP1X1XARqeoni D6tQOGMwpcMjj9shCMCV2U4NRDYzqes= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bVllhTjK; spf=pass (imf09.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335932; a=rsa-sha256; cv=none; b=ZR4um1BuPiwJRyosKk7TW/7MTdiyfxZYI2NNyUFRTlUG5ovVhAH5p7YYV5Ls82tOS1dwCq M5aIS4lN/LZSvznDSWdyq7oWcVL8aoRA/lnOWd7mjYnf3uV1LWnJ8TuWB8RTLTI+N7GN0A IqgXrs3e1xI/TDnZMZkT5/6GF4m91+M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335931; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=ZYdI/NkwLfDGmXJs7KnOK9vem40tm0vaN9MiHTPCcnM=; b=bVllhTjKFX6mvp0cM+qwus0uOcTolB2dd/eUBgF4A38ykFHcnFE1LCw0mTYxCuHMANpI4s k4GcRi7zI7YXUPmJuyufikA1JXPbdigZhsEgiLSoKNB4eU24jAZhwQLCYFRJytu40ksN1V dzv3fQGr9xg8IFr748+yCucMh4jRmCA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-138-7w4ECqAPP2uSVIr-cMmNMA-1; Mon, 20 Mar 2023 14:12:05 -0400 X-MC-Unique: 7w4ECqAPP2uSVIr-cMmNMA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 32B9B858289; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 062781121315; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id DAF1E403BC8B1; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180745.632709950@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:36 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 04/13] this_cpu_cmpxchg: S390: switch this_cpu_cmpxchg to locked, add _local function References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Stat-Signature: whfa33oqzpybapujp11smmmzausqyxpw X-Rspam-User: X-Rspamd-Queue-Id: 738AF140024 X-Rspamd-Server: rspam06 X-HE-Tag: 1679335932-41106 X-HE-Meta: U2FsdGVkX1/0zeDQXXvihwCJf8wFaut8jb/urTcdgqDdzwKdsEcO8KfQvo5NvdF7eJaFPl2yApeYt6CKWv63jlM44xzLMBeVMAw6xbwsb+u0xNUzjosl4GLkauwX4rwriWxLnnbtHwyf0ISUcPiX2OVhGcQVYf4Z7OQwo2/mMWyWG5S0D+dpzr7NENQsGAe3yR+J2HISvN2IGI1urY//CYGCohGmNZU72eYaNEg0UmN0soCZPd4Wv97a6LNcHaTSO0LduZRd1084N5uRMXNHQY/MUwxC5YpSjV16FKtxFte7u+llkkxcTq5U/m0ry1hERwjq8XMXPUtvCcDiol7SCddxKjf0drwWO+eHpKUWWUkjZPIf7D4k/rglST8pa29m3OtbCprX2jq+/JS/ABG298Tp9qeZ5PPbhfhMjoO3lJJ+NDLvkGpKzDyt6qLLApRnaUpl8TR47kgvYo4cEZ0wCLxntQwsjnPiV0RVT8yjHuLQFjQw+70SLVrIlpdaai7DFlQ3qJFnM6XOUKwoIkRl5kHLpUOZy+jubycMzlTm33uLn3Gz5NyEEflBz1XgwNpAZeXAd8milg1JMfJMa6vKUdSABr6a4lj9Q3C48D77EFyZGp6L42pq/q0ICkpG6ofqJcxiwCCZxHKgKgPmHmr+NL0CaR1IVM6SdMdsB9ok02nkhHi7HWqoiyhnd22yb724U+pOr7KK54J1Ym7navJLmZzNLSt693e71XTS511WFBqGEnuHRUz+yaBK5Sr7P5fUrwAPtGo/ChPPsNpsjX06RsJI5yHker9jaEypGPfKvOwg9JcPuGIV13gSJz9OLuHa5mF+y/uqK+gHr0jNQRBkTiGlJQMn59E8eRTdjzWce2Di2/5YDPkZg0zFzq+Yx1o6xhk70hLBO5IvF6IsexUTBT/NP0Jz+3z2nwbH0+Wu+EZu0Zd+viZMfDflJM3GceMf3s9t7Ho2h5DuQmDFvOK rn6n9hXZ 3/ELo+gMae1WENAC3g/Knz063sdNsw+sMkVvctNnFNz4N4rsZTQNS4/GaXpmh32ECs7T2wrB/ouR1p8MyDfFwSmEpFL5DW8O7ZR1++xbKlPN9pgFZIbSI8UA8CFq8wr52pu2NGZZjA4Wq5QH2qjeI4Ojqbmc4YwpHSy8aN5S4HIQnsEzR7fLNWeNc2r9RWGEYMitdzZlZnoc3YfRZL2QsGw5rNCZtd3kNGfskkxSvBKcRo5KwdMQqLSeHNA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add S390's this_cpu_cmpxchg_local. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/s390/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/s390/include/asm/percpu.h +++ linux-vmstat-remote/arch/s390/include/asm/percpu.h @@ -148,6 +148,11 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ From patchwork Mon Mar 20 18:03:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3100CC6FD1D for ; Mon, 20 Mar 2023 18:12:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD5066B0085; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8ADD66B0081; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 687576B0080; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4F9526B007D for ; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2CD1480D4E for ; Mon, 20 Mar 2023 18:12:12 +0000 (UTC) X-FDA: 80590070904.28.9D1410E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 7079A12001C for ; Mon, 20 Mar 2023 18:12:10 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VU1miWKG; spf=pass (imf29.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335930; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=F5ZFnFWd0bfXvbTVZzhDKmwHy0T6x+spMFYT61y9M6E=; b=ReHYP7msfG5A4wEe4K+Rn7cqBVhdBGdGEA9zx9nXE3L1z3psL/sretpJhxAQ/jbxwROPWI ad2j4HHjBc65qJRAB6I0nK5AJZrk0GgPMt/tjLKsfhRAe0B9Z77vWyT+4HCsyivWzZfCw+ 6/3RBdTksSN+zNlFojX5SHt8hVTm1Po= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VU1miWKG; spf=pass (imf29.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335930; a=rsa-sha256; cv=none; b=TYa4+41mmNYVaCbRNDiPq5oUvO6LwBGnhiA4UbZLqgWYLEnMF9XZtOABy2krYgE0u4KmGr 3xtvClR25ZT1CIrfPlPpCf6/c9DSCZN6fQbsXikXMLaJbEhvOQkjhAMKClx2n3dWaR3DZT vuuKxVuIdS++Fg5gKzwsInSCodnmF7A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=F5ZFnFWd0bfXvbTVZzhDKmwHy0T6x+spMFYT61y9M6E=; b=VU1miWKGQYLS5Zp6HvNbJabgeNimxkuUBuysVYSvbAcnImUW7/IXyEAKn74EE45S1ySXeT DxNG3dOl+vNCr6Hq5K67D/c2HjgtTr4Bguzq7oxVbxSrb+tt9calRVGs16FkLc7wmzrSCu gQTx1XT6LdquCuPh4zuMlCAOSDDlRO8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-248-VzJb7ShvP1CBrFE74a7ROQ-1; Mon, 20 Mar 2023 14:12:05 -0400 X-MC-Unique: VzJb7ShvP1CBrFE74a7ROQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 52C3585A5A3; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0108D40C6E67; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id DFEFB403BC8B3; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180745.658574087@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:37 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 05/13] this_cpu_cmpxchg: x86: switch this_cpu_cmpxchg to locked, add _local function References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7079A12001C X-Stat-Signature: ef5j31i6pst5fkzk9jy88hzyqih94r8y X-Rspam-User: X-HE-Tag: 1679335930-502584 X-HE-Meta: U2FsdGVkX18lO8hyejGpsrEQoaiTc7hXp5qZpKVhi63n2RdXM4pMSFAg0kQobADVFhT174gpBUq8jvi6pIhU30xfXF78mGSBklTZfNLiuG7rG89gfzHvRh8u+W8tWTg7mV5w3XnSynQUUi5C9f6tCrARYmn7YfWyc7CzPRK1DGhURz2Wr2q6Kfcwt6mx6pRgGhVZGdhySey8uGe0goDU3GTOETf90Lfj05XicOZFgYZ4K0gpZaw/c8LMIJ+IyEeocOm3jnvzMVZPupcr6H+wbl6Q49hBErr3YnVUplXT6hEQmUwGNaZ/VzRvAFoSH6D95WYRkuz8K3qt6mWlUW2C331oA5Phhk+8+uCKlJN9So16fl1YAfO+WIXepqCqQQerrCyFchRmOmWtNESmZwE336wD11Ca4ru2UcNyL53PCd9pGPJ+U5PMasgtk3hgkQYmOY9l7BT5cEIX7GA+UuOwlbii4FOmicG3+gUdTchGIR8ITpvSfNQ7p97x/2Z+Hg9TsIrY32HYfqilLeEPq5mtuSsH9bzd1c3twNATuqgGUhiG73XOZ/MAVRo0RCe+UzuXt1TxtqIUVqnBpJmWgl9KjncLg3kRuKsTJn8aVNV4WQuRcnjt3PfgzF3/Wn94kmqT2HoFMkboAMAM6JceyP3yVaKykAsseiab7Ou/wR/a4xZzcOvu/ocd665BjBBtUgqLRk0J98SrEKl3dJyvoXUL6LbQDOE0CvY7rRKVLq0FjdTSApetNGuQgalvTXWgQ8PYbfBN1aqAxcaJTw0ydFVI5gX1ZYDngoe6y+bLoJNs/sr7lvJINrzrg9yHZOtFYIE1IAF500c8U3Ha7UX21WDUiNW74YCnq6dGUkY1JK1IIVlNgCwIiAoD146pmKQBp3418dbl+dQDbS8qJQz8umU9kGW+hqu2ETvai312n00q17MpNwwke4+hdmENjNsODP/qeyXM+eTFxdzteCbdBV/ wXptpRzv K0RKn9ZhU/aC0JeYqLUUqzoCHngWFp3HFQi/eCHAJUhbzBZ6TsCJsG3zXiZ+7wXJcYfhYqRjiMP2BGs1lKQGmXTXXruaFkYjSc2tS6k/A9bAxlHQfrBlN9kpSyPFp6/pWezQ84oaMbz7XcGDdYMLWSzAsIAm7yj658okQFuNUHVLqxsEntA1mQ/HEoDzoAR42Dh7C9swDF8T90NdYeRT05jDJ8YlccZRuAlCSFy/As/sNcNcm23sPW3nYeQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change x86's this_cpu_cmpxchg_ helpers to be atomic. and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/x86/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/x86/include/asm/percpu.h +++ linux-vmstat-remote/arch/x86/include/asm/percpu.h @@ -197,11 +197,11 @@ do { \ * cmpxchg has no such implied lock semantics as a result it is much * more efficient for cpu local operations. */ -#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval) \ +#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval, lockp) \ ({ \ __pcpu_type_##size pco_old__ = __pcpu_cast_##size(_oval); \ __pcpu_type_##size pco_new__ = __pcpu_cast_##size(_nval); \ - asm qual (__pcpu_op2_##size("cmpxchg", "%[nval]", \ + asm qual (__pcpu_op2_##size(lockp "cmpxchg", "%[nval]", \ __percpu_arg([var])) \ : [oval] "+a" (pco_old__), \ [var] "+m" (_var) \ @@ -279,16 +279,20 @@ do { \ #define raw_cpu_add_return_1(pcp, val) percpu_add_return_op(1, , pcp, val) #define raw_cpu_add_return_2(pcp, val) percpu_add_return_op(2, , pcp, val) #define raw_cpu_add_return_4(pcp, val) percpu_add_return_op(4, , pcp, val) -#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval) -#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval) -#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval) +#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval, "") #define this_cpu_add_return_1(pcp, val) percpu_add_return_op(1, volatile, pcp, val) #define this_cpu_add_return_2(pcp, val) percpu_add_return_op(2, volatile, pcp, val) #define this_cpu_add_return_4(pcp, val) percpu_add_return_op(4, volatile, pcp, val) -#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, "") + +#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, LOCK_PREFIX) #ifdef CONFIG_X86_CMPXCHG64 #define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \ @@ -319,16 +323,17 @@ do { \ #define raw_cpu_or_8(pcp, val) percpu_to_op(8, , "or", (pcp), val) #define raw_cpu_add_return_8(pcp, val) percpu_add_return_op(8, , pcp, val) #define raw_cpu_xchg_8(pcp, nval) raw_percpu_xchg_op(pcp, nval) -#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval) +#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval, "") -#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) -#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) -#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) -#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) -#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) -#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) -#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) -#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval) +#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) +#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) +#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) +#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) +#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) +#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) +#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, LOCK_PREFIX) /* * Pretty complex macro to generate cmpxchg16 instruction. The instruction From patchwork Mon Mar 20 18:03:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D11DC6FD1D for ; Mon, 20 Mar 2023 18:12:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 924EA6B0081; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 80FA06B0082; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 639256B0083; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 520FB6B0081 for ; Mon, 20 Mar 2023 14:12:14 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F417FC042E for ; Mon, 20 Mar 2023 18:12:13 +0000 (UTC) X-FDA: 80590070946.08.E5E8471 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 2A2FB4001B for ; Mon, 20 Mar 2023 18:12:12 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IlM+vW4d; spf=pass (imf27.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335932; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=kXDHBNZWD1x4YyqOA6ZVeranXWbQxiEjKgF55Tb+rLk=; b=nky2m2k3Hg6LTxFeytLlhL7hkM71a45mbNNW/ygIXfVR3yINoOsjFTLA7zpTSoCB2WHwen Xnn+bE3hGOK/1bGVJ5BX7ht0qMPzAZ/7LOz5XOUO6URP9svYBcqLcgx+WDhkuHC8JVQc3a Xno65cjQLnYEc1bJ5bZnZRyJWqZvqz8= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IlM+vW4d; spf=pass (imf27.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335932; a=rsa-sha256; cv=none; b=PK/fnmibeqNIWoNeyuwwwBTP8C/r3q7aaiuZHrPnY2FAvFL68+NbBzenwFhP1mRW7FvrOJ UOCQLulR8q+aQk9GUw1NZ3f0f+8Hn5GAkgarMx6039JuyRsev7F9C/WFQO5UPdYp1YfCVx U0H6pIft36F0ra7YkRBEhvzh+yxesHM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335931; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=kXDHBNZWD1x4YyqOA6ZVeranXWbQxiEjKgF55Tb+rLk=; b=IlM+vW4d5Jkcfk+15yi4bUt1PwL2wP7lSl6q/azvgVoHqfOykY6NMrg4FS0YefHGb2a0N+ YBsHCsk+36skJR+zufgKXrIdXNfYolL4QkiRtvdaU66AupgDnXY6pVbtD5+DUvlR2QzPu0 KN+1uE4F3965MICGGUydHqP7INekSo0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-197-ptS4IJK8Mf6jxxec7kdteg-1; Mon, 20 Mar 2023 14:12:06 -0400 X-MC-Unique: ptS4IJK8Mf6jxxec7kdteg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4982185C069; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1568F2166B29; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id E4E79403BC8B5; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180745.683772749@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:38 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 06/13] add this_cpu_cmpxchg_local and asm-generic definitions References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2A2FB4001B X-Stat-Signature: y3fmmuejazc9r8f5ezbn8xz3ur5q7wbx X-Rspam-User: X-HE-Tag: 1679335932-861772 X-HE-Meta: U2FsdGVkX1//UHmy4C40MTJZTnn7QM4Xis8LUTVYyNg59OV51V9ZIVBVIZAA7NKTMp0NHDaF8JD6LGxWrN1DHWbKfpvR4JTAdm1KqU7DWwZKlQJhe/AJxkixhoPQ//aqrMvxKRRbByT/a1ix+XXa2RQpyz+/XEyTrjiy7pJ9HQSvvveGKdkydZvgN3uU5CQvuAbmCrTeJVwoWDsntrhLA9yzfXBC6CPm5W7gUt2u0F/vLaoP+SWwAPVt2DgfoXte5dCt1TJWkNZGpvAsMuE8ycPl7vWxWhkBZZ6tPR8mCVBClXumpQxIg7ZM6bwA7Zt0hnJYMOgL0/7N/g+q0/1BCvIRQxKV0YPIH3ict0hfZQOKFn59UGOsOfPI5CKkszZpFsnRADCPZpZEARBcsPpBaQ1AmdVxU5owzuS6t9o9CeitybElWkw+LxIReztxei/PD8r6GySBYEyxST4pkKYYjik8xsjUNQXvdFyzqx0V/DPSq9HY8Ay4DtA+JYWIeqjSaum7LmffALVue3It2ZL6YJQS8UxYHPUeubNE33h1Pf5PfxUu9XB7Wz/IgNd5UMN+CisdYuwrESiwWEQ4J5igOlxPLlbNWQ0s2qw23JMrP4WzFWqJ++xTGkuLYx5SrHQNbJxUxfvH9EuXB9YDaohtQ7aU+0soS41eRkXP31AHqDgTeNtj0dKQB9XS8TeNvuYMtTRm117GVLicpJL/IJA4lnTXQ1d55RK3XBb05w/vrVA49vNVN0Z9DK0tx/2iVLkESEW5IOnhNIZBygQgISTGGjjmBqlW376IIvSHXaBR7Uh/HcRHTn1NjvVjmSBrFCldOfUvte2Ndhc+U9Z2JLBJJcXoRxjI60eSPhHE6U5fZb01P/kmSJhsNqup5HzUrcOnuq9R0SMo0AQCh+sa156YFaoAL8daus5yc1o1CdPtvZ1WUXC6dzQg7RfGa77FUOeAbUDbCS+LWbr9VKpUpo6 fFwlK6hq LpB/+ep5KwvdJ7zB1CRRBbHJeCkMktLyUo1/Sir/si0+60w0r8lnkHSwpuXvb27SOiDJ/i3uCCtZZqB49om4abfCD3fWyJY1ERwnHF9LnPwXgZRZsfyCevSzd21UpO/o516nNCUGIxxbY4izy3vElkhnVg+BsJY4mq1Wk655CrjskqyzagYa/Wbd8hVbpX4Pde8ftG3S1ZR86dYMG9WCV1pNENJkLTzePw03evbuY1DbQKlKO8q/EDnIXhQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Add this_cpu_cmpxchg_local_ helpers to asm-generic/percpu.h. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/asm-generic/percpu.h =================================================================== --- linux-vmstat-remote.orig/include/asm-generic/percpu.h +++ linux-vmstat-remote/include/asm-generic/percpu.h @@ -424,6 +424,23 @@ do { \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg_local_1 +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_2 +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_4 +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_8 +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif + #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) Index: linux-vmstat-remote/include/linux/percpu-defs.h =================================================================== --- linux-vmstat-remote.orig/include/linux/percpu-defs.h +++ linux-vmstat-remote/include/linux/percpu-defs.h @@ -513,6 +513,8 @@ do { \ #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) +#define this_cpu_cmpxchg_local(pcp, oval, nval) \ + __pcpu_size_call_return2(this_cpu_cmpxchg_local_, pcp, oval, nval) #define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) From patchwork Mon Mar 20 18:03:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16E5EC7618A for ; Mon, 20 Mar 2023 18:12:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C31116B0088; Mon, 20 Mar 2023 14:12:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF3B76B0089; Mon, 20 Mar 2023 14:12:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 946CE6B008A; Mon, 20 Mar 2023 14:12:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 82D2E6B0088 for ; Mon, 20 Mar 2023 14:12:17 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 617511C67A0 for ; Mon, 20 Mar 2023 18:12:17 +0000 (UTC) X-FDA: 80590071114.29.383D56E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id AC64F16001C for ; Mon, 20 Mar 2023 18:12:15 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MXN+sq0N; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335935; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=wO6Kd7Y8EK9tLPzGQ8Uve+plh0sOV79AJsB8xhd25oc=; b=y3lF9ndzHCqh0TrQF0BjeRZBuLo88eWazRzWfsxaF7JjPROhh/AVUWhQsKO6pjBN3oBuPe T8YMq/eDT1U44WjBjz4HqQqk1vZH2NNnJu31WQ5Wr+LrXCgSArNj8Huz23KBa9CEtTqZAw rE6AkR/G5ZBFcyte/nU56i84kNG1KeM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MXN+sq0N; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335935; a=rsa-sha256; cv=none; b=z+9xYAB7FHa6CFCUcMDwMvOdhIkbVbst0tWlR9/eBik79P8i9O6u7ifDaaYlwGNm+4psjc x4FwNUaGaa+aqJtbnfznzyw6bj9rk1dx2NEmpCyq+AsRykE958la7kGXtRymhW/GPvEi3s n0fso3cNXwMmNlupYHcy/S99TfRLaGo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335935; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=wO6Kd7Y8EK9tLPzGQ8Uve+plh0sOV79AJsB8xhd25oc=; b=MXN+sq0NTZEXSaWQAGELNTCjVSmU6ico/xAhnaYW7cUqWcIhKwYv1lZVo6LcZI7eAlZyr4 dqhd0k0DoGQM0j69oEkZT1qhk+PjKwMAQhmZoiarcJclj7x6OW/yqyIhKDXmcDRGSZneJ2 cOnZRIiz4QyfHnyEr3G3mVwRz7IfVPE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-437-tjxg_1SWOlK-KVT9IU19TA-1; Mon, 20 Mar 2023 14:12:09 -0400 X-MC-Unique: tjxg_1SWOlK-KVT9IU19TA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 814FC100DEAD; Mon, 20 Mar 2023 18:12:08 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A6E57483EC4; Mon, 20 Mar 2023 18:12:07 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id E93FF403BC8B7; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180745.709381791@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:39 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Peter Xu , Marcelo Tosatti Subject: [PATCH v7 07/13] convert this_cpu_cmpxchg users to this_cpu_cmpxchg_local References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AC64F16001C X-Stat-Signature: 5uea7km1u7zu6oasttk8jxxijh5go8y6 X-HE-Tag: 1679335935-129782 X-HE-Meta: U2FsdGVkX1+YtB5ZWTja5AaVHFHtX5GFw4VpskG0PRY7btedS9TjfwD4vxOvRjj9ZlcA/G5rQpLsPaXH3/RzLs/0x14q/KFwAOvLekR/gKnMPDKhePY0o26WcxRpGFuzrHqXx5mAs1BcsXIkrgdpBUCrG3/vkMyQMY+ydpwbUlhdwfKnCpSWC4IgwqzVRnR56lncvnYKQOkYlv8OY4yFyVGMror+TddySEYPOOoH26fCtTcfxQklvS/GDCVgimEyru/AHlI61Xf8+OZw8AHSChMkPEwXFrf8SyIdHaD7ozas04XNay5tSuB52icrRC5B9fWveA73LeENq8Zl+S6qcZfcoipUi03Ix4wqfkY3m5ifWVIuVIytY8fmYN6IczIEs3WEWH/m3fTa/6NM24BmKRdkmtBOSQmH1mShIJyZI3ERcy17FKVPbZ3sYL2w/s23PXXQ9+GImnQS771UysjtBBhkHIl/CgxWQo4/vae+jQR6OllWrTD7kMMV9vjcb3xnCqHcbEwSg4jUxCZ20WhuK7ZoQdqX11ZOo1gmpeVzzRVHcN8ts8bN70UHQYLcdMx+x+RDa4FAO2q0c2ile4LKDntFJ0RV3umhHJ7Q6DFzw8VsfjiY/yaPVU0/iEgZmN5kYUUinQ5gIB7egF/jUHgKtQr5zcetIpvbfqDS8dwi3wth19XwszrXJ116t34xShSFK1KAgbfdBj4gMFDSaE0kJK+y9SDsWSoHXMDcS/EC7MhNXK7dT5Rp6lnA/Umg8Itnuso/6j8U6FvyRa5EVsXFgRZinhjevBbIiqvs21tjhqBpvBXaUxG7xEwyGiHKfhvGwYfrtGBKHXK/rDB+kU1fUTNe5iJZgHXpLIzxy9pz8yu+BtqFDJX5eKAT/0u675rIGmZdPwV9lshy9XNblIhmpMd5ybzvxyrn9oFj9F65wcVrTbeX+4EzBGm+WvEdDlpGQZx0dZPH4gJ89tlYIXj dxGWbdDm Oc67el438l3b0Q9ABewgx2HoDwerr8i1Aqij00ya06L/Z2RdrzTWSYxaI01oVqkPNiS3PE/vJ/F8m7b7nyUqum4DugFQxVHUv3dTq/1kMYidk2Uxipj156GLLn6kJSf8vcmKi7mlL0QvzbHyVJnnyl2LmzCpF8Pmww6/8EPFfYuAvR7xiP99QwU7hmr6d3Ou2NdvTmR0dADwfJptvEaody1GsAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: this_cpu_cmpxchg was modified to atomic version, which can be more costly than non-atomic version. Switch users of this_cpu_cmpxchg to this_cpu_cmpxchg_local (which preserves pre-non-atomic this_cpu_cmpxchg behaviour). Acked-by: Peter Xu Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/kernel/fork.c =================================================================== --- linux-vmstat-remote.orig/kernel/fork.c +++ linux-vmstat-remote/kernel/fork.c @@ -203,7 +203,7 @@ static bool try_release_thread_stack_to_ unsigned int i; for (i = 0; i < NR_CACHED_STACKS; i++) { - if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm) != NULL) + if (this_cpu_cmpxchg_local(cached_stacks[i], NULL, vm) != NULL) continue; return true; } Index: linux-vmstat-remote/kernel/scs.c =================================================================== --- linux-vmstat-remote.orig/kernel/scs.c +++ linux-vmstat-remote/kernel/scs.c @@ -83,7 +83,7 @@ void scs_free(void *s) */ for (i = 0; i < NR_CACHED_SCS; i++) - if (this_cpu_cmpxchg(scs_cache[i], 0, s) == NULL) + if (this_cpu_cmpxchg_local(scs_cache[i], 0, s) == NULL) return; kasan_unpoison_vmalloc(s, SCS_SIZE, KASAN_VMALLOC_PROT_NORMAL); From patchwork Mon Mar 20 18:03:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA6E6C7619A for ; Mon, 20 Mar 2023 18:12:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 099406B007D; Mon, 20 Mar 2023 14:12:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3B906B0082; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACF5B6B007E; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 76D236B007D for ; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4C458140DBA for ; Mon, 20 Mar 2023 18:12:12 +0000 (UTC) X-FDA: 80590070904.24.626DCAE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 7BD8A1C002B for ; Mon, 20 Mar 2023 18:12:10 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OqR3bF03; spf=pass (imf21.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335930; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=BypmsFU6x2b2VeziY+86v0g1rjEMHDAZ3JB5mq4dcFA=; b=0otXOpV/hzoerXGrPxuGapk06X8KLC0KJbWgm/6IxlZAwQ7/GGn2H2mRriE0UO2KAj/APL SN4wD3VYiwCvr+gRONO8KqtrwRf0Chihg88icYHj535rg/AkVabMkS0rWuCMODugc/+U9i 9sTQd/xqWs0vKbvDx3Hi+aNPc61L85s= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OqR3bF03; spf=pass (imf21.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335930; a=rsa-sha256; cv=none; b=vAlXUgIHPmAmu3EZR/2mZAEHfAZIU0ibyWf+2sp1y0qhLPNLbqx2Pvt63ZY6CQucjLSFfj i4BnV0Oj/KmKQhZdKupEhkdUpa4AVLQ2L5tklc/XdelZBgKLRAm4M9s0jua99fYXszvAb9 sNexGPELWg+lr9V3NSUqbDbBXTW4myk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=BypmsFU6x2b2VeziY+86v0g1rjEMHDAZ3JB5mq4dcFA=; b=OqR3bF036vISR2DAyfzQE+/Cnsvneh/aA4j/VQNMx0M67f+SC6JgOwrFxyYx4ABmd34tml caGU2E0fuS/lIjOoN3+xskwkbx0e/Z1z0xnOBvBkwxQ+wN1GDXOQgNViINW2CUb4ubbF+I s23y77hCxwbuFJhb52iJHFGcJgdjluw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-493-_-EFyM3VMJu8xKJY8JnBKA-1; Mon, 20 Mar 2023 14:12:06 -0400 X-MC-Unique: _-EFyM3VMJu8xKJY8JnBKA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7A10129A9CB2; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 27A6418EC2; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id ED5DA403BC8BE; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180745.733575720@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:40 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 08/13] mm/vmstat: switch counter modification to cmpxchg References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7BD8A1C002B X-Stat-Signature: t3jazm9mw4bi5gjw956mfkkauorfu63g X-HE-Tag: 1679335930-115448 X-HE-Meta: U2FsdGVkX19rEqM2aO+lJFuAcrDqR2AaOj1k0omoICAU3c8UrNS5J8PFQyI/Ktwkd9EzkVvKC6DsSR1xZpYBPJ57GDt7NdAbT54n9rFyqHehK6/+Ft1S4bgreTO9soJnDKhD5mIwT9pNH0e7VKy78wR96i2aS+gcWhDNzoJpDytvZ4JUYU72PUoJD1Z9LI/EktfKufk2aJTOh6EL9uJn50jQHY9/0bsPkWI5hgs/VOm8p9IQqHLh0bQr6D+bSzJv/q9h5f6a1HOo0IyJloSE7DsrFfgLTwAjiGSx2E9r+TBvxgnzzOJQmjqN/+GUr6OQkfDqKCavxoLU+oGchigXIPz7uHkBJWAImYaSnblNk/KqkoheLJuRTU61nE7F1g1R3P/0wfDTM3vFwLSxn/i04OHPkVJmNG7nyLejXIptEuUTIW7pferZksnPyEOKS0BpWgEcHp/XNbeiKg6faj9VQtxPB4U2pIRtV90YI7zVLeN3zNJ31DU6szx7FFuddWxuJk8lyFBiwPUsVbzKCuFllV8bSezQbbxbtTIX+PzxOjj2QcPEubblVbKcw0arUKcTCCLG6xQVjSC2DboH3LkbNkayLLte/iC/MkNNMS6vfWodQ8Usl6oC8L+USWJNPC3rV3+OI1KB8gJkG28mtlVGBraubncnHizB4bfis46VIWbJorR+XAHj35g4k8oiHhH97s3DfXpD22OrcNfZ7IPqIKoaH8NPYC7LYFmhvcwxRMRr1D8T3DmfLjV1f9mMPGZmDMO4lOeYFnkRvf1ur1Q2wETKFjHCWvmzwI7PeCCmFS2sWYv1GJPD+0yHLfAubxtvkwWYQmzbNnkDhCfCHrA4lyZi/vzhrkxmY2pFhpAPP/yRf4OLKPZA5R8SoGn4Du4RhlkLI9LOLdi6QvUBDZXt5z5uCrKKtKpBAkB5HW+htMcZRKE6k+Fa7bMSrrf6nc4p/U5jMdXqbQUDWyc7y3+ HS3yiAnk QqaHlwIBIvrhY5r3NeJbtLYBGd3g2xQHh+lrTYZyN+E0A335Lc5xATt3wFD4EtJ3M7F35pLMq2FTBGBIA5SmpaOUuvl4CYpSfsOzF/Xfb+zpVI4THzXg0F6LARSwrSUlSh7xkQtganZHxqJuk8k+1mUmN//mV9Id8dP9yvabK2annl+xljppg+Wq5nEz290nr0/Ud6bsDUWafZFibD9xHazFDaJwWpcdwSeFI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for switching vmstat shepherd to flush per-CPU counters remotely, switch the __{mod,inc,dec} functions that modify the counters to use cmpxchg. To facilitate reviewing, functions are ordered in the text file, as: __{mod,inc,dec}_{zone,node}_page_state #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state #endif This patch defines the __ versions for the CONFIG_HAVE_CMPXCHG_LOCAL case to be their non-"__" counterparts: #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state = {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state #endif To test the performance difference, a page allocator microbenchmark: https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/bench/page_bench01.c with loops=1000000 was used, on Intel Core i7-11850H @ 2.50GHz. For the single_page_alloc_free test, which does /** Loop to measure **/ for (i = 0; i < rec->loops; i++) { my_page = alloc_page(gfp_mask); if (unlikely(my_page == NULL)) return 0; __free_page(my_page); } Unit is cycles. Vanilla Patched Diff 115.25 117 1.4% Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -334,6 +334,188 @@ void set_pgdat_percpu_threshold(pg_data_ } } +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* + * If we have cmpxchg_local support then we do not need to incur the overhead + * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. + * + * mod_state() modifies the zone counter state through atomic per cpu + * operations. + * + * Overstep mode specifies how overstep should handled: + * 0 No overstepping + * 1 Overstepping half of threshold + * -1 Overstepping minus half of threshold + */ +static inline void mod_zone_state(struct zone *zone, enum zone_stat_item item, + long delta, int overstep_mode) +{ + struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; + s8 __percpu *p = pcp->vm_stat_diff + item; + long o, n, t, z; + + do { + z = 0; /* overflow to zone counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a zone. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to zone counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + zone_page_state_add(z, zone, item); +} + +void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(mod_zone_page_state); + +void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(__mod_zone_page_state); + +void inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_zone_page_state); + +void __inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_zone_page_state); + +void dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_zone_page_state); + +void __dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_zone_page_state); + +static inline void mod_node_state(struct pglist_data *pgdat, + enum node_stat_item item, + int delta, int overstep_mode) +{ + struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; + s8 __percpu *p = pcp->vm_node_stat_diff + item; + long o, n, t, z; + + if (vmstat_item_in_bytes(item)) { + /* + * Only cgroups use subpage accounting right now; at + * the global level, these items still change in + * multiples of whole pages. Store them as pages + * internally to keep the per-cpu counters compact. + */ + VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); + delta >>= PAGE_SHIFT; + } + + do { + z = 0; /* overflow to node counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a node. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to node counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + node_page_state_add(z, pgdat, item); +} + +void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(mod_node_page_state); + +void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(__mod_node_page_state); + +void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +{ + mod_node_state(pgdat, item, 1, 1); +} + +void inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_node_page_state); + +void __inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_node_page_state); + +void dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_node_page_state); + +void __dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_node_page_state); +#else /* * For use when we know that interrupts are disabled, * or when we know that preemption is disabled and that @@ -541,149 +723,6 @@ void __dec_node_page_state(struct page * } EXPORT_SYMBOL(__dec_node_page_state); -#ifdef CONFIG_HAVE_CMPXCHG_LOCAL -/* - * If we have cmpxchg_local support then we do not need to incur the overhead - * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. - * - * mod_state() modifies the zone counter state through atomic per cpu - * operations. - * - * Overstep mode specifies how overstep should handled: - * 0 No overstepping - * 1 Overstepping half of threshold - * -1 Overstepping minus half of threshold -*/ -static inline void mod_zone_state(struct zone *zone, - enum zone_stat_item item, long delta, int overstep_mode) -{ - struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - long o, n, t, z; - - do { - z = 0; /* overflow to zone counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a zone. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to zone counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - zone_page_state_add(z, zone, item); -} - -void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, - long delta) -{ - mod_zone_state(zone, item, delta, 0); -} -EXPORT_SYMBOL(mod_zone_page_state); - -void inc_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_zone_page_state); - -void dec_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_zone_page_state); - -static inline void mod_node_state(struct pglist_data *pgdat, - enum node_stat_item item, int delta, int overstep_mode) -{ - struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - long o, n, t, z; - - if (vmstat_item_in_bytes(item)) { - /* - * Only cgroups use subpage accounting right now; at - * the global level, these items still change in - * multiples of whole pages. Store them as pages - * internally to keep the per-cpu counters compact. - */ - VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); - delta >>= PAGE_SHIFT; - } - - do { - z = 0; /* overflow to node counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a node. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to node counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - node_page_state_add(z, pgdat, item); -} - -void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, - long delta) -{ - mod_node_state(pgdat, item, delta, 0); -} -EXPORT_SYMBOL(mod_node_page_state); - -void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) -{ - mod_node_state(pgdat, item, 1, 1); -} - -void inc_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_node_page_state); - -void dec_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_node_page_state); -#else /* * Use interrupt disable to serialize counter updates */ Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -8628,9 +8628,6 @@ static int page_alloc_cpu_dead(unsigned /* * Zero the differential counters of the dead processor * so that the vm statistics are consistent. - * - * This is only okay since the processor is dead and cannot - * race with what we are doing. */ cpu_vm_stats_fold(cpu); From patchwork Mon Mar 20 18:03:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C433C6FD1D for ; Mon, 20 Mar 2023 18:12:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0949F6B0078; Mon, 20 Mar 2023 14:12:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0441C6B007D; Mon, 20 Mar 2023 14:12:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E4E3E6B007E; Mon, 20 Mar 2023 14:12:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D22736B0078 for ; Mon, 20 Mar 2023 14:12:52 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A4C8AA0E08 for ; Mon, 20 Mar 2023 18:12:52 +0000 (UTC) X-FDA: 80590072584.18.C1F5DED Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id AF71180025 for ; Mon, 20 Mar 2023 18:12:50 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gTubR6DW; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335970; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=ZUaishK9kOHZ/QZsdg2n57rQlJ8J1fXXfaJxo7fVhG4=; b=loIeA1fJsiLQBp4lFguk0k45Id1TM5phCI1FB3MGFNyqgnVXoVo4Rgqky0NYz11vDbJDWu 6+8qtqe1SdurMOuYuXdmW0NMqEl/XC4WTDvLxN7VSU1AhFNqLQUR/FyNwEeooGFc5atbMX hSpYopqvmD7gsTXLaRONz9vYLbOYfVw= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gTubR6DW; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335970; a=rsa-sha256; cv=none; b=DvIyF6Xll80GOgcN7RAuSnkvFCC9SV7VWx8qaGoLS2xsvqJP0Zk4VkA70EE/2kkIhQVC0l ENenw6nV3I9M7dZ9lnAeuSZxwKkh3ftRMwG5DkBC6JiaD9RBZrOOWvgcgo5iJuS+h2qDGm KClrBlzOR4ZjBtaVKzGY69QdCRLCWZw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335969; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=ZUaishK9kOHZ/QZsdg2n57rQlJ8J1fXXfaJxo7fVhG4=; b=gTubR6DWffMN3aLEyp/g8lTbi8ZOJ9Idj4UU/uXGLrM30n3eZUyaXdk+3MFbFuXgOBfwGs W2F8rD+IZQXbZ3V4NCxQd12G/H1DfpPZ5WcsHjMDv3ujMDcR8LfDoqLKwxg4zBIeUSWPPc ask8G4lM9mvsYmg/vvihC63YRBAkA0I= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-494-WBTr80T6M62Z_oUDSgv-RA-1; Mon, 20 Mar 2023 14:12:29 -0400 X-MC-Unique: WBTr80T6M62Z_oUDSgv-RA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AFFC3800045; Mon, 20 Mar 2023 18:12:08 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A9D091121315; Mon, 20 Mar 2023 18:12:07 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id F12CE403BCD80; Mon, 20 Mar 2023 15:08:02 -0300 (-03) Message-ID: <20230320180745.758267946@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:41 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 09/13] vmstat: switch per-cpu vmstat counters to 32-bits References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: AF71180025 X-Stat-Signature: k3m3csguw3fhzat1nrcb4huthxrwdbkq X-HE-Tag: 1679335970-963833 X-HE-Meta: U2FsdGVkX19KoYKOIT9C8PJ0f8mbFm2Tj59g0WcQVEW57PCgU4/KeDonA97mOEfXMmbNF2qWcTcjLkoAO1ElSBADrr6y+9oqdbLsxp0RQluAPGPmdCcfilm1DtoP6ZdnUEe5vWsx/sOTSnTZqB67pE7n4OvlUDA0Oi1HXt65rso8Br5Ak6JrHjeGdaK7ZWQWI/Exhqo88/XcS1On4hDdxAfF/JOKzSXQCLY0q89fstWq3kLhf2V/LHSVnE8lCuLvTLBmtYnonbTUiiNaUO9l9QBy/Y173n7c/c0+oPbOqa/8xH7BVT8ivfX85IxqV3mj03QVZjtvOhAlyZnl0z+QbF85BV09RYT3lIehQSML5prIk6oxqwvUkKmE00Qo/iOeWx1AoKChRPXJ0SRg8TxI6D2+io3ZshCIQUN5rAoGrPUVT6494W5vrm15GfxldccIYTNXHW10c1DFIo8X9SUXD7Y/eJyFpE5e8od9Albd02LOH796pmGHYRWoNvq4DnUfCzFZIFE5tHt7UV5h6RWkDbI5lCq1Av+5i2jsJa7QiHHKc5YyUG03rOcR0/rhkJzp5Y/mns7pP8Vf8ziIu9iB7ki06aOAXnCUi6PmibomhrsFIoFQ9YsoO4weU59r+MSHp1R9nAmrts9+UJ6SEDPgmXW3maYlsZtSOxkiIULMyENHzwHJz1qOoGyMtxe+3eOLmsBIPOvUhnBCsoF7zJoCC3hv7fUtf+SaL2agxvo30+xogyLrN/Z3uqT+5IZEBvw4kSxvieFOpqVsFPA52UAUAd/167viioFXLkCOkSgXyPmiw21hDJeywYN+rr2Wl78c2Kiy1WfXISHEqbizrC3U55iYSfnGSMf+l8KV8iqce7YFAzaqQlLCzx2S3RLDu5owff1qgQsy+o1ia3wRBTe1WQEKd8+u/0udrZ8mn3E0Oc325ekTV35w+eYPYEwx/exjHmNXp/RsrKAZ/10inUK 9IBl+0o+ j8bll4XBLuYj9rCUEIccjAtcs3bs4ZlF+OQ6nwJPgfjzHSKfzWgi2ldryLWSr0Sx1h5SGCLxhf6YcpFG0rU9TdVc1bLk/16ETQfWMeYreNcYZqYUlTzOTWwTeTR8J7JbNsyf5BBhtc9QiBCwOnUS4HPX7XhtUHXbnueMJn2Kw6XY+nstxZMT/lzXfVa0Jx+M7KIDvTNcvIQhl1C8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures only provide xchg/cmpxchg in 32/64-bit quantities. Since the next patch is about to use xchg on per-CPU vmstat counters, switch them to s32. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/linux/mmzone.h =================================================================== --- linux-vmstat-remote.orig/include/linux/mmzone.h +++ linux-vmstat-remote/include/linux/mmzone.h @@ -689,8 +689,8 @@ struct per_cpu_pages { struct per_cpu_zonestat { #ifdef CONFIG_SMP - s8 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; - s8 stat_threshold; + s32 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; + s32 stat_threshold; #endif #ifdef CONFIG_NUMA /* @@ -703,8 +703,8 @@ struct per_cpu_zonestat { }; struct per_cpu_nodestat { - s8 stat_threshold; - s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; + s32 stat_threshold; + s32 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; }; #endif /* !__GENERATING_BOUNDS.H */ Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -351,7 +351,7 @@ static inline void mod_zone_state(struct long delta, int overstep_mode) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long o, n, t, z; do { @@ -428,7 +428,7 @@ static inline void mod_node_state(struct int delta, int overstep_mode) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long o, n, t, z; if (vmstat_item_in_bytes(item)) { @@ -525,7 +525,7 @@ void __mod_zone_page_state(struct zone * long delta) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long x; long t; @@ -556,7 +556,7 @@ void __mod_node_page_state(struct pglist long delta) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long x; long t; @@ -614,8 +614,8 @@ EXPORT_SYMBOL(__mod_node_page_state); void __inc_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -623,7 +623,7 @@ void __inc_zone_state(struct zone *zone, v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v + overstep, zone, item); __this_cpu_write(*p, -overstep); @@ -635,8 +635,8 @@ void __inc_zone_state(struct zone *zone, void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -646,7 +646,7 @@ void __inc_node_state(struct pglist_data v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v + overstep, pgdat, item); __this_cpu_write(*p, -overstep); @@ -670,8 +670,8 @@ EXPORT_SYMBOL(__inc_node_page_state); void __dec_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -679,7 +679,7 @@ void __dec_zone_state(struct zone *zone, v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v - overstep, zone, item); __this_cpu_write(*p, overstep); @@ -691,8 +691,8 @@ void __dec_zone_state(struct zone *zone, void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -702,7 +702,7 @@ void __dec_node_state(struct pglist_data v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v - overstep, pgdat, item); __this_cpu_write(*p, overstep); From patchwork Mon Mar 20 18:03:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07D4DC76195 for ; Mon, 20 Mar 2023 18:12:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D33A6B0078; Mon, 20 Mar 2023 14:12:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 882C66B007B; Mon, 20 Mar 2023 14:12:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74B176B007D; Mon, 20 Mar 2023 14:12:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 65B686B0078 for ; Mon, 20 Mar 2023 14:12:11 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 34CFA1C5B94 for ; Mon, 20 Mar 2023 18:12:11 +0000 (UTC) X-FDA: 80590070862.07.9A3885F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 70027A000B for ; Mon, 20 Mar 2023 18:12:08 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Mp3A37UD; spf=pass (imf15.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335928; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=Hcv+7yXWUE6dz6IV3pdXnHOeNZqOsphpnY/hyL0+XpA=; b=K/gJ02M07gMt3fWqZLDBgUDS3grs+h+PjLmwiu9lRXG/gGchbX6TExMgRVjb0IdZIKoxj+ sAf8WiN0CGlmrGyExINvCoSPNOXbnPQJrXZV98rHRT3yhh8XCaH4ZKQl7y4JbbxJIk4ojD IFsehHYxBr9Ycp/xG1EMg2rbEyzcxnY= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Mp3A37UD; spf=pass (imf15.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335928; a=rsa-sha256; cv=none; b=I9PHzJYyMGT/lhpEs3vOFZXqyamTsfpdWxBtU6iBSyAJvPZN+DL5GHmpHSL5ppoOd3zado VTeKB59Z4UcT+xvnSwb3xYDhsxHwsRQMZek2fqo2OhAEMZW1mA2uiIhskK2rN+xm+aaahG OzC0b43V/DDW0nAW1OFrDDJBRa9rV74= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Hcv+7yXWUE6dz6IV3pdXnHOeNZqOsphpnY/hyL0+XpA=; b=Mp3A37UDIH34B5E06Vba5VUikhPvTao2jlZ0bB6NPYtrDmF8We9SPTBi+ku3mTHJedIABL 58hAEOofRS3YJACqYi8oLZZKJXD6jQBI+5lO0D8VFUXzBu8Qv7oc/MN9L53jAVH6icfXIm Im2pcqo5gxBdw2e+d3AKg3v6v02mKdA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-607-5Z2d1MOjMb2V2UyjiOPjmg-1; Mon, 20 Mar 2023 14:12:04 -0400 X-MC-Unique: 5Z2d1MOjMb2V2UyjiOPjmg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7273338173CB; Mon, 20 Mar 2023 18:12:03 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2094F18EC2; Mon, 20 Mar 2023 18:12:03 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 0360D403BCD84; Mon, 20 Mar 2023 15:08:03 -0300 (-03) Message-ID: <20230320180745.783045641@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:42 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 10/13] mm/vmstat: use xchg in cpu_vm_stats_fold References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 70027A000B X-Rspam-User: X-Stat-Signature: o441j1nofjwpbg1hw7xn5xouegjxo48t X-HE-Tag: 1679335928-227600 X-HE-Meta: U2FsdGVkX18OP3hFL1BgGpl+kRyVuRNMtNQ7+IHQEQ5Vwa689NZb50neQzLNUwtdjqvu5QQNdNfBhzE1680acRNpEMzi+Y/KJZPqSurLCKEHyMUcKOSKScWC8U6j8NNurKIp2YprXajvGIESo8m/WU2DKfF3ezPyFR6lLaeBCA6rDjjgWex8AffnIQl6Eyo7UDTlyLuRNaxWseKgDOCv9UWswI+QYfQ2mqafHBPAHDsm7SVWIRzSXcJcMptS6S/DIe7sKjFbMSuitjiD3o85GjKHa/BeMDnyDKUSXPWnNEG9YYoHmtOULka90ZFVv7V0oREgS4Y/fpugpcGNJnmZ5jz+k4iNu4gNb2QNOt01vbFulZzbuTl1GUVqmn+IN51XTB44iU6nhzOlWDMnscZL2x5eYKQ1W7qiQg/vbgw7jpU3vXT+eNfTXSlzz8zhSnoQzFZ0lQi4Q8XTkTUQ8UVs7lsu/IQIqlAnERZ0Y/uDJteAuzASxenOVo6xPkx8SqGBRxF9wRk/lTdZD9n1SstvwDpiV8panx7vB8uMKPAEPcHVTwsFQq8ewS1xuQkrAkxtL7s2bTHtrMiDw97m2/U7QVFIm0O7WGLgD8iAUC9OHg2NtZ5FU43QKW2VSn6VcWsc7xv6rsQuhNA+QI+t8yyiBUb5lq2ILh1scM0v7+C8pCvJwI3IcdT5r4PrJtQOemnf/k8cpHtZIxz2OERbsXXs2sqfLZ8AL1HXBr9HoHUK5DGGy6Fz5WD63PG2XCTBOjbE0zlG1smi+yms0moNsXu+T1c2Yywq2aD2WsgJvBTrrB4htSZgy8dOgZ0LeNriDhIPewV/GYH9srhr/JppSXLiZsRZNi2aOL+id3rb5kVbjQ5svq2bXmtPTaKnKJtcZVtlvEE09ukZrZoVjci0AFUPkA3HX9zORe3ZGMc+9px5QXJDeIuQgTiK/CeW6Ch1olGJxU9nQbpvbh/6bN657N2 7N8j7Z1P VMLXrJ15KMA+Y4svJiXAqNUaCLZe5XflFUCQ+4Lj3rT2AnkMAo693fB0ka6C527pto3Kol0LWKvUa/TztUmI2+/pRbg42F/kH1DXb9m5l9+h4J+a+l14mSgAyzrFo03EJ0rMb4L0Gld4IhbpE4YoaSzXJVnPABxE5Gy6kPowD7eXowJSLddpuTCkPh1YPNyc8fvEsjc4AMWLvQpRtNIT/rHpKUL0rioxvkooLqMD0vUr+hdSDoM5XfkXxS7vS0RPYmow+u8ezDKLTgd37ej4aJmlmQg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation to switch vmstat shepherd to flush per-CPU counters remotely, use xchg instead of a pair of read/write instructions. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -924,7 +924,7 @@ static int refresh_cpu_vm_stats(bool do_ } /* - * Fold the data for an offline cpu into the global array. + * Fold the data for a cpu into the global array. * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ @@ -945,8 +945,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_stat_diff[i]) { int v; - v = pzstats->vm_stat_diff[i]; - pzstats->vm_stat_diff[i] = 0; + v = xchg(&pzstats->vm_stat_diff[i], 0); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; } @@ -956,8 +955,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_numa_event[i]) { unsigned long v; - v = pzstats->vm_numa_event[i]; - pzstats->vm_numa_event[i] = 0; + v = xchg(&pzstats->vm_numa_event[i], 0); zone_numa_event_add(v, zone, i); } } @@ -973,8 +971,7 @@ void cpu_vm_stats_fold(int cpu) if (p->vm_node_stat_diff[i]) { int v; - v = p->vm_node_stat_diff[i]; - p->vm_node_stat_diff[i] = 0; + v = xchg(&p->vm_node_stat_diff[i], 0); atomic_long_add(v, &pgdat->vm_stat[i]); global_node_diff[i] += v; } From patchwork Mon Mar 20 18:03:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BF82C7618A for ; Mon, 20 Mar 2023 18:12:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32E366B007E; Mon, 20 Mar 2023 14:12:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2497B6B0083; Mon, 20 Mar 2023 14:12:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7EB36B0080; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 996406B0082 for ; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4D784AB54F for ; Mon, 20 Mar 2023 18:12:12 +0000 (UTC) X-FDA: 80590070904.26.6DB8FDB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 7A3DEC001D for ; Mon, 20 Mar 2023 18:12:10 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UcGYbldq; spf=pass (imf22.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335930; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=BeFRTjFm0L142WsTkcDAje75c4PixN4nDF8ylAdfn+w=; b=DHvYq2ZVkMVc6Q745IbGS/KpJes7rqsc/0glOim41Z7DY14RhNZ8jPBpODWSPxb9FwCcGD /Wrc7x8SF5cuztHOSNYb3Pe27lm4Y2TG16YKFgejy0H2qR6SUbadkpKsNx1R+2Be0G3fwr wVHBcrhYs5KvpaBLYUkQZhVL6MpnS/I= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UcGYbldq; spf=pass (imf22.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335930; a=rsa-sha256; cv=none; b=HzsfuP7E/YJux2RM6IZz0WAwQhZ95RBpPueEH/39RbFeX3dd6DJ0789gvbtAjh++7BpavI FcMAKIbCPgu3vy0TZD6PfHDtlETsBzdTXSyABxG0J6RYQNXpGKiH/gs7LvJG4pV1l1aj8M /0a7tSlX0CndAlRSRKXzYx4k6SDkLfQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=BeFRTjFm0L142WsTkcDAje75c4PixN4nDF8ylAdfn+w=; b=UcGYbldqAuFvX++rBi3llAZk6L7oMwaMQ0saw3Vqc2ChD/bnh2qCrJZn1W6qtzy1OS45Yv xXumn5fWUM1X8eOOBlysGkbospqGBn8nNvkxEz6C8lmimv7ES9R20SXr9NWL6cnwV4nyNo ZiDMGfVPFBVDq8oA27uoAexvrfV4DLc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-446-UAD0kCRHMwmtXQvAe-NHhA-1; Mon, 20 Mar 2023 14:12:04 -0400 X-MC-Unique: UAD0kCRHMwmtXQvAe-NHhA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 72F5185C06E; Mon, 20 Mar 2023 18:12:03 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 20B4A140EBF4; Mon, 20 Mar 2023 18:12:03 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 092FE403BCD87; Mon, 20 Mar 2023 15:08:03 -0300 (-03) Message-ID: <20230320180745.807656081@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:43 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 11/13] mm/vmstat: switch vmstat shepherd to flush per-CPU counters remotely References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Stat-Signature: rifuhcxmxd9hwewdhkiwmyciwo75tkbr X-Rspam-User: X-Rspamd-Queue-Id: 7A3DEC001D X-Rspamd-Server: rspam06 X-HE-Tag: 1679335930-27922 X-HE-Meta: U2FsdGVkX19hKylqOHdMUYc5HO3i60E/vLpJTnLQjixsQRtS8InQ7nYXTDXWtL/Kdl1Dvn5L+lql7gTgtHLyuP3fAoX10MPaOMOOGlPcInrW8CjxvOYCd6rgjDF9Bsob+dl0em5YSzZc+csjn7fHfzdOqdQtSy1fTMvpe8Pg8P/fmc6cUl7bAF2Ofx4X6oJnWNz6pgzPlmLYAmw4YiiMzS52gYWDwH1LpVpJNeKVp6k6Gnim4QuI7HZl8m50AZrdGgj6FqzpxaaL5sYJuZDWXh/GP2MU8hTTuShkzlIGa/yHoUVVcQAeWIMb0I3j9qSTqt0yXxa2g710F4shNJq5wlTFlQ+zREfGCjamVPkAeOM7bSjFz7DPq+DqPioqy1SwA3sgZrLWDTzLigq/Hy7BCI7MnKzHl/yXdl2+iUv83rrDTZIEJzmdG4+kv9YPs7fEZSmQDyWp3xFSW1S6WBYgGa1Fe4ezpbaipeeBbqp1MVpu5Gwja08Ik+kzB3MPunf3RMTQtBKjZ6iw3KcRBZgcmbGemXPfXZvvQIpUxo7NgAzN0xOEE/vifhEwlA/ieMleGepaqaD7WsOv5ABSG/46wuox/VQTzTw4LOvkfFMHw/w4s8YPTFhDp7fDLvHTKkH2oChxgPcuFDzjJRQoM0jF9NbRFxcLwZ5+7fXFSS6RAhCJSJID9VNhcmWt7vYKzhkWFAtS9cJNNaglCR9AihsS750/5zKmWuEukS6FphnbIoAA9/yOwuHuq1FnJ8WAAqXyYyOjrdbD/3AyrUgKlvwueDcEMjr6FRpUe8bsjxwSEAHRP9Km8Iq17n/LBati3+bhj3CIKrWHzuEcHXY9CVBVM4uKzeHrFxdqiADPt1Xi5wQFODIU9M2O9tuQzrdFt7S6Xz4MpxkImRhVas/OBUK0sZIxnsbgwsXQyvJyk8jjVLbz+2+UFRCvC600CjKg/uOSTxXapH5knxfPXzdKvkr St7NMkSS ffcVKBHJBD5E1h3Eo2gak8QeIXUWr6J88Fy4AcUMyko0pEsYksEUFOHTPbkKTQISsBk4CP93ugFdyDlAw44Tq+vlbhxvnKqUrfHgAsBZ39j1rcWZuXXndmcb/bDnQi75FgSfBhkKQS4HpJL8NeHA2Q3Tlc7Bk8OHbz8PaYw4aR2xwbTYnrnKJ9HWrL7Pr+EZlBt9kb0euWEO0+7I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With a task that busy loops on a given CPU, the kworker interruption to execute vmstat_update is undesired and may exceed latency thresholds for certain applications. Performance details for the kworker interruption: oslat 1094.456862: sys_mlock(start: 7f7ed0000b60, len: 1000) oslat 1094.456971: workqueue_queue_work: ... function=vmstat_update ... oslat 1094.456974: sched_switch: prev_comm=oslat ... ==> next_comm=kworker/5:1 ... kworker 1094.456978: sched_switch: prev_comm=kworker/5:1 ==> next_comm=oslat ... The example above shows an additional 7us for the oslat -> kworker -> oslat switches. In the case of a virtualized CPU, and the vmstat_update interruption in the host (of a qemu-kvm vcpu), the latency penalty observed in the guest is higher than 50us, violating the acceptable latency threshold for certain applications. To fix this, now that the counters are modified via cmpxchg both CPU locally (via the account functions), and remotely (via cpu_vm_stats_fold), its possible to switch vmstat_shepherd to perform the per-CPU vmstats folding remotely. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -2043,6 +2043,23 @@ static void vmstat_shepherd(struct work_ static DECLARE_DEFERRABLE_WORK(shepherd, vmstat_shepherd); +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ +static void vmstat_shepherd(struct work_struct *w) +{ + int cpu; + + cpus_read_lock(); + for_each_online_cpu(cpu) { + cpu_vm_stats_fold(cpu); + cond_resched(); + } + cpus_read_unlock(); + + schedule_delayed_work(&shepherd, + round_jiffies_relative(sysctl_stat_interval)); +} +#else static void vmstat_shepherd(struct work_struct *w) { int cpu; @@ -2062,6 +2079,7 @@ static void vmstat_shepherd(struct work_ schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } +#endif static void __init start_shepherd_timer(void) { From patchwork Mon Mar 20 18:03:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6487C76195 for ; Mon, 20 Mar 2023 18:12:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 63CBA6B0080; Mon, 20 Mar 2023 14:12:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A7DA6B0082; Mon, 20 Mar 2023 14:12:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE2736B0081; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9D9686B0083 for ; Mon, 20 Mar 2023 14:12:12 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 52E9F120D47 for ; Mon, 20 Mar 2023 18:12:12 +0000 (UTC) X-FDA: 80590070904.25.485CF2B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 821621A001B for ; Mon, 20 Mar 2023 18:12:10 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V66n9QQg; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335930; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=s0eL+muEcWwaDlYOvrz0uKI6cYwztUvQb2dBAOG0XGM=; b=63oA+xyB7NEAyFKKzvsGT8zJ2um8eAjJaWrSegO8rTQ8Ye8HsqPVHUM4vbZTnzbR0xFSQG LtMA4u17YPYPG1B0DYHgP3SqQIQIKte2Lmf0U0k8JbIb8i8zZxfjSIVZ1HfH3T0CzaYW6m malvnBCyM1VgdZNNyo3ZmOyMGRosrgM= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V66n9QQg; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335930; a=rsa-sha256; cv=none; b=Q+ja+E07FeKi3BwTWGUB78sWqcYd4n2PiJmmPC1P7Kwn+i32ue6TejUbRTmnVUYgBAQhN3 Jt/9cfYAW+P0sO6CvZeyrAh6LEJKPvwhHxlaCfMnRraNzs1nBj0gB3KDGKwunT17YflrHW GuNBd8DVWLJs6ReJpKqLPhSr1gx1lNI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=s0eL+muEcWwaDlYOvrz0uKI6cYwztUvQb2dBAOG0XGM=; b=V66n9QQgzbB/UMw5q2jKrlD8dLRSaiEiXGB6UPIgQIpOKEbhvwixCKe+lZXCVDpeWkjMJc G0U4C5SROngooCmNu3K7mAjZsLLRcTY2QpeXdrVCBDA27X8SFCGKMbULD7h/qa50HBpc5v PTl4tAHFpDr8vxpHimmzq6XAGk5Ayvk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-488-GdUVvJhgPDKBvQym1NCNbg-1; Mon, 20 Mar 2023 14:12:04 -0400 X-MC-Unique: GdUVvJhgPDKBvQym1NCNbg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 88568855311; Mon, 20 Mar 2023 18:12:03 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5267A2027062; Mon, 20 Mar 2023 18:12:03 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 0EF05403BCD8B; Mon, 20 Mar 2023 15:08:03 -0300 (-03) Message-ID: <20230320180745.832991792@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:44 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 12/13] mm/vmstat: refresh stats remotely instead of via work item References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Stat-Signature: g98xq33cwm1ramwf39q18ju7goj3wxis X-Rspam-User: X-Rspamd-Queue-Id: 821621A001B X-Rspamd-Server: rspam06 X-HE-Tag: 1679335930-7991 X-HE-Meta: U2FsdGVkX1+HbzAFukaYZ7yADc4IHpKyv+yAMzrs+8BIOySjL1pCKGLxe52NGT3jlb9GUBr3rhdXEV6GRxc3k/ogXdKr2rt8FW3SIE43hKsvK03Tt2GeyJ5ZwDDJQBDUBfhY/r0Mgf++BIjiXvynS+J1/zLhb4mRKGw7Z54rGN4z64suM0xSiXY5Bt8zGOUoiaY2c3BOr+BXFT6dq3lB9e46L+rwn9J5NUlYlSIR+wBCo46IXmoJ1ExAR1+GYj3/eEVAiVD1cTxsj1q5Syy4lbwSt0YuHninFa9ItDGnABPWkNbBYVd3/n8j7UPnoKqhbSFS8tDV1bLAsEUb9iXxdJ6v9//MXtTWdFlEiIKb6l87RGgLaWmFKQQlGu18GMHvMw5ByKktqUnIqDOJzAMI3ei6VEk92Q1UnYvQdnzEcoc7sJu4kW+5EruYD5rK4SRiLEl5ZPdUXYemu5AEpD/hR7MECc38L4gmOP3N/OGTcEcq9VZ5XDZUt5++W85pmSknYo+++XAvFEgdIY95W2wet2KqyZAp1lC4oLONG3I7DhUItXLPC4CIWWQ8LO58E3t+uVwzjtiWxEW3cEBqzOz/Wjrx3kvvGeUsX0hcpCwveEooPZ5Lfo3noNMYxGyJYqwr/Z72HCuBjL4QJxzCa9Q84+wF/j9nRCdhu4KcsW/P6/Iw7y2U4Cayt1KIMH7tMr7FZE4eoDwpdrxdNO1QBCCEYb6D7koA6rjxD5zQVTor6muoB53EV9QNCWqnkkk5CmkXsgJ6Jkil4DWdAxI6OXLle8rv/VOBRtC2hre2WotpRhtnZeGq3s23Egr4Y6B1/Scoc38qNpBtYBSQSEu4swTdpAAP+2f3dmBh+ex7jdBNEg3127JdaTL0mENDwFD1FRwOwxJR7wj17oE1ligp2cNRJamp0Wak5mf6T2zHYvRUZlPtwraf8GaOGV65xKJ9GI6i1QJRKzqR7qtoo56l7gS UjPAS97R 1hQFasApMzg7dW0rONTHHXqI8dAc1y1bkmyGHywKzMv3J2rfpfbOLTQoAj0om9+x+Ey4NL+UcOCJhaEZQ+oPMBISkOlkyAM2DOWen5JKXpMz+mq42T9mmfj3RP/jV5QXS3BopFmAOETVep61kA4fm8u8eu4tk0ys8HsvJBWU4uXtR6Zu/RRZa5biBqlcy8qemAgZUvi0vOkRBXzk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Refresh per-CPU stats remotely, instead of queueing work items, for the stat_refresh procfs method. This fixes sosreport hang (which uses vmstat_refresh) with spinning SCHED_FIFO process. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -1901,11 +1901,20 @@ static DEFINE_PER_CPU(struct delayed_wor int sysctl_stat_interval __read_mostly = HZ; #ifdef CONFIG_PROC_FS +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +static int refresh_all_vm_stats(void); +#else static void refresh_vm_stats(struct work_struct *work) { refresh_cpu_vm_stats(true); } +static int refresh_all_vm_stats(void) +{ + return schedule_on_each_cpu(refresh_vm_stats); +} +#endif + int vmstat_refresh(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { @@ -1925,7 +1934,7 @@ int vmstat_refresh(struct ctl_table *tab * transiently negative values, report an error here if any of * the stats is negative, so we know to go looking for imbalance. */ - err = schedule_on_each_cpu(refresh_vm_stats); + err = refresh_all_vm_stats(); if (err) return err; for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { @@ -2045,7 +2054,7 @@ static DECLARE_DEFERRABLE_WORK(shepherd, #ifdef CONFIG_HAVE_CMPXCHG_LOCAL /* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ -static void vmstat_shepherd(struct work_struct *w) +static int refresh_all_vm_stats(void) { int cpu; @@ -2055,7 +2064,12 @@ static void vmstat_shepherd(struct work_ cond_resched(); } cpus_read_unlock(); + return 0; +} +static void vmstat_shepherd(struct work_struct *w) +{ + refresh_all_vm_stats(); schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } From patchwork Mon Mar 20 18:03:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13181697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62070C7619A for ; Mon, 20 Mar 2023 18:12:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8AC206B0074; Mon, 20 Mar 2023 14:12:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 810646B0089; Mon, 20 Mar 2023 14:12:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 613F06B008A; Mon, 20 Mar 2023 14:12:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4EDF26B0074 for ; Mon, 20 Mar 2023 14:12:22 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 27E9D402F3 for ; Mon, 20 Mar 2023 18:12:22 +0000 (UTC) X-FDA: 80590071324.14.895C6D9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 4441680015 for ; Mon, 20 Mar 2023 18:12:19 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ftA578Ru; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679335939; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=rZ4RHjC6A3UOUm+kzsUNThO0UFp9uz8rJmefi57nqXs=; b=clJ2n+Iyfas5xaVQnQymjUgFnyR2qt6Q92wpBPsyeo5zM4d4RGc+d9lEaRyGbMeyC4QT1o cmocl/7GRSAZq09VI4aqo/xn6T27njJR5SsJODmXYGcAX+rt0RjlG6wFy1/roruAKE51cV AJ5DTixpEqEEz0yw3tf4sl3RNG/9apc= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ftA578Ru; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679335939; a=rsa-sha256; cv=none; b=nyayXuOg/ViIiNQgVuc+wsyMW/mYK9Np0Tp5CRDmt1Ut8DL6nnXwSKqoBstihIBXYA/lj7 MCp/QzETvK8OhWRogPm8ifl8C0LTIPZ6DvMnSAr4SL296oUesrUjsiCFbRZlb3gsm7p5T1 a6DnZR+YAgzYB7tRAynnQWZ0M4CHYis= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679335938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=rZ4RHjC6A3UOUm+kzsUNThO0UFp9uz8rJmefi57nqXs=; b=ftA578RuZntSRybmvU8n0CP6yHuBpR2vJOHs/Dv7qAav3/7Cb3W8i3LKbI8kwUqsXUOEGY XAjhkHXJ9ZnLqg1vDx9oIQPA5/ydzNZSTswDrmkMSYmRkAocVEjiQl8lUADjGxZL4ntJhp P+hwJCisIFIcRDsxBQno5n9y+1FQz58= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-228-8t2wdVViOE22-Eq68lpdGQ-1; Mon, 20 Mar 2023 14:12:07 -0400 X-MC-Unique: 8t2wdVViOE22-Eq68lpdGQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2D4B7185A78F; Mon, 20 Mar 2023 18:12:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F2979492C13; Mon, 20 Mar 2023 18:12:04 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 13EFE403BCD8C; Mon, 20 Mar 2023 15:08:03 -0300 (-03) Message-ID: <20230320180745.858515310@redhat.com> User-Agent: quilt/0.67 Date: Mon, 20 Mar 2023 15:03:45 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v7 13/13] vmstat: add pcp remote node draining via cpu_vm_stats_fold References: <20230320180332.102837832@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 4441680015 X-Stat-Signature: i57goasdys65ubxqo5ntcfk7hby45x8k X-Rspam-User: X-HE-Tag: 1679335939-540216 X-HE-Meta: U2FsdGVkX19+PzHPaVA3pjOC71EAmTijJxf3bOkuhf/xB6CYyhph7/SORrp8rpMUrGiwxatqspMUB74h4nd6KtdlZN9Id08+ba9FlnkUk3TwcKtHdxTvCtIfUTBKFYwTK7Zm3QYkGNAyyS8qBClZx9XVIYoMgr1v6NBqt5/Z7iZrmQENYmqQrPfBrqCMbQp+1oFT1J0TAvDbbrrjuKCDDAFRULoIW+gPt3xGxh1ZjyiXnjfRHbpsR0fIpwD5nooFrVjcqX+rZbcZeZX382m5SlEcJSjwymoZUOaWZSxQMfumGZG+hJ1em4JByL6FDbzc8VWIFiQbbYLI0XEoWFPPN2qhdeY6BY2a2hDL8UTEHPbV0LGniw07Uc2FrLTA1CsiipIJ7vMi0ZxdzlPCbmpvs4J73qHIcPJIHPQlUqF/LakHjVG7CfWB20bpkgHtXfRyCej+7ShtcbSzeVRZ7DlGqXwqSvy5CQxAYrBkZscyJOE6eFTskbD9VGMQGFlYqRIXPYMM6ziB11tqYJDDrpAJxc4VKG2pXTTxDvPuW09DlyG6/jNTHO3eIItSeJmYpdi2pFO+s5H+2rsrY6x4Wv6ctbh2TgkimCt0cmZL7FBts7unHCNtNnQbTiKFwqrPW4tNCy16vG+/imwvTEPi/g1kAWYjC+bD/JZ1B69nGTcQHk8cw4NPTYmS26RD67LmHifCAXVavD+5dyUY20dZKXl1GaUG+DnEPDKw6YEO/kaGgFjeIs8egacYc8tDxkBldUpAKEM47I0/ENR3q1ckFJd1qOzAwaVKBIznNuvwtpoIyrlDSqkwdXXIX8u4+iYLxaEavpXRit91fvHhSuF26Tly5VePlU2dDIKBbReOOOR1V0ty5ApYZSCUHRtTkAoHJaLMAvs8mmz3FbmEPFJMzDoueR55JieVa5uuvkdFQrSIDAPI8//BkZmdQb6t/U0+wf1wx2e4zRRzOSrTOmbWMgT L5Vaope8 QWoyP6V3ZUWXhXPPJT43wOBTRSm2MI0WoBKKjRw1XkIkzf2qHZ5Vg18Y+9ZINytTHrSTMvcglf/JPgyDBujjHlCBzv5eEgTb3mAL3cPyvo3thXodpEgYs9tnbUB7ZSnMJGp1PXdPDhesOFvMNpo40UnJfE4+g/QnCndeREBAnc3Xiz0P7cMwRdqBW/6eGwJ8dvtnK9H3uE3IBYZTh1eWH7F2cjpVD5FflmGehBPOYVcRvP3AujVYfIi2hCA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Large NUMA systems might have significant portions of system memory to be trapped in pcp queues. The number of pcp is determined by the number of processors and nodes in a system. A system with 4 processors and 2 nodes has 8 pcps which is okay. But a system with 1024 processors and 512 nodes has 512k pcps with a high potential for large amount of memory being caught in them. Enable remote node draining for the CONFIG_HAVE_CMPXCHG_LOCAL case, where vmstat_shepherd will perform the aging and draining via cpu_vm_stats_fold. Suggested-by: Vlastimil Babka Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -928,7 +928,7 @@ static int refresh_cpu_vm_stats(bool do_ * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ -void cpu_vm_stats_fold(int cpu) +void cpu_vm_stats_fold(int cpu, bool do_pagesets) { struct pglist_data *pgdat; struct zone *zone; @@ -938,6 +938,9 @@ void cpu_vm_stats_fold(int cpu) for_each_populated_zone(zone) { struct per_cpu_zonestat *pzstats; +#ifdef CONFIG_NUMA + struct per_cpu_pages *pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); +#endif pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu); @@ -948,6 +951,11 @@ void cpu_vm_stats_fold(int cpu) v = xchg(&pzstats->vm_stat_diff[i], 0); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; +#ifdef CONFIG_NUMA + /* 3 seconds idle till flush */ + if (do_pagesets) + pcp->expire = 3; +#endif } } #ifdef CONFIG_NUMA @@ -959,6 +967,38 @@ void cpu_vm_stats_fold(int cpu) zone_numa_event_add(v, zone, i); } } + + if (do_pagesets) { + cond_resched(); + /* + * Deal with draining the remote pageset of a + * processor + * + * Check if there are pages remaining in this pageset + * if not then there is nothing to expire. + */ + if (!pcp->expire || !pcp->count) + continue; + + /* + * We never drain zones local to this processor. + */ + if (zone_to_nid(zone) == cpu_to_node(cpu)) { + pcp->expire = 0; + continue; + } + + WARN_ON(pcp->expire < 0); + /* + * pcp->expire is only accessed from vmstat_shepherd context, + * therefore no locking is required. + */ + if (--pcp->expire) + continue; + + if (pcp->count) + drain_zone_pages(zone, pcp); + } #endif } @@ -2060,7 +2100,7 @@ static int refresh_all_vm_stats(void) cpus_read_lock(); for_each_online_cpu(cpu) { - cpu_vm_stats_fold(cpu); + cpu_vm_stats_fold(cpu, true); cond_resched(); } cpus_read_unlock(); Index: linux-vmstat-remote/include/linux/vmstat.h =================================================================== --- linux-vmstat-remote.orig/include/linux/vmstat.h +++ linux-vmstat-remote/include/linux/vmstat.h @@ -291,7 +291,7 @@ extern void __dec_zone_state(struct zone extern void __dec_node_state(struct pglist_data *, enum node_stat_item); void quiet_vmstat(void); -void cpu_vm_stats_fold(int cpu); +void cpu_vm_stats_fold(int cpu, bool do_pagesets); void refresh_zone_stat_thresholds(void); struct ctl_table; Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -8629,7 +8629,7 @@ static int page_alloc_cpu_dead(unsigned * Zero the differential counters of the dead processor * so that the vm statistics are consistent. */ - cpu_vm_stats_fold(cpu); + cpu_vm_stats_fold(cpu, false); for_each_populated_zone(zone) zone_pcp_update(zone, 0);