From patchwork Tue Mar 14 18:59:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08D76C7618B for ; Tue, 14 Mar 2023 19:01:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23C4C8E0003; Tue, 14 Mar 2023 15:01:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EC188E0001; Tue, 14 Mar 2023 15:01:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE45B8E0006; Tue, 14 Mar 2023 15:01:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C9D3C8E0003 for ; Tue, 14 Mar 2023 15:01:12 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 83E35160D76 for ; Tue, 14 Mar 2023 19:01:11 +0000 (UTC) X-FDA: 80568421542.08.9059B71 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id B88EE1C000E for ; Tue, 14 Mar 2023 19:01:08 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fK+KM5+R; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820468; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=iUvNynje4NLXuVViJqGVXz34mfwfY1r+EmC29iXiBlQ=; b=Jg/RLVVN/F0Trf+PuCMoD1b9eH0tRkmDs3+VgQ4G5ckWacyPIK0xQpaLHxfVj8U/7ed4mc hI9P7YkSNTTIuZF9Yqz/T3CLqu/UD0jng6sqMo13m90KkRSQEaA6EfPMI/mWZVpV1Vo7Rl bBh/Ol7RKWFZywQLz8PN/APhU3Wn/Ao= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fK+KM5+R; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820468; a=rsa-sha256; cv=none; b=g0PBfi0niAKeIBbewIAVRRtl4BuwwJUdhSyBfnaqmehP6nnrtKAJ7uiAmZe+XAZWixDwYJ xxaZH0I6e95et2mAzByM42R/dwerkYb7DRjkZiI74AC3wK9gf6Otclhzk+Jo4UJcnUuNZ2 1w3gVIMLtrtLhyeQwcD8KKEF6ze+8kA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820467; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=iUvNynje4NLXuVViJqGVXz34mfwfY1r+EmC29iXiBlQ=; b=fK+KM5+R4E3aI5+1vDafYAkDkOYtzPSeqARJJMUdNwjLAoH+g4n62+St2V0BNNVP4aou+1 4S0bNZlIlYXE1ksZXuFLs7slAfqqlgEyhNxQhOtSZMEgtkfox7sW5DriadPVhWgKn+0/1H 6AaBKoe8g95qltD/Q5NjLufoTZpoxcI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-456-I9tNzClcMYS_EvR0ZH9ndw-1; Tue, 14 Mar 2023 15:01:04 -0400 X-MC-Unique: I9tNzClcMYS_EvR0ZH9ndw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5E48928004F8; Tue, 14 Mar 2023 19:01:03 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2D7681121315; Tue, 14 Mar 2023 19:01:03 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 555DF4039C284; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.551375754@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:15 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 01/12] this_cpu_cmpxchg: ARM64: switch this_cpu_cmpxchg to locked, add _local function References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: B88EE1C000E X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: q7rcueuxybn7ozgzifq81kedtw8d7b4g X-HE-Tag: 1678820468-42596 X-HE-Meta: U2FsdGVkX1/lxmBpkrq3B9OkGDRB18+vtd8j3FF+JeZ2i4bP7loNO0S03rFLnPXvsqpRw42bUyuQ2bjVrpVFAmnn2z+uYnfCgzIx6Q1itc711IX83fVuMJTefYrHT/xOI1cEgkmzsKn6v+Ky6dR+A/MpntmYvGlJ30PfE2aSCOezlG3qZ8dMtTt0DHzpPuWT2Jm32VbQTdjfZZlSMNBsmAoxiWsrJX4LDfTMS8nhPIT5xqPVXKFZcZ4Pi1MXDboEIIBbVAlWBypgLUSbq7Z1CteX/SOOZyE1eYvt3cGQSpeRjOdJNbpkaKO1x03ztF1Zfin1BpJDZBN+sjndCi1HTYljOh3c2kczFzc1/yT1TD/eUBZYKNE8i+B26h7hIQ0APLLa8yQMacg1OGHCybHLIsYuAufH57MG3B53BAO2DsMbeokUg2x1ZqAb31d5BbZvue9CQlVCxOXX2/GiDq4eCs0vZ1JMYPdurRToS/0zkr8grTHU/38sPsChqd4zJi9XjhmAVCNbedQKM1CD5wvPAVDsNia+HT2CRY6RptpoUiM7GrTtvBIZw7mfeGX+BvLeLDuPahhaXXRahFlwtqXQWtDpZZhU73QYmpKpYCpy9gPtbbxEVkkCIW4ZznawI/tw/cMcBwizZEyawYlODGHnMz1uw9mBxvISjggO+GhrJj5Dv+LElpegF00uBbeBr59yLTJjYT6rNxDUvTF52HygUroRqBaIpeMUwUapdv8FR+bM2aBYZNmChZfABek06+4o2yCckDss1MAqMx2nrS9bYrVpPNjvVb7KR7o6/ZndDkoBWnGLvVUa1OmcYlTV/E+VmU0rHnTe1W7LVfNFzCg6Lz2coZ25i8+ZIr5kA1htrPhv0BkBICVfjBXX2uGlcMFNvUyYg6jQLKtj3vcgjzVXQKAwRSEGQhpT+cJWz/TekN2OGRJ6JxhMYl8xrTWBA1Dn47NZeEXL8ALmfQXOemx cxTQTvzw /lgkJXrThoC5vyiWLaA9SpYHcqQ7Sws1yOx327lT6t31e8PUA072EClpW3mYi90erkma2J5+DgORcfsgxBLUGqo9Q+zJzDo8Cz7Eme4MPm7/nZiX0wI3p1VzKw1TZ5pIbtRF9h6MLzJ28kmyKhMzn0eCiXJt1ww0/vyO3N5Izz121nt+SOa11Wbtmr0wNmAE1UjCWtK9RX6udrzR0BjT9st58SIpziXoGXwKJBGv1fQZ17nh3G0Pdbiv2vA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change ARM's this_cpu_cmpxchg_ helpers to be atomic, and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/arm64/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/arm64/include/asm/percpu.h +++ linux-vmstat-remote/arch/arm64/include/asm/percpu.h @@ -232,13 +232,23 @@ PERCPU_RET_OP(add, add, ldadd) _pcp_protect_return(xchg_relaxed, pcp, val) #define this_cpu_cmpxchg_1(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_2(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_4(pcp, o, n) \ - _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + _pcp_protect_return(cmpxchg, pcp, o, n) #define this_cpu_cmpxchg_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg, pcp, o, n) + +#define this_cpu_cmpxchg_local_1(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_2(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_4(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_local_8(pcp, o, n) \ + _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); From patchwork Tue Mar 14 18:59:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94502C7618B for ; Tue, 14 Mar 2023 19:01:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDCB28E000D; Tue, 14 Mar 2023 15:01:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4CCD8E000A; Tue, 14 Mar 2023 15:01:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A76D88E000B; Tue, 14 Mar 2023 15:01:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8F4298E0001 for ; Tue, 14 Mar 2023 15:01:16 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F11C51C67BD for ; Tue, 14 Mar 2023 19:01:11 +0000 (UTC) X-FDA: 80568421584.25.79BE475 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 131A6140026 for ; Tue, 14 Mar 2023 19:01:09 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Umf33aeM; spf=pass (imf26.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820470; a=rsa-sha256; cv=none; b=WUTx6peXz5ond6Rtla+9ydobD1lCYszf37Wa6OiGO3PLp7Bz/2dlP8t+H22s9K3KgRtC5/ stNBhFwHSn2ClQcy/tFI3q4RtUwil0Ly/tQXWJE9s9W8uwhuB9w+3KRu8garfadhSH7RI9 HkE7mRNl4u/luxttI8Y+/9ce7zQfuR8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Umf33aeM; spf=pass (imf26.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=1qMXUw9dsgBVhjNMR0OfieQvkRg7GOetq5N/8/ayAmg=; b=TMJS3XhYflE4HatgeLm7ugKjpL1YxnCfpXSLubfgBR5XhNickJLxI0ghH0eEVkTskFgiGD GAFRGtSeoMKY1cSZJ4vg4jauKko7w9wU/uJkBfS2f49btIUKfHwesEbKlIdIAOcgi4Kb20 FNEZ4YSm307raMJXVu2A+I0bq74Erd0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820469; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=1qMXUw9dsgBVhjNMR0OfieQvkRg7GOetq5N/8/ayAmg=; b=Umf33aeMJ8vJ3UwvmYIJrPPSHClCx/MbwqMe7T8ItWiDUQix5D7dXvs9gih1Tv2Kz8vBi+ LyTOeKsrAL4HdhDfJ3CcAhccqUl/yYGs8+ehA+2OU6J8AEn55h3Sv7n/K8eaEz+q6hG8kI rU7E1FDfYYa/XKcy3na4zJ1x99AaJWY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-157-Qza4perdPzSYms_2eeHgDQ-1; Tue, 14 Mar 2023 15:01:04 -0400 X-MC-Unique: Qza4perdPzSYms_2eeHgDQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 665D83C1178D; Tue, 14 Mar 2023 19:01:03 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3F17A2166B26; Tue, 14 Mar 2023 19:01:03 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 57D0F4039C28C; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.577895774@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:16 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 02/12] this_cpu_cmpxchg: loongarch: switch this_cpu_cmpxchg to locked, add _local function References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspam-User: X-Rspamd-Queue-Id: 131A6140026 X-Rspamd-Server: rspam01 X-Stat-Signature: 5eyh11catrxumguayasg3jfrmexd5icd X-HE-Tag: 1678820469-491727 X-HE-Meta: U2FsdGVkX19htnTW/z3VeTDgzMb/W8hotNXnKkWLbQS7LnDng+ifEhnBSXVkQtERtrl1N2laWN5Wjn7iD9oJ1Rst6qmikHfgi3ubvNsnoBTchdQ1L5vsMVGrDi/KIanxPPOuqiSl/ysdGcfhC3tNKXJspzZcn4/4eauqWHR801tgB5sVy2nUf1PaVbpPeZM3e6PLZy9LY4Xl7L+SNbjha+0ROoTMvImCT5fvxdCgQ0vJ2Y4nkhBkcpP/LYG6CxCiAkj3+X6BFuz+KQ3AUxD0P6Zoss5hdu9gSrIotMl1+/48vJT81LV7umBR64qvMMBUcOe3ZnAV2dfp8Xc/wSe+wIbuABzOzEe6P/LONFwIceB8Fnb8Pe15w3dABZtTnU1f35qxSYmT/OoB0X1HMKqx83pNUyugaO/GA33G9VmgEtYLgBYXY0ovvxqfX6/ks71ZpilLY7PuKQPJIkNaxABVAkMk5k6PCeErPivt+A1EzkEtxvd8iNMggS8/QF4CQDI4WDMODCr8M3h1GmDrVEZFna/t6p5zGyqcTEIsh5zo0bmxQpcM/bslD3wG/6vC0OrWsdH1mJra2GvhT7eG0faqMpQsT/TeG8wVCiFTiOhjA/lhjhLCsTIYBi+AjQ6AoCE7uCkn9HyQ2sMK9fSGgQc1Nrk6AXvAjtTQdPnn/2DjqLYkCeHF44bA37BgEeCOtu8MUOV7yPvnqKV+F4w8Os8xW14w+3JasWM0UUD/FV4LRXGPnzh+01WUQznCTpQxqKpgOBU91iVQoYWyza5WqpAcv21OYygGxoggZHVA3hIkgEfl+OSt1xWtci5Rb8g35rbGArtqLuDZ0nUTPDcvkR6heYkQpvw3Oo6REmWPTMX7kl27hCbQWIxj/8OJn9lptICbIQlfDuUY3ifJcC+dqcZcGQoVrow6gf8Exhoue2XWEXHLr3ej22upfcxHxZV2/H3sdUXetxJJlhjjQKQnbew 8lYxQiCB LvsLtICuBThsrx4dpM0For8m67AGQFIpYq3U14c43ovdQRte63nO5jLoeaqEilte4PzPP3EQNykV2d0LPdIzHz1odsInFrod4E8vDARx8Q5iN8gQkxifTczgc3CdjS/rjs/PRTpdIBrZfr3U3VTDlUBMrLzub2+4WQOYnXJnFtLBQlXpKq9lw7nux2kGd4OTGKeO/7ndeGrbj15Y9STmtlU4Zx7JT9bVQKYouFkVn9jtxew0YC3JTpuEjyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add this_cpu_cmpxchg_local helpers to Loongarch. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/loongarch/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/loongarch/include/asm/percpu.h +++ linux-vmstat-remote/arch/loongarch/include/asm/percpu.h @@ -150,6 +150,16 @@ static inline unsigned long __percpu_xch } /* this_cpu_cmpxchg */ +#define _protect_cmpxchg(pcp, o, n) \ +({ \ + typeof(*raw_cpu_ptr(&(pcp))) __ret; \ + preempt_disable_notrace(); \ + __ret = cmpxchg(raw_cpu_ptr(&(pcp)), o, n); \ + preempt_enable_notrace(); \ + __ret; \ +}) + +/* this_cpu_cmpxchg_local */ #define _protect_cmpxchg_local(pcp, o, n) \ ({ \ typeof(*raw_cpu_ptr(&(pcp))) __ret; \ @@ -222,10 +232,15 @@ do { \ #define this_cpu_xchg_4(pcp, val) _percpu_xchg(pcp, val) #define this_cpu_xchg_8(pcp, val) _percpu_xchg(pcp, val) -#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) -#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_1(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_2(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_4(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) +#define this_cpu_cmpxchg_local_8(ptr, o, n) _protect_cmpxchg_local(ptr, o, n) + +#define this_cpu_cmpxchg_1(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_2(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_4(ptr, o, n) _protect_cmpxchg(ptr, o, n) +#define this_cpu_cmpxchg_8(ptr, o, n) _protect_cmpxchg(ptr, o, n) #include From patchwork Tue Mar 14 18:59:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42736C6FD1F for ; Tue, 14 Mar 2023 19:01:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B2316B0078; Tue, 14 Mar 2023 15:01:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 657E16B0075; Tue, 14 Mar 2023 15:01:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 435698E0001; Tue, 14 Mar 2023 15:01:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3129D6B0075 for ; Tue, 14 Mar 2023 15:01:11 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 037BC1A0597 for ; Tue, 14 Mar 2023 19:01:10 +0000 (UTC) X-FDA: 80568421542.17.470AEB6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id E41CF160030 for ; Tue, 14 Mar 2023 19:01:08 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=B2xKH1Dr; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820469; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=FePX2+VFBr01iZwaVbBM1HjdLAtX/eNoAgkcl35m848=; b=QaIw8+PtVlTCf5NmU8oJpwZS1c+YYofzb83zfw6/U6HkSr/0oH+tZ6tyhysdZhPmKgOEVL PYIuizNA58wUGMv+eyOg9fdPER/03yxz1sBkMIAON1alQtoZvKkXwISUrY5pY7iUUjuIXc RpI4/VIlUnlCfDEP9xJp2tJSfIfv3bw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=B2xKH1Dr; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820469; a=rsa-sha256; cv=none; b=PxtaNn82s8gjomcsOBjAmqt0crsjhnkN/Y6b23GcMz0PhOlT5tZiEjSOsIsVNyVjlVK3pc aR5uhwVsSvjUs/KcZqTrFZf+1a3FA66JtPXRDqD/s3aFDXU2gy4ZOEnYTE29c/LDVYdtg6 cSDs+qjc/SRH6jqBhLiJ69gd/gZR2UE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820468; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=FePX2+VFBr01iZwaVbBM1HjdLAtX/eNoAgkcl35m848=; b=B2xKH1DrgafkjMZlEfcadyLrpH/rsCejXqkO2GUykTUec7A1Of83Tq7yXGyuuImZXCT5iN TifRj4w5nRCDYX8TZ6/19MB7v/dkWtWZg4IVhayoR/kdSNQ5T5ArjreYWoNwmakdeyZ+Gp 6Lquf6X72rWbmbbWxuEJ6cefv+28mHQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-167-cFasFrcQOX2aPTg-iv9Y4g-1; Tue, 14 Mar 2023 15:01:04 -0400 X-MC-Unique: cFasFrcQOX2aPTg-iv9Y4g-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7B0181014708; Tue, 14 Mar 2023 19:01:03 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 490C8400F52; Tue, 14 Mar 2023 19:01:03 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 5B14E4039C28D; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.603655249@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:17 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 03/12] this_cpu_cmpxchg: S390: switch this_cpu_cmpxchg to locked, add _local function References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 9opru7rzt76dtz6gqxqcuw9xonnbp4w5 X-Rspamd-Queue-Id: E41CF160030 X-HE-Tag: 1678820468-237792 X-HE-Meta: U2FsdGVkX1+dIBeryf2vmT+itipAwUMGVhpvEjU4C4bZYPXPSaEXcoHJeXUERZ5mHC9JSyM1ZtAU2vSAJj9yZs/2NONYpxI59g2YBMG1RV7MFD2hBo8ihqJvXhJgWOefGEi7iZDoEUwyJGBKzZiJj4B4uhAP8EHvLVZyrFlNj+Dbuo5qscJYpWiMDPfX34XNKNsWT3/Gvhlfx5SZ7humpjtzZrI1pj7qcxnmceSIJhuDiiBkFUXXapkkS3LBGTXwyKU94iJKPXaWFv8WW/3v4PP0fPAct8H+62/XX8CU7P5Hu1wh3NXmTayZsvGqROhl+W/Fy9mw1+CSEmB42GXLXRyG3bIdqZ13h+clb9NmBjNalhPm2iQh54E5Uym/iQ80ZgG7sg6YYIwsZyY79IveLSGgmDv2obKj+70Gf3G9ZkE6zMhmBlX2T8YKTpkeuUFjN/pRployGqfmi/lqbw1Ln/OBDA2w3LE281qBqw2eUliet0+UM5GvgQfm3X3bcSgkLycQzKIn7A0DbLwfaWtkwbNKPNWOmng8XPzQl41uWokD8eiCw7Bi2oF3q/E1Xoh6NQP9MCcZlVvtB5+NXmuhGWTwp1nmkXa4VT0AFfkUq9gCh0MJtuTCccLnNgSilU1VFWxoR6ffD470EXldk1mTmwGsIPxAKwdpcJx1F5sCvDnRJzlrIffubZJVrG/rc4BoPPhz3592Zv84eD58CvIiJgXd4YvhpsKpTb0CLjUQoPmX0yH52M6XhakFRhE4qs25f4SOav3H0BgKRyKxpZW/fnKZzymFAs54qG5JPyeIxMNhjy3GWS8bFWrJ8+aHqqOfXhIHJp+BxKt9UeHTuMFZFUEtgj2NzvP0+mo5ZgR86DBJnp1e68Zy1Jby2Hb0XpYl+vmLlp24pxozeME4V31iX3S+OACeGkM/ZRtjSJCG2fJNTkGaC6XwnX7tWlZY+e0IVNwp9Q/hBWcFtmPOyVi ODoNpjlF pWMElH3WaqHDmJJp/rSKCWcWl5O3B8iiOVpOgDplm5SEY/ACBjOKYsnMFJFmTRmnW4Avvtxvu8YwCEUbYKnRpzbDyP/4Wa++0VY+EeTLoc26f+q4sleghc+0LY72TDq+VH7QXGf5Me3kEY+0yML6H9qNwM+n/FuZjRbg2866sj84OqgzcyR6buRz2aVcHNaS6x9By0hB4CCAImFl/YI22U7JXVOkxTabVPQAkd8xQD30m2aIcZe7NGbkllw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, add S390's this_cpu_cmpxchg_local. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/s390/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/s390/include/asm/percpu.h +++ linux-vmstat-remote/arch/s390/include/asm/percpu.h @@ -148,6 +148,11 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ From patchwork Tue Mar 14 18:59:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D73AAC6FD1F for ; Tue, 14 Mar 2023 19:01:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFFED8E0006; Tue, 14 Mar 2023 15:01:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE1DB8E0001; Tue, 14 Mar 2023 15:01:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2A7F8E0006; Tue, 14 Mar 2023 15:01:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9506E8E0001 for ; Tue, 14 Mar 2023 15:01:13 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 61C43120F16 for ; Tue, 14 Mar 2023 19:01:13 +0000 (UTC) X-FDA: 80568421626.25.AE017FE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 507CD16004A for ; Tue, 14 Mar 2023 19:01:11 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V25PWPGT; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820471; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=hjd5gtUB1FFqfcJWUv9qrW5kcd1bszAAtI+X6kwFBXk=; b=JJba8NaUO6vhhqj1++A1WWZIji4bvnDsFeefSXpayFeqLDOkh4o/nQs3IYN+OiBaAU5qyr QAYVuZ4kNsIp5ha6JtziDd9SEmdt3XdFRLMSHM6rh8FsXK1ZFvFCXrkt9G8b+fcJgwbh+3 h+WLivXWwpqCXxZAa/79tSLwu6tJ6R4= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=V25PWPGT; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820471; a=rsa-sha256; cv=none; b=AaTfgoZHAbBqkdR7dgf0twxna0WJtxEszZhpShvYUorF0raLrTfzupPvl3Iw/Ne6Gco5Jp moTDMp7ky9DWWjTiLI3tV5wHqWFHfe0IdPnSvS0dJ/BcyxeLlVT7saETkZUjYeZszjPiB7 tT/VOxRFk4v9Hxm+kJMqwgv6TYgr06s= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820470; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=hjd5gtUB1FFqfcJWUv9qrW5kcd1bszAAtI+X6kwFBXk=; b=V25PWPGT7yxpHh/ZqYMyjgCF83B+ak/aTeWoWlhB0IptuCKwJwviPe+u9zUL5Wam94g61i FMJXiKllPjBv0XMypVh3Cjv2X35TAsDVvPLy5pqFf1NYNwj25oVAD04zptOE97oVitOZSR jf6WDzFQRKa3ThX0pi8B2AXKDc7s41I= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-592-Nza2RIASPx-JqNnIHRoL4Q-1; Tue, 14 Mar 2023 15:01:04 -0400 X-MC-Unique: Nza2RIASPx-JqNnIHRoL4Q-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 87B1D803462; Tue, 14 Mar 2023 19:01:03 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1830D400F4F; Tue, 14 Mar 2023 19:01:03 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 5DBBB4039C28E; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.628024107@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:18 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 04/12] this_cpu_cmpxchg: x86: switch this_cpu_cmpxchg to locked, add _local function References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 507CD16004A X-Rspam-User: X-Stat-Signature: jg4t1dcdx8ffy7agjfb4ewteb8qt8hzu X-HE-Tag: 1678820471-184955 X-HE-Meta: U2FsdGVkX19B6s2jpETdnunMFTb2YppSBUkth2V+HSjFjbhS13ESzeiLEPbPn/265NoPyPbN993HAZNuMDJ9rKSv+K0FllBmT8LFTNVNLq1PzyoILwLUCG2AsYdRvjCGuvzFJw89pT+9mCeMJtsU8m/vglMV4H4/+VNTgo4iBnJQLkwgBTCVEhOyYVGcouPFBEZKKbjJWA/YtuZqKXiUjA/NSIwIJZ9SUAn5jIIn1lvW58k+VHViRALp1oQy0MkTdInWjlbthYFh6ERxhryGWNKSTA61O5NRTdnGyudvYopxIGiA2YcGU17WTaHucqjEeCmsdrvgSYP0lar21hXAABfbSK8TfOoX7r51lNddQ4URlVIx5TWiM5Q13oeO+CR6dqefPtYsolIXezodVZTONUBBJ0fsEVf9TVWb7eLY1i4amPjXv4GO45RDhhGwHdVH0soFk1nwax6J7iUvFqVGhqof01+YvZjd4xHTv5SEDLr3X54tuVH04pbL10+vLAyRxRPla03hz6YLamCbeBdvtQ8ZrJqJGmiPCaS4yE/uGKBN8Zhswy4RXTlVTO26KB8lDpVu4eq70o8oACrHaqhmbUCSfY3uCa04iDpa64WGjSuPZHPNP+t2YFLGF1dslv3nsxAQpiMpUP8BzFXgRCAK0e6skHlonzrq6DQArZfJvObzL+GMzc6zSBuZ7DvgTKEV3HqI2cOV2Qn0LmfzZSZgQ/yoQUCLq7jB56eNIwHbLIv9/KMWcUMMXBwza6gRQUx/3fwe2YlTDlyn11frwdMstaTObYbl+KLYc/EvzdaHFIpITQriEeZCDfS6ifuyAPEQHURaTeLIw8lODkZn4C7ZDuiMsRXAqkzX9YDdBJF2YwHY2lxCjKsh2nn5rXDO5Ur7Wpj6ul5PGc1mbsZ525mJYFpUqUN+LtNeUrxSRrthNg6f1/ktbn6hiPpAmhiHhicOHPRRGm6M1n0vzFK8vi5 0fYJBKoS sVD81kEAaNeZEVwnfoTGyWGyX04M8Svu8Rf2tk86nOX4+SB17snl4CrWs0V8n0WKLCrXp4mTeuUrtVoqN8uWB707uOlM9XatNkYkaN1tCAl/mmi+iQ/UsN7wm13C4zUC6uhCaAucxL+aWkdaPbqeMJo03Xbx6mfADIOen0GJ6Dsj/74vxDEtprtptk4LO+yJjoV9ohOvk67E4fhiaSKSb0jKXgAZKfwMMA8E6p+QYgZ2Y0WwiKHepec7AqA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Following the kernel convention for cmpxchg/cmpxchg_local, change x86's this_cpu_cmpxchg_ helpers to be atomic. and add this_cpu_cmpxchg_local_ helpers which are not atomic. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/arch/x86/include/asm/percpu.h =================================================================== --- linux-vmstat-remote.orig/arch/x86/include/asm/percpu.h +++ linux-vmstat-remote/arch/x86/include/asm/percpu.h @@ -197,11 +197,11 @@ do { \ * cmpxchg has no such implied lock semantics as a result it is much * more efficient for cpu local operations. */ -#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval) \ +#define percpu_cmpxchg_op(size, qual, _var, _oval, _nval, lockp) \ ({ \ __pcpu_type_##size pco_old__ = __pcpu_cast_##size(_oval); \ __pcpu_type_##size pco_new__ = __pcpu_cast_##size(_nval); \ - asm qual (__pcpu_op2_##size("cmpxchg", "%[nval]", \ + asm qual (__pcpu_op2_##size(lockp "cmpxchg", "%[nval]", \ __percpu_arg([var])) \ : [oval] "+a" (pco_old__), \ [var] "+m" (_var) \ @@ -279,16 +279,20 @@ do { \ #define raw_cpu_add_return_1(pcp, val) percpu_add_return_op(1, , pcp, val) #define raw_cpu_add_return_2(pcp, val) percpu_add_return_op(2, , pcp, val) #define raw_cpu_add_return_4(pcp, val) percpu_add_return_op(4, , pcp, val) -#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval) -#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval) -#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval) +#define raw_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, , pcp, oval, nval, "") +#define raw_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, , pcp, oval, nval, "") #define this_cpu_add_return_1(pcp, val) percpu_add_return_op(1, volatile, pcp, val) #define this_cpu_add_return_2(pcp, val) percpu_add_return_op(2, volatile, pcp, val) #define this_cpu_add_return_4(pcp, val) percpu_add_return_op(4, volatile, pcp, val) -#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval) -#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval) +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, "") + +#define this_cpu_cmpxchg_1(pcp, oval, nval) percpu_cmpxchg_op(1, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval, LOCK_PREFIX) +#define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval, LOCK_PREFIX) #ifdef CONFIG_X86_CMPXCHG64 #define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \ @@ -319,16 +323,17 @@ do { \ #define raw_cpu_or_8(pcp, val) percpu_to_op(8, , "or", (pcp), val) #define raw_cpu_add_return_8(pcp, val) percpu_add_return_op(8, , pcp, val) #define raw_cpu_xchg_8(pcp, nval) raw_percpu_xchg_op(pcp, nval) -#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval) +#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval, "") -#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) -#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) -#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) -#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) -#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) -#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) -#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) -#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval) +#define this_cpu_read_8(pcp) percpu_from_op(8, volatile, "mov", pcp) +#define this_cpu_write_8(pcp, val) percpu_to_op(8, volatile, "mov", (pcp), val) +#define this_cpu_add_8(pcp, val) percpu_add_op(8, volatile, (pcp), val) +#define this_cpu_and_8(pcp, val) percpu_to_op(8, volatile, "and", (pcp), val) +#define this_cpu_or_8(pcp, val) percpu_to_op(8, volatile, "or", (pcp), val) +#define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) +#define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, "") +#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval, LOCK_PREFIX) /* * Pretty complex macro to generate cmpxchg16 instruction. The instruction From patchwork Tue Mar 14 18:59:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7974FC05027 for ; Tue, 14 Mar 2023 19:01:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B26C28E000B; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A03C28E000C; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 853A88E000E; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2FA948E000E for ; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A399E404B7 for ; Tue, 14 Mar 2023 19:01:16 +0000 (UTC) X-FDA: 80568421752.08.8EC4762 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id F280A100029 for ; Tue, 14 Mar 2023 19:01:13 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=adOBaLO3; spf=pass (imf05.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=X5A6sr5GdS/CRp9e0AjLg+A82o3GBDzuPA8yAWqGcJk=; b=RXQZGtm6jJZ7RhOhPrEeaWRhVBpTEccxEbTOWGeYexh4lVe0CYPSUzVLy4ZBur1TvT3Dm/ wJZqnbXmCgHLFTZiLHsQMEw2aM/Kij6RjyKel4LEmtS7/vIche++mVwrVUJfZZJBvN6z1p pvwa+sePF4FWM7iBQhJ3/FI73X3X3kY= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=adOBaLO3; spf=pass (imf05.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820474; a=rsa-sha256; cv=none; b=Pv5412VWwfDReDmXVy30FP/Rhvq+Z7a5Gl4MZHpTFGagcMO+4+OfxYjsv5Ypnr2dRq82Dw qrH2a9nOvE2hhMQs107+QOddvyU7fz7nL6Q9bswRt5SyzFwZd48mzZ/Sj6T9yt3mgJb9PA fQksDwhdF6FZrkIwC+zkNGYJ1O67S1Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=X5A6sr5GdS/CRp9e0AjLg+A82o3GBDzuPA8yAWqGcJk=; b=adOBaLO3u8PBtxscE/KKtKvb/Jps+VsNcJw3B9iNnGs51g9n3E9VkANZ0mKyqtbAd6HG11 b/D0vl7ZP6vrFXuZ5kEwNLZQZMc0d6DSAiCGVjrvDqqu4mE4+vcni8+jU3qglXzNjte3Db Hz8YyFTfTXsb+9PL5d9kGH8a68KiDC8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-386-e5iFwDmNPK6iXBSIkZqWfA-1; Tue, 14 Mar 2023 15:01:08 -0400 X-MC-Unique: e5iFwDmNPK6iXBSIkZqWfA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E190A1024D07; Tue, 14 Mar 2023 19:01:04 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B63FF140EBF4; Tue, 14 Mar 2023 19:01:04 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 643694039C28F; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.654080886@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:19 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 05/12] add this_cpu_cmpxchg_local and asm-generic definitions References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Queue-Id: F280A100029 X-Stat-Signature: rteipf6qfhnagtwbo7txduoejuaopfdo X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678820473-538984 X-HE-Meta: U2FsdGVkX1+Eb3BSJYavx9HdNjVsitZLp9j8/6Zd6A4JzPbcVweH+lYk2J4aYEtMTNLWtTeOI5ls8UYrYWzkOaaXALMvjIrL6nfeej7VZ8uxAsDRvw+IHvOZ6Sys/0vLK4XD5fQvUnA3+9KoyAP+nonQwQxUShzzWBDdwNvjf5t5Sve+O8xapV1zBJGWtqhqJGIDCikZj4aEtVmmxFxw3dZvnSDKIfHqiRxdrgJj2A8nrI21N2NjgIs5nmdG6EG3yzfYJGT+iVbI1uYvE4HNmFUMS170ZVfOBWFctEYLZTKY4H4GejxWwJXYuSbPStFTqsEhIVBKL22b3VTgTbjNZWkn2C6/+64riWjRp4L8ttUipjzsNBxG7Ny33AacV9bfD7oOA1tPjL4pQQrb2/Dda76a+YM0z88Wq3a85v8Mxt7kKcqXMLxsIx4iIurpJY9ofpjrb+fstYSV+aQY3Gh0oP+RfFYqJemZKELy08qXtyxApG7QQ0gnx22rGviNzhElQmdXYTcE9nocFUuIKMnhHpEgWrdOa6+LDvQZaJsq4ta7lTpH7MHzgo8eXXa2XIS26tB1ftm71GNJFL/BYB5r86WmOI/bK/Lj45YzDb8+GcYk4Y+sN+BSCsrhfjpKBVfz/oYJTlFolUj3Lo/hZiVm7Z0A4Apufnk2Q/YAQm/3w2J2nnDoN+66gbeUOvAFk39thjg4bmapYDGPc7xBvkZwVWAC6TFSDcR0rQGDPEUpmB5HwjPfKhiK5Hk6QKEytz4P6PeuEThsLel3bZtVl6GyJDF5m0CJxfmrgnOE9y7HmDuEBgd4E0MEtTT8aPBuWFJwzJWhy0DL3FALx3JozYVT/Ti91uGiVJiWNhM+8/54UEZD5ZPtHeo8DkbDkYSybMu4kDHxJfhNbYkmyKf/N+Vcd9ivyYc2XG875NqDMLsPB8s1lyQLIF3fgsKhwEBkJ7B1A/V7EgqLDHaMkVpbH7G Ap4XwWXH R3x/3yVn7hCPlKfFOEV5er7yo3lKp9pNna23sUOuJdgNHaz0yrgd1JtKMMy0Ipv+oiFZeSzBaD8XXgUUHLRlpq5vEB5atyeQgMA3Yj9Q9AcZDBYc79SLg/d2gDZ7Hr4480nH67BGEVlQkJs+SloIMw0MmFaZ+Ii1jeWObLx5hzRrAPSJtnrVE4NMVwWTHpP/Wbjn+gANzvLxTq4L1tBPS24bfzARL0Ij4ELuN+6iW7E9BCv8qJXXc+VHI4A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Goal is to have vmstat_shepherd to transfer from per-CPU counters to global counters remotely. For this, an atomic this_cpu_cmpxchg is necessary. Add this_cpu_cmpxchg_local_ helpers to asm-generic/percpu.h. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/asm-generic/percpu.h =================================================================== --- linux-vmstat-remote.orig/include/asm-generic/percpu.h +++ linux-vmstat-remote/include/asm-generic/percpu.h @@ -424,6 +424,23 @@ do { \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg_local_1 +#define this_cpu_cmpxchg_local_1(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_2 +#define this_cpu_cmpxchg_local_2(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_4 +#define this_cpu_cmpxchg_local_4(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg_local_8 +#define this_cpu_cmpxchg_local_8(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif + #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) Index: linux-vmstat-remote/include/linux/percpu-defs.h =================================================================== --- linux-vmstat-remote.orig/include/linux/percpu-defs.h +++ linux-vmstat-remote/include/linux/percpu-defs.h @@ -513,6 +513,8 @@ do { \ #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) +#define this_cpu_cmpxchg_local(pcp, oval, nval) \ + __pcpu_size_call_return2(this_cpu_cmpxchg_local_, pcp, oval, nval) #define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) From patchwork Tue Mar 14 18:59:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35AA8C6FD1F for ; Tue, 14 Mar 2023 19:01:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 858A08E000F; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 791BB8E000B; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 609428E000C; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 19BCD8E000B for ; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EB655AB2A1 for ; Tue, 14 Mar 2023 19:01:16 +0000 (UTC) X-FDA: 80568421752.04.5774E93 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 7F8D918002F for ; Tue, 14 Mar 2023 19:01:14 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AMiVm1q7; spf=pass (imf16.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=4jas7+kZOWWtovsxw7rARjyOL6nO08FiGsNhXOwYk0U=; b=YODf8AXpgD6ykZzDcEsYwcp2gHdgesDwmIA0wTIRMH29aUR+5JzzAvvl9z4kzSy+f1vjQ1 FVKy3hXqEOy3/8Sdfz5zoYLMOG39+aZPcAaH81Ekd8cVZuaBGX4bcc+MIA7CZh+kp2yJl9 fL54nb0UhOFlNxCMcLDl5y0y7IFwXuA= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AMiVm1q7; spf=pass (imf16.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820474; a=rsa-sha256; cv=none; b=BUOd++/Qg+koj8DKskEA6XH1lG4ik0s5X2FJ3sZcWtMZJXmEJTbd/Rl1WRcjbBJc4AQJuq G7saJg9TO38H+aid0u2zAs7zS44ckhxstLbwdh8feQm9ayZBvlH2ak2Wzcy6dU5eS2IE// B+UnyHSqcsrk7XVY00rjFbP9OnBbqd0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=4jas7+kZOWWtovsxw7rARjyOL6nO08FiGsNhXOwYk0U=; b=AMiVm1q731P0Dk5sNb0ibUxDC+5mRqVunEmedQLy11RHdScubCv6bTeYDQOWzfys4+Tj38 Rub4+TEWx/v9A5tQvNoRzm8TQ2TydDl8wPQ/wixaFh5dO1uF2A4ulKD1lIhmAK3FusS5gW 7C0zNMSaRniJPzrXmz8pEC8oQtgjKyc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-359-nFSnaC_BP_mQcwnztKYEjw-1; Tue, 14 Mar 2023 15:01:07 -0400 X-MC-Unique: nFSnaC_BP_mQcwnztKYEjw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 05ED6101A55E; Tue, 14 Mar 2023 19:01:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C894B140E95F; Tue, 14 Mar 2023 19:01:04 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 67D854039C290; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.678940741@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:20 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Peter Xu , Marcelo Tosatti Subject: [PATCH v6 06/12] convert this_cpu_cmpxchg users to this_cpu_cmpxchg_local References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Queue-Id: 7F8D918002F X-Stat-Signature: di36jttu5miwh5npsmoz66nxajinsnsw X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678820474-1108 X-HE-Meta: U2FsdGVkX18u7IlIXEAp1/iJ3UVz4M2fwcejsEbCVVTCuouHRSU0gv3qb541X2GNLcS+WwwWQznBi8EJ2ooOPppxauPoDH+KUAn/LjBABuO+Au6lB1q4zmf95vAyvx+ARip5LD1ClUMnqI4Sj7VuAuNhvvwU5kwNjXYRYIdCjrgnPe/qFu/HLcto+vMjA4bTF5ikBc3hBN9d6Mi5CQQaqaaA0gzcMWx0cSwzHR74DLpWA1sCWQfpd+SqPQSHuc6QMnidzTv+b7KnzBiP/HGMSOrU+SniEJi6PCdiWSFMpL+Cg0r1nkgAQMmZWNAHTlFKM3JlcAjpN83VRIJsr/T5vThcJIM0hccEaqvjI5ivOgILPwpkSUwFN/YqlyzqdoEzrLCpaHq2ItSrEZaLOTg/mmu6pFysTrIWymck3du9HYaMHFMa/xtfEraaAq5pvcpad0hU6UHuU9YVk7yXnrh8XNXr1QxXpvcgh+bdGS0HsyKeAD4Cz1hua0bRcR0W4QLA3YlZlmEqPsny47IWVpYxbBpP0oCr/xw5ZPfZHHWeVIEf04AKi+yUfBcPTERnv/RQDTMJCTndYBjGyYos6xvbMipt5/gmCYVUJe6aRJjF85xhr3z6fBcfePSTKdb4CLPSqq6hkKfUCMsViHdka6I31UHoN9npBkadMKUG9IjsVHh5QplfWkk8bra15/7c6/d5pab0Cu2sREZbaRL4suWmgroyIbWm0PwDBMfva2dMjJ+w4cdGD9oUOXCW50yFJCUaTWfuQsuB9XG8/kJz86GYE6hytJ1Hjnj42qTyrkLiN+003Zkm6SBoJ6S8VCWVBVpEzt/aFv21bPCnE9JfFowuljM4zBIDONb3vzsCAAjAZ3zFbaRP9G0mKl1hOPEnF+tPlOmCAEBz1mWis9ucJNYkULALRjbMzuoVIQMpTxDkCnMBrkB2peltmgUoig6IYn6YwVMmo77fnNXFGN1nIHT XBo1TGlw CgtmESKQz1uhkgmgylR0krUNMGmIIrIoeJikhdgw2x7YXa3fWf9O0WAYD+O7E8vH3ZNjfA8J+ugseEP5Dw6SE3SDyVqEl1JMIoavqitsjWoLAYjzPhDqDs7nnRNQuEpxpEP8s1sCzclxopfvwV+3awKZEhktZdBZP24Oq9WqSnfahLV/DtP6QLb7M34ytlh89BxcuKYYq8j3V4HKxfvhtGu/8Ht6/FYGRxPoRAh9xC6hqf4t/DKh+poyKtw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: this_cpu_cmpxchg was modified to atomic version, which can be more costly than non-atomic version. Switch users of this_cpu_cmpxchg to this_cpu_cmpxchg_local (which preserves pre-non-atomic this_cpu_cmpxchg behaviour). Acked-by: Peter Xu Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/kernel/fork.c =================================================================== --- linux-vmstat-remote.orig/kernel/fork.c +++ linux-vmstat-remote/kernel/fork.c @@ -203,7 +203,7 @@ static bool try_release_thread_stack_to_ unsigned int i; for (i = 0; i < NR_CACHED_STACKS; i++) { - if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm) != NULL) + if (this_cpu_cmpxchg_local(cached_stacks[i], NULL, vm) != NULL) continue; return true; } Index: linux-vmstat-remote/kernel/scs.c =================================================================== --- linux-vmstat-remote.orig/kernel/scs.c +++ linux-vmstat-remote/kernel/scs.c @@ -83,7 +83,7 @@ void scs_free(void *s) */ for (i = 0; i < NR_CACHED_SCS; i++) - if (this_cpu_cmpxchg(scs_cache[i], 0, s) == NULL) + if (this_cpu_cmpxchg_local(scs_cache[i], 0, s) == NULL) return; kasan_unpoison_vmalloc(s, SCS_SIZE, KASAN_VMALLOC_PROT_NORMAL); From patchwork Tue Mar 14 18:59:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C59ABC74A44 for ; Tue, 14 Mar 2023 19:01:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6AAAD8E0007; Tue, 14 Mar 2023 15:01:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 630928E0001; Tue, 14 Mar 2023 15:01:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FBD58E0007; Tue, 14 Mar 2023 15:01:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 411638E0001 for ; Tue, 14 Mar 2023 15:01:14 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DBEBF804C0 for ; Tue, 14 Mar 2023 19:01:13 +0000 (UTC) X-FDA: 80568421626.16.D9296E1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 157361A0033 for ; Tue, 14 Mar 2023 19:01:10 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XOxy4pBJ; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820471; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=HRaKPhoc2o/dyfmAGwpM6cYdTDahEMUEDBWaTmQOBWk=; b=ilS+YELi1iSV5wnLF9AqhcsS8i39j4TNmpt/nZX+D0aSR01alhsjayY2lWTsNn316h7XiM XvAASOMRkANtPwW+LucG/k9gnnJAanQe8JUQS3Ur1Dn3YycBUraZBbURljsxVjWC766KuC q8ixG6Jan4vZmdCKXuaDIek84VEM54g= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XOxy4pBJ; spf=pass (imf19.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820471; a=rsa-sha256; cv=none; b=WMwPwz3d+/ikdgfpQGFaf/mdQXSJB6rdPLRg4LYoMSth0t256h+ynO1TGrlo+ORyiZqpqy MkXHqEDMVwvCZpE0JCit+SpgImUU+e2mUh646oDKlXAb2rz+3SrNvQtx0shHfHfVukeQNe ioKIorzi33CukoOj9nievf08KYq4UPU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820470; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=HRaKPhoc2o/dyfmAGwpM6cYdTDahEMUEDBWaTmQOBWk=; b=XOxy4pBJHCPuXcI8Qrz+phO+GDubRgcDie3It2H+iq2Y7Sjib3C6LJ0MBl6+xWZ7p2Mekb TOzylHaNZSbYz8vG4IObiiS//lKEQQTM3ipfl97zpyuhOWLy+LVZg2Pqwq1PixGQYhd0J/ 4dC35SWhMwBcyUeuPbaAFShm+NHBCoM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-117-82bnoofIMDyQIZinJMuk6w-1; Tue, 14 Mar 2023 15:01:06 -0400 X-MC-Unique: 82bnoofIMDyQIZinJMuk6w-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1355B3C11781; Tue, 14 Mar 2023 19:01:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B8D792166B26; Tue, 14 Mar 2023 19:01:04 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 6B8274039C291; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.704136431@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:21 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 07/12] mm/vmstat: switch counter modification to cmpxchg References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 157361A0033 X-Rspam-User: X-Stat-Signature: pxhq1mgpr6rs1uqxg911ho5xyddfgumf X-HE-Tag: 1678820470-25960 X-HE-Meta: U2FsdGVkX1+1CS857/2jGrSWTLF/nwC8CcWTLJsmzbSkopmlYBeQapC/fK6/QAV6wJ/A7LbBUqF+3aboYqHHPthzDVcVV4J5U2VdqoXOEWZEqzz6TDNS94uAtLlq6nMxy6eCeZwzvzzLUADZnY7DysKlPvfYCDx2HdL572lrC/8wIim29BZTpGeN7gCLI8zv/jnbKW98DohtxsvNmW+Uakisy5LkgbkOiwLvw6uCzO6uZCRe4I6RgWId/ml5xU6L/x2//U+Wb4zDF1rQIb36SVhs88+kQueoVfSsLoe+5r+X0J+f5XYpwl6YMR4gQqUyR1zN2e4JniBvOFCFa+XrFPL9b/SozU9R/ZUsCNn8uVjzRQh9L0C1ooy0x5ZaRaZST7O4EthqrYU8tU91WPyg3XSu41LFzxA4s3+J2s4r3orTTj3oEWCb3StGr/DCbiyvOMf9z+AfsF5dDQFCflSQ4ZsWMeKkFVBCOin0FNAM+4CnWtrfKa8p78Yit+awS6JQ4x9CMbcWDBVnlKGcsGJOtr5DT1a8/3a2583gySY92BLn9miFD80BFXBMtApbnGBeCF5dMO2dEV/vo+Y58zc27sSf1UpnoqQ9fmRubYBRdlny6vOMUBolBppP6osda0zkcj26upv/cwahF2JGUZnwtAI2xKf60drI3nPi6BwAMRRDMKz+MeE+9ZAv1v0gjx1Th5PaDvmCP+kIx8dsLrcEtxh/n7fSsIK6Hld1FkGhG06YlsF+RvqdECvmEGH0XC8lMWmURAZdyItqEbg/YGtEkd6xVHVTvixw1vX17xLdSX2sBRv0nkVOYL3/qOac8bzd0jeVCJP6+/X3ICAoU/ecQiRKM4J4F/cZPtHhPHMkiBQCKV8f5LvIG7cNIuYxcnVDl2u+CNRh1RIna4S5F8N+g2LbuaBTxvj4xGOhRvWQ3sw8G2fr0Madbxw2EJXDRc2yJ4S68G+dj/2Kn50rc/Z tDO/OEIB ZWFBM2Xvg6QPszphi/u4LyZB6fKTWXsKR+qJXY/LR93BQCO76+0XoZDYBC7HFgEEEADYzW7OeEu4IBQNL7098d9h6wf8nkaAsT4EjXH2UZ8XDzpTAZvjvCrglx1uOVE3vZnD/LBnvEihURJoN+jyOa8z1ZhlQSpm0HigengF1uYsT+d5j2pGPQOcanrmM7IDnjZkXLfZTjORRnvqxDaaTSGUs9DYqZeeRRnfN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for switching vmstat shepherd to flush per-CPU counters remotely, switch the __{mod,inc,dec} functions that modify the counters to use cmpxchg. To facilitate reviewing, functions are ordered in the text file, as: __{mod,inc,dec}_{zone,node}_page_state #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state #endif This patch defines the __ versions for the CONFIG_HAVE_CMPXCHG_LOCAL case to be their non-"__" counterparts: #ifdef CONFIG_HAVE_CMPXCHG_LOCAL {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state = {mod,inc,dec}_{zone,node}_page_state #else {mod,inc,dec}_{zone,node}_page_state __{mod,inc,dec}_{zone,node}_page_state #endif To test the performance difference, a page allocator microbenchmark: https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/bench/page_bench01.c with loops=1000000 was used, on Intel Core i7-11850H @ 2.50GHz. For the single_page_alloc_free test, which does /** Loop to measure **/ for (i = 0; i < rec->loops; i++) { my_page = alloc_page(gfp_mask); if (unlikely(my_page == NULL)) return 0; __free_page(my_page); } Unit is cycles. Vanilla Patched Diff 115.25 117 1.4% Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -334,6 +334,188 @@ void set_pgdat_percpu_threshold(pg_data_ } } +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* + * If we have cmpxchg_local support then we do not need to incur the overhead + * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. + * + * mod_state() modifies the zone counter state through atomic per cpu + * operations. + * + * Overstep mode specifies how overstep should handled: + * 0 No overstepping + * 1 Overstepping half of threshold + * -1 Overstepping minus half of threshold + */ +static inline void mod_zone_state(struct zone *zone, enum zone_stat_item item, + long delta, int overstep_mode) +{ + struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; + s8 __percpu *p = pcp->vm_stat_diff + item; + long o, n, t, z; + + do { + z = 0; /* overflow to zone counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a zone. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to zone counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + zone_page_state_add(z, zone, item); +} + +void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(mod_zone_page_state); + +void __mod_zone_page_state(struct zone *zone, enum zone_stat_item item, + long delta) +{ + mod_zone_state(zone, item, delta, 0); +} +EXPORT_SYMBOL(__mod_zone_page_state); + +void inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_zone_page_state); + +void __inc_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_zone_page_state); + +void dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_zone_page_state); + +void __dec_zone_page_state(struct page *page, enum zone_stat_item item) +{ + mod_zone_state(page_zone(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_zone_page_state); + +static inline void mod_node_state(struct pglist_data *pgdat, + enum node_stat_item item, + int delta, int overstep_mode) +{ + struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; + s8 __percpu *p = pcp->vm_node_stat_diff + item; + long o, n, t, z; + + if (vmstat_item_in_bytes(item)) { + /* + * Only cgroups use subpage accounting right now; at + * the global level, these items still change in + * multiples of whole pages. Store them as pages + * internally to keep the per-cpu counters compact. + */ + VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); + delta >>= PAGE_SHIFT; + } + + do { + z = 0; /* overflow to node counters */ + + /* + * The fetching of the stat_threshold is racy. We may apply + * a counter threshold to the wrong the cpu if we get + * rescheduled while executing here. However, the next + * counter update will apply the threshold again and + * therefore bring the counter under the threshold again. + * + * Most of the time the thresholds are the same anyways + * for all cpus in a node. + */ + t = this_cpu_read(pcp->stat_threshold); + + o = this_cpu_read(*p); + n = delta + o; + + if (abs(n) > t) { + int os = overstep_mode * (t >> 1); + + /* Overflow must be added to node counters */ + z = n + os; + n = -os; + } + } while (this_cpu_cmpxchg(*p, o, n) != o); + + if (z) + node_page_state_add(z, pgdat, item); +} + +void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(mod_node_page_state); + +void __mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, + long delta) +{ + mod_node_state(pgdat, item, delta, 0); +} +EXPORT_SYMBOL(__mod_node_page_state); + +void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) +{ + mod_node_state(pgdat, item, 1, 1); +} + +void inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(inc_node_page_state); + +void __inc_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, 1, 1); +} +EXPORT_SYMBOL(__inc_node_page_state); + +void dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(dec_node_page_state); + +void __dec_node_page_state(struct page *page, enum node_stat_item item) +{ + mod_node_state(page_pgdat(page), item, -1, -1); +} +EXPORT_SYMBOL(__dec_node_page_state); +#else /* * For use when we know that interrupts are disabled, * or when we know that preemption is disabled and that @@ -541,149 +723,6 @@ void __dec_node_page_state(struct page * } EXPORT_SYMBOL(__dec_node_page_state); -#ifdef CONFIG_HAVE_CMPXCHG_LOCAL -/* - * If we have cmpxchg_local support then we do not need to incur the overhead - * that comes with local_irq_save/restore if we use this_cpu_cmpxchg. - * - * mod_state() modifies the zone counter state through atomic per cpu - * operations. - * - * Overstep mode specifies how overstep should handled: - * 0 No overstepping - * 1 Overstepping half of threshold - * -1 Overstepping minus half of threshold -*/ -static inline void mod_zone_state(struct zone *zone, - enum zone_stat_item item, long delta, int overstep_mode) -{ - struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - long o, n, t, z; - - do { - z = 0; /* overflow to zone counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a zone. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to zone counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - zone_page_state_add(z, zone, item); -} - -void mod_zone_page_state(struct zone *zone, enum zone_stat_item item, - long delta) -{ - mod_zone_state(zone, item, delta, 0); -} -EXPORT_SYMBOL(mod_zone_page_state); - -void inc_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_zone_page_state); - -void dec_zone_page_state(struct page *page, enum zone_stat_item item) -{ - mod_zone_state(page_zone(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_zone_page_state); - -static inline void mod_node_state(struct pglist_data *pgdat, - enum node_stat_item item, int delta, int overstep_mode) -{ - struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - long o, n, t, z; - - if (vmstat_item_in_bytes(item)) { - /* - * Only cgroups use subpage accounting right now; at - * the global level, these items still change in - * multiples of whole pages. Store them as pages - * internally to keep the per-cpu counters compact. - */ - VM_WARN_ON_ONCE(delta & (PAGE_SIZE - 1)); - delta >>= PAGE_SHIFT; - } - - do { - z = 0; /* overflow to node counters */ - - /* - * The fetching of the stat_threshold is racy. We may apply - * a counter threshold to the wrong the cpu if we get - * rescheduled while executing here. However, the next - * counter update will apply the threshold again and - * therefore bring the counter under the threshold again. - * - * Most of the time the thresholds are the same anyways - * for all cpus in a node. - */ - t = this_cpu_read(pcp->stat_threshold); - - o = this_cpu_read(*p); - n = delta + o; - - if (abs(n) > t) { - int os = overstep_mode * (t >> 1) ; - - /* Overflow must be added to node counters */ - z = n + os; - n = -os; - } - } while (this_cpu_cmpxchg(*p, o, n) != o); - - if (z) - node_page_state_add(z, pgdat, item); -} - -void mod_node_page_state(struct pglist_data *pgdat, enum node_stat_item item, - long delta) -{ - mod_node_state(pgdat, item, delta, 0); -} -EXPORT_SYMBOL(mod_node_page_state); - -void inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) -{ - mod_node_state(pgdat, item, 1, 1); -} - -void inc_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, 1, 1); -} -EXPORT_SYMBOL(inc_node_page_state); - -void dec_node_page_state(struct page *page, enum node_stat_item item) -{ - mod_node_state(page_pgdat(page), item, -1, -1); -} -EXPORT_SYMBOL(dec_node_page_state); -#else /* * Use interrupt disable to serialize counter updates */ Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -8628,9 +8628,6 @@ static int page_alloc_cpu_dead(unsigned /* * Zero the differential counters of the dead processor * so that the vm statistics are consistent. - * - * This is only okay since the processor is dead and cannot - * race with what we are doing. */ cpu_vm_stats_fold(cpu); From patchwork Tue Mar 14 18:59:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18F0CC05027 for ; Tue, 14 Mar 2023 19:01:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A03C78E0009; Tue, 14 Mar 2023 15:01:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B4628E0001; Tue, 14 Mar 2023 15:01:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82DD28E0009; Tue, 14 Mar 2023 15:01:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6FAB48E0001 for ; Tue, 14 Mar 2023 15:01:15 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 11911AB19A for ; Tue, 14 Mar 2023 19:01:15 +0000 (UTC) X-FDA: 80568421710.13.A6E65F0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 27BCB140033 for ; Tue, 14 Mar 2023 19:01:12 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cgmdfe94; spf=pass (imf09.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=eQHLLQ5PDSU2xDgYL5oReXWI8s5CvUHAEWJjGfc5xCc=; b=ZK5ZIeQg8aD8srGMa4/UH7BL+F6+k9Bdqt0/A415ZRhyVLvgcqApKoe4FtYW7DcRaP+Vhn iPVS9g/C7jaf745XGRAeGOeswlcLb/R4loNb72tAs3D8GIyZaE5rq6eWBxwddhZvdLxEfB OFoe39CVs3mCzIS26x9XC2LqfJ6aREY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cgmdfe94; spf=pass (imf09.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820473; a=rsa-sha256; cv=none; b=UUF1hx6nXQYXgQFdVKzKwRu4W9SkiXVgDp1pA1HR5R3WBGxd3RcOlKT2YZM3+x0GrOstf/ fG9poI3i94ABIhcVQKJIKR+nsOliA06Vf20pE4Jih6Eg39wVkrslGBJpES/fuFWUFoeyDF x/lqPHdCeVc6JhhaasJNIGmRLUnDbcg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820472; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=eQHLLQ5PDSU2xDgYL5oReXWI8s5CvUHAEWJjGfc5xCc=; b=cgmdfe94QxyMF7bevKU3GpY00arYt5gAAElP69HeytU73qI71kIj4tY7bhFSoZ8hC0qIXU 1BNkEDIHVfTWIGKqIzWdDGZI6nLgYZyA4sz3HBIxG7lb1Y5tTeFf+DrHYv6zHSBPSiMITy ftdEuPxLDBPfNhzKpo1FnTm+JAQIdWQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-661-LIT0suwmN2y9uMxy61V8SA-1; Tue, 14 Mar 2023 15:01:07 -0400 X-MC-Unique: LIT0suwmN2y9uMxy61V8SA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 38A56100F909; Tue, 14 Mar 2023 19:01:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DB7542027040; Tue, 14 Mar 2023 19:01:04 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 70A8E4039C292; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.729445598@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:22 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 08/12] vmstat: switch per-cpu vmstat counters to 32-bits References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: xwz7obuyefsqt7fytxob4jtgniayjues X-Rspamd-Queue-Id: 27BCB140033 X-HE-Tag: 1678820472-623101 X-HE-Meta: U2FsdGVkX19mQzozKziJm37YbGILcK+fWBshj0iwZqgsaNbJN8vrss6i+4Aepw/T2yZaImM9D01Jx8EoJkgDocAu+Dnl7EkIKG71h8jkohhz2C05mYtl6WxrasjsoyTqkb87XZSwG2l6gayMuCXGMd0c+Vv2C7+IHNxzwRE5tVFuUDay4ckbrOowQ+pZCVgPJYr6CB1018iB2sU6IzFpxJwwsnJnyaOZ2C2GnbI7hnTdMWHrd1FnrxYmK1u2V8l0ZGldbqnisjzRiGsXUG9TVEnAaQJjzJ/BCLa98CIrjMWfoVIbdQ/6yqekvetVPVSm3vjXIgT42PYU9+o/i3FLLe9mQt2bWMB+R/Ap+P8UYDhvvO3ijIAOpzner3CQLJIpQx76zVpf32TkAVaW9qjcwNOU8k1fl6fEMqtWr+S8GkfeRzZrL/HhN3QTERB0pR4Ftlww+dsTUR/oe9p+VCkNv/X7/PbXHDBh/5Oeo72ZN2nP8HPhBAKy485Agkk0FbNr6wubLSvPHunB1/NR008ScgsFQatIclhYmjGElUB/GshW9A+D8kIMdLHJnKoZzHusofsB8agpVdcXIlWYw5FR3vQKl+L/CWoesyS17iD4vDTlnskYvrVMnK4EfIcpobq51ElbGiqOtknYnxcH7AZIrPw9DgAe4W1LAKwcRcbGtMX/G9LYq78BGacUlRYwkUyrDIC1SibqiAv0nEzI+/AE8++i2JhA2yMS1TRXomM4hf8xO8S+PjqxIfPGJ7YzY/NeZJKrFdoZoHXfHHGCGNvtD6Fgf0+Vs7WVBRWBpLH7X+zYdoT+ThbellcnzbgGItB/31fk6O7y4o0rfyuByCGuTOKdNqqI7pjKVrljnI79kimxIaARJ+IsSKlVF1E0HDp15zB4KP9cKgjVH9CBUry4kPCDmqvzjMCALrtwFpefr5DMFR8gGhmTJpT3JLDWRQGGXyu7iXKPOcbzG7lh4P2 3go6t7Lx /IfyY4B/sllscsttdxBjhttmCXuZNjJO3JU55IHp8S1cJJM577DYm1hpwO9f3FHRnT/iM3dzWAbayjTxjb0W3JP8LliXxvdFTH+6D/l0aMw0C7GVRi3YN1Why4N8XPapMyJHijpVonjhmTevtT0E2a/9zVUs1MF29A0INV461JXbsep17qV28H56EcF1A+M4pA2TOtSeSycZ+NNE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures only provide xchg/cmpxchg in 32/64-bit quantities. Since the next patch is about to use xchg on per-CPU vmstat counters, switch them to s32. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/include/linux/mmzone.h =================================================================== --- linux-vmstat-remote.orig/include/linux/mmzone.h +++ linux-vmstat-remote/include/linux/mmzone.h @@ -689,8 +689,8 @@ struct per_cpu_pages { struct per_cpu_zonestat { #ifdef CONFIG_SMP - s8 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; - s8 stat_threshold; + s32 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS]; + s32 stat_threshold; #endif #ifdef CONFIG_NUMA /* @@ -703,8 +703,8 @@ struct per_cpu_zonestat { }; struct per_cpu_nodestat { - s8 stat_threshold; - s8 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; + s32 stat_threshold; + s32 vm_node_stat_diff[NR_VM_NODE_STAT_ITEMS]; }; #endif /* !__GENERATING_BOUNDS.H */ Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -351,7 +351,7 @@ static inline void mod_zone_state(struct long delta, int overstep_mode) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long o, n, t, z; do { @@ -428,7 +428,7 @@ static inline void mod_node_state(struct int delta, int overstep_mode) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long o, n, t, z; if (vmstat_item_in_bytes(item)) { @@ -525,7 +525,7 @@ void __mod_zone_page_state(struct zone * long delta) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; + s32 __percpu *p = pcp->vm_stat_diff + item; long x; long t; @@ -556,7 +556,7 @@ void __mod_node_page_state(struct pglist long delta) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; + s32 __percpu *p = pcp->vm_node_stat_diff + item; long x; long t; @@ -614,8 +614,8 @@ EXPORT_SYMBOL(__mod_node_page_state); void __inc_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -623,7 +623,7 @@ void __inc_zone_state(struct zone *zone, v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v + overstep, zone, item); __this_cpu_write(*p, -overstep); @@ -635,8 +635,8 @@ void __inc_zone_state(struct zone *zone, void __inc_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -646,7 +646,7 @@ void __inc_node_state(struct pglist_data v = __this_cpu_inc_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v > t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v + overstep, pgdat, item); __this_cpu_write(*p, -overstep); @@ -670,8 +670,8 @@ EXPORT_SYMBOL(__inc_node_page_state); void __dec_zone_state(struct zone *zone, enum zone_stat_item item) { struct per_cpu_zonestat __percpu *pcp = zone->per_cpu_zonestats; - s8 __percpu *p = pcp->vm_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_stat_diff + item; + s32 v, t; /* See __mod_node_page_state */ preempt_disable_nested(); @@ -679,7 +679,7 @@ void __dec_zone_state(struct zone *zone, v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; zone_page_state_add(v - overstep, zone, item); __this_cpu_write(*p, overstep); @@ -691,8 +691,8 @@ void __dec_zone_state(struct zone *zone, void __dec_node_state(struct pglist_data *pgdat, enum node_stat_item item) { struct per_cpu_nodestat __percpu *pcp = pgdat->per_cpu_nodestats; - s8 __percpu *p = pcp->vm_node_stat_diff + item; - s8 v, t; + s32 __percpu *p = pcp->vm_node_stat_diff + item; + s32 v, t; VM_WARN_ON_ONCE(vmstat_item_in_bytes(item)); @@ -702,7 +702,7 @@ void __dec_node_state(struct pglist_data v = __this_cpu_dec_return(*p); t = __this_cpu_read(pcp->stat_threshold); if (unlikely(v < - t)) { - s8 overstep = t >> 1; + s32 overstep = t >> 1; node_page_state_add(v - overstep, pgdat, item); __this_cpu_write(*p, overstep); From patchwork Tue Mar 14 18:59:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17640C74A44 for ; Tue, 14 Mar 2023 19:01:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3B758E0005; Tue, 14 Mar 2023 15:01:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C72B38E0001; Tue, 14 Mar 2023 15:01:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B3A3F8E0003; Tue, 14 Mar 2023 15:01:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A19E88E0001 for ; Tue, 14 Mar 2023 15:01:12 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 74D49AB325 for ; Tue, 14 Mar 2023 19:01:12 +0000 (UTC) X-FDA: 80568421584.19.12078A5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 5DA9280025 for ; Tue, 14 Mar 2023 19:01:10 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="K/+c7SNo"; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820470; a=rsa-sha256; cv=none; b=JIeu1NpP53MyXn3s3jA6mPOHFqLkkbmmr4CS7AtHiKlll/HrYDqUjwfjnjmag5AT2+Lt6R Ps01gyJd6mbWnY+5NGyL4L3NTCgCTVHsUZPr1IDHgWC6z4MMylb+JTOSIjtwJvTbt2kRbz F6PCDCKFiNLw3rA4qEAy9PhwpLM6PuQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="K/+c7SNo"; spf=pass (imf02.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820470; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=/KQqYUQDwTZyYbpRZbwn7x1mgss03C//4Fvg72sL2vY=; b=R9lAmuvonPE3PMaCoibwPhbrsC8LN/QzQXcNdR9Diqfp/O4wtMoKkXRcPkItyrY9y6XdXL AXnpFeUDyiFhOXSuCb9iwi7ouJ2zzbf6Fp4TygoB5tYinEMT28w9YG5AkCrGGmKCBDvROC 3CKMxB1AOTMT7J8U7JxQQEVv9g0bxZk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820469; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=/KQqYUQDwTZyYbpRZbwn7x1mgss03C//4Fvg72sL2vY=; b=K/+c7SNoXt6BqJh7A8DAgfMCsqZSI0eX+n6Woc8u9ohaWgQw45S06o3JBn8nDasoN9K0TN Wn/MVEVZUpAd9ZiOw0KkSPS7fxkHDALLZ2IHI+LtFWQ3AYuWLIs5bMGVsxqVK+WJ+DD+C3 A7Oo6IAZzGf8891wdt1efqJ2UNJAnNA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-416-vzkJw1CyPTS8_TahwjoFcg-1; Tue, 14 Mar 2023 15:01:06 -0400 X-MC-Unique: vzkJw1CyPTS8_TahwjoFcg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 136261024D28; Tue, 14 Mar 2023 19:01:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DB2631410F1B; Tue, 14 Mar 2023 19:01:04 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 7585B4039C293; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.754032715@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:23 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 09/12] mm/vmstat: use xchg in cpu_vm_stats_fold References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspam-User: X-Rspamd-Queue-Id: 5DA9280025 X-Rspamd-Server: rspam01 X-Stat-Signature: h1ktyx9qx7qkqrg1q3ncqo1y1wm7qbij X-HE-Tag: 1678820470-610940 X-HE-Meta: U2FsdGVkX1/Lm44lllO4t2p8BpB1dH66dhqGCGz9VISVy0Zwwafh3kfluCJqs9gH9yLjgWnDuznP2GxDSgegz/JAeHle/Yvl4hr3b7Zd/xmgHtpzo9OHNAfsDie6ZXdAi9EaTGDj2KKwKktN8kV6mJf40F4tSfUed5mOwe6mgrIJvKIVCpeZgvBFmFu2RwdxfW5m0wBTiLunp38VS20cTFDO0wExPkwrvMDArzo+T4Jes0slwqv4XVhWv2advbsm+RTuMxgRx8XuPSZD0imxmr+l9KG6xFo7CL23zpoH7T8MAIk5ld3f6d6m4J0xZt/XDpmZssc6h6uSnYfPmCDA/dP1IQgA5Jl0BusMfwxCoDVZi7Rl90j7EePNPf06A5W0E0G4h3rAUcdXYWEsXIK8TLDMkkJY7wi5A20t1bvu+dnzq4kNWXDI+o2LjWDzxjDJHNrEvrugpfMJ20+t2fwLLs5v/qWWFv5gmYkJ6HYBY9Nq9RgMwLNUN9xtsSLXAw/yKhRxGXS10PPLAl5nErjIk+ppN8Cgcw6ET62GKwHTH2ZxV4p0Xo7ex9B/H81P112FOwpaCR5FdHUhfqkxiKNW5/u2OjzfcQk+MkR+XDKZZuDcoC+YSn9SvtWLuxzBBXg0ytBjPvo8hroYWx07Y2Mj0io77vCq19uWH0sfSXfDJ0H6/jW5PrAGdGseQK5wK9XEIKafJttaoQ1ore+X94C6z9l2IVxRxhYAmhIg5Na9DvRrGQyCeIsgiW9gjAJAMk8cQ6923E8Le0gI0q/MI7IbS0VGXTSf2+2MRXfkkqrdAQwU1iVLhp0GJI2Qj2Mfqo7BOsZPUuCv73iA3fzdNPKk5MH0IGHQirrGUl7vNWiepMxGIm21IjL0vXjP5ifXDnfKlTkjfommKxG3P1QBqqft/rN6ge7O/yw/mgkAb1ip6FIlVOyplJ8+qSpihjMLgeDBdNoxEFEBHMl0dgknx3h f+vTtO53 131OAdcky027Ljnweg+GKPFcPN7lOgE8QhCnArFMx4D8DkbA6hxadtUGfRn/lPUZItUMXJs5eXx6vhbN4UJy3w4Uq3nEpqtRkDx/yMBpv99LIoEhI9s1YmzlaMp7GgJNiVJQgjLRqfWKbFqlCszXgXEhHilBJT2ufMj8Hc0P31xqmyWQEY8lnBzQ7p+UoYFubK14mp/8BWSwfUHC8qxzQscyBQ7uOZqshRYM45PrWOLkZE62/OAagYQO3IlOjoXOI5Iv1jPucOrNg0G6k6drzfgf9Nw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation to switch vmstat shepherd to flush per-CPU counters remotely, use xchg instead of a pair of read/write instructions. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -924,7 +924,7 @@ static int refresh_cpu_vm_stats(bool do_ } /* - * Fold the data for an offline cpu into the global array. + * Fold the data for a cpu into the global array. * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ @@ -945,8 +945,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_stat_diff[i]) { int v; - v = pzstats->vm_stat_diff[i]; - pzstats->vm_stat_diff[i] = 0; + v = xchg(&pzstats->vm_stat_diff[i], 0); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; } @@ -956,8 +955,7 @@ void cpu_vm_stats_fold(int cpu) if (pzstats->vm_numa_event[i]) { unsigned long v; - v = pzstats->vm_numa_event[i]; - pzstats->vm_numa_event[i] = 0; + v = xchg(&pzstats->vm_numa_event[i], 0); zone_numa_event_add(v, zone, i); } } @@ -973,8 +971,7 @@ void cpu_vm_stats_fold(int cpu) if (p->vm_node_stat_diff[i]) { int v; - v = p->vm_node_stat_diff[i]; - p->vm_node_stat_diff[i] = 0; + v = xchg(&p->vm_node_stat_diff[i], 0); atomic_long_add(v, &pgdat->vm_stat[i]); global_node_diff[i] += v; } From patchwork Tue Mar 14 18:59:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C217EC6FD1F for ; Tue, 14 Mar 2023 19:01:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D8468E0008; Tue, 14 Mar 2023 15:01:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 262E08E0001; Tue, 14 Mar 2023 15:01:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 128C18E0008; Tue, 14 Mar 2023 15:01:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0516C8E0001 for ; Tue, 14 Mar 2023 15:01:15 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BB520160D9C for ; Tue, 14 Mar 2023 19:01:14 +0000 (UTC) X-FDA: 80568421668.24.ED242BE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 3307E16002A for ; Tue, 14 Mar 2023 19:01:12 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="dyT/QMqq"; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820472; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=Nr9fYbpF3v+q6jpxCLVFeOy0r5BagqTS2hiKgCt5CBQ=; b=Y+F3cag9ySJWN6yC3pZX1TN8VKV5Q/tKmQ7Dm/e9HFhPEKgZONAenif9yh2Zhx0Us76DRd 5c59yAdGHeSE8RLbk9kVCmfWu2I7lrejYgYw9sXuohJq82ZP6Rt9CghG00v24HoupId/P9 +ilimtyRwN81reqCHrl6NS0ac1LOdAY= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="dyT/QMqq"; spf=pass (imf08.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820472; a=rsa-sha256; cv=none; b=NRgY2YAplvCGQSnpI8jhHG47itCpmqqrhZjlR7kKwlK6/oFBo5aXrVbHxFX6N6/bB2Vi4i L6SMGcsza0QrKz+FWdonpgCYmFjeaO6vYTqRjRnxUtyS0h3yzAnCWsIVwNvdJ73v6YEntP FND5gqOrgWhLBe44EgKgYEkOLQxQIP4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820471; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=Nr9fYbpF3v+q6jpxCLVFeOy0r5BagqTS2hiKgCt5CBQ=; b=dyT/QMqqWnMd91A1deuQH4Q9NLmWHZ3GRY0GMvAxruDa2c4/CxoC/+fALfgzSjIZaopxao XSTiXnQTCgOuR0tt7hHc53tfdCHy1fL06KWeylCHJ3MOF7MzLeA0VrrQdS8mDHD9wErvNn kEHoedpS6CJ7Zjs4eVGeLSvVxwXYEXE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-117-FUxLZ_pZN0q1wQRTUiKORQ-1; Tue, 14 Mar 2023 15:01:06 -0400 X-MC-Unique: FUxLZ_pZN0q1wQRTUiKORQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1088728004F0; Tue, 14 Mar 2023 19:01:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D9360C164E7; Tue, 14 Mar 2023 19:01:04 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 78F4D4039C294; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.779596601@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:24 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 10/12] mm/vmstat: switch vmstat shepherd to flush per-CPU counters remotely References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3307E16002A X-Stat-Signature: txibzy34o87pk6rstfcgcw61wc54wr5d X-HE-Tag: 1678820472-815001 X-HE-Meta: U2FsdGVkX18ArmtTe6GRbgicAo7P2rgNLOrsS3sigYvGyoTXlGaWTTaBGth6HrJ5qsXUEnrAV/Mnr6vD0s65vTofP+NVjru292KNhFN/lmaiI6HVPVxt/wmW1JeySJDOpTxdpgtWtPQ6NCKI5HKpTsDjKnu7Iv267tAUT/AL6lrdzxO1bHW2PP46FhKMyNCEy1xRLReKBUo1OluXEh1xnIogqqFExWzb7f5qiESHwrFZoSgOTpGiTATFl91fgfUEKJiZPGJdXg+DvBsOOspYGiOFxoO1UD7azwbzEYsBQ4CikyxBrPXMkqKHxXdA8rlKSm30JoBkisVzJQZliT09K4CqJ8hcLe4KsNl7n9mjPBuFcNq2zwyVFym2U0WMUusbtBX/LyvjUy5zNe7w240bvRWhtOgVWkkuh1aK6kn3Ee+RuR7NRkEv55ggH2zhpS0HoYdsUkG2jtxnyu/Wjk31+ukFvtKV3bNsmL7581lRhKfbTlHluMw9PK9eN453S5C4f18zOeVVxf0bIPz5Bqj9wWKNk114qUVh4fkJr5md4kCwtJC+cYFQZVl+BreBipfA1jQ0JMt27SavmoSzPjX8AlgM3AFAGqyacqkIp8DJJS3RIhqpRbu680BbrIeqkX/BlZO5DqhaCZ2ZcnrrxsatUc2Ihndqf6gCOg9FkmpIFjfZDKeBYvA2yc80jo+13FqcURi/8guRuksx8FC+cLe/XJUbWJ+kDaEBka+geqvAz2C8Ybu3/LKI+otDOqZTE8eqHJoWl40sispqHrM5AjIixgv32wtMa8VBAbAMayvHaUvthifEElYX4wHZiMPdHxaUwsUFpax1pk4Sj+373Vtu/x7ttqMtklMyAw1Sl9jCSak3E0I58jDT+LBcg12hL8oTbCxEZ2CHSIZk45Cn1yvTWnGK70Wf1zrQ8/9j8Kh2ycNJcZLk31+s5n32y9Y9KnUB367T4Oi4y/Mcc0cmHex rNXiVZdq XcqWb3ez2OKEJ77GLGjyzQ2D/iPsiNKzbydpPDl2B5HsCnzBfDuP1F6fPY8WtOylwnKsyr2hIJNXbD6W1LBlbhMgWfPVkMKgtmGcCCEty2/nAOeP74SpKWoCdPKTrSJTWLCQjoAh4dCDadVW7kyRQ+H+76bZZcqgbAy8INQ9/hZFslhfCEm7Ul5gD73b6XEY1T2iq0IV1dA+Hnnw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that the counters are modified via cmpxchg both CPU locally (via the account functions), and remotely (via cpu_vm_stats_fold), its possible to switch vmstat_shepherd to perform the per-CPU vmstats folding remotely. This fixes the following two problems: 1. A customer provided evidence which indicates that the idle tick was stopped; albeit, CPU-specific vmstat counters still remained populated. Thus one can only assume quiet_vmstat() was not invoked on return to the idle loop. If I understand correctly, I suspect this divergence might erroneously prevent a reclaim attempt by kswapd. If the number of zone specific free pages are below their per-cpu drift value then zone_page_state_snapshot() is used to compute a more accurate view of the aforementioned statistic. Thus any task blocked on the NUMA node specific pfmemalloc_wait queue will be unable to make significant progress via direct reclaim unless it is killed after being woken up by kswapd (see throttle_direct_reclaim()). The evidence is: - The process was trapped in throttle_direct_reclaim(). The function wait_event_killable() was called to wait condition allow_direct_reclaim(pgdat) for current node to be true. The allow_direct_reclaim(pgdat) examined the number of free pages on the node by zone_page_state() which just returns value in zone->vm_stat[NR_FREE_PAGES]. - On node #1, zone->vm_stat[NR_FREE_PAGES] was 0. However, the freelist on this node was not empty. - This inconsistent of vmstat value was caused by percpu vmstat on nohz_full cpus. Every increment/decrement of vmstat is performed on percpu vmstat counter at first, then pooled diffs are cumulated to the zone's vmstat counter in timely manner. However, on nohz_full cpus (in case of this customer's system, 48 of 52 cpus) these pooled diffs were not cumulated once the cpu had no event on it so that the cpu started sleeping infinitely. I checked percpu vmstat and found there were total 69 counts not cumulated to the zone's vmstat counter yet. - In this situation, kswapd did not help the trapped process. In pgdat_balanced(), zone_wakermark_ok_safe() examined the number of free pages on the node by zone_page_state_snapshot() which checks pending counts on percpu vmstat. Therefore kswapd could know there were 69 free pages correctly. Since zone->_watermark = {8, 20, 32}, kswapd did not work because 69 was greater than 32 as high watermark. 2. With a SCHED_FIFO task that busy loops on a given CPU, and kworker for that CPU at SCHED_OTHER priority, queuing work to sync per-vmstats will either cause that work to never execute, or stalld (i.e. stall daemon) boosts kworker priority which causes a latency violation Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -2043,6 +2043,23 @@ static void vmstat_shepherd(struct work_ static DECLARE_DEFERRABLE_WORK(shepherd, vmstat_shepherd); +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +/* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ +static void vmstat_shepherd(struct work_struct *w) +{ + int cpu; + + cpus_read_lock(); + for_each_online_cpu(cpu) { + cpu_vm_stats_fold(cpu); + cond_resched(); + } + cpus_read_unlock(); + + schedule_delayed_work(&shepherd, + round_jiffies_relative(sysctl_stat_interval)); +} +#else static void vmstat_shepherd(struct work_struct *w) { int cpu; @@ -2062,6 +2079,7 @@ static void vmstat_shepherd(struct work_ schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } +#endif static void __init start_shepherd_timer(void) { From patchwork Tue Mar 14 18:59:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F124C05027 for ; Tue, 14 Mar 2023 19:01:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E5348E000A; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD2558E000C; Tue, 14 Mar 2023 15:01:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B89BB8E0001; Tue, 14 Mar 2023 15:01:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9CA4D8E000A for ; Tue, 14 Mar 2023 15:01:16 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 43261160D99 for ; Tue, 14 Mar 2023 19:01:16 +0000 (UTC) X-FDA: 80568421752.25.78997D3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 0BA718001B for ; Tue, 14 Mar 2023 19:01:13 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=I0moCVyX; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=BjYqqe7MvmRoNdsZ3mRmEvGJpIWPHc0RoB3EQFJD7pE=; b=lhj3IWSZuEs/JMJM0XIlknwlVBbUg0Rg+4M7Wf596y4WEsC9vqdfjF97zmJH68/IOcfGPV QzjFRyj17PNaP7ROTz8BIa8KKp8kDD+1/8k3ffhqMU+hfU533auHgz0LEmkk3IZMB75eb0 E5Kv6s4EJ0xtPHHcksuE8JztEgno37M= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=I0moCVyX; spf=pass (imf30.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820474; a=rsa-sha256; cv=none; b=tyPp18chmn0b5nuPcVGDB5F/ZrU3YGZ2+Arh9b+mk5ksl1yR7dIO8QDdC8DDxR+dcCQ881 AKVNNitvwkmOl89JbvYKmLjT2L4dUphZIH37hxD6zNA+sV5SE5eGk+AVFZ+Ro8rEbhXF9O iJs+eoAPCPeyZ3Qw4WmaTDY11pJ6fW4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=BjYqqe7MvmRoNdsZ3mRmEvGJpIWPHc0RoB3EQFJD7pE=; b=I0moCVyXjV+b7xtxbNQCPA5LLWOUywDzsOrppZbpHhZdGaqcBHITxVL0TqJOI/tGcMle71 o/nTsn+Za+mgYwfyDQku8QwpyJXAyqu1tMajh1GdbIw+t/5/gX4z8k25LjvJreTBmDoMj4 u+qcgRmEP+1v5T0/csRgmHBi2x+OQYE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-661-eTB4kV2KNn62pW4FAutmSg-1; Tue, 14 Mar 2023 15:01:07 -0400 X-MC-Unique: eTB4kV2KNn62pW4FAutmSg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 30F6B18E0046; Tue, 14 Mar 2023 19:01:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F26262A68; Tue, 14 Mar 2023 19:01:04 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 7C4E04039C295; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.804121555@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:25 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 11/12] mm/vmstat: refresh stats remotely instead of via work item References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0BA718001B X-Rspam-User: X-Stat-Signature: wu4frpjwzyw6hqbskbxk8azhx1yrddn5 X-HE-Tag: 1678820473-120386 X-HE-Meta: U2FsdGVkX1/zCgvQCcd/vcNM0/+2R+qfqy5p29WKdqK8n8QdGKYZpgecnG+fT5ljAA4uaN2h3Z2qGJiPeotnMwl1H+Kad0Hic56I3p+UDc3q+tudBb9nG5DFUvo7637DvdkDPqP40uxyot4fDgmuGCPixJbMbkY6fXaTZC48vxm/C8gkbzQd/2vINKuNyfQiLOTrGA5UM+jV0ysBtdIiNi/TIQIl6vIzsqywtwAc2AWzSM/VKqrDQxfoW45dpyLaMVEEDdiUyHLQWZhdw3Z92XsrSgeOfkDUxJ78VB7D7IzlhoZYuUkc1w8TQ6CSLYrb0JiHaqqR4gMqg4y/44CZbMeNM7x0tF6A8TxvAxTCddxm04MOwTDAuguUkxNpFFz/X7QcgamlCrIwr2x0M0eNVIymciVUtBTcRQCrpOyEksZmhclBbMqDLO9BH4pZboA0S0tGSMYQdMZzHYfa3FCyGvF0V/bP3WotvNtjM/H+d8wabZSnCSiKpHUh3JaWR+gSUSjUzC45HM3zI88tIbqfPMbw1LZNGEyp39ZLMB5AUMRGLTahVDb8h9VoTEzWuCGoxTJ8ygE+ZaoVq84OGSgP6HMagbtpx2Xmo5lnw1TuSso3VgxyOhldgE97l8ToRkM1dPc/OXDHsjv4pN8+CWxkMhwVjMIRdCRdxrxmheZ9Sq8JxZDtj7/sQAaTBb3Z3kCmG36kxcf7IjmcUxjyw9Xh7vBRRVLsh/IrVDIdjHD6ovtySG6zCyurnjErxYILUJURoHSgjBnjC9G1sMlZ6IQ65FZHss2AP0D+zn+QzvCeZyssChNVvEECSqJ9DSR9wF9qI8ekNyjT5n6LxHoqc9h7tYWY3WDVuNypDu4cpuqgM1tHNobMYQU8od0MmPmUhllaiQkM6emzBv/whDkLL/yP3g5KLwcYtYEnWplKty0tlUfUfQcnSaBAGoVm8MMIRU4bEzyc+b6G1P3E67S9a42 v2lYeVe6 qaOXgNryRGLlfa2/G9Cozw8ROGM29axOJ/GIR0OmD7Ku5YfhKNnJYrBPir4PkTP90THf/CFZ5tdmTZl1NPwyWxVAwTWOy4q6wtNLY3JDgHMXnIHhih4vDiUJ+47/yloolGyqer7/jsTNv6qPvPvBqKJ2Mrvp3KbcJxqD/I4CE5TZbTZyOSb2Va2MDltg3nT4ZSy0hfvzKQGscAgA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Refresh per-CPU stats remotely, instead of queueing work items, for the stat_refresh procfs method. This fixes sosreport hang (which uses vmstat_refresh) with spinning SCHED_FIFO process. Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -1901,11 +1901,20 @@ static DEFINE_PER_CPU(struct delayed_wor int sysctl_stat_interval __read_mostly = HZ; #ifdef CONFIG_PROC_FS +#ifdef CONFIG_HAVE_CMPXCHG_LOCAL +static int refresh_all_vm_stats(void); +#else static void refresh_vm_stats(struct work_struct *work) { refresh_cpu_vm_stats(true); } +static int refresh_all_vm_stats(void) +{ + return schedule_on_each_cpu(refresh_vm_stats); +} +#endif + int vmstat_refresh(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { @@ -1925,7 +1934,7 @@ int vmstat_refresh(struct ctl_table *tab * transiently negative values, report an error here if any of * the stats is negative, so we know to go looking for imbalance. */ - err = schedule_on_each_cpu(refresh_vm_stats); + err = refresh_all_vm_stats(); if (err) return err; for (i = 0; i < NR_VM_ZONE_STAT_ITEMS; i++) { @@ -2045,7 +2054,7 @@ static DECLARE_DEFERRABLE_WORK(shepherd, #ifdef CONFIG_HAVE_CMPXCHG_LOCAL /* Flush counters remotely if CPU uses cmpxchg to update its per-CPU counters */ -static void vmstat_shepherd(struct work_struct *w) +static int refresh_all_vm_stats(void) { int cpu; @@ -2055,7 +2064,12 @@ static void vmstat_shepherd(struct work_ cond_resched(); } cpus_read_unlock(); + return 0; +} +static void vmstat_shepherd(struct work_struct *w) +{ + refresh_all_vm_stats(); schedule_delayed_work(&shepherd, round_jiffies_relative(sysctl_stat_interval)); } From patchwork Tue Mar 14 18:59:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 13174935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6D1FC6FD1F for ; Tue, 14 Mar 2023 19:01:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF19A8E000C; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEB428E000E; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F2758E0010; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7B8AB8E000C for ; Tue, 14 Mar 2023 15:01:17 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 40B07140307 for ; Tue, 14 Mar 2023 19:01:17 +0000 (UTC) X-FDA: 80568421794.27.28F54AC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 92C66120038 for ; Tue, 14 Mar 2023 19:01:13 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dtJQ9sBk; spf=pass (imf29.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678820473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=s8sJ/cpBUh6LGMZ+lcAYSBuSqrBbcVYVRwc+yqWmsjw=; b=6ZYQvQV6K9WhBeisdiHj6fIMV8hSK7MYRLhxBwN0nCi2Iy0L2weqxSQHe8wxPT7rwcaaPO 1E7cy/c5D6e4pQyrfhE/StKuqM7CnG+dUQy1XRzfKYMLy5x1vQu1Dt6VTBk6AD6JFfYGRQ VH1K28B8oe+2D/4GFA9Gx/vLbrf8O8Y= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dtJQ9sBk; spf=pass (imf29.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678820473; a=rsa-sha256; cv=none; b=IYKzxqnkEC+Gdax7ESMgZJYLXqLHOcX6Tu6woed5pB+31UUNMHddmOGwqe+vhv82ZaBMFM 76Ch1CiAz/fPn+Cd3l7/L7ZJORt+XDCF81aBN3MdRdfv5dlt2hjgz6SVBDDKFCtQtNVhV8 HuUJBT0AeK7SeKKwgu1iBjvSmTVSP24= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1678820473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=s8sJ/cpBUh6LGMZ+lcAYSBuSqrBbcVYVRwc+yqWmsjw=; b=dtJQ9sBkE9yY1t03lxbdIvSA9pHjGCSOhl3632/cVIAytga51RspzXxb0JZj05urN49b5e TV7977WxBojJh2XUCk/rJdhpV91h7jvBzN5HRnMcbnaOAYkzwNUqg1Jfrl621myD5GXfiz Av4Yugxj8RYyJ7wXphIK9kj5uLVFXpg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-287-Jb4Ci1vOPbS-aOjTw-mf6g-1; Tue, 14 Mar 2023 15:01:06 -0400 X-MC-Unique: Jb4Ci1vOPbS-aOjTw-mf6g-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 30E6128004F3; Tue, 14 Mar 2023 19:01:05 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 07BDF4042AC1; Tue, 14 Mar 2023 19:01:05 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 80AD44039C296; Tue, 14 Mar 2023 16:00:44 -0300 (-03) Message-ID: <20230314185951.829632491@redhat.com> User-Agent: quilt/0.67 Date: Tue, 14 Mar 2023 15:59:26 -0300 From: Marcelo Tosatti To: Christoph Lameter Cc: Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Russell King , Huacai Chen , Heiko Carstens , x86@kernel.org, Vlastimil Babka , Michal Hocko , Marcelo Tosatti Subject: [PATCH v6 12/12] vmstat: add pcp remote node draining via cpu_vm_stats_fold References: <20230314185914.836510860@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 92C66120038 X-Rspam-User: X-Stat-Signature: mu585jmquugtmh4hxdsnobcgpbg56yd8 X-HE-Tag: 1678820473-630458 X-HE-Meta: U2FsdGVkX1/Ot+YdMEFliCpImXLT5/oYvRuY3aoZb0P8rP/i9E8S0Wx8lk07YJ1acPHft5GP50rAAhQXAFLjYpehnjWQ0jZ6rtFZTJ1CnuwVKUNkl89YXlZ62LqYuHTp403v/1tfqMZuLUo7pv28NCCJHEd2LYYmgRMXRNCqFBPn+6geOobLSalqZIi66n5CPqPi3Z3zoYKBWWiRZ8/LqIRZqoIojbRE46HjgdBwgByzUbWgwJC0UhLRZER5KcfIynQBuK2KyshQh2OvgaoDlqchNBKGoDH2ZCh62isw8IZ9kvzJSLsSJB6tiNi5cmf/5+kqVBdZOuQleJaTej/WnjgmdznoeUEpdZDwU3DWnEHG2UrU7ik04I9YetXTlICyxEb4g7HyDxGV7suTbxzQcWZWETGNV02XvMcJ0PrANj2R2r+aKPB1bkZWb4eTK/qwJFqHDSu3RtbElC7a8QGs1a/9j3RDWFdbP9M76F3CBBt8LRR6ZjcqY+uVvYvnKJ6huAqLIumOigSy2qxisiQYqLJB8K0JMO+/EH8h/UkXJ9dICoyUHRvsW2CXQmmfwzANrEPD2z3YMnmc97QFfU2BwNw33SmhgkZLxrh6fReC0XOmOc7AAGiuE3H0zA5WQTJXdU0h37fo7boZas3LDfKqwRmd9+9NQujYoE/rgZfXC/Y2DIFNL39SVAtBMic1IQ6qhxkj9Qp1BEpLRpUTAaZrMXro842jJfnySpc9otqNTWmDdDg+OhE/sKGmZL5DfJP2WuZWlJ9VxRFCgN1dfSY5mFEPYO2SzjEy7APejzRdOhSFPVseQ1vwXwhCcn1ffeogjaYnNBH3XxYFF5iZBhNJwyO74NhI6gnoBCzwtodCSJJiRBFarOqujRJT/Ta5xfP1a+0reu6ce0ZM98ujUs7DHt9Bh/OAgL+gLly57d+Qna94fgb2+doVOoyOwJLiOhjbtXPMhPdpQHzKTRzzRG4 MESdOCBE ue/XyVS9R10XnHsUiEwpcfqOZXho5tf2cMlS8lYqoIKvV574WuGfWoTbsODAMeUChmZs7Ms5JZQESllrGQ24ZiKbav6aqTtV5aefXvCCwZfIAxP7zoArEseC3V1kgAN8/8nGNvTq3cTZhiqzhCdC7Z4txv/p0vUDJVsO1vTaFrHFXTlvRFdWHm9Xp6nx4nMd1ILtuANGR29c3sCg6omyVe0vR7QFSw5/sMj78p2Rrp8SOR/GgX6Mp8Y7ddQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Large NUMA systems might have significant portions of system memory to be trapped in pcp queues. The number of pcp is determined by the number of processors and nodes in a system. A system with 4 processors and 2 nodes has 8 pcps which is okay. But a system with 1024 processors and 512 nodes has 512k pcps with a high potential for large amount of memory being caught in them. Enable remote node draining for the CONFIG_HAVE_CMPXCHG_LOCAL case, where vmstat_shepherd will perform the aging and draining via cpu_vm_stats_fold. Suggested-by: Vlastimil Babka Signed-off-by: Marcelo Tosatti Index: linux-vmstat-remote/mm/vmstat.c =================================================================== --- linux-vmstat-remote.orig/mm/vmstat.c +++ linux-vmstat-remote/mm/vmstat.c @@ -928,7 +928,7 @@ static int refresh_cpu_vm_stats(bool do_ * There cannot be any access by the offline cpu and therefore * synchronization is simplified. */ -void cpu_vm_stats_fold(int cpu) +void cpu_vm_stats_fold(int cpu, bool do_pagesets) { struct pglist_data *pgdat; struct zone *zone; @@ -938,6 +938,9 @@ void cpu_vm_stats_fold(int cpu) for_each_populated_zone(zone) { struct per_cpu_zonestat *pzstats; +#ifdef CONFIG_NUMA + struct per_cpu_pages *pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); +#endif pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu); @@ -948,6 +951,11 @@ void cpu_vm_stats_fold(int cpu) v = xchg(&pzstats->vm_stat_diff[i], 0); atomic_long_add(v, &zone->vm_stat[i]); global_zone_diff[i] += v; +#ifdef CONFIG_NUMA + /* 3 seconds idle till flush */ + if (do_pagesets) + pcp->expire = 3; +#endif } } #ifdef CONFIG_NUMA @@ -959,6 +967,38 @@ void cpu_vm_stats_fold(int cpu) zone_numa_event_add(v, zone, i); } } + + if (do_pagesets) { + cond_resched(); + /* + * Deal with draining the remote pageset of a + * processor + * + * Check if there are pages remaining in this pageset + * if not then there is nothing to expire. + */ + if (!pcp->expire || !pcp->count) + continue; + + /* + * We never drain zones local to this processor. + */ + if (zone_to_nid(zone) == cpu_to_node(cpu)) { + pcp->expire = 0; + continue; + } + + WARN_ON(pcp->expire < 0); + /* + * pcp->expire is only accessed from vmstat_shepherd context, + * therefore no locking is required. + */ + if (--pcp->expire) + continue; + + if (pcp->count) + drain_zone_pages(zone, pcp); + } #endif } @@ -2060,7 +2100,7 @@ static int refresh_all_vm_stats(void) cpus_read_lock(); for_each_online_cpu(cpu) { - cpu_vm_stats_fold(cpu); + cpu_vm_stats_fold(cpu, true); cond_resched(); } cpus_read_unlock(); Index: linux-vmstat-remote/include/linux/vmstat.h =================================================================== --- linux-vmstat-remote.orig/include/linux/vmstat.h +++ linux-vmstat-remote/include/linux/vmstat.h @@ -291,7 +291,7 @@ extern void __dec_zone_state(struct zone extern void __dec_node_state(struct pglist_data *, enum node_stat_item); void quiet_vmstat(void); -void cpu_vm_stats_fold(int cpu); +void cpu_vm_stats_fold(int cpu, bool do_pagesets); void refresh_zone_stat_thresholds(void); struct ctl_table; Index: linux-vmstat-remote/mm/page_alloc.c =================================================================== --- linux-vmstat-remote.orig/mm/page_alloc.c +++ linux-vmstat-remote/mm/page_alloc.c @@ -8629,7 +8629,7 @@ static int page_alloc_cpu_dead(unsigned * Zero the differential counters of the dead processor * so that the vm statistics are consistent. */ - cpu_vm_stats_fold(cpu); + cpu_vm_stats_fold(cpu, false); for_each_populated_zone(zone) zone_pcp_update(zone, 0);