From patchwork Wed Jan 25 07:34:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13115045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D4A6C54E94 for ; Wed, 25 Jan 2023 07:35:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B18E06B0073; Wed, 25 Jan 2023 02:35:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AC9296B0075; Wed, 25 Jan 2023 02:35:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9440F6B0078; Wed, 25 Jan 2023 02:35:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7DDA56B0073 for ; Wed, 25 Jan 2023 02:35:50 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 439C6C05E1 for ; Wed, 25 Jan 2023 07:35:50 +0000 (UTC) X-FDA: 80392512060.09.483247D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 2CE4EC0011 for ; Wed, 25 Jan 2023 07:35:47 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WHDFeTJF; spf=pass (imf10.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674632148; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=fnKT1MJD+ucJIxHKvxLLC0LjfVH19PUsoy+1ammqfUQ=; b=R5mNXW/q+M8ez5a9IhOJ6ugf4hdfGNduAkzEarCRmLmxf0rTkEb206letrb2zXdYCAXJkx jJ7RU63fpCmcQ04bcwXOd+WOspWWC7axFDenuTWiDXz7hxDVHLyhyD94fVnlYPG/RE3hNT IMLneKES6xadM94GDphOFXz+EK77ZLU= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WHDFeTJF; spf=pass (imf10.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674632148; a=rsa-sha256; cv=none; b=Ad1fwIjcxIzEqmAKnJqVI6QyMF6wLbnwOYrVKot9B1cwBjMOACcpluHl9K2OJYG/IA5ZfA u/R9oBQd2uR/4Atse2yGEhW8btvZu5ijIvbwGEnISoznqZoC6DapwwEmaF5B/6lPAA7Nnz By+oolvBtUrZpwWbI1/+aGbeJwlNepE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632147; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=fnKT1MJD+ucJIxHKvxLLC0LjfVH19PUsoy+1ammqfUQ=; b=WHDFeTJFBlfz0CicRG5NH5PHDWKHtgewABqGF9oDZEDDpM3yoE9g2nqpZsVxpwH52nV1CE leYjS+Uv2BHnVUBmvFJSrYfQK18nx8RfMaO+pTZRMxmUTdKLCIfkAo5v3VCEF+8HEjrpF+ sOGwNqFq+8u7dOCC9q4JjozdFNE2R1M= Received: from mail-oa1-f70.google.com (mail-oa1-f70.google.com [209.85.160.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-385-USYUjizGPSW15RkJZAhAJA-1; Wed, 25 Jan 2023 02:35:46 -0500 X-MC-Unique: USYUjizGPSW15RkJZAhAJA-1 Received: by mail-oa1-f70.google.com with SMTP id 586e51a60fabf-1631c656f97so1570954fac.1 for ; Tue, 24 Jan 2023 23:35:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=fnKT1MJD+ucJIxHKvxLLC0LjfVH19PUsoy+1ammqfUQ=; b=lkbuG19rcz5QdbOGDwrjxbhlFkFZ3h+nkFOGV4u1vB/tEDmPGdj78zYZPCZ3EB3X/U 8cJqdLnLAWvAF/JyOdBTJFUPsLbDyrMrzM/1ouDD34G8vkuunsD48RzwR0i2+XJHToNf U8evgXUDubyRfS+ilNI0AqUJZ0GHlUExJu0a2B6pRRCFQsT0EMCDMhql44X4FLuKeI9m vyDsscQmUG8vvnEPKRjxI1Vz2JxUSEt46zyNs8HNIaQy4uAVuOoyDO6AuhTtgxSZKzAT O31lf/WEkyCeP6p+sL4GPC5bUkct48GrOohkmAhQv/0nW9yZmTiY+z6QkgZQt3nvaobH QsBw== X-Gm-Message-State: AFqh2kpkOFLNMG7rwYKgNDDc5J+tNbF2Y7fXXoe35OB4lV3H3WraQ7Rd +9OE0RX0eHvthabCLd2fRrS7hMSzP8pPFTXMAkE7q39CsmbTYgcZuvfSWsYxR24QXUNklPmXjhV 9WZZAjd9A9x4= X-Received: by 2002:a4a:b542:0:b0:502:a732:f8f5 with SMTP id s2-20020a4ab542000000b00502a732f8f5mr7489704ooo.5.1674632144803; Tue, 24 Jan 2023 23:35:44 -0800 (PST) X-Google-Smtp-Source: AMrXdXvlSfi5XHt02LX8y2cukx3jlCSkr0tODzZuof9xvw5qoVzLqkPz/nSgbEX8rYko34VuDssNUw== X-Received: by 2002:a4a:b542:0:b0:502:a732:f8f5 with SMTP id s2-20020a4ab542000000b00502a732f8f5mr7489692ooo.5.1674632144483; Tue, 24 Jan 2023 23:35:44 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:35:43 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 0/5] Introduce memcg_stock_pcp remote draining Date: Wed, 25 Jan 2023 04:34:57 -0300 Message-Id: <20230125073502.743446-1-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: 4hu715sjn5gw5818uxa88q48byfpdz98 X-Rspam-User: X-Rspamd-Queue-Id: 2CE4EC0011 X-Rspamd-Server: rspam06 X-HE-Tag: 1674632147-441548 X-HE-Meta: U2FsdGVkX1/yorqRdkDqVohaxb1nAdtS6FVUJS84gJEg0SE6I+jag3egOqXQF+R1HkOIqs02dvUGgkUhJ7nY/CeJx8rG+TJL0i96X7+cvhSPBNL4+HGe/nc60VLMJ8p5tUbeun77lJjL7LWfWCoEBNIz7LJDctVmo0t8q9zlAuiPqMRo71RNWBtzx8CqUJStvGD1LxVdrGg9msvsr8FvQ+dR8qWPGb5jO5pDteUEtSe5Zzto2RsvnOFIwe65mVgfO1cNt16cLCK/fK3SmuRxZ4keCUaEUfIajn/65y+fF6V62xgJYUAyaRcPeH1ZW3I248lAlo5vhyyxb88jqg+K+x+D5ETBbS7duKZzerBxizc8xTqvrdVrJJxoX7cjOQJ9oIHiUr00CPUWHW6O0KhdCVymKp3gsb2bLgq+ZaYyfkK9NbkBalAFVQdxxKq5s5VF6R+smCuKx6j+DKr8TQYNMWztSACjpzWul1wKR8E7lnCTH6NLvlww9sizDUb931sREgH/VTzKycqAudjxUzlSYrzZj8QqHPqM+CUdzP3vprjLLXB81+uQzaqoGib6x9V02PPb4zD7+qyq8+gpPqsnAdYYjUKUw9yshZqmnCKhCpyDtHwuREpk07UtKik0BwSxuiqVQEI+0LJv9oblseJnWtc+gOHMjs/Xm3Ndrz6pjvCU4cIVzjDaq8pKXl3quFFDn4QbBP+lXTUyFdgWR0spNeySs1niPXsNivUo/Gk577kApw5Sbp0w6HgiZOH/32byNG7D4j9tsUT0le05Guy5y9dDDxZnradIXY6AO0mWCFbJRGixKbOW/Zh7JWOjyrn1vry74IlRCgSU6wPvTCZCbZYm6rDHwgB8lgqPU254XYEMWi9noXXT8O/gur7w9uLFVJP1vPyCzBrB1mSuGUD2f9w12dRI9WQqaqHV9D1iRSjcgzxc+M2nvlIRFdjzR25tU+nXmzPOAAZgXvVCLX/ e1TsSgfZ QEPzL4dbRFq65LelSML01zPCwC2fzfNGeqMOMcEgj3u7RuBeanFra8NbrJPX5m6FB6EWeQHs/tHwdXqFFH3gIgAjw24SGuIWGeL6YQZECWY2+AV/lD9rsYl+D/JWYMDa+IEBk+1jmGyLslQBDRi46+ZTR4Eznbf+8GepZ03Z3UqBCP4lO1o+iLSvbRjD4C5Ri3Q8T8FaZVIwlE8b/h8v5gsJe9CmGlG3PM99CQmCCPIYbbL2LBew4XZY5cDPn8EQXlcR/lK/i+FLcJsyhvFM7BQxbPdcnsEel+/T0RdWiKCuawx878qOgfOo/xAHv1lAA+YX43feNjEndbLBj7iyReo/xK3lTx/D6adQ9HrzTkJGgcLSpGwGVYS7Ir8AmxWY8axm3IrIHHHdmo9WcvKioCdbRTwLjEDSzIF2F6eckGzjjqHllkR9Qatte+vxYwkDA+zcgnfsl9fjci6mt/tdHIzZjpYjVr12vwW6TLI9AXk4xaGZGSlIjYPDRgQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Disclaimer: a - The cover letter got bigger than expected, so I had to split it in sections to better organize myself. I am not very confortable with it. b - Performance numbers below did not include patch 5/5 (Remove flags from memcg_stock_pcp), which could further improve performance for drain_all_stock(), but I could only notice the optimization at the last minute. 0 - Motivation: On current codebase, when drain_all_stock() is ran, it will schedule a drain_local_stock() for each cpu that has a percpu stock associated with a descendant of a given root_memcg. This happens even on 'isolated cpus', a feature commonly used on workloads that are sensitive to interruption and context switching such as vRAN and Industrial Control Systems. Since this scheduling behavior is a problem to those workloads, the proposal is to replace the current local_lock + schedule_work_on() solution with a per-cpu spinlock. 1 - Possible performance losses: With a low amount of shedule_work_on(), local_locks are supposed to perform better than spinlocks, given cacheline is always accessed by a single CPU and there is no contention. The impact on those areas is analyzed bellow: 1.1 - Cacheline usage In current implementation drain_all_stock() will be remote reading the percpu memcg_stock cacheline of every online CPU, and remote writing to all cpus that succeed the mem_cgroup_is_descendant() test. (stock->flags, stock->work) With spinlocks, drain_all_stock() will be remote reading the percpu memcg_stock cacheline of every online CPU, and remote writing to all cpus that succeed the mem_cgroup_is_descendant() test. (stock->stock_lock, on top of the above) While the spinlock may require extra acquire/release writes on some archs, they will all happen on an exclusive cacheline, so not much overhead. In both cases, the next local cpu read will require a fetch from memory (as it was invalidated on the remote write) and cacheline exclusivity get before the local write. So about cacheline usage, there should not be much impact. 1.2 - Contention We can safely assume that drain_all_stock() will not run oftenly. If it was not the case, there would be a lot of scheduled tasks and kill cpu performance. Since it does not run oftenly, and it is the only function that acesses remote percpu memcg_stock, contention should not be too expressive, and should cause less impact than scheduling (remote) and running the scheduled work (local). 2 - Performance test results: 2.1 - Test System: - Non-virtualized AMD EPYC 7601 32-Core Processor, 128 CPUs distributed in 8 NUMA nodes: 0-7,64-71 on node0, 8-15,72-79 on node1, and so on. - 256GB RAM, 2x32GB per NUMA node - For cpu isolation: use kernel cmdline: isolcpus=8-15,72-79 nohz_full=8-15,72-79 - Memcg group created with: [Slice] MemoryAccounting=true MemoryLimit=1G MemoryMax=1G MemorySwapMax=1M - For pinning runs on given CPU, I used 'taskset -c $cpunum command' - For restarting the memcg, it was used: restart_memcg(){ systemctl stop test.slice systemctl daemon-reload systemctl start test.slice } 2.2 - Impact on functions that use memcg_stock: 2.2.1 - Approach: Using perf or tracepoints get a very course result, so it was preferred to count the total cpu clocks between entering and exiting the functions that use the memcg_stock, on an isolated cpu. Something like this was used on x86: fun_name(){ u64 clk = rdtsc_ordered(); ... clk = rdtsc_ordered() - clk; dbg->fun_name.cycles += clk; dbg->fun_name.count++; } The percpu statistics were then acquired via a char device after the test finished, and an average function clock usage was calculated. For the stress test, run "cat /proc/interrupts > /dev/null" in a loop of 1000000 iterations inside the memcg for each cpu tested: for each cpu in $cpuset; do systemd-run --wait --slice=test.slice taskset -c $cpu bash -c " for k in {1..100000} ; do cat /proc/interrupts > /dev/null; done" For the drain_all_stock() test, it was necessary to restart the memcg (or cause an OOM) to call the function. 2.2.2 - Results For 1 isolated CPU, pinned on cpu 8, with no drain_all_stock() calls, being STDDEV the standard deviation between the average on 6 runs, and Call Count the sum of calls on the 6 runs: Patched Average clk STDDEV Call Count consume_stock: 63.75983475 0.1610502136 72167768 refill_stock: 67.45708322 0.09732816852 23401756 mod_objcg_state: 98.03841384 1.491628532 181292961 consume_obj_stock: 63.2543456 0.04624513799 94846454 refill_obj_stock: 78.56390025 0.3160306174 91732973 Upstream Average clk STDDEV Call Count consume_stock: 53.51201046 0.05200824438 71.866536 refill_stock: 65.46991584 0.1178078417 23401752 mod_objcg_state: 84.95365055 1.371464414 181.292707 consume_obj_stock: 60.03619438 0.05944582207 94.846327 refill_obj_stock: 73.23757912 1.161933856 91.732787 Patched - Upstream Diff (cycles) Diff % consume_stock: 10.24782429 19.15051258 refill_stock: 1.987167378 3.035237411 mod_objcg_state: 13.08476328 15.40223781 consume_obj_stock: 3.218151217 5.360351785 refill_obj_stock: 5.326321123 7.272661368 So in average the above patched functions are 2~13 clocks cycles slower than upstream. On the other hand, drain_all_stock is faster on the patched version, even considering it does all the draining instead of scheduling the work to other CPUs: drain_all_stock cpus Upstream Patched Diff (cycles) Diff(%) 1 44331.10831 38978.03581 -5353.072507 -12.07520567 8 43992.96512 39026.76654 -4966.198572 -11.2886198 128 156274.6634 58053.87421 -98220.78915 -62.85138425 (8 cpus being in the same NUMA node) 2.3 - Contention numbers 2.3.1 - Approach On top of the patched version, I replaced the spin_lock_irqsave() on functions that use the memcg_stock with spin_lock_irqsave_cc(), which is defined as: #define spin_lock_irqsave_cc(l, flags) \ if (!spin_trylock_irqsave(l, flags)) { \ u64 clk = rdtsc_ordered(); \ spin_lock_irqsave(l, flags); \ clk = rdtsc_ordered() - clk; \ pr_err("mydebug: cpu %d hit contention :" \ " fun = %s, clk = %lld/n", \ smp_processor_id(), __func__, clk); \ } So in case of contention (try_lock failing) it would record an approximate clk usage before getting the lock, and print this to dmesg. For the stress test, run "cat /proc/interrupts > /dev/null" in a loop of 1000000 iterations for each of the 128 cpus inside the memcg (limit set to 20G): for each cpu in {1..128}; do restart_memcg() systemd-run --wait --slice=test.slice taskset -c $cpu bash -c " for k in {1..100000} ; do cat /proc/interrupts > /dev/null; done" This loop was repeated for over 8 hours 2.3.2- Results Function # calls consume_stock: 15078323802 refill_stock: 2495995683 mod_objcg_state: 39765559905 consume_obj_stock: 19882888224 refill_obj_stock: 21025241793 drain_all_stock: 592 Contentions hit: 0 (Other more aggressive synthetic tests were run, and even in this case contention was hit just a couple times over some hours.) 2.4 - Syscall time measure 2.4.1- Approach To measure the patchset effect on syscall time, the following code was used: (copied/adapted from https://lore.kernel.org/linux-mm/20220924152227.819815-1-atomlin@redhat.com/ ) ################ #include #include #include #include #include typedef unsigned long long cycles_t; typedef unsigned long long usecs_t; typedef unsigned long long u64; #ifdef __x86_64__ #define DECLARE_ARGS(val, low, high) unsigned long low, high #define EAX_EDX_VAL(val, low, high) ((low) | ((u64)(high) << 32)) #define EAX_EDX_ARGS(val, low, high) "a" (low), "d" (high) #define EAX_EDX_RET(val, low, high) "=a" (low), "=d" (high) #else #define DECLARE_ARGS(val, low, high) unsigned long long val #define EAX_EDX_VAL(val, low, high) (val) #define EAX_EDX_ARGS(val, low, high) "A" (val) #define EAX_EDX_RET(val, low, high) "=A" (val) #endif static inline unsigned long long __rdtscll(void) { DECLARE_ARGS(val, low, high); asm volatile("cpuid; rdtsc" : EAX_EDX_RET(val, low, high)); return EAX_EDX_VAL(val, low, high); } #define rdtscll(val) do { (val) = __rdtscll(); } while (0) #define NRSYSCALLS 30000000 #define NRSLEEPS 100000 #define page_mmap() mmap(NULL, 4096, PROT_READ|PROT_WRITE, \ MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) #define page_munmap(x) munmap(x, 4096) void main(int argc, char *argv[]) { unsigned long a, b, cycles; int i, syscall = 0; int *page; page = page_mmap(); if (page == MAP_FAILED) perror("mmap"); if (page_munmap(page)) perror("munmap"); if (argc != 2) { printf("usage: %s {idle,syscall}\n", argv[0]); exit(1); } rdtscll(a); if (strncmp("idle", argv[1], 4) == 0) syscall = 0; else if (strncmp("syscall", argv[1], 7) == 0) syscall = 1; else { printf("usage: %s {idle,syscall}\n", argv[0]); exit(1); } if (syscall == 1) { for (i = 0; i < NRSYSCALLS; i++) { page = page_mmap(); if (page == MAP_FAILED) perror("mmap"); #ifdef MY_WRITE page[3] = i; #endif if (page_munmap(page)) perror("munmap"); } } else { for (i = 0; i < NRSLEEPS; i++) usleep(10); } rdtscll(b); cycles = b - a; if (syscall == 1) printf("cycles / syscall: %d\n", (b-a)/(NRSYSCALLS*2)); else printf("cycles / idle loop: %d\n", (b-a)/NRSLEEPS); } ################ Running with ./my_test syscall will cause it to print the average clock cycles usage of the syscall pair (page_mmap() and page_munmap()); It was compiled with two versions: With -DMY_WRITE and without it. The difference is writing to the allocated page, causing it to fault. Each version was run 200 times, pinned to an isolated cpu. Then an average and standard deviation was calculated on those results. 2.4.2- Results Patched: no_write write AVG 2991.195 5746.495 STDEV 27.77488427 40.55878512 STDEV % 0.9285547838 0.7058004073 Upstream: no_write write AVG 3012.22 5749.605 STDEV 25.1186451 37.26206223 STDEV % 0.8338914522 0.6480803851 Pat - Up: no_write write Diff -21.025 -3.11 Diff % -0.6979901866 -0.05409067232 Meaning the pair page_mmap() + page_munmap() + pagefault runs a tiny bit faster on the patched version, compared to upstream. 3 - Discussion on results On 2.2 we see every function that uses memcg_stock on local cpu gets a slower by some cycles on the patched version, while the function that accesses it remotely (drain_all_stock()) gets faster. The difference is more accentuated as we raise the cpu count, and consequently start dealing with sharing memory across NUMA. On 2.3 we see contention is not a big issue, as expected in 1.2. This probably happens due to the fact that drain_all_stock() runs quite rarely on normal operation, and the other functions are quite fast. On 2.4 we can see that page_mmap() + page_munmap() + pagefault ran a tiny bit faster. This is probably due to the fact that drain_all_stock() does not schedule work on the running cpu, causing it not to get interrupted, and possibly making up for the increased time in local functions. 4- Conclusion Scheduling work on isolated cpus can be an issue for some workloads. Reducing the issue by replacing the local_lock in memcg_stock with a spinlock should not cause much impact on performance. Leonardo Bras (5): mm/memcontrol: Align percpu memcg_stock to cache mm/memcontrol: Change stock_lock type from local_lock_t to spinlock_t mm/memcontrol: Reorder memcg_stock_pcp members to avoid holes mm/memcontrol: Perform all stock drain in current CPU mm/memcontrol: Remove flags from memcg_stock_pcp mm/memcontrol.c | 75 ++++++++++++++++++++----------------------------- 1 file changed, 31 insertions(+), 44 deletions(-) Acked-by: Roman Gushchin Reported-by: Leonardo Bras Acked-by: Roman Gushchin Signed-off-by: Michal Hocko