From patchwork Fri Apr 21 17:40:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13220533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA6D9C7618E for ; Fri, 21 Apr 2023 17:40:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233160AbjDURkj (ORCPT ); Fri, 21 Apr 2023 13:40:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233184AbjDURk0 (ORCPT ); Fri, 21 Apr 2023 13:40:26 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AE4A125B2 for ; Fri, 21 Apr 2023 10:40:25 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-2474877cc18so1388770a91.3 for ; Fri, 21 Apr 2023 10:40:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098824; x=1684690824; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=abvwWBWOn0vLHIokwPNC/tHyu1Fa1R/g8ZqTWIaeK3o=; b=XiTBjh0uy9kBhESmjZrsmo27Fd2heaOUEeGy+wI7JgY+55BFi83KpJfUElFkBAdtm2 /9Y+dOWIDQQOwVTZr/9l2zpPw4NhLuSad5LwurXNF9rM5PxMETajnAk6IdSR9itjleO5 5RCZOU6uWoQn/XlBfG6L6Ut73QqfPkSzpxRKMZq9c5dNC03EQysfXGAmQsX7vQIaMlXI wrYrPEcgEz4/6R89vEXhItTLbmEid9U911aGVkkoSTQdm4KxH3ssyAravljY6M6dDyLy P40RfGznmXP1ykBLl7M1nQBK+thu7rJX9Fwoe+MatBRuu3PDwmPEoG+B+msi8iGqptyL Fwng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098824; x=1684690824; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=abvwWBWOn0vLHIokwPNC/tHyu1Fa1R/g8ZqTWIaeK3o=; b=dgT3YjoLsXWIwtENFUkUPUeSRtP0NC+ejtrFYDBqFLamhGaX41YLTo8FgCFfb1/yQA 79gwsXhTCmZNB1fdqHoQSsFhUMqVfLbCQiOxDnYGLYL+bkIUhBnokoXxhXVWRkPIihMa NYOuyY4rUmK43kBZD0QdVWhPyC1baFsHMm4Deqt1swV5LQqQ95gqVVnzlABsZ2FsbWox 1mIedwk6te4sA4lhKo59rd8R1tCRDvQgNl6Xj+ezScdhyUYYEmJTpKBlpjTG1xqVz18I nY8/9vs4VxNSnY3HDVQvNaFX5cgL4nqEJeqnGsboCDYxa2vb8zVniT0THNJDObw0RaIg +B2w== X-Gm-Message-State: AAQBX9cpzorSNTgNxAOmHA4gvGfheSqUkpPfHxkokbBv1Xkxrq1y8y7q A7uf3M7CEs1asB91CVGWzQYyBpX41TbJQQ/1 X-Google-Smtp-Source: AKy350YbjANQ50zXR9lpXjPZgwQ40mwqLJ7O/YqrPonNpx/xppRCva3J0ueIG/FDAbEMT7LPHpueSbGCVpFSyjJ8 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:290:b0:247:2677:77b with SMTP id w16-20020a17090a029000b002472677077bmr1455398pja.8.1682098824669; Fri, 21 Apr 2023 10:40:24 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:16 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-2-yosryahmed@google.com> Subject: [PATCH v5 1/5] writeback: move wb_over_bg_thresh() call outside lock section From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , " =?utf-8?q?Micha?= =?utf-8?q?l_Koutn=C3=BD?= " , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org wb_over_bg_thresh() calls mem_cgroup_wb_stats() which invokes an rstat flush, which can be expensive on large systems. Currently, wb_writeback() calls wb_over_bg_thresh() within a lock section, so we have to do the rstat flush atomically. On systems with a lot of cpus and/or cgroups, this can cause us to disable irqs for a long time, potentially causing problems. Move the call to wb_over_bg_thresh() outside the lock section in preparation to make the rstat flush in mem_cgroup_wb_stats() non-atomic. The list_empty(&wb->work_list) check should be okay outside the lock section of wb->list_lock as it is protected by a separate lock (wb->work_lock), and wb_over_bg_thresh() doesn't seem like it is modifying any of wb->b_* lists the wb->list_lock is protecting. Also, the loop seems to be already releasing and reacquring the lock, so this refactoring looks safe. Signed-off-by: Yosry Ahmed Reviewed-by: Michal Koutný Reviewed-by: Jan Kara Acked-by: Shakeel Butt Acked-by: Tejun Heo --- fs/fs-writeback.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 1db3e3c24b43..11aa1652fb84 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -2024,7 +2024,6 @@ static long wb_writeback(struct bdi_writeback *wb, struct blk_plug plug; blk_start_plug(&plug); - spin_lock(&wb->list_lock); for (;;) { /* * Stop writeback when nr_pages has been consumed @@ -2049,6 +2048,9 @@ static long wb_writeback(struct bdi_writeback *wb, if (work->for_background && !wb_over_bg_thresh(wb)) break; + + spin_lock(&wb->list_lock); + /* * Kupdate and background works are special and we want to * include all inodes that need writing. Livelock avoidance is @@ -2078,13 +2080,19 @@ static long wb_writeback(struct bdi_writeback *wb, * mean the overall work is done. So we keep looping as long * as made some progress on cleaning pages or inodes. */ - if (progress) + if (progress) { + spin_unlock(&wb->list_lock); continue; + } + /* * No more inodes for IO, bail */ - if (list_empty(&wb->b_more_io)) + if (list_empty(&wb->b_more_io)) { + spin_unlock(&wb->list_lock); break; + } + /* * Nothing written. Wait for some inode to * become available for writeback. Otherwise @@ -2096,9 +2104,7 @@ static long wb_writeback(struct bdi_writeback *wb, spin_unlock(&wb->list_lock); /* This function drops i_lock... */ inode_sleep_on_writeback(inode); - spin_lock(&wb->list_lock); } - spin_unlock(&wb->list_lock); blk_finish_plug(&plug); return nr_pages - work->nr_pages; From patchwork Fri Apr 21 17:40:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13220534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F4D3C77B78 for ; Fri, 21 Apr 2023 17:40:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233331AbjDURkl (ORCPT ); Fri, 21 Apr 2023 13:40:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233235AbjDURk3 (ORCPT ); Fri, 21 Apr 2023 13:40:29 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEF5112588 for ; Fri, 21 Apr 2023 10:40:26 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-24736ac595aso2200462a91.1 for ; Fri, 21 Apr 2023 10:40:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098826; x=1684690826; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=ze/kPlw8gotaEo1sJnB2fYrZx8MA/zlu9x0xYHZ+g7E=; b=evkdgDpSyqlLAXON2ZuU2lSGfLM1dQnhdPP+fmyZWeKwGNZTWFCJKPVGhM2bZZGKOC f7YfSGCo2Nf3oXwwYJmGjEDgBvFsw6i76b70QgcRaecrvTKD295qL9CqwnO4UztiMSAR wca4o1fBiZOb2DZIrqp+VDQfAwIcLci5K41MPeROy0/47HRyQgXuMC8OBwZyCxCgwh/S 8f4JUGNY1xeou8G6F0i+W3dKhEWQ3Zv+nckYooRRkUzwRcW77nDMDOwXtSb1IOOWcx0Y jIqwdtbmW/rhz4VKzZZ8k/jeg5TikDCeSv0Wmg2wr9UElKoOoUpGS9AakxlgHuAC0y31 KsJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098826; x=1684690826; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=ze/kPlw8gotaEo1sJnB2fYrZx8MA/zlu9x0xYHZ+g7E=; b=jbdkLZoJ3WGfnBx6Y7dxPE/qDzXK7agySBQNsp/coW+MawnoHATDIhw3t8kE+PYz1i wwQh9ePC4C64BPN4zI+QPLAdQcrE7acjuKv32GKFMWKRUz2H5gzF6YXPzcI0dvbomRbv z9+czNk37vs6ix9075Iw7B9SVp2feSu7+P+Oxv+D/yFEmxtqWzm53+pRNW2iJF78j3Ju bR1cHY3sP6KrG7w80cVqCqCGgA+vn71TVpOUYbDQ7Z+f/MbU6HVUly/eB3KXWTlr3Vwn TW8q5rktP1WH6INJcOCat1PTf3zrGZlDtJcOI+DM+Hal8MJn+cdRRBy0tOWN56ZVC2cc l2BQ== X-Gm-Message-State: AAQBX9cpHqUBZgp3htfVBJAA6nhzCmkocEupbjoSeQ07z/Ypto7yZCR3 jE/UqEI56sRlU6pBHbo8GM2zAXjMr7Wgase0 X-Google-Smtp-Source: AKy350aoZj3p9w7xLo2ekZAFCbJRpr+XbYHOfgfJjD/OnpfpL06mwTgu6bL4JQHk39cy27rMag7V9/fdDGHimXJS X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:3fcd:b0:24b:27db:3cfc with SMTP id u13-20020a17090a3fcd00b0024b27db3cfcmr1409596pjm.8.1682098826505; Fri, 21 Apr 2023 10:40:26 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:17 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-3-yosryahmed@google.com> Subject: [PATCH v5 2/5] memcg: flush stats non-atomically in mem_cgroup_wb_stats() From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , " =?utf-8?q?Micha?= =?utf-8?q?l_Koutn=C3=BD?= " , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The previous patch moved the wb_over_bg_thresh()->mem_cgroup_wb_stats() code path in wb_writeback() outside the lock section. We no longer need to flush the stats atomically. Flush the stats non-atomically. Signed-off-by: Yosry Ahmed Reviewed-by: Michal Koutný Acked-by: Shakeel Butt Acked-by: Tejun Heo --- mm/memcontrol.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4b27e245a055..5e79fdf8442b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4648,11 +4648,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - /* - * wb_writeback() takes a spinlock and calls - * wb_over_bg_thresh()->mem_cgroup_wb_stats(). Do not sleep. - */ - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); From patchwork Fri Apr 21 17:40:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13220535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7256EC7618E for ; Fri, 21 Apr 2023 17:40:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233178AbjDURkn (ORCPT ); Fri, 21 Apr 2023 13:40:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233279AbjDURkb (ORCPT ); Fri, 21 Apr 2023 13:40:31 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C26AB125B2 for ; Fri, 21 Apr 2023 10:40:28 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-517bb11770bso1473409a12.0 for ; Fri, 21 Apr 2023 10:40:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098828; x=1684690828; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=ZwYNf0IQKL13Uvalja3XtUe1fz98Sj9XyCNuQ2xKGcs=; b=Vz7q7OzmTKVhjKuL1Z10tx4sjzaZzQc1RfkV5JIKDa4QF+qc7PE1nRpKj4JbTW26CL 6HWg0Bk/Qn7IeGOg2v+aAU068B55qLODSnp8tJGuESSICbp+6Gjpfa3HASvT4P/INIIX FNJAyhgiBW8P7szSQRrC+9yXtrHR4LwEf5c+uGkIF6nWGq5t81yHqYblmUSNV6/Pj4a5 x4hPCDxoK7EryyUSzUifkrvbv6JClx5/LSyTe0bPdsXuYCvvh5cuu7sfplfgaUZyWnve cPw/375qmd31nmxG3Xi1PRi7CTxei8qHjgER9ienP6JVZlSSk1rMC1yFsifBD6+eDFuk /A6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098828; x=1684690828; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=ZwYNf0IQKL13Uvalja3XtUe1fz98Sj9XyCNuQ2xKGcs=; b=PHHlqUuw0L8wGeyfUfKhIRpCy61Vk3YUc0zCG+n+zsehT1MYCNI1zOJBPwsgY92WSz XEaGKoHh72wWghldroNfsr7MkxE8e9TJsrDlJvVLt5hKrB6gkUQr62ebeG+pS8YFIDep OVTOPzUqZc0rv+b5vu3NpDX+Zx5gV++OHLjsRaMJYG97M+5KOJ1/LRzQB1o/MqDV3O6x wcwhwqaBqt6ylhswkjsOc0PTkqz/a4GBvlqG3tytysZ3IMd5gn37ySxUzMEk0W5x6ABR ddoj3fughUeo2figOdhIMNha7cNqRL0jjHEyKbjuq2Dk0ro6Q2JErHbef8Zzb9tCiezk h9Uw== X-Gm-Message-State: AAQBX9cEDirGIQ96KQnyYd5UMjKIDLEQCtQy51U/ivabpnBLJDXiNUHq /e7Y33CXczyd3Dx6x8+wdEmwm3CLUwH1lWkI X-Google-Smtp-Source: AKy350aINzx6WzxMD77WpFdXBe8LNCyI2424id24uJ5dWpjeDUwupScbzRciRo/Uc9LOO+muamaNQYT7/vZDgFTY X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:903:2447:b0:1a9:2c3e:b087 with SMTP id l7-20020a170903244700b001a92c3eb087mr1970101pls.0.1682098828387; Fri, 21 Apr 2023 10:40:28 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:18 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-4-yosryahmed@google.com> Subject: [PATCH v5 3/5] memcg: calculate root usage from global state From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , " =?utf-8?q?Micha?= =?utf-8?q?l_Koutn=C3=BD?= " , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Currently, we approximate the root usage by adding the memcg stats for anon, file, and conditionally swap (for memsw). To read the memcg stats we need to invoke an rstat flush. rstat flushes can be expensive, they scale with the number of cpus and cgroups on the system. mem_cgroup_usage() is called by memcg_events()->mem_cgroup_threshold() with irqs disabled, so such an expensive operation with irqs disabled can cause problems. Instead, approximate the root usage from global state. This is not 100% accurate, but the root usage has always been ill-defined anyway. Signed-off-by: Yosry Ahmed Reviewed-by: Michal Koutný Acked-by: Shakeel Butt --- mm/memcontrol.c | 24 +++++------------------- 1 file changed, 5 insertions(+), 19 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5e79fdf8442b..cb78bba5b4a4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3699,27 +3699,13 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) if (mem_cgroup_is_root(memcg)) { /* - * We can reach here from irq context through: - * uncharge_batch() - * |--memcg_check_events() - * |--mem_cgroup_threshold() - * |--__mem_cgroup_threshold() - * |--mem_cgroup_usage - * - * rstat flushing is an expensive operation that should not be - * done from irq context; use stale stats in this case. - * Arguably, usage threshold events are not reliable on the root - * memcg anyway since its usage is ill-defined. - * - * Additionally, other call paths through memcg_check_events() - * disable irqs, so make sure we are flushing stats atomically. + * Approximate root's usage from global state. This isn't + * perfect, but the root usage was always an approximation. */ - if (in_task()) - mem_cgroup_flush_stats_atomic(); - val = memcg_page_state(memcg, NR_FILE_PAGES) + - memcg_page_state(memcg, NR_ANON_MAPPED); + val = global_node_page_state(NR_FILE_PAGES) + + global_node_page_state(NR_ANON_MAPPED); if (swap) - val += memcg_page_state(memcg, MEMCG_SWAP); + val += total_swap_pages - get_nr_swap_pages(); } else { if (!swap) val = page_counter_read(&memcg->memory); From patchwork Fri Apr 21 17:40:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13220536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C1A2C77B7E for ; Fri, 21 Apr 2023 17:40:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233273AbjDURko (ORCPT ); Fri, 21 Apr 2023 13:40:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233294AbjDURkc (ORCPT ); Fri, 21 Apr 2023 13:40:32 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A886C12582 for ; Fri, 21 Apr 2023 10:40:30 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-63b68f53fd8so1794224b3a.2 for ; Fri, 21 Apr 2023 10:40:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098830; x=1684690830; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2xVbU1EXT3F/5uSzRhbvwRdAhlPEIhLbnGkOO/O2kc0=; b=VsD7UvvlQQthDrDrmrxfA+6j9bxs+fBOtiWfjPuVqKRncTMVYwHyMK7lQKw4z7d6OO 8f3IhH6XmGqVDVmFY8Cyv6LrhOYmMlAEwO97OjVKp4EfQ7GvmXRUQBA2elyZi2HdAWT8 tC3JhABpjBI1uwry+YoMsz8jQ+5FjhTBlcxyLfQuh9MawFFcJmVXlkCxxE83KRAGfIRN izUhapvRy65QRBOUgMasmNbiSkr8HJo7bR2eGmnfXoyNgGu2rdwIe+txD2JLoEKv7Tja ItjbWxT/dXoPWnPJUuXxJqFt3IyOH6lsotFnhAXC0TTQySKSNCNtSjYyYHqymn6WRMKQ FgWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098830; x=1684690830; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2xVbU1EXT3F/5uSzRhbvwRdAhlPEIhLbnGkOO/O2kc0=; b=GUB18kL6iCDo+FTXZcQAbHNCdqb6jWRXRpZnASDEFxt+7oeiHKjprRYZ86P8Nb993o fpmNyhBJC/Uxv+UEeRzhlrWb9ERFV8u+qUEaIZ58DBxeivgnMXl9XxlKaOQ1ncv2rWLy HvPvwKOj6KnccS3fAPY2MKxa76sQ2yL2AWOuVJAmUfP+EwQFxrRDZBjITO4O7+WQpt7t W5yj0PYBweWPcr/mEgL86JRaKADc2wqcdtItFsArxagJUJeR7dPkfaaHxhOIbGtkeUbU PLq2aaI2/Ewrh5PTiPEF3PYiXCZ/Qa3colVbgVjEKJOXzxMYhBOC18LybWaBawjTc6PU x+CA== X-Gm-Message-State: AAQBX9d/d3A4n062PZxwtrK3K1ud531mHgsd369tzyLXQajMM8/5rI3K 6iY1GeTrJTGVyi4zmu6vLGtxRCfu7/4kbBiT X-Google-Smtp-Source: AKy350a5A112WsBcfM4E2oTlYZXb/Q86xb8XlqVuh7OqAqi/kCn371in9+BxtMemeJoGBiiS6Eu5MnjLStpGGXAY X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:2e97:b0:62d:9bea:2a0c with SMTP id fd23-20020a056a002e9700b0062d9bea2a0cmr2546450pfb.4.1682098830169; Fri, 21 Apr 2023 10:40:30 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:19 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-5-yosryahmed@google.com> Subject: [PATCH v5 4/5] memcg: remove mem_cgroup_flush_stats_atomic() From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , " =?utf-8?q?Micha?= =?utf-8?q?l_Koutn=C3=BD?= " , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Previous patches removed all callers of mem_cgroup_flush_stats_atomic(). Remove the function and simplify the code. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt --- include/linux/memcontrol.h | 5 ----- mm/memcontrol.c | 24 +++++------------------- 2 files changed, 5 insertions(+), 24 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 222d7370134c..00a88cf947e1 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1038,7 +1038,6 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_atomic(void); void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, @@ -1537,10 +1536,6 @@ static inline void mem_cgroup_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_atomic(void) -{ -} - static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cb78bba5b4a4..eadfe371a8f2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -639,7 +639,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void do_flush_stats(bool atomic) +static void do_flush_stats(void) { /* * We always flush the entire tree, so concurrent flushers can just @@ -652,30 +652,16 @@ static void do_flush_stats(bool atomic) WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - if (atomic) - cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); - else - cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + cgroup_rstat_flush(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } -static bool should_flush_stats(void) -{ - return atomic_read(&stats_flush_threshold) > num_online_cpus(); -} - void mem_cgroup_flush_stats(void) { - if (should_flush_stats()) - do_flush_stats(false); -} - -void mem_cgroup_flush_stats_atomic(void) -{ - if (should_flush_stats()) - do_flush_stats(true); + if (atomic_read(&stats_flush_threshold) > num_online_cpus()) + do_flush_stats(); } void mem_cgroup_flush_stats_ratelimited(void) @@ -690,7 +676,7 @@ static void flush_memcg_stats_dwork(struct work_struct *w) * Always flush here so that flushing in latency-sensitive paths is * as cheap as possible. */ - do_flush_stats(false); + do_flush_stats(); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } From patchwork Fri Apr 21 17:40:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13220537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E272C77B76 for ; Fri, 21 Apr 2023 17:40:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233370AbjDURkq (ORCPT ); Fri, 21 Apr 2023 13:40:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233316AbjDURkd (ORCPT ); Fri, 21 Apr 2023 13:40:33 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A291012589 for ; Fri, 21 Apr 2023 10:40:32 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-b98f3ca02b5so2075899276.1 for ; Fri, 21 Apr 2023 10:40:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682098832; x=1684690832; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xEOFJLcgICBTmNTLnyQD1ooBVf7w9cMBHK4zBwAxhps=; b=2LLI0uS0HCGgUG+To5FG73zbxk+tuzk5tTcOBZNGKwnVbYk9yq2r0zkyTl0MD4wfQ3 kKiggVfa7NNclULg8NuxAqHV3zYnq7LZjfMrcVD62yChQCNHWbk54zVsBhxomTb0t0SN +XgJr7wul3fQD2JopF/MZfktxPtuUWKER8IL9ZU+RSEl7sW66jFXxgj6K5ErZztO7Nce l4ZWAyWNanyDDUgKfIgOcWOkpHNwJKJvYyr6/84gH8i9GqoBVPKeVaSusHPk+GdeSMBt A9hidHBqS6YCIVpadKtHiQ8uYxCY8mC3l7iI3bjfr9izY/KQoToKFIDt3lP79eIF321D fMOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682098832; x=1684690832; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xEOFJLcgICBTmNTLnyQD1ooBVf7w9cMBHK4zBwAxhps=; b=fwAqsuICJanVqlRRsgUembDrXgTm3LO0cWxy75i0LdKqA3DkGowANnPRdS/rtTnwf3 mHyx2m5InoTASm0h234fptxcHLn2A0X9cjF+IFDYCXRFv2NjIy68i0glF9TyRKXjUUKC sIjP+tOHO+MY+tJcJZLjxJmXmNYIkUmaVhw+oo8BbS/19uvauX7nbJTrOqSwCeiQg3d3 syq0QNzCeiuxN5g0ssC78OvWBIYPRegWzPSTJsNUcMKnje0eOQUL2jfbDnUd2FUN5xwX jmCGZwDzQMwWXYBg9spwfH/lZVippeClEJz6/UfJdoxcCp6OzEoVEOrFdhqsXNs9JW1Z t+fA== X-Gm-Message-State: AAQBX9dgN7RTR0nQ0IRspBg7BUhbHb8P8XH0VypM1uflU3Udjzidaj2S M+1rxCjHIIVBZqILM0WoGX++myNBUlPzwQXm X-Google-Smtp-Source: AKy350ZZ8ts19Uuwp1KvgVZn+d+WSE2zsbcOTm9kgrCL3PJuuy6na6YJxn3jgGZppB+bAg1Vc1I9v8GZpcBWJ5vX X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:bc4a:0:b0:b33:531b:3dd4 with SMTP id d10-20020a25bc4a000000b00b33531b3dd4mr1583021ybk.1.1682098831943; Fri, 21 Apr 2023 10:40:31 -0700 (PDT) Date: Fri, 21 Apr 2023 17:40:20 +0000 In-Reply-To: <20230421174020.2994750-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230421174020.2994750-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog Message-ID: <20230421174020.2994750-6-yosryahmed@google.com> Subject: [PATCH v5 5/5] cgroup: remove cgroup_rstat_flush_atomic() From: Yosry Ahmed To: Alexander Viro , Christian Brauner , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Tejun Heo Cc: Jan Kara , Jens Axboe , " =?utf-8?q?Micha?= =?utf-8?q?l_Koutn=C3=BD?= " , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Previous patches removed the only caller of cgroup_rstat_flush_atomic(). Remove the function and simplify the code. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Tejun Heo --- include/linux/cgroup.h | 1 - kernel/cgroup/rstat.c | 26 +++++--------------------- 2 files changed, 5 insertions(+), 22 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 885f5395fcd0..567c547cf371 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -692,7 +692,6 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) */ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_atomic(struct cgroup *cgrp); void cgroup_rstat_flush_hold(struct cgroup *cgrp); void cgroup_rstat_flush_release(void); diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index d3252b0416b6..f9ad33f117c8 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -171,7 +171,7 @@ __weak noinline void bpf_rstat_flush(struct cgroup *cgrp, __diag_pop(); /* see cgroup_rstat_flush() */ -static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) +static void cgroup_rstat_flush_locked(struct cgroup *cgrp) __releases(&cgroup_rstat_lock) __acquires(&cgroup_rstat_lock) { int cpu; @@ -207,9 +207,8 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) } raw_spin_unlock_irqrestore(cpu_lock, flags); - /* if @may_sleep, play nice and yield if necessary */ - if (may_sleep && (need_resched() || - spin_needbreak(&cgroup_rstat_lock))) { + /* play nice and yield if necessary */ + if (need_resched() || spin_needbreak(&cgroup_rstat_lock)) { spin_unlock_irq(&cgroup_rstat_lock); if (!cond_resched()) cpu_relax(); @@ -236,25 +235,10 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) might_sleep(); spin_lock_irq(&cgroup_rstat_lock); - cgroup_rstat_flush_locked(cgrp, true); + cgroup_rstat_flush_locked(cgrp); spin_unlock_irq(&cgroup_rstat_lock); } -/** - * cgroup_rstat_flush_atomic- atomic version of cgroup_rstat_flush() - * @cgrp: target cgroup - * - * This function can be called from any context. - */ -void cgroup_rstat_flush_atomic(struct cgroup *cgrp) -{ - unsigned long flags; - - spin_lock_irqsave(&cgroup_rstat_lock, flags); - cgroup_rstat_flush_locked(cgrp, false); - spin_unlock_irqrestore(&cgroup_rstat_lock, flags); -} - /** * cgroup_rstat_flush_hold - flush stats in @cgrp's subtree and hold * @cgrp: target cgroup @@ -269,7 +253,7 @@ void cgroup_rstat_flush_hold(struct cgroup *cgrp) { might_sleep(); spin_lock_irq(&cgroup_rstat_lock); - cgroup_rstat_flush_locked(cgrp, true); + cgroup_rstat_flush_locked(cgrp); } /**