From patchwork Wed Feb 19 18:12:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 11392281 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 468FB138D for ; Wed, 19 Feb 2020 18:12:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EFBAD2465D for ; Wed, 19 Feb 2020 18:12:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=cmpxchg-org.20150623.gappssmtp.com header.i=@cmpxchg-org.20150623.gappssmtp.com header.b="GkTPctej" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EFBAD2465D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 083D06B0003; Wed, 19 Feb 2020 13:12:23 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 034436B0006; Wed, 19 Feb 2020 13:12:22 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8D866B0007; Wed, 19 Feb 2020 13:12:22 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0076.hostedemail.com [216.40.44.76]) by kanga.kvack.org (Postfix) with ESMTP id D17156B0003 for ; Wed, 19 Feb 2020 13:12:22 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 8269A2DFA for ; Wed, 19 Feb 2020 18:12:22 +0000 (UTC) X-FDA: 76507671324.07.scene81_890ae2fc9993e X-Spam-Summary: 2,0,0,21e5aad5cd267cb3,d41d8cd98f00b204,hannes@cmpxchg.org,:akpm@linux-foundation.org:mhocko@suse.com:tj@kernel.org:guro@fb.com::cgroups@vger.kernel.org:linux-kernel@vger.kernel.org:kernel-team@fb.com,RULES_HIT:2:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1437:1515:1535:1605:1730:1747:1777:1792:1969:2195:2199:2393:2559:2562:2693:2904:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4049:4120:4250:4321:4605:5007:6119:6261:6299:6653:7808:7875:7903:9592:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13869:13894:14096:14394:21080:21433:21444:21451:21627:21740:21990:30005:30054:30056:30070,0,RBL:209.85.222.193:@cmpxchg.org:.lbl8.mailshell.net-66.201.201.201 62.14.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:71,LUA_SUMMARY:none X-HE-Tag: scene81_890ae2fc9993e X-Filterd-Recvd-Size: 9371 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Wed, 19 Feb 2020 18:12:21 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id b7so996798qkl.7 for ; Wed, 19 Feb 2020 10:12:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=nyHLiC8XkwcyLqR3rLu/BwGv+w1mXfzRj/lTpFIDBKk=; b=GkTPctej1MwDF6qrQTa6Bd5b3GEMii0/4ZyyoxUUxSVK4yrAFiTPNUkV9khKirjNjO lFDwLo4wsvoC6/B1DDUizFlvfN1+WWtaqgWi/dqZJ9Z8CZveyMpxwBLgytas4Yn+2Uc3 Zuurs21A/VIiEmkLD9OohUUj/FiGGL+hG1hnIdL9AoWmoc/gil5ltV68GjMX+g1fnDq+ i/8uY7ZU3xANLsmzcJI/zwX+vq8svSYZikeiMmC/vdmpTR2gYM5p/7jvug5Ciks+H39r O1M7vDcsrpHuOqOaZuNZINoAIhGcM6fltA2fDVsAeeAxuAIvKOxGrtQ4FgMm+OFKpFWL P3Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=nyHLiC8XkwcyLqR3rLu/BwGv+w1mXfzRj/lTpFIDBKk=; b=dw/EvUnvB4yJ1WiXZI7S3JSNiL4BNpiKTMwKO1lmDW3O6tidC8Ng0sPRLpS94aFwUZ YFGRJ2pawgUjSYjTs65wXNntTXZbY3n5vFExa5r9gD/gs1r0E/qyL1inxZtiQCkiNHkM f9EES6/CR56Er8YuCxBtHQXgNcxdb0Drkk+j1D1dWy5K3uP9lrf06Ls3LWJSQK2FWTF9 FOS3O+URbWwJ7n5JCJmL73WWTdYqoQAs3TZBENLXa4+YNrT4f5+D6DeqkJbce/7UyTfl lIJiLqmnZvwp4jFP2P1SmU8MERLgGkUx2rEYUaHeMCGd7fXMWkR1P1MkfYSQtNjx+uf1 QDNg== X-Gm-Message-State: APjAAAVOe97uANktLODYD9lPdDyyfuMnlbnZkPgOKOwLTbCjW3Uy54L6 kwwAomiQQxcu3fnuJvwZrQ6Amg== X-Google-Smtp-Source: APXvYqxQP+2JZAwmgpbQaOmHN+mZFY9huQzZ0AFJMd+VQkazsqk7kclEYBq/LhoeD2GTcppz0tsQvw== X-Received: by 2002:a05:620a:222d:: with SMTP id n13mr24920245qkh.268.1582135940510; Wed, 19 Feb 2020 10:12:20 -0800 (PST) Received: from localhost ([2620:10d:c091:500::2:3bde]) by smtp.gmail.com with ESMTPSA id t23sm361021qto.88.2020.02.19.10.12.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2020 10:12:19 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Michal Hocko , Tejun Heo , Roman Gushchin , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: [PATCH] mm: memcontrol: asynchronous reclaim for memory.high Date: Wed, 19 Feb 2020 13:12:19 -0500 Message-Id: <20200219181219.54356-1-hannes@cmpxchg.org> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We have received regression reports from users whose workloads moved into containers and subsequently encountered new latencies. For some users these were a nuisance, but for some it meant missing their SLA response times. We tracked those delays down to cgroup limits, which inject direct reclaim stalls into the workload where previously all reclaim was handled my kswapd. This patch adds asynchronous reclaim to the memory.high cgroup limit while keeping direct reclaim as a fallback. In our testing, this eliminated all direct reclaim from the affected workload. memory.high has a grace buffer of about 4% between when it becomes exceeded and when allocating threads get throttled. We can use the same buffer for the async reclaimer to operate in. If the worker cannot keep up and the grace buffer is exceeded, allocating threads will fall back to direct reclaim before getting throttled. For irq-context, there's already async memory.high enforcement. Re-use that work item for all allocating contexts, but switch it to the unbound workqueue so reclaim work doesn't compete with the workload. The work item is per cgroup, which means the workqueue infrastructure will create at maximum one worker thread per reclaiming cgroup. Signed-off-by: Johannes Weiner --- mm/memcontrol.c | 60 +++++++++++++++++++++++++++++++++++++------------ mm/vmscan.c | 10 +++++++-- 2 files changed, 54 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cf02e3ef3ed9..bad838d9c2bb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1446,6 +1446,10 @@ static char *memory_stat_format(struct mem_cgroup *memcg) seq_buf_printf(&s, "pgsteal %lu\n", memcg_events(memcg, PGSTEAL_KSWAPD) + memcg_events(memcg, PGSTEAL_DIRECT)); + seq_buf_printf(&s, "pgscan_direct %lu\n", + memcg_events(memcg, PGSCAN_DIRECT)); + seq_buf_printf(&s, "pgsteal_direct %lu\n", + memcg_events(memcg, PGSTEAL_DIRECT)); seq_buf_printf(&s, "%s %lu\n", vm_event_name(PGACTIVATE), memcg_events(memcg, PGACTIVATE)); seq_buf_printf(&s, "%s %lu\n", vm_event_name(PGDEACTIVATE), @@ -2235,10 +2239,19 @@ static void reclaim_high(struct mem_cgroup *memcg, static void high_work_func(struct work_struct *work) { + unsigned long high, usage; struct mem_cgroup *memcg; memcg = container_of(work, struct mem_cgroup, high_work); - reclaim_high(memcg, MEMCG_CHARGE_BATCH, GFP_KERNEL); + + high = READ_ONCE(memcg->high); + usage = page_counter_read(&memcg->memory); + + if (usage <= high) + return; + + set_worker_desc("cswapd/%llx", cgroup_id(memcg->css.cgroup)); + reclaim_high(memcg, usage - high, GFP_KERNEL); } /* @@ -2304,15 +2317,22 @@ void mem_cgroup_handle_over_high(void) unsigned long pflags; unsigned long penalty_jiffies, overage; unsigned int nr_pages = current->memcg_nr_pages_over_high; + bool tried_direct_reclaim = false; struct mem_cgroup *memcg; if (likely(!nr_pages)) return; - memcg = get_mem_cgroup_from_mm(current->mm); - reclaim_high(memcg, nr_pages, GFP_KERNEL); current->memcg_nr_pages_over_high = 0; + memcg = get_mem_cgroup_from_mm(current->mm); + high = READ_ONCE(memcg->high); +recheck: + usage = page_counter_read(&memcg->memory); + + if (usage <= high) + goto out; + /* * memory.high is breached and reclaim is unable to keep up. Throttle * allocators proactively to slow down excessive growth. @@ -2325,12 +2345,6 @@ void mem_cgroup_handle_over_high(void) * overage amount. */ - usage = page_counter_read(&memcg->memory); - high = READ_ONCE(memcg->high); - - if (usage <= high) - goto out; - /* * Prevent division by 0 in overage calculation by acting as if it was a * threshold of 1 page @@ -2369,6 +2383,16 @@ void mem_cgroup_handle_over_high(void) if (penalty_jiffies <= HZ / 100) goto out; + /* + * It's possible async reclaim just isn't able to keep + * up. Before we go to sleep, try direct reclaim. + */ + if (!tried_direct_reclaim) { + reclaim_high(memcg, nr_pages, GFP_KERNEL); + tried_direct_reclaim = true; + goto recheck; + } + /* * If we exit early, we're guaranteed to die (since * schedule_timeout_killable sets TASK_KILLABLE). This means we don't @@ -2544,13 +2568,21 @@ static int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, */ do { if (page_counter_read(&memcg->memory) > memcg->high) { + /* + * Kick off the async reclaimer, which should + * be doing most of the work to avoid latency + * in the workload. But also check in on its + * progress before resuming to userspace, in + * case we need to do direct reclaim, or even + * throttle the allocating thread if reclaim + * cannot keep up with allocation demand. + */ + queue_work(system_unbound_wq, &memcg->high_work); /* Don't bother a random interrupted task */ - if (in_interrupt()) { - schedule_work(&memcg->high_work); - break; + if (!in_interrupt()) { + current->memcg_nr_pages_over_high += batch; + set_notify_resume(current); } - current->memcg_nr_pages_over_high += batch; - set_notify_resume(current); break; } } while ((memcg = parent_mem_cgroup(memcg))); diff --git a/mm/vmscan.c b/mm/vmscan.c index 74e8edce83ca..d6085115c7f2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1947,7 +1947,10 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken); reclaim_stat->recent_scanned[file] += nr_taken; - item = current_is_kswapd() ? PGSCAN_KSWAPD : PGSCAN_DIRECT; + if (current_is_kswapd() || (cgroup_reclaim(sc) && current_work())) + item = PGSCAN_KSWAPD; + else + item = PGSCAN_DIRECT; if (!cgroup_reclaim(sc)) __count_vm_events(item, nr_scanned); __count_memcg_events(lruvec_memcg(lruvec), item, nr_scanned); @@ -1961,7 +1964,10 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, spin_lock_irq(&pgdat->lru_lock); - item = current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; + if (current_is_kswapd() || (cgroup_reclaim(sc) && current_work())) + item = PGSTEAL_KSWAPD; + else + item = PGSTEAL_DIRECT; if (!cgroup_reclaim(sc)) __count_vm_events(item, nr_reclaimed); __count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed);