From patchwork Fri Dec 11 20:21:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11969341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35056C4361B for ; Fri, 11 Dec 2020 20:21:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9E03423F38 for ; Fri, 11 Dec 2020 20:21:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E03423F38 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E342F6B0068; Fri, 11 Dec 2020 15:21:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E10BD6B006C; Fri, 11 Dec 2020 15:21:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C38046B006E; Fri, 11 Dec 2020 15:21:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0052.hostedemail.com [216.40.44.52]) by kanga.kvack.org (Postfix) with ESMTP id ADE446B0068 for ; Fri, 11 Dec 2020 15:21:49 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 685B63633 for ; Fri, 11 Dec 2020 20:21:49 +0000 (UTC) X-FDA: 77582122338.04.rock57_0400d3e27403 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 4304B8019A9C for ; Fri, 11 Dec 2020 20:21:49 +0000 (UTC) X-HE-Tag: rock57_0400d3e27403 X-Filterd-Recvd-Size: 7270 Received: from mail-qt1-f194.google.com (mail-qt1-f194.google.com [209.85.160.194]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Fri, 11 Dec 2020 20:21:48 +0000 (UTC) Received: by mail-qt1-f194.google.com with SMTP id z9so7419235qtn.4 for ; Fri, 11 Dec 2020 12:21:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=wsBBzNzDskmm1L92habwo+bQrCRo6cUbscDHID44jtI=; b=Xw6EwkP5NGT8ojgRINybMgV0CEA/32rfK6VpqwScYHQTiSjgDDDgIAv1ObyIa5yf8C 64a4xj8I9BmfJOgiscgJ3oQ4YecLGRFL6TiNq902N1woOP6EDuQJnsCJ6dOfH9ewifOR MHvhW9NkUCciATgOXVgOufjCPnXSSmSJhqlDa/Xt8iehA/LPGNT8fPO9d/qJwAiqIv6z 7QOagbaZpnlnuSh9xVI5MZS35bpwCt5Wr4lUmMv/8P1oYw6DPaKVMtbez3yebFuipSB5 wAtry6eKK77Kd7hnv+WqLe+JNzdZdVMEoyeZksgW4rPzcNJ8PFJGSFromgAIr7KTjIfZ paSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wsBBzNzDskmm1L92habwo+bQrCRo6cUbscDHID44jtI=; b=a3QRUBKJqdXlcAT/oguYKTtm4cajjrRpal7xO1rNArVZtLOu04Z8efXIkU2tOoewK3 855i7alK0m9tGJodgJh7uVd5hB5US6CFnqUNGOhtxjHg4NpsjI8EVaYBgLDrHjSRJZD5 w1Elws+PHcruHSR4ysJaVqE3xWg9oQxxzs9/lSvTzdCIfprTQB44XnEh8nNUOq3X3QKL feZHtDwJ1VCri43OrBkNk/I7ZLx1xtTQJhnEAv6KEZ+pbcisKYdVFz0En929B7laOtbY fx/50RkWkLzCDnAOBxT/zpcWsrNOa37lo9pqSYDKubQOjsL84zyYsunXvRK8fcyUgV98 XIgg== X-Gm-Message-State: AOAM530laxwZfbeDOxltaT1pPHfbp07z/Ed/iRwVG+detO1ykLLupri1 W9kYS5/NlTz1Jm0WbcleQ+xtpQ== X-Google-Smtp-Source: ABdhPJwZBDj+bDwR2xyRmaL+PBvHpV7akfdLiClVp+ecUemviam551kOC6gczZy6ZbRQQi66Jhb9zA== X-Received: by 2002:aed:29a5:: with SMTP id o34mr17294060qtd.379.1607718108108; Fri, 11 Dec 2020 12:21:48 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y192sm8514455qkb.12.2020.12.11.12.21.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Dec 2020 12:21:47 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v3 3/6] mm: apply per-task gfp constraints in fast path Date: Fri, 11 Dec 2020 15:21:37 -0500 Message-Id: <20201211202140.396852-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201211202140.396852-1-pasha.tatashin@soleen.com> References: <20201211202140.396852-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Function current_gfp_context() is called after fast path. However, soon we will add more constraints which will also limit zones based on context. Move this call into fast path, and apply the correct constraints for all allocations. Also update .reclaim_idx based on value returned by current_gfp_context() because it soon will modify the allowed zones. Note: With this patch we will do one extra current->flags load during fast path, but we already load current->flags in fast-path: __alloc_pages_nodemask() prepare_alloc_pages() current_alloc_flags(gfp_mask, *alloc_flags); Later, when we add the zone constrain logic to current_gfp_context() we will be able to remove current->flags load from current_alloc_flags, and therefore return fast-path to the current performance level. Suggested-by: Michal Hocko Signed-off-by: Pavel Tatashin --- mm/page_alloc.c | 15 ++++++++------- mm/vmscan.c | 10 ++++++---- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ec05396a597b..c2dea9ad0e98 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4976,6 +4976,13 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, } gfp_mask &= gfp_allowed_mask; + /* + * Apply scoped allocation constraints. This is mainly about GFP_NOFS + * resp. GFP_NOIO which has to be inherited for all allocation requests + * from a particular context which has been marked by + * memalloc_no{fs,io}_{save,restore}. + */ + gfp_mask = current_gfp_context(gfp_mask); alloc_mask = gfp_mask; if (!prepare_alloc_pages(gfp_mask, order, preferred_nid, nodemask, &ac, &alloc_mask, &alloc_flags)) return NULL; @@ -4991,13 +4998,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, if (likely(page)) goto out; - /* - * Apply scoped allocation constraints. This is mainly about GFP_NOFS - * resp. GFP_NOIO which has to be inherited for all allocation requests - * from a particular context which has been marked by - * memalloc_no{fs,io}_{save,restore}. - */ - alloc_mask = current_gfp_context(gfp_mask); + alloc_mask = gfp_mask; ac.spread_dirty_pages = false; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 469016222cdb..d9546f5897f4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3234,11 +3234,12 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist, unsigned long try_to_free_pages(struct zonelist *zonelist, int order, gfp_t gfp_mask, nodemask_t *nodemask) { + gfp_t current_gfp_mask = current_gfp_context(gfp_mask); unsigned long nr_reclaimed; struct scan_control sc = { .nr_to_reclaim = SWAP_CLUSTER_MAX, - .gfp_mask = current_gfp_context(gfp_mask), - .reclaim_idx = gfp_zone(gfp_mask), + .gfp_mask = current_gfp_mask, + .reclaim_idx = gfp_zone(current_gfp_mask), .order = order, .nodemask = nodemask, .priority = DEF_PRIORITY, @@ -4158,17 +4159,18 @@ static int __node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned in { /* Minimum pages needed in order to stay on node */ const unsigned long nr_pages = 1 << order; + gfp_t current_gfp_mask = current_gfp_context(gfp_mask); struct task_struct *p = current; unsigned int noreclaim_flag; struct scan_control sc = { .nr_to_reclaim = max(nr_pages, SWAP_CLUSTER_MAX), - .gfp_mask = current_gfp_context(gfp_mask), + .gfp_mask = current_gfp_mask, .order = order, .priority = NODE_RECLAIM_PRIORITY, .may_writepage = !!(node_reclaim_mode & RECLAIM_WRITE), .may_unmap = !!(node_reclaim_mode & RECLAIM_UNMAP), .may_swap = 1, - .reclaim_idx = gfp_zone(gfp_mask), + .reclaim_idx = gfp_zone(current_gfp_mask), }; trace_mm_vmscan_node_reclaim_begin(pgdat->node_id, order,