From patchwork Wed Mar 4 14:22:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mateusz Nosek X-Patchwork-Id: 11420217 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E042921 for ; Wed, 4 Mar 2020 14:23:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5A92E20866 for ; Wed, 4 Mar 2020 14:23:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AjBLDLRa" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5A92E20866 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 576076B0003; Wed, 4 Mar 2020 09:23:15 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5264A6B0005; Wed, 4 Mar 2020 09:23:15 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43B8C6B0006; Wed, 4 Mar 2020 09:23:15 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0250.hostedemail.com [216.40.44.250]) by kanga.kvack.org (Postfix) with ESMTP id 2A63F6B0003 for ; Wed, 4 Mar 2020 09:23:15 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id BB51A49960F for ; Wed, 4 Mar 2020 14:23:14 +0000 (UTC) X-FDA: 76557897108.10.bikes85_7398ee827b137 X-Spam-Summary: 2,0,0,d59e5e0ed9d165ea,d41d8cd98f00b204,mateusznosek0@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:981:988:989:1260:1345:1437:1535:1543:1711:1730:1747:1777:1792:2393:2553:2559:2562:2693:2910:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3874:4321:4362:5007:6120:6261:6653:7514:7576:9413:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12895:12986:13255:14096:14181:14721:21080:21444:21451:21627:21666:30012:30054:30090,0,RBL:209.85.208.196:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: bikes85_7398ee827b137 X-Filterd-Recvd-Size: 5940 Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Wed, 4 Mar 2020 14:23:14 +0000 (UTC) Received: by mail-lj1-f196.google.com with SMTP id x7so2240726ljc.1 for ; Wed, 04 Mar 2020 06:23:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=DuM4SHf1sVSRFd0iAfPg/3FUWAci4sdx8YlPTtxLHUA=; b=AjBLDLRaVZo2u3eVMtEp8eF/n6W+A+BSXG1x0tQJO17cbACepFUj9lAAQYvLH+nwIT kFtbfwM6ADxCwKKha/V5bMdPvRNkj1PBa2ds4tOgWDCS6UxryiCc2PLfDFGA1mg+/qtj tu/segWSBZ4AjX0KJ2fF8H9CqDT3NMzYEHWfBrZnJOumiDvXYN+MFgseacU3MoUd5rmR 80fqvqmWZzMTG7ig66oQpPjGKJ/7xyzSmlbRFGGcWWmn/lg+anX8O1BnCn3ZI5Yy4HL7 LwIbnYpjhvzSEUpQbMAxEbvCmBmG/Y2P2vba5qmJ9i7wmARKsj4ZCCkHv+I1LYEuEpDI 6ffw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=DuM4SHf1sVSRFd0iAfPg/3FUWAci4sdx8YlPTtxLHUA=; b=pv4S43rFcZ4i3JPNEEJgGmnH6w+r8BPwRoXugJeNur55SSUahDVTWQl7pJxS08rQJm L3oycs4/eB0J6Z7f2LxFHxdS5vfoJJD34e88ndw6P9Xil/fkKoxvld+ShUMXoasyR46t ku/2aQEUYheOmAIq18i9xbaO9gwhJvWcLCbQE3DNiqgKApRxXDeFMpZ6ee6Wax/TGj07 rjhKo+0zRVsg8LIZV/673NWkfHS3r1ZrJHOhqYOkGVGWHJDpWWj9CCVGDEDnEO1/UPAw o08qdpK71InL+EFZCqGfWGhgWRNRBsZOqC+DkzZP1215kin7BQ1rpaaw1NoLmYDyjMe0 AL5g== X-Gm-Message-State: ANhLgQ1+ZJ9NuKeP+CPpMU93MjcbfXQLT+4uWq9U3zTYmf182BjYJDs4 tG/cUVXcSNaBLcx8OEPpMWVEFGd0sjU= X-Google-Smtp-Source: ADFU+vvKA4imZdpFdR2Wcr5JQLvnEaAwkgmbc8Utlu+oNaDX0sfIzjqzNhUBEv1GqpQkCoaeD+Fm+Q== X-Received: by 2002:a2e:3807:: with SMTP id f7mr2123355lja.103.1583331791595; Wed, 04 Mar 2020 06:23:11 -0800 (PST) Received: from localhost.localdomain (188.146.98.66.nat.umts.dynamic.t-mobile.pl. [188.146.98.66]) by smtp.gmail.com with ESMTPSA id z23sm9575712ljg.99.2020.03.04.06.23.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Mar 2020 06:23:11 -0800 (PST) From: mateusznosek0@gmail.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Mateusz Nosek , akpm@linux-foundation.org Subject: [RFC PATCH] mm: Micro-optimisation: Save two branches on hot - page allocation path Date: Wed, 4 Mar 2020 15:22:30 +0100 Message-Id: <20200304142230.8753-1-mateusznosek0@gmail.com> X-Mailer: git-send-email 2.17.1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mateusz Nosek This patch makes ALLOC_KSWAPD equal to __GFP_KSWAPD_RACLAIM (cast to 'int'). Thanks to that code like: ```if (gfp_mask & __GFP_KSWAPD_RECLAIM) alloc_flags |= ALLOC_KSWAPD;``` can be changed to: ```alloc_flags |= (__force int) (gfp_mask &__GFP_KSWAPD_RECLAIM);``` Thanks to this one branch less is generated in the assembly. In case of ALLOC_KSWAPD flag two branches are saved, first one in code that always executes in the beggining of page allocation and the second one in loop in page allocator slowpath. Signed-off-by: Mateusz Nosek Acked-by: Vlastimil Babka Acked-by: Mel Gorman --- mm/internal.h | 2 +- mm/page_alloc.c | 23 +++++++++++++++-------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 86372d164476..7fb724977743 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -535,7 +535,7 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone, #else #define ALLOC_NOFRAGMENT 0x0 #endif -#define ALLOC_KSWAPD 0x200 /* allow waking of kswapd */ +#define ALLOC_KSWAPD 0x800 /* allow waking of kswapd */ enum ttu_flags; struct tlbflush_unmap_batch; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 79e950d76ffc..73afd883eab5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3609,10 +3609,14 @@ static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone) static inline unsigned int alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) { - unsigned int alloc_flags = 0; + unsigned int alloc_flags; - if (gfp_mask & __GFP_KSWAPD_RECLAIM) - alloc_flags |= ALLOC_KSWAPD; + /* + * __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD + * to save a branch. + */ + BUILD_BUG_ON(__GFP_KSWAPD_RECLAIM != (__force gfp_t) ALLOC_KSWAPD); + alloc_flags = (__force int) (gfp_mask & __GFP_KSWAPD_RECLAIM); #ifdef CONFIG_ZONE_DMA32 if (!zone) @@ -4248,8 +4252,13 @@ gfp_to_alloc_flags(gfp_t gfp_mask) { unsigned int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET; - /* __GFP_HIGH is assumed to be the same as ALLOC_HIGH to save a branch. */ + /* + * __GFP_HIGH is assumed to be the same as ALLOC_HIGH + * and __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD + * to save two branches. + */ BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH); + BUILD_BUG_ON(__GFP_KSWAPD_RECLAIM != (__force gfp_t) ALLOC_KSWAPD); /* * The caller may dip into page reserves a bit more if the caller @@ -4257,7 +4266,8 @@ gfp_to_alloc_flags(gfp_t gfp_mask) * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will * set both ALLOC_HARDER (__GFP_ATOMIC) and ALLOC_HIGH (__GFP_HIGH). */ - alloc_flags |= (__force int) (gfp_mask & __GFP_HIGH); + alloc_flags |= (__force int) + (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); if (gfp_mask & __GFP_ATOMIC) { /* @@ -4274,9 +4284,6 @@ gfp_to_alloc_flags(gfp_t gfp_mask) } else if (unlikely(rt_task(current)) && !in_interrupt()) alloc_flags |= ALLOC_HARDER; - if (gfp_mask & __GFP_KSWAPD_RECLAIM) - alloc_flags |= ALLOC_KSWAPD; - #ifdef CONFIG_CMA if (gfpflags_to_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA;