From patchwork Fri Oct 11 12:07:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Fleming X-Patchwork-Id: 13832454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 451C9CFD35C for ; Fri, 11 Oct 2024 12:07:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B48A6B0092; Fri, 11 Oct 2024 08:07:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9647F6B00AA; Fri, 11 Oct 2024 08:07:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82CEE6B00AD; Fri, 11 Oct 2024 08:07:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6558C6B0092 for ; Fri, 11 Oct 2024 08:07:44 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BB4B81411A8 for ; Fri, 11 Oct 2024 12:07:39 +0000 (UTC) X-FDA: 82661197164.22.3AB1399 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) by imf18.hostedemail.com (Postfix) with ESMTP id 65C0C1C0014 for ; Fri, 11 Oct 2024 12:07:41 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=readmodwrite-com.20230601.gappssmtp.com header.s=20230601 header.b=uruVdL+q; spf=none (imf18.hostedemail.com: domain of matt@readmodwrite.com has no SPF policy when checking 209.85.128.41) smtp.mailfrom=matt@readmodwrite.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728648324; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=Al1zJR4+ZVpH1JbQFkIWa/uoqZRxnaZwnAqoh4d34IM=; b=JHCKGrIpgRTDXhg7Scyt7FLcpoeQh9gGSK8w8CpYb4Qtrj462f0ZZUWshuM9jfeTmptCdk jgCMZRs0STWGbJnBkJqpc3xNFZuNFY61cDCm7AP2mllkpP0k14QxciG4nR8m+GFobLQx+g TStEPtopvJufe/m3IH5JGm8/zN2Rf18= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728648324; a=rsa-sha256; cv=none; b=6mbaJebcvfjV7oQ/v4H5JiCrBcCuHZH+h9cSC7b7ozsFsZTS+PXw+EhUiMULjVpubjiwdq NIlJtqQpFd6UDSIuc7oJC281zA+yyhcHB4sePZU54K4Qg84FXj5lhUKtyyVpaugvroi8MV psttXOwYBwM+zP0dzlECRZs/HSPah3U= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=readmodwrite-com.20230601.gappssmtp.com header.s=20230601 header.b=uruVdL+q; spf=none (imf18.hostedemail.com: domain of matt@readmodwrite.com has no SPF policy when checking 209.85.128.41) smtp.mailfrom=matt@readmodwrite.com; dmarc=none Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-4311ac1994dso11157045e9.1 for ; Fri, 11 Oct 2024 05:07:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=readmodwrite-com.20230601.gappssmtp.com; s=20230601; t=1728648461; x=1729253261; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Al1zJR4+ZVpH1JbQFkIWa/uoqZRxnaZwnAqoh4d34IM=; b=uruVdL+qn46C3gWhxyRrZuZQrcfD0OIAWI+7TtutsbolzUtbY4o7YEmm0GEIBm+3Bq ZTQ+nUd2BIdMz83Vr8emnPDDT1clMxPBKRY2j4pPSYD1Y6/YZUTD7JLEHcK6gURwshcw V0jBRRljmAXVDaDU+XQ9xl1AHJClED7UGbKHWvhXJ9CxFBgZPT2fsgKnzuUGmcHCPHf+ artodbh3gNaO/Uvkx23RROr6KKuLTcq4My7KmoHoWj6SIwADVMkpW5mAtDbUT/IimP2c hfidhpVD6Ow7tPVC3wLmrfkzefiytBFtjKwPxtUWcuy/gNnw/yBqF5DC7KLJuDqw9B5/ UpCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728648461; x=1729253261; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Al1zJR4+ZVpH1JbQFkIWa/uoqZRxnaZwnAqoh4d34IM=; b=gFOCHQRJufw53kQHylCtdLrBr4/QdB0g+BxdKwkWdQZMkv3bD/GdJuL4e9s3FOKlML oxLfMwgTYzB9o8OF8hD7xvzWE8f8Yzje4SlMpr8mdsJGwf2G6NB2DTyPsHi1klCZaC+B m2E+mFXhH6rIYy8BGj/TwDj2GyQ2wIIvxSeEUn4pzBDTBBHCsBrvkXEIcM7nqngF6buZ zdyEkvPg+t4OZpjojbNZrjeePPVMNI02W9KGCG+bfOTiUBsOVBSY8b8Rl9cqFSekkCTJ vkYV61NcsVYH7NYP8oMFhrcCezO0Eqd5HH+ndNr/UzZlR/V3TCE7o0EE8J3Vi6OsZYsP Lk0w== X-Forwarded-Encrypted: i=1; AJvYcCW49s57+eN9gWF7PbIWk6ZNyfXWdMlVbhaM720ReTY6f3JqnUl1BDURwU1Nugibt012ewTBHK4KFg==@kvack.org X-Gm-Message-State: AOJu0Yy2Slh5euS1qWEiMn4oCFAB3vftYhO18lsZaM4K73tNppt+jZ21 45GXRTRecxqbZOqllQL1U7djsWP2x19uMNTqsSoZlJHCJ0zl2jEuzULg9j/yv1g= X-Google-Smtp-Source: AGHT+IEE5Py2nNm8l22l9+7wDPQ3SVtWve+Jv5RgOHbELCq+J5dilgxxm8KuJJzcQBPXjSaoSaW9pA== X-Received: by 2002:a05:600c:3b0f:b0:42c:bcc8:5877 with SMTP id 5b1f17b1804b1-4311dee7281mr21889985e9.13.1728648460522; Fri, 11 Oct 2024 05:07:40 -0700 (PDT) Received: from matt-Precision-5490.. ([2a09:bac1:2880:f0::179:225]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-430ccf1f696sm73098755e9.5.2024.10.11.05.07.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Oct 2024 05:07:40 -0700 (PDT) From: Matt Fleming To: Vlastimil Babka Cc: linux-kernel@vger.kernel.org, kernel-team@cloudflare.com, Matt Fleming , Andrew Morton , Mel Gorman , linux-mm@kvack.org Subject: [PATCH] mm/page_alloc: Let GFP_ATOMIC order-0 allocs access highatomic reserves Date: Fri, 11 Oct 2024 13:07:37 +0100 Message-Id: <20241011120737.3300370-1-matt@readmodwrite.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 65C0C1C0014 X-Stat-Signature: y7j9n6ce53bbg1oei8t57mednh8qr3ce X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1728648461-84450 X-HE-Meta: U2FsdGVkX18Z9NLITEztRWq0RjctYnppwF1Ui15h1VtE4AAYKmdYrBaiZGOA8SZxlvEezO0g5EDsP39FXznI8q+AblO3pU+XX+Kgwu/VnkmdzKV3O3ysgWRXXRAHFhzhuy+1vSziWJ/lunoo4kyaQry7Qeq+JtyZeJcZ2jsQCzGEN33qV9XMCb01U7UJaiWgpl40TV39V7uSsKeSs1175tJCjrfKU3XhINQ49NRiWkxOWRITgYZ5DB0yj52Bkxi24JJ5Tou4jI/tN/blvsE2IPbc1AiCmepTsAq/lNiiu9NefJWe4Xgs7yk6YECL6EEFvj6xPlhcuOIre5NKxuFemHCWokyARFrz+WfYnTFbFhe4ca+omdOor+VuPmT4VTIeCC1UKPXKHFYZRB8j35h/lmnPHUAjJjUAuJv/18iwfqUstDhFziHNrJZbTnQ7wS1CPZwHAhKGOFsllX5hJ9OCBCI3LdwQn4ohd1ZkFBntiitTuGDPDdbj1oF7r0G+Ko1AGBGCcyBFsOkCDJtlP3orQanjVm8CMV0z8WfDLza8ALUeiWPR0GMRFSUW891TikTbyhH7mE4Q9ZWD9on65pvkHyjCBSXoKzDq432e23AvXs/UlcqMskvmH6VxqscXUywMXlIaHjxNoo9mZcZD5868/zHjWhOBrr4dIfX/q6Z6gxbk7aQ3DdWoC2l0vwNrFVrdPnPFfAgR3ydbtdu2Mfo+NYtxU8H68GpLA/qtMmW+V6oW+UNmJ0rVy2o2MDPHwCxadI+foF4kRhWpjtKuPOA60xu29vTNSbQEwGO24NEP37Y87rBPIqSwf/ut6txccfsMpoVbczHGujwjWWUZbIY9wvuRi975FIwhJGHvXaL8N6BvJ7IRiPK/2ALpq57HL04U+TNxTE4J7Mv+3qJq0yybcxU09Ikc4XumTbG+MwXcNMVfoTSrJ2IXwDfLGIIYz8pMcZCD/FIZW4CETicRBOF 1IBEhp2r D6gxdU6XTcm/tABR3dB0mCUrs7Kye9BsKJs+Dw6TM9jw/J53mgP7ZK9omE4GLNosCcdJDoJyRa6DGu3OPH8C29WJNbSYPhE05+BuydtOdi0gov8AtAhIvL2vOZAh8iW7ZWqTHDrOlQDoIXfsPhxY3KGm/GPS7l3GfjKtmNbJXmkTVOD2cboBWw6wPVKKC3OKqP1C7esYlpqPPsD3SEoUlK+8RwkmCgtN4X5/gEacHtFpAfl+RYX+RE/XJmJXJ5y1LxUHA3WTQl97LYMiLogOl8zbhJnjJgZmDoliitDS9ThUr0gj75gsHgPtCdOQLC7an7I7Rop6wBOVsncIniMVK3gAOHOnrhTRhRoQzUIUR7GEPfxAOYFa/sRUN2QSX1eAKX793NhLIBV4rFKNxAmA1EcaQBmpkQMgQnKVvkI6mKaoQnTDhtRW/rF66IOUzU8iMH9JZqmji2FlOKtOUzDtFflsFdqMyvB8/RVrC962Cli6MW2SdP41lH+bPFeEddTbhv/3yh8NK/LqmGZ81AHmMy81hNOBRnU1MTmab0FR9alUy5HqyfKdIxlyBX4VZfUZ2bm3bSxnab8KQwvKlt1h5ckazi6MGeynwB1Xto/q6+NXrugE/hhjWnYR0Otg4RaAC3uGF2/ViPhlt9JwOilZ4ZxpJwimaxylWvM2XsFtN4Pi2MnpFCKrtj/BKpQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Matt Fleming Under memory pressure it's possible for GFP_ATOMIC order-0 allocations to fail even though free pages are available in the highatomic reserves. GFP_ATOMIC allocations cannot trigger unreserve_highatomic_pageblock() since it's only run from reclaim. Given that such allocations will pass the watermarks in __zone_watermark_unusable_free(), it makes sense to fallback to highatomic reserves the same way that ALLOC_OOM can. This fixes order-0 page allocation failures observed on Cloudflare's fleet when handling network packets: kswapd1: page allocation failure: order:0, mode:0x820(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0-7 CPU: 10 PID: 696 Comm: kswapd1 Kdump: loaded Tainted: G O 6.6.43-CUSTOM #1 Hardware name: MACHINE Call Trace: dump_stack_lvl+0x3c/0x50 warn_alloc+0x13a/0x1c0 __alloc_pages_slowpath.constprop.0+0xc9d/0xd10 __alloc_pages+0x327/0x340 __napi_alloc_skb+0x16d/0x1f0 bnxt_rx_page_skb+0x96/0x1b0 [bnxt_en] bnxt_rx_pkt+0x201/0x15e0 [bnxt_en] __bnxt_poll_work+0x156/0x2b0 [bnxt_en] bnxt_poll+0xd9/0x1c0 [bnxt_en] __napi_poll+0x2b/0x1b0 bpf_trampoline_6442524138+0x7d/0x1000 __napi_poll+0x5/0x1b0 net_rx_action+0x342/0x740 handle_softirqs+0xcf/0x2b0 irq_exit_rcu+0x6c/0x90 sysvec_apic_timer_interrupt+0x72/0x90 Suggested-by: Vlastimil Babka Cc: Andrew Morton Cc: Mel Gorman Cc: linux-mm@kvack.org Link: https://lore.kernel.org/all/CAGis_TWzSu=P7QJmjD58WWiu3zjMTVKSzdOwWE8ORaGytzWJwQ@mail.gmail.com/ Signed-off-by: Matt Fleming Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8afab64814dc..0c4c359f5ba7 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2898,7 +2898,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, * failing a high-order atomic allocation in the * future. */ - if (!page && (alloc_flags & ALLOC_OOM)) + if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK))) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) {