From patchwork Thu Oct 24 15:12:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13849339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2287ACE8E89 for ; Thu, 24 Oct 2024 15:12:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 544036B0082; Thu, 24 Oct 2024 11:12:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F3416B00B6; Thu, 24 Oct 2024 11:12:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 344A16B00B8; Thu, 24 Oct 2024 11:12:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0FC5A6B0082 for ; Thu, 24 Oct 2024 11:12:50 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4F84840F89 for ; Thu, 24 Oct 2024 15:12:39 +0000 (UTC) X-FDA: 82708837470.26.35E8DB5 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf16.hostedemail.com (Postfix) with ESMTP id B96B818001B for ; Thu, 24 Oct 2024 15:12:28 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=w4NtCV+1; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=cHtBXh6T; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=w4NtCV+1; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=cHtBXh6T; spf=pass (imf16.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729782690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3E+SnM9nTIdJirG48OGIur8Xm4drVI2DTc9+V4rDTS0=; b=TM//Y1FoitlGwdhPHwwJ9XCdp5Zce8N8S6KnAKnhjRyQklixjFo5qWqloB5KQnV3qNEWjC Nqu0x+urOK10ryA2Fdr5izDaZ5pgYh6m1C5kXOc4eJqSP3YWQ123qCY1kJ/hKt1mO0brtQ Ll4r3jjjFL71D4xfVxxvUGyI2/nC4kI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=w4NtCV+1; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=cHtBXh6T; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=w4NtCV+1; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=cHtBXh6T; spf=pass (imf16.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729782690; a=rsa-sha256; cv=none; b=UCa1EV6xjDAO6wq1+MOjwXmC3EpvsOdOzGD1YBEpVbvQ2Lpab7NaCbFnFukYpnhgeG57lq 9qnQ2/7zHRvHhpqYa2gw3sz/mGA0xCFbL3E4JGUUfn06UJic4ESFdo8MqZUU2yxsepSEK+ 5RlnOCUB/8pYOAbSbFZunYFGUWBPfHI= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 241C722153; Thu, 24 Oct 2024 15:12:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1729782765; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3E+SnM9nTIdJirG48OGIur8Xm4drVI2DTc9+V4rDTS0=; b=w4NtCV+1PXhRANojgclEsLkM44CgDPAFrn96mLtWcW6qnRmFTQKcozW1Dz7vlMgENemK/1 cbV3ECymKdxQxrNLrGkmtsu+/+m0Hy9fn7Kd4TWpDdtmG4UHdJ3TX9/KjUuFfNsak5mhin xjo7jbcztY8JT360seRaAhz/26JC8io= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1729782765; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3E+SnM9nTIdJirG48OGIur8Xm4drVI2DTc9+V4rDTS0=; b=cHtBXh6Tz4HByMzeg7ITW0IwfohzkLaV6JNhFogCv4LeQf2cc666QVxm6eU4aBKNHkSp5l A73bdgqmmvZl+/CQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1729782765; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3E+SnM9nTIdJirG48OGIur8Xm4drVI2DTc9+V4rDTS0=; b=w4NtCV+1PXhRANojgclEsLkM44CgDPAFrn96mLtWcW6qnRmFTQKcozW1Dz7vlMgENemK/1 cbV3ECymKdxQxrNLrGkmtsu+/+m0Hy9fn7Kd4TWpDdtmG4UHdJ3TX9/KjUuFfNsak5mhin xjo7jbcztY8JT360seRaAhz/26JC8io= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1729782765; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3E+SnM9nTIdJirG48OGIur8Xm4drVI2DTc9+V4rDTS0=; b=cHtBXh6Tz4HByMzeg7ITW0IwfohzkLaV6JNhFogCv4LeQf2cc666QVxm6eU4aBKNHkSp5l A73bdgqmmvZl+/CQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 076881368E; Thu, 24 Oct 2024 15:12:45 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id d4UWAe1jGmcDPwAAD6G6ig (envelope-from ); Thu, 24 Oct 2024 15:12:45 +0000 From: Vlastimil Babka To: Andrew Morton , "Liam R. Howlett" , Lorenzo Stoakes Cc: Jann Horn , Thorsten Leemhuis , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Petr Tesarik , Vlastimil Babka , Michael Matz , Gabriel Krisman Bertazi , Matthias Bodenbinder , stable@vger.kernel.org, Rik van Riel , Yang Shi Subject: [PATCH hotfix 6.12] mm, mmap: limit THP aligment of anonymous mappings to PMD-aligned sizes Date: Thu, 24 Oct 2024 17:12:29 +0200 Message-ID: <20241024151228.101841-2-vbabka@suse.cz> X-Mailer: git-send-email 2.47.0 In-Reply-To: <2050f0d4-57b0-481d-bab8-05e8d48fed0c@leemhuis.info> References: <2050f0d4-57b0-481d-bab8-05e8d48fed0c@leemhuis.info> MIME-Version: 1.0 X-Rspamd-Action: no action X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: B96B818001B X-Stat-Signature: xx7kq14qjabozyi8omd9xnuwyfwh3mzw X-HE-Tag: 1729782748-257267 X-HE-Meta: U2FsdGVkX1+zGbuBHCqoNF6WBxc5rXS2WMj3MoTKBHXGeGX6MjCB3/Yh2SRbu/ydH4jt/gKgQqV3lk+SDcY52CF9l2Sz5EgpxLV4qzX6SIGafU/FDXPQAE1z0/FpG9tHUGYOvo27BAkzRtySMcz9Wb6s3X3s24srz4kjREGOSY1xUWQYzXzFf5f+rdkcppxe3jul2vQ5i59zw74AxJ1Kk+2ml7n7QKP5THkSNn3a7HgMYCPRenUS8LnL/R7qPeUVoNJCfPM5zEzIYEyC4l99bRyIMyG6Shxu2xS3TfWOOhM9MfY+pwVLjXI/lLXdjQMyxrETEA5ATOeh8AFy3VErgSmmvKU6XQb0349+7lbK84chdB0/HqQP6Bsj7UVp1W7M+IeGoZXAtMuLY1TdIh9mEEIcAOW4ackKlfRqPhNEM1p9EVBn9BnJ1RGnfcQ6t+/mbEpjksMo8UTMOxV6neMZEHZRoVKUe7XEQp88ZcsKm4Ctkiu4bwSKst9uoKxV6luI78JMMPhzs7uKP3I+uC7fJpPgbVT3Jom+1kYOvEG4DaWLQosIZyoTwqVEXzet8gEnRvRJieXMktkHbVPb8szKyH/t+s4he+z9sVWdqvbcwhkZg+mP7vaRu50X1FqfxrpLUw04xJ2AErUTRJ31aw0yiRpbTCo0GG66hvhAZJK5thk14RbJ3j56rK3c9y/gIdGjXzEiCzA3ePanSvditqa4R/Nqka603grEW/HDIFyBON0b7JowvaVwZZVf4eV6TKRtXMVUHEyaeMjqcJI63Ido2m0D5Xh8A489mHPPjKFWs2c7tN3IY646JPmKT8+cjbsr0qH4ZxWbynhduPf+XAphkJCPp37wCUxLP2mlft8h8giiG6VGGm4Jh66krMA9VOfuUEKAWrpho/Kb72PuahpAbtpTfz4ArBZuWKoYDU98nu87W8WevPdnFWT26riyAv5E7Bu76yObKeFI8Y307xC l3btC8pw pncEHmCmdeOIN3pA0vOiNBNlN5OIFd/B6uUS8Ee4DI0S3K7vQkhHvGdWv2pABUitDyPTjg2+hdLr6zFAOO6XnzbAbWDUa4XP+LVT01ujwe6U0SEqncPNmjCY4c2geKl+RBmjH68Rl91kmNmlEFqPXnZXpTmp5wu3fv4dHr3Il1445pHYNoNUdedSedH4YdbG7bR0CHhp4DifVanxYf7t6Y70uHUJp635zW4RTj0xWT2zXB477Up3DIJwNSdeTMkJrjnxxgY37KY3xwcyDlHjoFBp2FKAFaaj3qtYrr4zG1kBfhKB7zb6fXpF0JeDW+SHYBgJhzISPMXE7lWh/DldqAAkaClrTJqr+c3cuKNxShziDRiFmHjqohm+JQUopu4XKFVgWRSpkBtjNT5ilVPS9fc46oUImd03qD1i/nYkmrWorlIeWbk8blrkb6LI5N7gcYu5BYBs4Baw7+3LRXWsViTpFGXs3/cCjoKeaYBHHoJI2yFIAyfFm5JtFHQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") a mmap() of anonymous memory without a specific address hint and of at least PMD_SIZE will be aligned to PMD so that it can benefit from a THP backing page. However this change has been shown to regress some workloads significantly. [1] reports regressions in various spec benchmarks, with up to 600% slowdown of the cactusBSSN benchmark on some platforms. The benchmark seems to create many mappings of 4632kB, which would have merged to a large THP-backed area before commit efa7df3e3bb5 and now they are fragmented to multiple areas each aligned to PMD boundary with gaps between. The regression then seems to be caused mainly due to the benchmark's memory access pattern suffering from TLB or cache aliasing due to the aligned boundaries of the individual areas. Another known regression bisected to commit efa7df3e3bb5 is darktable [2] [3] and early testing suggests this patch fixes the regression there as well. To fix the regression but still try to benefit from THP-friendly anonymous mapping alignment, add a condition that the size of the mapping must be a multiple of PMD size instead of at least PMD size. In case of many odd-sized mapping like the cactusBSSN creates, those will stop being aligned and with gaps between, and instead naturally merge again. Reported-by: Michael Matz Debugged-by: Gabriel Krisman Bertazi Closes: https://bugzilla.suse.com/show_bug.cgi?id=1229012 [1] Reported-by: Matthias Bodenbinder Closes: https://bugzilla.kernel.org/show_bug.cgi?id=219366 [2] Closes: https://lore.kernel.org/all/2050f0d4-57b0-481d-bab8-05e8d48fed0c@leemhuis.info/ [3] Fixes: efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") Cc: Cc: Rik van Riel Cc: Yang Shi Signed-off-by: Vlastimil Babka Reviewed-by: Lorenzo Stoakes --- mm/mmap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index 9c0fb43064b5..a5297cfb1dfc 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -900,7 +900,8 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, if (get_area) { addr = get_area(file, addr, len, pgoff, flags); - } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) + && IS_ALIGNED(len, PMD_SIZE)) { /* Ensures that larger anonymous mappings are THP aligned. */ addr = thp_get_unmapped_area_vmflags(file, addr, len, pgoff, flags, vm_flags);