From patchwork Mon Aug 21 18:33:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13359736 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76797EE4996 for ; Mon, 21 Aug 2023 18:37:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7BE48E000F; Mon, 21 Aug 2023 14:37:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E2CA38E0002; Mon, 21 Aug 2023 14:37:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF3428E000F; Mon, 21 Aug 2023 14:37:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C13E48E0002 for ; Mon, 21 Aug 2023 14:37:45 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6C683B2280 for ; Mon, 21 Aug 2023 18:37:45 +0000 (UTC) X-FDA: 81148970490.12.58BE263 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) by imf01.hostedemail.com (Postfix) with ESMTP id 48ED240014 for ; Mon, 21 Aug 2023 18:37:43 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=ZMy9LilU; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.44 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692643063; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=hxW29R58ZmFpg/rPJgqQ3VkRYdS419AYIG4Hav75F/o=; b=UxlYMFYdPCymhEOmrnAEzg3Hc4KQ7VA0C+BYLHZAXCsb/8vVqpoZMzFChHOa8hmYCgrHVY JE4g+0B7aZKS64nQBRlMOiul8Q6PVC/PncGGZtAh+t9v9qw/wrHSum/o5D/exAkriwR7AU y60NfG231OA9+sPlWwC3gxS9JaIUS3g= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20221208.gappssmtp.com header.s=20221208 header.b=ZMy9LilU; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.44 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692643063; a=rsa-sha256; cv=none; b=SeEBpgvAzWKp9k1I6XTpJ5Zs+YEcBq1K5jUePxNZXnonjngiVZmWNMyiuB4DQiUJqVrNZB y35l+TOwOwO069r7PrZagVKE8vt0qpw/KUnneTd0xYg7IkeliM5c3b/jRcdFfhL1pRgVzy D+iyaf4qGA1vbJyyW78Qmru2bzR7IDo= Received: by mail-qv1-f44.google.com with SMTP id 6a1803df08f44-64f42fcd809so6586946d6.1 for ; Mon, 21 Aug 2023 11:37:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20221208.gappssmtp.com; s=20221208; t=1692643062; x=1693247862; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=hxW29R58ZmFpg/rPJgqQ3VkRYdS419AYIG4Hav75F/o=; b=ZMy9LilUbtHaMB8a2Bc9S+CTYD8of9JLuHTQfeCD4NwFL27z0kKncoczddhT5Hkg0z x5QqUUE/AhnBHqO/PTpGJukrdQd3Eqobf1Ghno3MwWAKR+zNZZcB1BAlcriQMWvPUExr xqo9fFWzITkM0TDqdec52HjfHdgkDP3lZp0NV0FjzjO5HvOQa08PUmglBsDMUyPhMh29 tBmxeC3Jr7ot3YQNZGIvnRCo/+UBOuEJM4ibv7jFsftN5kMb3sX3nBaHH3LN8OuxsiXU sKqbovAAYS7ZDY0aVjDgxSPjTLLUIz64y30G25tdyNzmndtcEBtCaUnLl735iwhhfqXT lJbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692643062; x=1693247862; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hxW29R58ZmFpg/rPJgqQ3VkRYdS419AYIG4Hav75F/o=; b=G5zKj9Z94DZM4Ww9dyaJg8qpTm19689y4GpEnjLU/7TkL4F4sElxa/41tJxfQWteGc Tyuh6Sw7dfgcNwm2Yr4zMMGGjChLe6im2laDTvO1UK4tgZzU7D7rfI3CNHOA9aYYxKjs z40gyHglXOoPyL0WEOWK+O81L9vi+wapV7zrj8BpgysqRRuR1l3fZQSzvIQdy27d5Mfh 1m2c17pVCxjS9MsQ2+Zx4mH00PwY6mkzTgFOhbJioKeIqcwMk5zjGqAYjfrJFmq5SlCQ pFqUBzpwZ6YMfxCkPMDBBglfteK0KKgtNYfHtjTyK5TEvQ9M82eZobfZD3uGaApOfO5r 7V6A== X-Gm-Message-State: AOJu0YyLP4oLeImkQl908JdcpmA9HenJ+D2hpyCmaPN1TahueCJVcHO+ BFqOgtEtbacijBofisc4nY3xyA== X-Google-Smtp-Source: AGHT+IGy8J3qOalYFiRMZVkh+1jZEMs5H+0oIK7/2greDFHUnnDFEMOD0EUG29cPhWcOuKvVixoW0w== X-Received: by 2002:a0c:c991:0:b0:63d:70f6:8f6f with SMTP id b17-20020a0cc991000000b0063d70f68f6fmr9159374qvk.43.1692643062274; Mon, 21 Aug 2023 11:37:42 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-699c-6fe1-d2a8-6a30.res6.spectrum.com. [2603:7000:c01:2716:699c:6fe1:d2a8:6a30]) by smtp.gmail.com with ESMTPSA id m5-20020a0cf185000000b00649d9781acesm1453832qvl.2.2023.08.21.11.37.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Aug 2023 11:37:41 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 0/8] mm: page_alloc: freelist migratetype hygiene Date: Mon, 21 Aug 2023 14:33:32 -0400 Message-ID: <20230821183733.106619-1-hannes@cmpxchg.org> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: 48ED240014 X-Rspam-User: X-Stat-Signature: p8zjtwiots1wsqb7nq6raccq4iiwmhg8 X-Rspamd-Server: rspam01 X-HE-Tag: 1692643063-439765 X-HE-Meta: U2FsdGVkX19gZhyjvCZoJMgvYGmbBYRpaCZcsnND6Q2veGsVnk5Rm9vdpQDFN07LuNbfD9mal/s2tbMBeomTa+yHUWLl/YewC2/Zx4w+fI1C8vpKHPeH7kdPMZtR2Hbk179ua1ygDMadgL87eFJTfszvuhe0mq+dSsvBJ6pgMihejrWyvx16uIGU41A33h5qrFGIqZScBexKUlXE7NbKDtHG5XowmviAgEovTsEQbjuiDcB3nZzSDFeLjKASP6NNEwwNqB1/fXurS7thR8wacMIGwEx/+tTeOw5AGLfgWigTmNX9tnNIqYMFd/BnyfteDJfVXUqalZoGiAZlPBThgJH09mH0HsgJgOqLEdYNh3ixkK/6WWAFbHlrgbnf/XKFUuZXnpl0cFSl9Kz6ICLZvmTnUpzAkaUsR1IiBIqZXdA8mhwYIYBpCOBBFfXPDt0NZHuxhaoTmHEFbjDdUl1n3BP72mU93NNZPHyhoxoWB8m7tjCwRYpmbPxZiCwz/h8f+PTxo/p46PbBtQKHHwGh+vgfzISW6i0bvDzAa/BZo5V5GwruvvIRVY/kGyTlUvn3/RyXUrWfDEA6+7Mydprz+WkFLA/YRDn5WDG+YASzRZNhQ6yEwsUJfovlZwsmHaQ/YqGl9Mde5uFNrSBZmELS6+wP0tylYdZVgjWJcS/7xERHuYF7oqUKpQ4EV5WQHh3nj8+bYG3x2JHcjDg7XYn57qd+KJVQYFcAxYYeZ3by332/kmvCqLiho9Fn8xghETv+SA4P7KIgVQqzeuk71e1xHaJ15icJ/HdM/eYvrdhUvcZ7Dt7FIDYVwl+pKS6Odf3zPJCTLId4EqawJZfr0Jex8W4/9O0OwC3f0EvTRjT+dxrtywZfOlmQS0JJeyi5nsXi5b9cMVPMh7PbESS79Gic3Vne/DftkUgsmdHNjAjPcA5hbVCzsd+mP7AeTNxeL+n3h7vUm48a/W+caw9yE1m HHokkJ3t QLwrPB4wCOkPSZnjzqsUlcmmUnoVWX6fDbEyM25L7fA/ooiQyotAZuooYf/QgsZAyLHTWpxQvSDqRCP2SWxa//1M6SLnGrCqYcGYYv2r0OkXsaQihMeoeoHwCp9ozYclAXeDfZ+YBv5b4xsppkYNCeJkEZV09Y8TU1G0U1yT5Wzl/eCu3FoYceBGDvKTPza47t8Kj6AsKIztsxXXYqrk0tJmQB+N2xJUW3GY5JARzc60zyyzuDIxthFXw+4tb1PZ+1M+5KNKX0aRXIRf8uxDA8avv2PMIaCzKcIrHNXRqdyyFg4n7vVW2b02aywQlEQR+oV9Vqei++oLHO7l636w15IvIpRqX73t4R5U7BkDlz/SFtn2KLQbFwGGSu6ugSn/re6VkKzGZBOxgsrH+OYe4L+vPfoEm059y72wQRdfFoD+bz+lfmCvIsgjjE937qOpm13VjYuzN6vsezvJPNlryA8T4A/U5U4soBKaohqNcNY2P++9cYNrkjzi7IwtehWcJH5/kUa2e4LYiqnmZ4exXE+vNIZKeIsqoXF3gYet0tbP+J04= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a breakout series from the huge page allocator patches[1]. While testing and benchmarking the series incrementally, as per reviewer request, it became apparent that there are several sources of freelist migratetype violations that later patches in the series hid. Those violations occur when pages of one migratetype end up on the freelists of another type. This encourages incompatible page mixing down the line, where allocation requests ask for one migrate type, but receives pages of another. This defeats the mobility grouping. The series addresses those causes. The last patch adds type checks on all freelist movements to rule out any violations. I used these checks to identify the violations fixed up in the preceding patches. The series is a breakout, but has merit on its own: Less type mixing means improved grouping, means less work for compaction, means higher THP success rate and lower allocation latencies. The results can be seen in a mixed workload that stresses the machine with a kernel build job while periodically attempting to allocate batches of THP. The data is averaged over 50 consecutive defconfig builds: VANILLA PATCHED-CLEANLISTS Hugealloc Time median 14642.00 ( +0.00%) 10506.00 ( -28.25%) Hugealloc Time min 4820.00 ( +0.00%) 4783.00 ( -0.77%) Hugealloc Time max 6786868.00 ( +0.00%) 6556624.00 ( -3.39%) Kbuild Real time 240.03 ( +0.00%) 241.45 ( +0.59%) Kbuild User time 1195.49 ( +0.00%) 1195.69 ( +0.02%) Kbuild System time 96.44 ( +0.00%) 97.03 ( +0.61%) THP fault alloc 11490.00 ( +0.00%) 11802.30 ( +2.72%) THP fault fallback 782.62 ( +0.00%) 478.88 ( -38.76%) THP fault fail rate % 6.38 ( +0.00%) 3.90 ( -33.52%) Direct compact stall 297.70 ( +0.00%) 224.56 ( -24.49%) Direct compact fail 265.98 ( +0.00%) 191.56 ( -27.87%) Direct compact success 31.72 ( +0.00%) 33.00 ( +3.91%) Direct compact success rate % 13.11 ( +0.00%) 17.26 ( +29.43%) Compact daemon scanned migrate 1673661.58 ( +0.00%) 1591682.18 ( -4.90%) Compact daemon scanned free 2711252.80 ( +0.00%) 2615217.78 ( -3.54%) Compact direct scanned migrate 384998.62 ( +0.00%) 261689.42 ( -32.03%) Compact direct scanned free 966308.94 ( +0.00%) 667459.76 ( -30.93%) Compact migrate scanned daemon % 80.86 ( +0.00%) 83.34 ( +3.02%) Compact free scanned daemon % 74.41 ( +0.00%) 78.26 ( +5.10%) Alloc stall 338.06 ( +0.00%) 440.72 ( +30.28%) Pages kswapd scanned 1356339.42 ( +0.00%) 1402313.42 ( +3.39%) Pages kswapd reclaimed 581309.08 ( +0.00%) 587956.82 ( +1.14%) Pages direct scanned 56384.18 ( +0.00%) 141095.04 ( +150.24%) Pages direct reclaimed 17055.54 ( +0.00%) 22427.96 ( +31.50%) Pages scanned kswapd % 96.38 ( +0.00%) 93.60 ( -2.86%) Swap out 41528.00 ( +0.00%) 47969.92 ( +15.51%) Swap in 6541.42 ( +0.00%) 9093.30 ( +39.01%) File refaults 127666.50 ( +0.00%) 135766.84 ( +6.34%) The series is based on v6.5-rc7. include/linux/mm.h | 18 +- include/linux/page-isolation.h | 2 +- include/linux/vmstat.h | 8 - mm/debug_page_alloc.c | 12 +- mm/internal.h | 5 - mm/page_alloc.c | 382 +++++++++++++++++++++------------------ mm/page_isolation.c | 23 ++- 7 files changed, 230 insertions(+), 220 deletions(-) [1] https://lore.kernel.org/lkml/20230418191313.268131-1-hannes@cmpxchg.org/