From patchwork Mon Sep 11 19:41:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13379605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE2E8EE57DF for ; Mon, 11 Sep 2023 19:50:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E52526B02DB; Mon, 11 Sep 2023 15:50:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E02C66B02DC; Mon, 11 Sep 2023 15:50:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF2376B02DD; Mon, 11 Sep 2023 15:50:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C0BF26B02DB for ; Mon, 11 Sep 2023 15:50:35 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8C5311A0AF4 for ; Mon, 11 Sep 2023 19:50:35 +0000 (UTC) X-FDA: 81225358830.14.B519957 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf05.hostedemail.com (Postfix) with ESMTP id A5F61100023 for ; Mon, 11 Sep 2023 19:50:33 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=c5OGRHoo; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.179 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694461833; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=bUhpLn3tgSPZYhOpcPmf1kQLHZfyHW/pDHKf5NypG+s=; b=WyJJoQDXjb+Ym2oBm7cSsYfoGT7dXDUMtf9vXeT4RrChOVx81cMxsN7zSrtGEHz1yPZ7tV ++/dIBxewIFv6voaWwtI8RSJMS8kbz48VYKeTnXoFDMCIiqG9pQ3FqbCr7eBHf/4jGb6Gk 9SrkLBnuCnQoE2+jXScASMOnsHDyzVo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694461833; a=rsa-sha256; cv=none; b=IUdWZcBCGFTw/NXHegW/Fz+BZNFazlDLYQyHY/ON9TEv0JZ/rYtkVN91M9B9UfrDkS6D79 LLhdUo5DKzrKjagkhRFw17puWtd0oOKzhtxIhYSRJxymkBcEJHj3h0pPszTSsDFVt/dgc5 QoIFP62AZOTCCxJDmt9L4gGbW7buaKQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=c5OGRHoo; spf=pass (imf05.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.179 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-76ef6d98d7eso257962485a.0 for ; Mon, 11 Sep 2023 12:50:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1694461832; x=1695066632; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=bUhpLn3tgSPZYhOpcPmf1kQLHZfyHW/pDHKf5NypG+s=; b=c5OGRHooXlNV7aqdTQERADVwHm+siKGoOboWxQcqw0i+KeIfHJ5OqUKfO4Bt6RajwJ b8z82lU9mNXJtR4U1reLhYwVhk49ClI34vfY1tL7hm/Nn/IaCaCk6aadABYJoRO06FxA ls0fs6lyfsE7CL6Fwznc3apPS2pZc08en6rKTdMfBt7aaiaC/R98lu9pAsJTmUrAVeJM 4lbaXT3v2GgAcU1NDDdKYub6ieqHMsJ7nBOGih8Alhq4fscYhwUybJ2EkXuvdLnuKFM/ r8jVrsB7RQNNonXOxp23fWBXgybl/TUC3xOZ4Nyltd0r3SLo4xPD9bH4hAfee2Tzfau6 vb/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694461832; x=1695066632; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=bUhpLn3tgSPZYhOpcPmf1kQLHZfyHW/pDHKf5NypG+s=; b=PTIWET8SFH71StFv0d31zogL9mbqXcwhzBuo/J4OwJDBJOLObQnViFF+qgcZ0leNgW speNQk8oWsn7Y7wWFM7uffGLJsYAWbmuyEAv4KjCoLgP1YewgyuoBWyhLKTKXTfN1kkZ f39wvbov3lppLYPUFCbGCIgY4lKeIgHy+KHnTqba1UqQMy0aNR30auMVX1aVvmdmpicx d6ROkkEjk6d/oa7wIKcOcPrrfU5e3UahcEwSDoacpqaAp8Lac4F7TyqhZmXDimxV7pAZ VVYuOg/zHFIzCVo0qeAn7cWN01fBjoVd4CHetmIqlxoRGRN+b6m32XKqUZPBFlKGbx6j sjAw== X-Gm-Message-State: AOJu0YyVGo2W/eO127isqaeTGYDJHYGAkxBPJ4GMXcEN0uy6eghXlJRt CVid4rjJibypvBkvYO2somRCyA== X-Google-Smtp-Source: AGHT+IH6zew5o97PM9FqBHRV0vZ420ai/jzzZn1v0Pm7AewGkf/LsavwKwaGmeu3UPe/OWVG6DMtOg== X-Received: by 2002:a05:6214:5342:b0:64f:4e7a:bc4d with SMTP id kv2-20020a056214534200b0064f4e7abc4dmr9236075qvb.43.1694461832568; Mon, 11 Sep 2023 12:50:32 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-3012-16a2-6bc2-2937.res6.spectrum.com. [2603:7000:c01:2716:3012:16a2:6bc2:2937]) by smtp.gmail.com with ESMTPSA id y2-20020a0cc542000000b0064733ac9a9dsm3161975qvi.122.2023.09.11.12.50.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Sep 2023 12:50:32 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Miaohe Lin , Kefeng Wang , Zi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH V2 0/6] mm: page_alloc: freelist migratetype hygiene Date: Mon, 11 Sep 2023 15:41:41 -0400 Message-ID: <20230911195023.247694-1-hannes@cmpxchg.org> X-Mailer: git-send-email 2.42.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: A5F61100023 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: k1hagjsbuat6eeft9a7x6ud9yro7q6ui X-HE-Tag: 1694461833-640815 X-HE-Meta: U2FsdGVkX194aZlGfi7O/ptKH7oGVFD1bZIh5FKrjqBul1vWeF6WbJQAX+s07nt9GIpeWXKWvMDRf+UBS5pUq0YaCgjGe8sqaElhLVlHPZCduYoBUkOnu+A4qOiHGwXxckNPiZp8pzw3lWl3RbfTYto9QBi11ACdX3EGSyFD3FTdjbrfGclZCtiUL+J5YPByc9HXZi8oaV8PCanqhkX/wjC5WXUjRY3NwkEq1zEUFjnsamoA1Ix0YJKVxvcoL2c5xuNPgnMYe/BSGiHs7y5+V/dfEN7LEs40DJCcFqgM0hrHu1Z0juhdG5SjJt4nG2pplfvyiXO7GmPlO91X2n6Pl2VIO/sirWdDeCWGhgad8ZH24ARAjBpnw3clGRiGbbUAhjlRhXFNxgf0Z9ytoQnxZdeiftHmYrEmNRacqKVWnaYpo2J0ZpAOX2ap6x6/+sBZ24RoJphkgRa3aLB0P3EqUUcTp7u8M7vyy6EyRaRVTfPVROwvG/jMTzmt0VQOBa/Ku59uwoIC9Q6+O5sg+vheSYEWLO9wg5enRPGqj53TFtc4MiyFm58CyeNlqXIjjQ4wbkUwiZ4KmaW6TwjLUVKdqCttfA5hyYKC5OicmyLmBzvD2BTP85JtEnlp096TT4trROncCyw288Ux6IR3h1jv16YBu1Xa0SNRbCDuNeQ3cUGcM1G8KRxn0/nxuf2oIRSusbPWf5kNSaqi78VNoGmgtLl3C7K4VuLsLmMgfJr4qhs1cZmi+1Wn/Am6fq4x6UOno2yJ9oLNHoQNMENMrK7TP9jNBBBBesIAtT+ld3Ybeqm46BPbzw52TdBCDPF/J719TdwnKtkz17WZZeMRuyuvpTpOhnS3oejxq0G8Y/xhQ0iJ7acZI5PacWbRvvzbIoHH9VK0r2lRMwLK/QrZVdBawY0UckmetP14Ggfg5KeVOUcWDEqUEuK/SRZ4w6ghVVqeBX5U8dGs6KX2xWfX/4X SCFvnIxl TfT+vV1dpBdqmu4cmEBtwUsWoi5R2P2zD07CJuKSQETzsp5VYkXtX7+irmX2ck+4iFk3rT4r2YM6rdd9O3Z4VofZ3FlKwZD+/jK/SI7oR+K7sxrHOmdWDoab+TGRcbORfqDGIYdomfqCLver47tWYybahzppMfYzg/7vzTYKEns5vWsqXT/4Dpzda1wif+FhSbBiY7O2dDo9TWUGWh33GRBkKQOB688OuOFsqsqCYlE8UDmbHxenjeqYPYpJsvHFlmSjRKQY5GrTrdebtImaP/JPDkMMeuehtM5sK0NFvcR9NT2+Y/6BsyNzPJ0c3ZKI8VFS3pJMP1VaU5Nq07t4sgltuIQ5GT26TCtz/zFmkUw6bP8yah4YHPAcZ9Pnvy8l63qbG3t2fk05mc5XVNifVFgH9twDi6Fz/kqJqj8PGy9yatJ+7YNURkUBBgvReqTG4X/q8KIRy6/sqXSVZQctaILlf4ZbUC4Pcs57/4P0SAuaQSd01vzGttVXdhIF8x2d0PM3HzyvFrBYBu0qqegpGCJlzs603chEnWOpEbQ98eEBU4ZU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: V2: - dropped the get_pfnblock_migratetype() optimization patchlet since somebody else beat me to it (thanks Zi) - broke out pcp bypass fix since somebody else reported the bug: https://lore.kernel.org/linux-mm/20230911181108.GA104295@cmpxchg.org/ - fixed the CONFIG_UNACCEPTED_MEMORY build (lkp) - rebased to v6.6-rc1 The series is based on v6.6-rc1 plus the pcp bypass fix above ^ Reported-by: Mike Kravetz Signed-off-by: Johannes Weiner Reported-by: Vlastimil Babka Signed-off-by: Johannes Weiner Signed-off-by: Zi Yan Signed-off-by: Zi Yan Signed-off-by: Zi Yan Signed-off-by: Zi Yan Signed-off-by: Johannes Weiner Reviewed-by: Zi Yan --- This is a breakout series from the huge page allocator patches[1]. While testing and benchmarking the series incrementally, as per reviewer request, it became apparent that there are several sources of freelist migratetype violations that later patches in the series hid. Those violations occur when pages of one migratetype end up on the freelists of another type. This encourages incompatible page mixing down the line, where allocation requests ask for one migrate type, but receives pages of another. This defeats the mobility grouping. The series addresses those causes. The last patch adds type checks on all freelist movements to rule out any violations. I used these checks to identify the violations fixed up in the preceding patches. The series is a breakout, but has merit on its own: Less type mixing means improved grouping, means less work for compaction, means higher THP success rate and lower allocation latencies. The results can be seen in a mixed workload that stresses the machine with a kernel build job while periodically attempting to allocate batches of THP. The data is averaged over 50 consecutive defconfig builds: VANILLA PATCHED-CLEANLISTS Hugealloc Time median 14642.00 ( +0.00%) 10506.00 ( -28.25%) Hugealloc Time min 4820.00 ( +0.00%) 4783.00 ( -0.77%) Hugealloc Time max 6786868.00 ( +0.00%) 6556624.00 ( -3.39%) Kbuild Real time 240.03 ( +0.00%) 241.45 ( +0.59%) Kbuild User time 1195.49 ( +0.00%) 1195.69 ( +0.02%) Kbuild System time 96.44 ( +0.00%) 97.03 ( +0.61%) THP fault alloc 11490.00 ( +0.00%) 11802.30 ( +2.72%) THP fault fallback 782.62 ( +0.00%) 478.88 ( -38.76%) THP fault fail rate % 6.38 ( +0.00%) 3.90 ( -33.52%) Direct compact stall 297.70 ( +0.00%) 224.56 ( -24.49%) Direct compact fail 265.98 ( +0.00%) 191.56 ( -27.87%) Direct compact success 31.72 ( +0.00%) 33.00 ( +3.91%) Direct compact success rate % 13.11 ( +0.00%) 17.26 ( +29.43%) Compact daemon scanned migrate 1673661.58 ( +0.00%) 1591682.18 ( -4.90%) Compact daemon scanned free 2711252.80 ( +0.00%) 2615217.78 ( -3.54%) Compact direct scanned migrate 384998.62 ( +0.00%) 261689.42 ( -32.03%) Compact direct scanned free 966308.94 ( +0.00%) 667459.76 ( -30.93%) Compact migrate scanned daemon % 80.86 ( +0.00%) 83.34 ( +3.02%) Compact free scanned daemon % 74.41 ( +0.00%) 78.26 ( +5.10%) Alloc stall 338.06 ( +0.00%) 440.72 ( +30.28%) Pages kswapd scanned 1356339.42 ( +0.00%) 1402313.42 ( +3.39%) Pages kswapd reclaimed 581309.08 ( +0.00%) 587956.82 ( +1.14%) Pages direct scanned 56384.18 ( +0.00%) 141095.04 ( +150.24%) Pages direct reclaimed 17055.54 ( +0.00%) 22427.96 ( +31.50%) Pages scanned kswapd % 96.38 ( +0.00%) 93.60 ( -2.86%) Swap out 41528.00 ( +0.00%) 47969.92 ( +15.51%) Swap in 6541.42 ( +0.00%) 9093.30 ( +39.01%) File refaults 127666.50 ( +0.00%) 135766.84 ( +6.34%) include/linux/mm.h | 18 +- include/linux/page-isolation.h | 2 +- include/linux/vmstat.h | 8 - mm/debug_page_alloc.c | 12 +- mm/internal.h | 5 - mm/page_alloc.c | 357 ++++++++++++++++++--------------- mm/page_isolation.c | 23 ++- 7 files changed, 217 insertions(+), 208 deletions(-)